Collaborative Research: RI: Medium: Expert-in-the-Loop Neural Summarization for Consequential Domains

合作研究：RI：中：结果领域的专家在环神经摘要

• 批准号: 2211954
• 负责人: Byron Wallace
• 金额: $ 60万
• 依托单位: Northeastern University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-07-01 至 2026-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2211954&HistoricalAwards=false
• 关键词: Collaborative; Research; RI; Medium; Expert

Automatic summarization methods aim to create shortened versions of texts (for example, news or scientific articles) that still accurately communicate their main points. Summarization methods provide a potential means to counteract the problem of “information overload” which is prevalent across many areas. But much of the research on automatic summarization has focussed largely on just one type of data: news articles. This is not because summarizing news articles is seen as particularly important. Rather, it is a result of there being conveniently available large datasets that can be used to “train” machine learning models to perform summarization. However, the resultant focus on approaches that assume a setting in which one has access to large volumes of “training data” to use to train summarization models has warped research priorities; little work has been done on investigating how automatic summarization methods might be used in important but specialized domains such as medicine or law. In these kinds of areas one is unlikely to have access to a massive dataset of manually written summaries. Furthermore, domain experts in such areas are not likely to blindly trust a system-generated summary (nor should they). This motivates a need for transparency with respect to how the model generated a particular summary, and for approaches that permit the expert to interact with the model more generally. This project aims to address these issues by investigating and extending the capabilities of modern, pre-trained, neural summarization models in the context of domains and tasks in which one has limited explicit supervision, and where there is a heightened need for factually accurate summaries. The project will involve critically evaluating state-of-the-art models when fine-tuned for summarization in domains like medicine under limited supervision; a specific aim is to characterize their behavior with respect to the factuality of model outputs. The idea is then to extend these models to permit interactive and efficient supervision, via active learning methods, alternative types of supervision (e.g., expert “highlights”), and novel pre-training objectives. Finally, the investigators will design architectures that afford increased transparency and controllability; this will be accomplished using latent variable summarization models, which will in turn allow one to inspect which input segments informed particular outputs. This will provide a natural means for the end-user (domain expert) to verify model outputs, and it will also provide a means to “debug” summarization systems. The hope is that these technical innovations will allow domain experts to benefit from automated summarization technology.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

自动摘要方法旨在创建缩短的文本版本（例如新闻或科学文章），这些文本仍然可以准确地传达其要点。摘要方法提供了一种潜在的手段来抵消“信息过载”的问题，而“信息过载”的问题在许多领域都普遍存在。但是，关于自动摘要的许多研究主要集中在一种类型的数据上：新闻文章。这不是因为总结新闻文章被认为特别重要。相反，这是有方便的大型数据集的结果，可以用来“训练”机器学习模型以执行摘要。但是，最终关注的方法是假定一个设置的方法，在这种方法中，人们可以使用大量的“培训数据”来训练摘要模型。研究如何在重要但专业领域（例如医学或法律）中使用自动摘要方法几乎没有完成。在这类领域，人们不太可能访问大量的手动书面摘要数据集。此外，在这些领域的领域专家不太可能盲目信任系统生成的摘要（也不应该）。这激发了对模型如何生成特定摘要的透明度的需求，并且对于使专家可以更普遍地与模型进行交互的方法。该项目旨在通过调查和扩展现代，预训练的，神经摘要模型的能力来解决这些问题，这些域名和任务有限的明确监督，以及对实际准确摘要的需求增强。当在有限的监督下，在像医学这样的域中进行了微调以在医学之类的领域进行汇总，该项目将涉及批判性评估最先进的模型；一个具体的目的是表征他们相对于模型输出的事实的行为。然后，这个想法是扩展这些模型，以通过主动学习方法，替代类型的监督（例如专家“亮点”）和新颖的培训预培训目标，以允许交互式和有效的监督。最后，调查人员将设计可提高透明度和可控性的体系结构。这将使用潜在变量摘要模型来完成，这又可以允许一个人检查哪些输入段了解特定的输出。这将为最终用户（域专家）验证模型输出提供一种自然手段，并且还将提供一种“调试”摘要系统的手段。希望这些技术创新将使领域专家能够从自动汇总技术中受益。该奖项反映了NSF的法定任务，并被认为是使用基金会的知识分子优点和更广泛的影响评估标准来通过评估来获得的支持。

项目成果

Automatically Summarizing Evidence from Clinical Trials: A Prototype Highlighting Current Challenges

• DOI: 10.48550/arxiv.2303.05392
• 发表时间: 2023-03
• 期刊: Proceedings of the conference. Association for Computational Linguistics. Meeting
• 作者: S. Ramprasad;Denis Jered McInerney;Iain J. Marshal;Byron Wallace

Overview of MSLR2022: A Shared Task on Multi-document Summarization for Literature Reviews

• 发表时间: 2022
• 作者: Lucy Lu Wang;Jay DeYoung;Byron Wallace