RI: Small: Extracting and Representing Commonsense Knowledge Using Language Models

RI：小：使用语言模型提取和表示常识知识

基本信息

批准号：
2006851
负责人：
Douglas Downey
金额：
$ 47万
依托单位：
Northwestern University
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2020
资助国家：
美国
起止时间：
2020-10-01 至 2024-09-30
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2006851&HistoricalAwards=false
关键词：
RI Small Extracting Representing Commonsense

项目摘要

As computers advance from serving as our tools to becoming our helpers and collaborators, they must be capable of commonsense reasoning--for example, knowing that a person needs to use their hand to open a door (and that this might be a problem if they are carrying groceries). This kind of commonsense reasoning is a longstanding, elusive goal of artificial intelligence, but is becoming within reach today due to the availability of vast amounts of data and more powerful computational models for learning from those data. This project is aimed at a key step in enabling commonsense reasoning by machines: the automatic acquisition of common sense knowledge. The project’s approach builds upon recent breakthroughs in language models that learn by reading large amounts of text, and combines these in novel ways with explicit commonsense knowledge gathered from humans. The project probes new, scalable methods for humans to impart their commonsense knowledge to the system, by using existing dictionaries and encyclopedias, building curricula that help machines build to commonsense mastery step by step, and by directly enforcing key logical constraints (for example, that if one item is bigger than another, then the second item must be smaller than the first). Success in this project could help power new virtual assistants, medical diagnosis and treatment systems, improved search engines, and other important applications of AI. The work also aims to enable the development of better language models themselves---improving current commercial technologies such as speech recognition and machine translation, and ultimately helping to power the next generation of computer systems capable of communicating with people more naturally using language. Along the way, the project will help train the next generation of students about these approaches and technologies, via education and outreach activities.The technical strategy used in the project involves learning unsupervised neural language models (LMs) to capture textual distributions, and then extracting common sense knowledge from those models. This approach is challenging because common sense knowledge is multifarious and massive, and yet is not often explicitly stated in text. The project aims to overcome this challenge using several methods for scalably incorporating human input in concert with neural language models. First, the project studies how to use explicit lexical knowledge found in dictionaries to improve LMs, extending prior work in modeling the definitions of terms with neural LMs. Next, the project is investigating a “scaffold” of semantic tasks (a task curriculum of increasing complexity) incrementally constructing models for each task in turn in a way that aims to improve the learning of each subsequent task. Third, the project is developing methods for encoding commonsense logical constraints within neural language models. Lastly, because time and energy cost is a potential barrier to the application of the proposed techniques, the project is also studying how to make its approaches efficient. In particular, the project is investigating ways to scale-up LMs to larger corpora while reducing the significant computational and energy cost in LM training, by learning how to automatically identify text that will be more informative for training.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

随着计算机从充当我们的工具发展成为我们的帮助者和合作者，它们必须能够进行常识推理——例如，知道一个人需要用手来开门（如果他们这种常识性推理是人工智能长期存在的、难以捉摸的目标，但由于大量数据的可用性和用于从这些数据中学习的更强大的计算模型，现在已经变得可以实现。是实现这一目标的关键一步机器常识推理：该项目的方法建立在通过阅读大量文本进行学习的语言模型的最新突破之上，并以新颖的方式将这些知识与从人类那里收集的明确常识知识相结合。人类向系统传授常识知识的新的、可扩展的方法，通过使用现有的词典和百科全书，构建帮助机器逐步掌握常识的课程，并直接强制执行关键逻辑约束（例如，如果一个项目比另一个项目大，那么第二个项目必须比第一个项目小）。该项目的成功可以帮助推动新的虚拟助手、医疗诊断和治疗系统、改进的搜索引擎和其他重要应用程序。这项工作还旨在开发更好的语言模型本身——改进语音识别和机器翻译等当前的商业技术，并最终帮助推动能够更自然地与人交流的下一代计算机系统。在此过程中，该项目将帮助培养下一代学生的语言能力。这些方法和技术，通过教育和推广活动。该项目中使用的技术策略包括学习无监督神经语言模型（LM）来捕获文本分布，然后从这些模型中提取常识知识。这种方法具有挑战性，因为常识知识。该项目的目标是使用多种方法将人类输入与神经语言模型相结合来克服这一挑战。词典有待改进语言模型，扩展了之前使用神经语言模型对术语定义进行建模的工作接下来，该项目正在研究语义任务的“支架”（复杂性不断增加的任务课程），逐步为每个任务构建模型，旨在第三，该项目正在开发在神经语言模型中编码常识逻辑约束的方法。最后，由于时间和精力成本是所提出技术应用的潜在障碍，该项目还在研究如何实现。使其方法有效。特别是，该项目正在研究如何通过学习如何自动识别为训练提供更多信息的文本，将 LM 扩展到更大的语料库，同时减少 LM 训练中的大量计算和能源成本。该奖项反映了 NSF 的法定使命通过使用基金会的智力优点和更广泛的影响审查标准进行评估，并被认为值得支持。

项目成果

期刊论文数量（4）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

Who’s on First?: Probing the Learning and Representation Capabilities of Language Models on Deterministic Closed Domains

谁在第一？：探索确定性封闭域上语言模型的学习和表示能力

DOI：
10.18653/v1/2021.conll-1.16
发表时间：
2024-09-14
期刊：
影响因子：
0
作者：
David Demeter;Doug Downey
通讯作者：
Doug Downey

“It doesn’t look good for a date”: Transforming Critiques into Preferences for Conversational Recommendation Systems

“约会看起来不太好”：将批评转化为对话推荐系统的偏好

DOI：
10.18653/v1/2021.emnlp-main.145
发表时间：
2021-01
期刊：
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
影响因子：
0
作者：
Bursztyn, Victor;Healey, Jennifer;Lipka, Nedim;Koh, Eunyee;Downey, Doug;Birnbaum, Larry
通讯作者：
Birnbaum, Larry

CODE: Compiler-based Neuron-Aware Ensemble Training

代码：基于编译器的神经元感知集成训练