RI: Small: New tools for studying structural and inductive bias in NLP models

RI：小：研究 NLP 模型中的结构和归纳偏差的新工具

• 批准号: 2128145
• 负责人: Daniel Jurafsky
• 金额: $ 50万
• 依托单位: Stanford University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2128145&HistoricalAwards=false
• 关键词: RI; Small; New; tools; studying

Modern natural language processing systems, based on neural networks trained using large amounts of text, are a key part of the infrastructure of the nation and the world. These systems power practical tools like machine translation, web search, or automatic question answering, as well as research tools that help scientists and policy makers. These language processing models have made enormous progress in many ways, yet systems still fail unexpectedly, their successes cannot be explained, and their blind spots lead to biases. This project develops new tools for studying language models: why they work as well as they do, what their limitations are, and what distortions they introduce into language understanding, with the goal of improved systems and helping mitigate negative impacts on society.This project develops and investigates four kinds of new analytic tools for studying the inductive biases of language models - the structural tendencies that determine what they can learn. The structural transfer-learning paradigm involves training language models on artificial languages that can be manipulated, to see which structural aspects improve performance on natural language. The challenge-task paradigm brings humans in the loop to develop new evaluations to study why and how language processing systems fail, such as on aspect of language that change over time. The new theoretical framework of sensitivity models the complexity of language processing tasks by measuring how responsive the classification is to minor changes in the input, demonstrating which tasks or examples are easy or hard. And new tools are introduced to measure how embeddings of words introduce structural distortions - exaggerations or understatements in word relationships - that can cause models to fail. Understanding the limitations of technology and what makes one system better or one task or dataset harder than another is a crucial step toward building better language processing systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

基于使用大量文本训练的神经网络的现代自然语言处理系统是国家和世界基础设施的关键部分。这些系统的实用工具，例如机器翻译，网络搜索或自动问答，以及帮助科学家和政策制定者的研究工具。这些语言处理模型在许多方面取得了巨大的进步，但是系统仍然出乎意料地失败，无法解释其成功，并且它们的盲点导致了偏见。该项目开发了用于研究语言模型的新工具：为什么它们能和他们这样做，局限性是什么，以及他们将哪些扭曲引入语言理解中，目的是改善系统并帮助减轻对社会的负面影响。该项目发展并研究了四种新的分析工具，以研究模型的象征性倾向，这些模型的结构倾向 - 可以确定什么可以确定他们可以学习什么。结构转移学习范式涉及可以操纵的人造语言的培训语言模型，以查看哪些结构方面可以改善自然语言的性能。挑战任务范式使人类在循环中开发新的评估，以研究语言处理系统的失败，例如随着时间的流逝而变化的语言方面。灵敏度的新理论框架通过测量分类对输入中的较小变化的响应方式来模拟语言处理任务的复杂性，并证明哪些任务或示例很容易或很难。并引入了新工具，以衡量单词的嵌入如何引入结构扭曲 - 单词关系中的夸张或轻描淡写 - 可能导致模型失败。了解技术的局限性以及使一个系统更好，一个任务或数据集比另一个系统更难的是迈向建立更好的语言处理系统的关键步骤。该奖项反映了NSF的法定任务，并被认为是通过基金会的智力优点和更广泛的影响来通过评估来获得支持的审查标准。