Collaborative Research: RI:Medium:MoDL:Mathematical and Conceptual Understanding of Large Language Models

合作研究：RI:Medium:MoDL:大型语言模型的数学和概念理解

• 批准号: 2211780
• 负责人: Tengyu Ma
• 金额: $ 40万
• 依托单位: Stanford University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2025-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2211780&HistoricalAwards=false
• 关键词: Collaborative; Research; RI; Medium; MoDL

Large language models (LLMs) have achieved unprecedented success in natural language processing (NLP). Since language models are being seen as a cornerstone of artificial intelligence in the near future, there is a need to be able to understand them, and to convey that understanding to regulators as well as the general public. These models are based on deep neural networks that are trained from vast quantities of text and have been demonstrated to be highly useful in performing tasks such as question answering, text classification, machine translation and summarization. Despite the huge empirical success, there is little understanding about their inner workings. This project seeks to bridge the gap by developing conceptual and mathematical understanding about training and using LLMs. The project will advance such understanding. The project will also seek to develop and disseminate instructional materials and draw on ideas from the project to impact ongoing programs at their institution to help increase participation in computing by individuals from underrepresented groups. The project has three components. (1) We will first build simplified generative models that capture the intrinsic structures of text, and analyze language models that are trained on texts from such generative models. (2) We then analyze why the learned language models can encode useful information that helps a wide range of downstream tasks. (3) Finally, we analyze and design new adaptation methods for downstream tasks with quantitative sample and computational efficiency guarantees. Education and outreach plans are integrated into this project: the investigators will develop a new introductory course in machine learning and disseminate instructional materials, mentor graduate and undergraduate students from underrepresented groups (through Princeton Freshman Scholars Institute, Stanford Summer Teacher Research Program, REU’s) and organize research workshops to promote conversations between the theoretical machine learning and NLP community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

大型语言模型（LLM）在自然语言处理（NLP）方面取得了前所未有的成功。由于语言模型在不久的将来被视为人工智能的基石，因此有必要能够理解它们，并将这种理解传达给监管机构和公众。这些模型基于从大量文本训练的深度神经网络，并已被证明在执行问题回答，文本分类，机器翻译和摘要等任务方面非常有用。尽管经验取得了巨大的成功，但对他们的内部运作几乎没有理解。该项目旨在通过对培训和使用LLM的概念和数学理解来建立构成差距的弥合。该项目将提高这种理解。该项目还将寻求开发和传播教学材料，并利用项目的想法，以影响其机构正在进行的计划，以帮助增加代表性不足的群体的个人参与计算。该项目有三个组成部分。 （1）我们将首先构建简化的通用模型，以捕获文本的内在结构，并分析对此类通用模型的文本进行培训的语言模型。 （2）然后我们分析为什么学习的语言模型可以编码有用的信息来帮助广泛的下游任务。 （3）最后，我们分析并设计了具有定量样本和计算效率保证的下游任务的新适应方法。教育和宣传计划已融入该项目：研究人员将开发一门新的介绍课程，并传播教学材料，智力研究生和本科生（通过普林斯顿大学新生学者研究所，斯坦福大学暑期教师研究计划，REU's）的新介绍课程，并促进了宣传机器学习和NLP的统计信息。通过基金会的智力优点和更广泛的影响评估标准通过评估来支持。