CAREER: Building Next-Generation Language Models Based on Retrieval

职业：基于检索构建下一代语言模型

• 批准号: 2239290
• 负责人: Danqi Chen
• 金额: $ 60万
• 依托单位: Princeton University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-02-15 至 2028-01-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2239290&HistoricalAwards=false
• 关键词: CAREER; Building; Next; Generation; Language

Large language models (LMs) have revolutionized the field of natural language processing, achieving state-of-the-art performance in a wide range of downstream tasks. Despite the success, these LMs are trained on enormous amounts of text data and cost a massive amount to create and run. Additionally, they are inherently difficult to interpret, challenging to update with ever-changing real-world information, and may leak private user information. This proposal seeks to develop an alternative paradigm to standard language modeling: retrieval-based language models, with the aim of reducing training and inference costs while also providing benefits such as better interpretability, adaptability, and privacy. This research will be integrated into education through new teaching modules in developing undergraduate and graduate natural language processing courses, promoting education for undergraduate research, and outreach to K-12 students and teachers from underrepresented communities.This project addresses a full pipeline including training, scaling, adapting, and using retrieval-based language models and is organized into four components, including (1) building a general learning framework for retrieval-based LMs and developing scalable algorithms to support end-to-end learning; (2) investigating the scaling law of retrieval-based LMs and developing better data quantization to improve inference efficiency; (3) devising methods to quickly update retrieval-based LMs and adapt them to unseen and privacy-sensitive domains; (4) designing effective approaches to use retrieval-based LMs on downstream tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

大型语言模型 (LM) 彻底改变了自然语言处理领域，在各种下游任务中实现了最先进的性能。尽管取得了成功，但这些语言模型需要接受大量文本数据的训练，并且创建和运行成本高昂。此外，它们本质上难以解释，难以根据不断变化的现实世界信息进行更新，并且可能会泄露私人用户信息。该提案旨在开发标准语言建模的替代范式：基于检索的语言模型，目的是减少训练和推理成本，同时提供更好的可解释性、适应性和隐私性等好处。这项研究将通过新的教学模块融入教育，开发本科生和研究生自然语言处理课程，促进本科生研究教育，以及向来自代表性不足社区的 K-12 学生和教师进行推广。该项目涉及包括培训、扩展在内的完整管道。 、适应和使用基于检索的语言模型，分为四个部分，包括（1）为基于检索的语言模型构建通用学习框架，并开发可扩展的算法以支持端到端学习； （2）研究基于检索的语言模型的缩放规律并开发更好的数据量化以提高推理效率； （3）设计快速更新基于检索的语言模型并使其适应不可见和隐私敏感领域的方法； (4) 设计有效的方法，在下游任务中使用基于检索的 LM。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions

MQuAKE：通过多跳问题评估语言模型中的知识编辑

• DOI: 10.48550/arxiv.2305.14795
• 发表时间: 2023-05-24
• 期刊: 2022 14th International Conference on Knowledge and Smart Technology (KST)
• 作者: Zexuan Zhong;Zhengxuan Wu;Christopher D. Manning;Christopher Potts;Danqi Chen
• 通讯作者: Danqi Chen

Retrieval-based Language Models and Applications

基于检索的语言模型和应用

• DOI: 10.18653/v1/2023.acl-tutorials.6
• 发表时间: 2024-09-13
• 期刊: ArXiv
• 作者: Akari Asai;Sewon Min;Zexuan Zhong;Danqi Chen
• 通讯作者: Danqi Chen

Privacy Implications of Retrieval-Based Language Models

基于检索的语言模型的隐私影响

• DOI: 10.48550/arxiv.2305.14888
• 发表时间: 2023-05-24
• 期刊: ISPRS Int. J. Geo Inf.
• 作者: Yangsibo Huang;Samyak Gupta;Zexuan Zhong;K. Li;Danqi Chen
• 通讯作者: Danqi Chen

Enabling Large Language Models to Generate Text with Citations

启用大型语言模型来生成带引文的文本

• DOI: 10.18653/v1/2023.emnlp-main.398
• 发表时间: 2023-12
• 期刊: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
• 作者: Gao, Tianyu;Yen, Howard;Yu, Jiatong;Chen, Danqi
• 通讯作者: Chen, Danqi

Adapting Language Models to Compress Contexts

调整语言模型以压缩上下文

• DOI: 10.48550/arxiv.2305.14788
• 发表时间: 2023-05-24
• 期刊: ArXiv
• 作者: Alexis Chevalier;Ale;er Wettig;er;Anirudh Ajith;Danqi Chen
• 通讯作者: Danqi Chen