CAREER: Insertion-Based Natural Language Generation

职业：基于插入的自然语言生成

• 批准号: 2339766
• 负责人: Nanyun Peng
• 金额: $ 58.55万
• 依托单位: University of California-Los Angeles
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-05-01 至 2029-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2339766&HistoricalAwards=false
• 关键词: CAREER; Insertion; Based; Natural; Language

AbstractLanguage models (LMs) have become the foundations of most natural language processing (NLP) applications nowadays. However, existing language models predominantly follow the auto-regressive paradigm, which trains models to predict the next word given the left-side context. Then, they generate sentences word-by-word, left-to-right. The simplicity of this paradigm is attractive but it has several limitations, including inefficiency in generation, lack of reliable control for human-machine collaboration, and more importantly, it significantly deviates from how humans interpret and compose sentences. The proposed research explores a fundamentally distinct paradigm of language modeling --- insertion-based models. Such models formulate the generation process as iteratively inserting words into an incomplete context, offering significantly more flexibility and controllability compared to the auto-regressive models. The insertion-based formulation better mimics human writing behaviors and thus provides a tool for computational linguistics and cognitive science to study the structure of languages. The controllability of the model will benefit communication researchers, content creators, and creative composers for applications such as creative content generation, tailored communication, and personalized user experiences. The research results will be integrated in teaching materials to disseminate generative artificial intelligence (AI) for K-12 and undergraduate. This project aims to advance our understandings of the benefits and capability of insertion-based LMs. The proposed research will explore different generation orders supported by the flexibility of the insertion-based formulation, with the goal to discover optimal insertion orders guided by linguistic theories. Additionally, the project will introduce a novel model architecture for a variation of insertion-based LMs that incorporates deletion operations, enabling the models to rectify previous generation errors. This enhancement brings added flexibility and controllability to insertion-based LMs. Furthermore, an ambitious goal of this research is to investigate the scaling law and scale up the pretraining of insertion-based LMs. The success of these explorations has the potential to lead to a family of large language models that exhibit enhanced flexibility, controllability, and inference efficiency surpassing the auto-regressive LMs, resulting in benefits for a wide range of natural language generation and general NLP tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

如今，抽象语言模型（LMS）已成为大多数自然语言处理（NLP）应用程序的基础。但是，现有的语言模型主要遵循自动回归范式，该范式训练模型以预测给定左侧上下文的下一个单词。然后，他们从左到右生成句子。这种范式的简单性很有吸引力，但它有几个局限性，包括发电的效率低下，缺乏可靠的人机协作控制，更重要的是，它与人类的解释和撰写句子有很大的偏离。拟议的研究探讨了语言建模的根本不同的范式 - 基于插入的模型。这样的模型将迭代的单词插入不完整的上下文中，与自动回归模型相比，具有更大的灵活性和可控性。基于插入的公式可以更好地模仿人类写作行为，因此为计算语言学和认知科学提供了一种研究语言结构的工具。该模型的可控性将使沟通研究人员，内容创建者和创意作曲家受益，例如创意内容，量身定制的沟通和个性化的用户体验。研究结果将集成到教材中，以分发K-12和本科生的生成人工智能（AI）。该项目旨在提高我们对基于插入LMS的好处和能力的理解。拟议的研究将探索以基于插入的配方灵活性支持的不同产生订单，其目标是发现以语言理论为指导的最佳插入顺序。此外，该项目将引入一种新型的模型体系结构，以用于插入的LMS的变化，该插入的LMS结合了删除操作，从而使模型能够纠正上一代错误。这种增强为基于插入的LMS增添了灵活性和可控性。此外，这项研究的一个雄心勃勃的目标是研究规模定律并扩大基于插入的LMS的预处理。 The success of these explorations has the potential to lead to a family of large language models that exhibit enhanced flexibility, controllability, and inference efficiency surpassing the auto-regressive LMs, resulting in benefits for a wide range of natural language generation and general NLP tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.