Collaborative Research: CIF:Medium:Theoretical Foundations of Compositional Learning in Transformer Models

合作研究：CIF：Medium：Transformer 模型中组合学习的理论基础

• 批准号: 2403075
• 负责人: Samet Oymak
• 金额: $ 40万
• 依托单位: Regents of the University of Michigan - Ann Arbor
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-07-01 至 2028-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2403075&HistoricalAwards=false
• 关键词: Collaborative; Research; CIF; Medium; Theoretical

Large Language Models (LLMs) based on transformer architectures, such as GPT-4, Llama 2, and Claude 3, have demonstrated remarkable emergent capabilities in compositional reasoning, allowing them to tackle complex tasks by decomposing them into simpler intermediate steps. Examples to these tasks include text and code generation, basic arithmetic and problem solving, and answering complex questions. Despite these empirical advances, the underlying mechanics of these capabilities remain largely unexplored. This collaborative research project aims to investigate the theoretical foundations of compositional learning in transformer models, focusing on three key areas: model expressivity, statistical learning theory, and optimization, aiming to develop novel learning guarantees, algorithms, architectures, and design principles that significantly advance the development of more capable and interpretable Artificial Intelligence (AI) and LLM systems. The research findings will be incorporated into educational curricula, fostering a diverse community around transformers, compositional learning, and their applications. The project will also engage the broader public through workshops and outreach activities, promoting responsible AI practices and AI education for undergraduate and K-12 students.The first thrust will explore the expressive capacity of transformers augmented with loops, memory, and external tools, which are essential for compositional reasoning. The second thrust will examine the statistical properties of autoregressive training using compositional data to understand its limits, benefits, and ability to generalize to novel problem instances. This is expected to lead to new theories of compositional learning that will highlight the role of skill acquisition and composition. The third thrust will investigate the optimization principles of compositional learning with transformers. This research will shed light on the optimization landscape and identify techniques for more efficient training of transformers through compositional techniques.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

大型语言模型（LLM）基于变压器体系结构，例如GPT-4，Llama 2和Claude 3，在组成推理中表现出了出色的新兴功能，从而使它们可以通过将它们分解为简单的中间步骤来解决复杂的任务。这些任务的示例包括文本和代码生成，基本算术和解决问题以及回答复杂问题。尽管有这些经验的进步，但这些能力的基本机制在很大程度上尚未得到探索。该协作研究项目旨在调查变形金刚模型中构图学习的理论基础，重点关注三个关键领域：模型表达性，统计学习理论和优化，旨在开发新颖的学习保证，算法，体系结构和设计原理，从而显着促进更有能力和可解释的人工智能（AI）和LLM系统的发展。研究发现将纳入教育课程中，培养围绕变形金刚，组成学习及其应用的多样化社区。该项目还将通过研讨会和外展活动吸引更广泛的公众，促进负责任的AI实践和针对本科生和K-12学生的AI教育。第一个推力将探索具有循环，记忆和外部工具增强变压器的表达能力，这对于组成推理至关重要。第二个推力将使用组成数据检查自回归训练的统计特性，以了解其限制，收益和推广到新的问题实例的能力。预计这将导致构图学习的新理论，这些理论将突出技能获取和组成的作用。第三个推力将研究变压器组成学习的优化原理。这项研究将阐明优化景观，并通过组成技术确定对变压器进行更有效培训的技术。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子和更广泛影响的评估评估标准的评估值得支持的。