Collaborative Research: CIF:Medium:Theoretical Foundations of Compositional Learning in Transformer Models

合作研究：CIF：Medium：Transformer 模型中组合学习的理论基础

• 批准号: 2403074
• 负责人: Dimitrios Papailiopoulos
• 金额: $ 50万
• 依托单位: University of Wisconsin-Madison
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-07-01 至 2028-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2403074&HistoricalAwards=false
• 关键词: Collaborative; Research; CIF; Medium; Theoretical

Large Language Models (LLMs) based on transformer architectures, such as GPT-4, Llama 2, and Claude 3, have demonstrated remarkable emergent capabilities in compositional reasoning, allowing them to tackle complex tasks by decomposing them into simpler intermediate steps. Examples to these tasks include text and code generation, basic arithmetic and problem solving, and answering complex questions. Despite these empirical advances, the underlying mechanics of these capabilities remain largely unexplored. This collaborative research project aims to investigate the theoretical foundations of compositional learning in transformer models, focusing on three key areas: model expressivity, statistical learning theory, and optimization, aiming to develop novel learning guarantees, algorithms, architectures, and design principles that significantly advance the development of more capable and interpretable Artificial Intelligence (AI) and LLM systems. The research findings will be incorporated into educational curricula, fostering a diverse community around transformers, compositional learning, and their applications. The project will also engage the broader public through workshops and outreach activities, promoting responsible AI practices and AI education for undergraduate and K-12 students.The first thrust will explore the expressive capacity of transformers augmented with loops, memory, and external tools, which are essential for compositional reasoning. The second thrust will examine the statistical properties of autoregressive training using compositional data to understand its limits, benefits, and ability to generalize to novel problem instances. This is expected to lead to new theories of compositional learning that will highlight the role of skill acquisition and composition. The third thrust will investigate the optimization principles of compositional learning with transformers. This research will shed light on the optimization landscape and identify techniques for more efficient training of transformers through compositional techniques.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

基于 Transformer 架构的大型语言模型 (LLM)，例如 GPT-4、Llama 2 和 Claude 3，在组合推理方面表现出了卓越的涌现能力，使它们能够通过将复杂任务分解为更简单的中间步骤来处理复杂的任务。这些任务的示例包括文本和代码生成、基本算术和问题解决以及回答复杂问题。尽管取得了这些实证进展，但这些能力的基本机制在很大程度上仍未得到探索。该合作研究项目旨在研究 Transformer 模型中组合学习的理论基础，重点关注三个关键领域：模型表达性、统计学习理论和优化，旨在开发新颖的学习保证、算法、架构和设计原则，从而显着推进开发能力更强、可解释的人工智能（AI）和法学硕士系统。研究结果将被纳入教育课程，围绕变压器、组合学习及其应用培育一个多元化的社区。该项目还将通过研讨会和外展活动吸引更广泛的公众，促进本科生和 K-12 学生负责任的人工智能实践和人工智能教育。第一个重点将探索通过循环、内存和外部工具增强的变压器的表达能力，对于组合推理至关重要。第二个重点将使用组合数据检查自回归训练的统计特性，以了解其局限性、好处以及推广到新问题实例的能力。预计这将导致新的作曲学习理论，强调技能习得和作曲的作用。第三个重点将研究变压器组合学习的优化原理。这项研究将揭示优化前景，并确定通过组合技术更有效地训练变压器的技术。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。