CCF: SHF: Small: Transformer synthesis

CCF：SHF：小型：变压器综合

• 批准号: 2203399
• 负责人: Niraj Jha
• 金额: $ 60万
• 依托单位: Princeton University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-02-01 至 2025-01-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2203399&HistoricalAwards=false
• 关键词: CCF; SHF; Small; Transformer; synthesis

Just within four years of being first proposed, transformers have had a dramatic impact on the natural language processing (NLP) field and are also beginning to have an impact on other fields, such as computer vision. This success has largely been driven by large-scale pre-training datasets, increasing computational power, and robust training techniques. However, a major challenge that remains is efficient optimal transformer model synthesis for a specific task and set of user requirements. However, this is not easy to do since the design space of transformer models is vast. This project addresses this challenge through the development of transformer-synthesis methodologies and tools. Given the importance of transformers, such tools are likely to have a transformative impact on many application areas. The research will be disseminated to industry via tech transfer e.g., via open-source online distribution of source codes, summer internships, and by leveraging PIs involvement with local companies. Outreach and curriculum development plans will also be undertaken within the context of the proposed research.There is currently no universal framework that can navigate the vast transformer hyperparameter design space. Previously proposed transformer models are homogeneous in terms of data flow through the network. Unfortunately, this leads to very suboptimal transformer architectures. This project expands the transformer design space to incorporate heterogeneous architectures that venture beyond self-attention by employing other operations like convolutions and linear transforms. It will also explore novel projection layers and positional encodings to make hidden sizes flexible across various transformer layers. It will use a dense embedding to capture model similarity to significantly enhance search efficiency. It will develop a heteroscedastic surrogate model to further speed up search. It will include operations that optimize long-range interactions for long input sequences. It will also explore skipped connections and block-level grow-and-prune synthesis to improve architectural search efficiency.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在首次提出的四年之内，变形金刚对自然语言处理（NLP）领域产生了巨大影响，并且也开始对其他领域（例如计算机视觉）产生影响。这一成功在很大程度上是由大规模的预训练数据集驱动的，增加了计算能力和强大的训练技术。但是，仍然存在的主要挑战是针对特定任务和一组用户需求的有效最佳变压器模型综合。但是，这并不容易，因为变压器模型的设计空间很大。 该项目通过开发变压器合成方法和工具来应对这一挑战。 鉴于变压器的重要性，此类工具可能会对许多应用领域产生变革性的影响。这项研究将通过技术转移（例如，通过开放源代码在线分发来源代码，暑期实习以及利用PIS与当地公司的参与来传播到行业）。外展和课程开发计划也将在拟议的研究的背景下实施。目前，没有可以浏览庞大的变压器超级参数设计空间的通用框架。先前提出的变压器模型在通过网络的数据流方面是均匀的。不幸的是，这导致了非常优化的变压器体系结构。该项目扩大了变压器设计空间，以通过采用其他操作（例如卷积和线性变换）来融合异质体系结构，这些架构超越自我发挥。它还将探索新颖的投影层和位置编码，以使隐藏尺寸在各种变压器层上柔韧。它将使用密集的嵌入来捕获模型相似性，以显着提高搜索效率。它将开发出异质的替代模型，以进一步加速搜索。它将包括为长输入序列优化长距离交互的操作。它还将探索跳过的连接和块级的成长和促进综合，以提高建筑搜索效率。该奖项反映了NSF的法定任务，并且使用基金会的知识分子优点和更广泛的影响评估标准，被认为值得通过评估来提供支持。