Collaborative Research: Hardware-Aware Matrix Computations for Deep Learning Applications

协作研究：深度学习应用的硬件感知矩阵计算

• 批准号: 2247014
• 负责人: Atri Rudra
• 金额: $ 37.7万
• 依托单位: SUNY at Buffalo
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-05-01 至 2026-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2247014&HistoricalAwards=false
• 关键词: Collaborative; Research; Hardware; Aware; Matrix

Deep Learning (DL) systems these days are ubiquitous, and arguably affect our everyday lives more than any other computational system. Recently, such deep models (e.g., GPT-3) have increasingly become large and unwieldy with a large computational footprint. Given the ever increasing computational requirements, it has become nearly impossible to make progress on cutting edge research in learning such DL models outside of a few large technological companies. This project will explore principled ways to create DL systems that are as expressive as the large deep models but at a fraction of the computational cost. On the practical front these improvements are expected to expand the possibility of creating such powerful DL models to larger parts of society. On the educational front, this project will train undergraduate (UG) researchers and will integrate responsible computing into UG curriculum.This project will study how one can use structured matrices in concert with modern hardware constraints to achieve similar performance as these really large models but at a fraction of size and computational cost. Specifically, the investigators focus on the following two thrusts: (i) Design the ‘holy grail’ of structured matrices that satisfy all properties that are desirable in DL applications (including having an efficient projection problem as well as having efficient parallel and/or hardware friendly learning algorithms); and (ii) Thinking of new applications that our new theory can unlock. This DL lens exposes new problems to consider when studying structured matrices. In turn, the new family of structured matrices studied in this project will not only have immediate practical applications but will also unlock new twists on classical theoretical problems in matrix computations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

如今，深度学习 (DL) 系统无处不在，并且比任何其他计算系统对我们日常生活的影响更大，最近，由于计算量大，此类深度模型（例如 GPT-3）变得越来越庞大且笨重。随着计算需求的不断增加，除了少数大型科技公司之外，在学习此类深度学习模型方面的前沿研究几乎不可能取得进展，该项目将探索创建与大型深度模型一样具有表现力的深度学习系统的原则性方法。在一个在实际方面，这些改进有望将创建如此强大的深度学习模型的可能性扩大到社会的更大范围。在教育方面，该项目将培训本科生（UG）研究人员，并将负责任的计算融入其中。 UG 课程。该项目将研究如何使用结构化矩阵与现代硬件约束相结合，以较小的尺寸和计算成本实现与这些大型模型类似的性能。具体而言，研究人员重点关注以下两个重点：（ i) 设计结构化的“圣杯”满足深度学习应用所需的所有属性的矩阵（包括高效的投影问题以及高效的并行和/或硬件友好的学习算法）；以及（ii）思考我们的新理论可以解锁的新应用。 Lens 揭示了研究结构化矩阵时需要考虑的新问题。反过来，该项目中研究的新的结构化矩阵系列不仅将具有直接的实际应用，而且还将为矩阵计算中的经典理论问题带来新的变化。该奖项反映了 NSF 的法定奖项。使命和通过使用基金会的智力价值和更广泛的影响审查标准进行评估，该项目被认为值得支持。

项目成果

Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutions

Laughing Hyena Distillery：从卷积中提取紧凑递归

• DOI: 10.48550/arxiv.2310.18780
• 发表时间: 2023-10-28
• 期刊: ArXiv
• 作者: Stefano Massaroli;Michael Poli;Daniel Y. Fu;Hermann Kumbong;Rom N. Parnichkun;Aman Timalsina;David W. Romero;Quinn McIntyre;Beidi Chen;A. Rudra;Ce Zhang;Christopher Ré;Stefano Ermon;Y. Bengio
• 通讯作者: Y. Bengio

Zoology: Measuring and Improving Recall in Efficient Language Models

动物学：测量和提高高效语言模型的召回率

• 发表时间: 2024-05
• 期刊: Proceedings of 12th International Conference on Learning Representations (ICLR
• 作者: Arora, Simran;Eyuboglu, Sabri;Timalsina, Aman;Johnson, Isys;Poli, Michael;Zou, James;Rudra, Atri;Ré, Christopher
• 通讯作者: Ré, Christopher

Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture

Monarch Mixer：基于简单次二次 GEMM 的架构

• 发表时间: 2023-12
• 期刊: Proceedings of the 36th Neural Information Processing Systems Conference (NeurIPS
• 作者: Fu, Daniel Y.;Arora, Simran;Grogan, Jessica;Johnson, Isys;Eyuboglu, Sabri;Thomas, Armin W.;Spector, Benjamin;Poli, Michael;Rudra, Atri;Ré, Christopher
• 通讯作者: Ré, Christopher