CAREER: Efficient and Scalable Large Foundational Model Training on Supercomputers for Science

职业：科学超级计算机上高效且可扩展的大型基础模型训练

• 批准号: 2340011
• 负责人: Zhao Zhang
• 金额: $ 59.97万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-07-01 至 2029-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2340011&HistoricalAwards=false
• 关键词: CAREER; Efficient; Scalable; Large; Foundational

Deep learning (DL) methods, especially the large foundational models, enable exciting new approaches to problems in many science and engineering disciplines, such as genomics, bioinformatics, meteorology, and natural language processing. Training foundational models at extreme scales is time-consuming, prone to low utilization with limited scalability, and human-effort demanding. This NSF CAREER project addresses the convergence, performance, and scalability gaps of large foundational model pre-training on supercomputers with innovative algorithms, systems, and interface design. In addition to the algorithm and computer system innovation, this project contributes to translational computer science by lowering the barrier of sizeable foundational model training and the time consumption of scientific deep learning, thus enabling significantly more scientific research to embrace large foundational models. The research results will be publicly available as open-source software to the broader community, with comprehensive documentation on the design and usage to help users from all domains.Technically, this NSF CAREER project has four research and educational thrusts: The first thrust focuses on new optimization techniques such as first-, second-, and mixed-order optimizers with potential approximation techniques to enhance time-to-convergence. The second thrust aims to enhance the scaling efficiency by designing novel sparsification algorithms that leverage the spatial and temporal patterns of gradients. The third thrust considers a new complex parallelism abstraction that transparently deploys large models across processors with near-optimal performance given the present capability of compute, interconnect, and I/O on a supercomputer. The fourth thrust designs educational activities, including a distributed DL system course, a DL tutorial, and a DL bootcamp targeting students and practitioners with different levels of expertise.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习 (DL) 方法，尤其是大型基础模型，为许多科学和工程学科（例如基因组学、生物信息学、气象学和自然语言处理）中的问题提供了令人兴奋的新方法。在极端规模下训练基础模型非常耗时，容易导致利用率低、可扩展性有限，并且需要人力。该 NSF CAREER 项目通过创新的算法、系统和接口设计，解决了超级计算机上大型基础模型预训练的收敛性、性能和可扩展性方面的差距。除了算法和计算机系统的创新之外，该项目还通过降低大规模基础模型训练的障碍和科学深度学习的时间消耗，为转化计算机科学做出贡献，从而使更多的科学研究能够拥抱大型基础模型。研究结果将作为开源软件向更广泛的社区公开，并提供有关设计和使用的全面文档，以帮助所有领域的用户。从技术上讲，这个 NSF 职业项目有四个研究和教育重点：第一个重点是新的优化技术，例如一阶、二阶和混合阶优化器，以及潜在的近似技术，可缩短收敛时间。第二个重点旨在通过设计利用梯度的空间和时间模式的新颖的稀疏化算法来提高缩放效率。第三个推力考虑了一种新的复杂并行抽象，考虑到超级计算机上当前的计算、互连和 I/O 能力，该抽象可以跨处理器透明地部署大型模型，并具有接近最佳的性能。第四个重点是设计教育活动，包括分布式深度学习系统课程、深度学习教程以及针对不同专业水平的学生和从业者的深度学习训练营。该奖项反映了 NSF 的法定使命，并通过使用基金会的评估进行评估，认为值得支持。智力价值和更广泛的影响审查标准。