CAREER: Efficient and Scalable Large Foundational Model Training on Supercomputers for Science

职业：科学超级计算机上高效且可扩展的大型基础模型训练

• 批准号: 2340011
• 负责人: Zhao Zhang
• 金额: $ 59.97万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-07-01 至 2029-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2340011&HistoricalAwards=false
• 关键词: CAREER; Efficient; Scalable; Large; Foundational

Deep learning (DL) methods, especially the large foundational models, enable exciting new approaches to problems in many science and engineering disciplines, such as genomics, bioinformatics, meteorology, and natural language processing. Training foundational models at extreme scales is time-consuming, prone to low utilization with limited scalability, and human-effort demanding. This NSF CAREER project addresses the convergence, performance, and scalability gaps of large foundational model pre-training on supercomputers with innovative algorithms, systems, and interface design. In addition to the algorithm and computer system innovation, this project contributes to translational computer science by lowering the barrier of sizeable foundational model training and the time consumption of scientific deep learning, thus enabling significantly more scientific research to embrace large foundational models. The research results will be publicly available as open-source software to the broader community, with comprehensive documentation on the design and usage to help users from all domains.Technically, this NSF CAREER project has four research and educational thrusts: The first thrust focuses on new optimization techniques such as first-, second-, and mixed-order optimizers with potential approximation techniques to enhance time-to-convergence. The second thrust aims to enhance the scaling efficiency by designing novel sparsification algorithms that leverage the spatial and temporal patterns of gradients. The third thrust considers a new complex parallelism abstraction that transparently deploys large models across processors with near-optimal performance given the present capability of compute, interconnect, and I/O on a supercomputer. The fourth thrust designs educational activities, including a distributed DL system course, a DL tutorial, and a DL bootcamp targeting students and practitioners with different levels of expertise.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习（DL）方法，尤其是大型基础模型，为许多科学和工程学科的问题提供了令人兴奋的新方法，例如基因组学，生物信息学，气象学和自然语言处理。极端尺度上的训练基础模型耗时，可伸缩性有限，易于使用人力，并要求人力胜任。这个NSF职业项目介绍了具有创新算法，系统和界面设计的超级计算机预培训的大型基础模型的融合，性能和可伸缩性差距。除了算法和计算机系统创新外，该项目还通过降低了较大的基础模型培训的障碍和科学深度学习的时光，从而有助于转化计算机科学，从而实现了更大的科学研究，以采用大型基础模型。该研究结果将作为开源软件公开提供给更广泛的社区，并提供有关设计和用法的全面文档，以帮助来自所有领域的用户。技术上，该NSF职业项目具有四个研究和教育性的推力：第一个推力集中在新的优化技术上，例如第一，第二，第二，第二和混合级优化的近似近似时间，以增强近似时间来增强时间效果。第二个推力旨在通过设计利用梯度的空间和时间模式的新型稀疏算法来提高缩放效率。第三个推力考虑了一种新的复杂并行性抽象，鉴于当前的计算，互连和I/O在超级计算机上的当前功能，透明地在处理器上透明地部署了大型模型。第四个推力设计教育活动，包括分布式DL系统课程，DL教程以及DL训练营的目标，以不同水平的专业知识的学生和从业人员进行针对的学生和从业人员。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子和更广泛影响的评估来通过评估来获得支持的。