Collaborative Research: OAC Core: ScaDL: New Approaches to Scaling Deep Learning for Science Applications on Supercomputers

协作研究：OAC 核心：ScaDL：在超级计算机上扩展深度学习科学应用的新方法

• 批准号: 2106661
• 负责人: Zhao Zhang
• 金额: $ 22.64万
• 依托单位: University of Texas at Austin
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2023-11-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2106661&HistoricalAwards=false
• 关键词: Collaborative; Research; OAC; Core; ScaDL

Today's deep learning (DL) revolution is enabled by efficient deep neural network (DNN) training methods that capture important patterns within large quantities of data in compact, easily usable DNN models. DL methods are applied routinely to tasks like natural language translation and image labeling--and, in science and engineering, to problems as diverse as drug design, environmental monitoring, and fusion energy. Yet as data sizes increase and DL methods grow in sophistication, the time required to train new models often emerges as a major challenge. The Scalable Deep Learning (ScaDL) project will address this challenge by making it possible to use specialized high-performance computing (HPC) systems to train bigger models more rapidly. Efficient use of the thousands of powerful processors in modern HPC systems for DNN training has previously been stymied by communication costs that grow rapidly with the number of processors used. ScaDL will overcome this obstacle by developing new DNN training methods that reduce communication requirements by performing additional computation, by validating the effectiveness of these new methods in a range of scientific applications that use DL in different ways, and by integrating the new methods into scalable DL software for use by domain scientists, computer scientists, and engineers supporting DL application in HPC centers. By permitting the use of powerful HPC systems to train DNN models thousands of times faster than on a single computer, ScaDL will enable advances in many areas of science and engineering. The project will also contribute to educational outcomes by engaging PhD students in project goals, by using ScaDL tools in a new DL systems engineering class at the University of Chicago, and by enlisting participants in summer schools at the Texas Advanced Computing Center (TACC) and U. Chicago, both of which target recruitment of students from underserved communities at the graduate, undergraduate, and high-school levels, to apply the tools to scientific problems. ScaDL's focus on science applications and education aligns the project with NSF's mission of promoting the progress of science.The ScaDL project contributes to science in two ways. First, it explores new techniques for enhancing the speed and scalability of commonly used optimization methods without losing model performance, by: 1) exploiting scalable algorithms for second-order information approximation; 2) developing methods for adapting to different computer hardware by tuning computation and communication to maximize training speed; 3) exploring compression techniques to reduce communication overheads; 4) using well-known benchmark applications to evaluate the convergence of ScaDL; and 5) applying its new algorithms and systems to science applications. Second, it will release an open-source implementation of the proposed algorithms and system. The implementation will be available on a variety of hardware platforms and capable of choosing the ratio of computation and communication required to make efficient use of the computation and communication hardware on a particular HPC system. The resulting algorithms and system will help disseminate ScaDL research results to a wide spectrum of research domains and users, and promote the adoption of the new methods in practical settings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

当今的深度学习 (DL) 革命是通过高效的深度神经网络 (DNN) 训练方法实现的，这些方法可以在紧凑、易于使用的 DNN 模型中捕获大量数据中的重要模式。深度学习方法通​​常应用于自然语言翻译和图像标记等任务，在科学和工程领域，也应用于药物设计、环境监测和聚变能源等各种问题。然而，随着数据规模的增加和深度学习方法的复杂化，训练新模型所需的时间往往成为一个重大挑战。可扩展深度学习 (ScaDL) 项目将通过使用专门的高性能计算 (HPC) 系统更快地训练更大的模型来应对这一挑战。现代 HPC 系统中数千个强大的处理器的高效使用用于 DNN 训练，此前曾因通信成本随着所使用的处理器数量而快速增长而受到阻碍。 ScaDL 将克服这一障碍，开发新的 DNN 训练方法，通过执行额外的计算来减少通信需求，验证这些新方法在以不同方式使用 DL 的一系列科学应用中的有效性，以及将新方法集成到可扩展的 DL 中供领域科学家、计算机科学家和工程师使用的软件，支持 HPC 中心的 DL 应用。通过允许使用强大的 HPC 系统以比单台计算机快数千倍的速度训练 DNN 模型，ScaDL 将推动许多科学和工程领域的进步。该项目还将通过让博士生参与项目目标、在芝加哥大学新的深度学习系统工程课程中使用 ScaDL 工具以及招募德克萨斯州高级计算中心 (TACC) 暑期学校的参与者来促进教育成果芝加哥大学，这两个学校的目标都是从服务匮乏的社区招收研究生、本科生和高中生，将这些工具应用于科学问题。 ScaDL 对科学应用和教育的关注使该项目与 NSF 促进科学进步的使命保持一致。ScaDL 项目通过两个方式为科学做出贡献。首先，它探索了在不损失模型性能的情况下提高常用优化方法的速度和可扩展性的新技术，方法是：1）利用可扩展算法进行二阶信息近似； 2）通过调整计算和通信来开发适应不同计算机硬件的方法，以最大限度地提高训练速度； 3）探索压缩技术以减少通信开销； 4）使用著名的基准应用程序来评估ScaDL的收敛性； 5）将其新算法和系统应用于科学应用。 其次，它将发布所提出的算法和系统的开源实现。该实现可在各种硬件平台上使用，并且能够选择有效利用特定 HPC 系统上的计算和通信硬件所需的计算和通信比率。由此产生的算法和系统将有助于将 ScaDL 研究成果传播给广泛的研究领域和用户，并促进新方法在实际环境中的采用。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准。

项目成果

Deep Neural Network Training With Distributed K-FAC

使用分布式 K-FAC 进行深度神经网络训练

• DOI: 10.1109/tpds.2022.3161187
• 发表时间: 2022-12
• 期刊: IEEE Transactions on Parallel and Distributed Systems
• 影响因子: 5.3
• 作者: Pauloski, J. Gregory;Huang, Lei;Xu, Weijia;Chard, Kyle;Foster, Ian T.;Zhang, Zhao
• 通讯作者: Zhang, Zhao

KAISA: an adaptive second-order optimizer framework for deep neural networks

KAISA：深度神经网络的自适应二阶优化器框架

• DOI: 10.1145/3458817.3476152
• 发表时间: 2021-11
• 期刊: Storage and Analysis
• 作者: Pauloski, J. Gregory;Huang, Qi;Huang, Lei;Venkataraman, Shivaram;Chard, Kyle;Foster, Ian;Zhang, Zhao
• 通讯作者: Zhang, Zhao