Collaborative Research: OAC Core: ScaDL: New Approaches to Scaling Deep Learning for Science Applications on Supercomputers

协作研究：OAC 核心：ScaDL：在超级计算机上扩展深度学习科学应用的新方法

• 批准号: 2401246
• 负责人: Zhao Zhang
• 金额: $ 22.64万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2024-10-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2401246&HistoricalAwards=false
• 关键词: Collaborative; Research; OAC; Core; ScaDL

Today's deep learning (DL) revolution is enabled by efficient deep neural network (DNN) training methods that capture important patterns within large quantities of data in compact, easily usable DNN models. DL methods are applied routinely to tasks like natural language translation and image labeling--and, in science and engineering, to problems as diverse as drug design, environmental monitoring, and fusion energy. Yet as data sizes increase and DL methods grow in sophistication, the time required to train new models often emerges as a major challenge. The Scalable Deep Learning (ScaDL) project will address this challenge by making it possible to use specialized high-performance computing (HPC) systems to train bigger models more rapidly. Efficient use of the thousands of powerful processors in modern HPC systems for DNN training has previously been stymied by communication costs that grow rapidly with the number of processors used. ScaDL will overcome this obstacle by developing new DNN training methods that reduce communication requirements by performing additional computation, by validating the effectiveness of these new methods in a range of scientific applications that use DL in different ways, and by integrating the new methods into scalable DL software for use by domain scientists, computer scientists, and engineers supporting DL application in HPC centers. By permitting the use of powerful HPC systems to train DNN models thousands of times faster than on a single computer, ScaDL will enable advances in many areas of science and engineering. The project will also contribute to educational outcomes by engaging PhD students in project goals, by using ScaDL tools in a new DL systems engineering class at the University of Chicago, and by enlisting participants in summer schools at the Texas Advanced Computing Center (TACC) and U. Chicago, both of which target recruitment of students from underserved communities at the graduate, undergraduate, and high-school levels, to apply the tools to scientific problems. ScaDL's focus on science applications and education aligns the project with NSF's mission of promoting the progress of science.The ScaDL project contributes to science in two ways. First, it explores new techniques for enhancing the speed and scalability of commonly used optimization methods without losing model performance, by: 1) exploiting scalable algorithms for second-order information approximation; 2) developing methods for adapting to different computer hardware by tuning computation and communication to maximize training speed; 3) exploring compression techniques to reduce communication overheads; 4) using well-known benchmark applications to evaluate the convergence of ScaDL; and 5) applying its new algorithms and systems to science applications. Second, it will release an open-source implementation of the proposed algorithms and system. The implementation will be available on a variety of hardware platforms and capable of choosing the ratio of computation and communication required to make efficient use of the computation and communication hardware on a particular HPC system. The resulting algorithms and system will help disseminate ScaDL research results to a wide spectrum of research domains and users, and promote the adoption of the new methods in practical settings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

当今的深度学习（DL）革命是通过有效的深度神经网络（DNN）训练方法来实现的，该方法在紧凑，易于使用的DNN模型中捕获大量数据中的重要模式。 DL方法常规应用于自然语言翻译和图像标签等任务，以及在科学和工程中，以及像药物设计，环境监测和融合能量一样多样化的问题。然而，随着数据尺寸的增加和DL方法的增长，训练新模型所需的时间通常会成为一个主要的挑战。可扩展的深度学习（SCADL）项目将通过使用专业的高性能计算（HPC）系统来更快地训练更大的模型，从而解决这一挑战。在现代HPC系统中，有效利用成千上万的强大处理器进行DNN培训，以前被沟通成本迅速增长，随着所使用的处理器数量而迅速增长。 SCADL将通过开发新的DNN训练方法来克服这一障碍，从而通过在一系列以不同方式使用DL的科学应用中验证这些新方法的有效性来减少交流需求，并通过将新方法集成到可扩展的DL软件中，以供域科学家，计算机科学家，计算机科学家，计算机科学家，以及支持HPC Centers中的DL DL应用程序。通过允许使用强大的HPC系统训练DNN型号数千倍的速度要比一台计算机快数千倍，SCADL将在许多科学和工程领域的进步。该项目还将通过在芝加哥大学的新DL系统工程课程中使用SCADL工具，以及在得克萨斯州高级计算中心（TACC）和U. Chicago的暑期学校中使用SCADL工具，并通过在毕业生不熟悉的培训中招聘人员招聘贫困的学生，并将其招募到较高的级别的学生中，这两个项目还将通过在芝加哥大学的新DL系统工程课程中使用SCADL工具来为教育成果做出贡献。 Scadl对科学应用和教育的关注使该项目与NSF促进科学进步的使命保持一致。SCADL项目通过两种方式为科学做出了贡献。首先，它探讨了新技术，以提高常用优化方法的速度和可扩展性而不会丢失模型性能，1）利用可伸缩算法来实现二阶信息近似； 2）通过调整计算和通信以最大化训练速度来开发适应不同计算机硬件的方法； 3）探索压缩技术以减少沟通开销； 4）使用众所周知的基准应用来评估SCADL的收敛性； 5）将其新算法和系统应用于科学应用。 其次，它将发布提出的算法和系统的开源实现。该实现将在各种硬件平台上可用，能够选择有效利用特定HPC系统上计算和通信硬件所需的计算和通信比率。由此产生的算法和系统将有助于将SCADL研究结果传播到各种研究领域和用户，并在实际环境中促进采用新方法。该奖项反映了NSF的法定任务，并被视为值得通过基金会的知识分子优点和更广泛影响的评估来审查审查的审查标准。