Collaborative Research: OAC Core: CropDL - Scheduling and Checkpoint/Restart Support for Deep Learning Applications on HPC Clusters

合作研究：OAC 核心：CropDL - HPC 集群上深度学习应用的调度和检查点/重启支持

• 批准号: 2403090
• 负责人: Wei Niu
• 金额: $ 15万
• 依托单位: University of Georgia Research Foundation Inc
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-10-01 至 2027-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2403090&HistoricalAwards=false
• 关键词: Collaborative; Research; OAC; Core; CropDL

Machine Learning (ML) and Deep Learning (DL) (more specifically, Deep Neural Network (DNN)) workloads are beginning to dominate the High-Performance Computing (HPC) arena. Today, massive computational resources are required to train even a single state-of-the-art deep learning model (e.g., large language models or LLMs). As the need for training massive DNN models continues and expands from the private sector to NSF-supported scientists and engineers (who are more likely to use shared computing resources), efficient checkpointing is emerging as a critical need. Checkpointing not only helps deal with failures but also provides more scheduling flexibility on shared HPC resources, as a very long-running job can be broken into several shorter ones. The premise of the CropDL project is that efficient and automated application-level checkpoint and restart will be critical to facilitating the use of shared HPC clusters for long-running ML training tasks, drastically increasing the number of researchers that can successfully train large ML models for various applications. This project also contributes to education and diversity in multiple aspects, for example, 1) introducing courses (or course material) to bring attention to ML-related workloads in computer systems undergraduate and graduate education; 2) integrating research tasks from this project with synergistic research programs at universities to increase the participation of women and underrepresented minority groups; and 3) supporting and training PhD students in their research, creating momentum on systems and cyberinfrastructure research related to emerging ML workloads and popularizing integrative research that combines the properties of these workloads with the complexities of modern HPC hardware.The overarching goal of CropDL is to support application-level checkpoints/restarts of deep learning applications for better resiliency, faster average completion time, and higher resource utilization. Particularly, several properties of DL workloads (as compared to scientific computations) create distinct sets of opportunities and challenges for checkpointing: 1) limited communication patterns during parallel execution, which can enable efficient coordinated checkpoints, 2) many unique opportunities for compression of checkpoints, and possibly taking uncoordinated checkpoints, and 3) malleable execution, where restarting from a different number of nodes is possible. Based on this observation, the first direction of this project is to exploit the properties of the DNN model(s) to be trained during checkpointing. This includes asynchronous versioned checkpointing for DL applications under a wide variety of parallelism models as well as content-based data reduction (compression and sparsification) techniques to reduce checkpoint volumes. The second direction of research focuses on using current and upcoming HPC systems' resources efficiently while checkpointing. It formulates tasks, data, and I/O requirements from DL applications into DAG representations and develops methods to schedule them. It also supports efficient I/O for deep learning applications with emerging I/O platforms. The last direction is to automate checkpointing through a compilation system based on the computational graph of DL workloads. All these efforts consider a variety of parallelization schemes for DNNs, i.e., data, model, and/or pipelined parallelism.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

机器学习 (ML) 和深度学习 (DL)（更具体地说，深度神经网络 (DNN)）工作负载开始在高性能计算 (HPC) 领域占据主导地位。如今，即使是训练一个最先进的深度学习模型（例如大型语言模型或法学硕士）也需要大量计算资源。随着训练大规模 DNN 模型的需求持续增长，并从私营部门扩展到 NSF 支持的科学家和工程师（他们更有可能使用共享计算资源），高效的检查点正在成为一项关键需求。检查点不仅有助于处理故障，而且还为共享 HPC 资源提供了更大的调度灵活性，因为可以将长时间运行的作业分解为多个较短的作业。 CropDL 项目的前提是，高效且自动化的应用程序级检查点和重启对于促进使用共享 HPC 集群进行长期运行的 ML 训练任务至关重要，从而大大增加能够成功训练大型 ML 模型的研究人员数量各种应用程序。该项目还在多个方面为教育和多样性做出了贡献，例如，1）引入课程（或课程材料）以引起人们对计算机系统本科生和研究生教育中与机器学习相关的工作量的关注； 2) 将本项目的研究任务与大学的协同研究计划相结合，以增加妇女和代表性不足的少数群体的参与； 3) 支持和培训博士生的研究，为与新兴 ML 工作负载相关的系统和网络基础设施研究创造动力，并推广将这些工作负载的特性与现代 HPC 硬件的复杂性相结合的综合研究。CropDL 的总体目标是支持深度学习应用程序的应用程序级检查点/重启，以实现更好的弹性、更快的平均完成时间和更高的资源利用率。特别是，深度学习工作负载的几个属性（与科学计算相比）为检查点创建了不同的机会和挑战：1）并行执行期间有限的通信模式，这可以实现高效的协调检查点，2）许多独特的检查点压缩机会，并可能采用不协调的检查点，以及 3) 可延展的执行，其中可以从不同数量的节点重新启动。基于这一观察，该项目的第一个方向是利用检查点期间要训练的 DNN 模型的属性。这包括在各种并行模型下为深度学习应用程序提供异步版本化检查点，以及用于减少检查点量的基于内容的数据缩减（压缩和稀疏化）技术。第二个研究方向侧重于在检查点时有效地使用当前和即将推出的 HPC 系统资源。它将 DL 应用程序的任务、数据和 I/O 需求制定为 DAG 表示形式，并开发调度它们的方法。它还通过新兴 I/O 平台支持深度学习应用程序的高效 I/O。最后一个方向是通过基于深度学习工作负载计算图的编译系统自动设置检查点。所有这些努力都考虑了 DNN 的各种并行化方案，即数据、模型和/或流水线并行性。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。