Collaborative Research: Elements: VLCC-States: Versioned Lineage-Driven Checkpointing of Composable States

协作研究：元素：VLCC-States：可组合状态的版本化谱系驱动检查点

• 批准号: 2411387
• 负责人: M Mustafa Rafique
• 金额: $ 30万
• 依托单位: Rochester Institute of Tech
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-10-01 至 2027-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2411387&HistoricalAwards=false
• 关键词: Collaborative; Research; Elements; VLCC; States

Checkpointing is a fundamental pattern used by a variety of scientific applications at both small and large computing scales. Widely adopted for resilience purposes by long-running applications (i.e., checkpoint-restart), it has seen an explosion of additional use cases that directly help applications progress faster and reduce time-to-solution even in the absence of failures: adjoint computations (essential in financial modeling, weather prediction, computational fluid dynamics, seismic imaging, and control theory) need to capture a history of checkpoints in a forward pass, which are then revisited in a backward pass. Training artificial intelligence models, increasingly used by scientific applications, often results in trajectories that do not lead to convergence or may lead to undesirable patterns, prompting the need to backtrack to an earlier checkpoint of the learning model to try an alternative. Transfer learning and fine-tuning using a previous checkpoint of a learning model can be used to adapt the training more quickly, avoiding expensive training from scratch. Many other use cases are important in scientific computing: suspend-resume (e.g., to preempt a long-running job in favor of a higher priority job), migration (checkpoint on one machine, restart on another), debugging (replay a problematic code region to reproduce errors without starting from scratch), and reproducibility (checkpoint and compare intermediate data during repeated runs). Despite broad applicability, current state-of-the-art solutions lack the flexibility, performance, and scalability needed to address these scenarios efficiently. The Versioned Lineage-Driven Checkpointing of Composable States (VLCC-States) project aims to fill this gap. It will streamline the development and use of checkpointing patterns for scientific applications, which simplifies and improves the reusability of integration efforts across different communities, improves awareness of the multitude of checkpointing scenarios, reduces development effort and cost, and enables flexible customization to extract the best performance and scalability for the desired application scenario.VLCC-States provides technical innovation in three areas. First, it introduces composable providers of intermediate states, which hide the complexity of capturing and assembling checkpoints of distributed data structures and their transformations across different modules and programming languages while optimizing their layout to eliminate redundancies, reduce sizes, and improve performance. Second, it provides multi-level co-optimized caching and prefetching techniques, which enable scalable management of the life cycle of checkpoints for interleavings of capture and reuse operations on heterogeneous storage stacks under concurrency. Third, it develops specialized checkpointing tools for large Artificial Intelligence models, with a focus on integration with PyTorch and DeepSpeed, to enable users to transparently take advantage of high-performance and scalable checkpointing using a familiar API. This project will engage partners in industry and national research laboratories to co-design VLCC-States, tune its capabilities, and evaluate its implementation. This project will undertake educational and broadening participation activities to improve community awareness and understanding of challenges in scientific data management.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

检查点是小型和大型计算规模的各种科学应用程序使用的基本模式。它被长期运行的应用程序（即检查点重启）广泛用于弹性目的，它已经看到了额外用例的爆炸式增长，即使在没有故障的情况下，也可以直接帮助应用程序更快地进展并缩短解决方案的时间：伴随计算（在金融建模、天气预报、计算流体动力学、地震成像和控制理论中至关重要）需要在前向传递中捕获检查点的历史记录，然后在后向传递中重新访问这些历史记录。训练人工智能模型越来越多地被科学应用所使用，通常会产生不会导致收敛或可能导致不良模式的轨迹，从而促使需要回溯到学习模型的早期检查点以尝试替代方案。使用学习模型的先前检查点进行迁移学习和微调可以更快地适应训练，从而避免从头开始进行昂贵的训练。许多其他用例在科学计算中也很重要：挂起-恢复（例如，抢占长时间运行的作业以支持更高优先级的作业）、迁移（在一台机器上设置检查点，在另一台机器上重新启动）、调试（重放有问题的代码）区域来重现错误而无需从头开始）和重现性（在重复运行期间检查点和比较中间数据）。尽管具有广泛的适用性，但当前最先进的解决方案缺乏有效解决这些场景所需的灵活性、性能和可扩展性。版本化沿袭驱动的可组合状态检查点 (VLCC-States) 项目旨在填补这一空白。它将简化科学应用程序检查点模式的开发和使用，从而简化和提高不同社区之间集成工作的可重用性，提高对多种检查点场景的认识，减少开发工作量和成本，并实现灵活定制以提取最佳性能VLCC-States 在三个领域提供技术创新。首先，它引入了可组合的中间状态提供程序，隐藏了捕获和组装分布式数据结构检查点及其跨不同模块和编程语言的转换的复杂性，同时优化其布局以消除冗余、减小大小并提高性能。其次，它提供了多级协同优化的缓存和预取技术，可以对检查点的生命周期进行可扩展的管理，以实现并发下异构存储堆栈上交错的捕获和重用操作。第三，它为大型人工智能模型开发专门的检查点工具，重点是与 PyTorch 和 DeepSpeed 的集成，使用户能够使用熟悉的 API 透明地利用高性能和可扩展的检查点。该项目将吸引行业和国家研究实验室的合作伙伴共同设计 VLCC-States、调整其能力并评估其实施情况。该项目将开展教育和扩大参与活动，以提高社区对科学数据管理挑战的认识和理解。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。