Collaborative Research: Elements: VLCC-States: Versioned Lineage-Driven Checkpointing of Composable States

协作研究：元素：VLCC-States：可组合状态的版本化谱系驱动检查点

• 批准号: 2411387
• 负责人: M Mustafa Rafique
• 金额: $ 30万
• 依托单位: Rochester Institute of Tech
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-10-01 至 2027-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2411387&HistoricalAwards=false
• 关键词: Collaborative; Research; Elements; VLCC; States

Checkpointing is a fundamental pattern used by a variety of scientific applications at both small and large computing scales. Widely adopted for resilience purposes by long-running applications (i.e., checkpoint-restart), it has seen an explosion of additional use cases that directly help applications progress faster and reduce time-to-solution even in the absence of failures: adjoint computations (essential in financial modeling, weather prediction, computational fluid dynamics, seismic imaging, and control theory) need to capture a history of checkpoints in a forward pass, which are then revisited in a backward pass. Training artificial intelligence models, increasingly used by scientific applications, often results in trajectories that do not lead to convergence or may lead to undesirable patterns, prompting the need to backtrack to an earlier checkpoint of the learning model to try an alternative. Transfer learning and fine-tuning using a previous checkpoint of a learning model can be used to adapt the training more quickly, avoiding expensive training from scratch. Many other use cases are important in scientific computing: suspend-resume (e.g., to preempt a long-running job in favor of a higher priority job), migration (checkpoint on one machine, restart on another), debugging (replay a problematic code region to reproduce errors without starting from scratch), and reproducibility (checkpoint and compare intermediate data during repeated runs). Despite broad applicability, current state-of-the-art solutions lack the flexibility, performance, and scalability needed to address these scenarios efficiently. The Versioned Lineage-Driven Checkpointing of Composable States (VLCC-States) project aims to fill this gap. It will streamline the development and use of checkpointing patterns for scientific applications, which simplifies and improves the reusability of integration efforts across different communities, improves awareness of the multitude of checkpointing scenarios, reduces development effort and cost, and enables flexible customization to extract the best performance and scalability for the desired application scenario.VLCC-States provides technical innovation in three areas. First, it introduces composable providers of intermediate states, which hide the complexity of capturing and assembling checkpoints of distributed data structures and their transformations across different modules and programming languages while optimizing their layout to eliminate redundancies, reduce sizes, and improve performance. Second, it provides multi-level co-optimized caching and prefetching techniques, which enable scalable management of the life cycle of checkpoints for interleavings of capture and reuse operations on heterogeneous storage stacks under concurrency. Third, it develops specialized checkpointing tools for large Artificial Intelligence models, with a focus on integration with PyTorch and DeepSpeed, to enable users to transparently take advantage of high-performance and scalable checkpointing using a familiar API. This project will engage partners in industry and national research laboratories to co-design VLCC-States, tune its capabilities, and evaluate its implementation. This project will undertake educational and broadening participation activities to improve community awareness and understanding of challenges in scientific data management.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

检查点是小型和大型计算量表的各种科学应用都使用的基本模式。通过长期运行的应用（即检查点位均值）广泛用于弹性目的，它看到了其他用例的爆炸爆炸，这些案例直接有助于应用程序更快地进展并减少更快的进展，甚至在没有故障的情况下减少时间到期：邻居计算：在财务建模，天气模型，天气预测，计算动力学，seist seist toction the Inception中的经济模型中必不可少） 经过。培训人工智能模型越来越多地由科学应用使用，通常会导致轨迹不会导致收敛或可能导致不良模式，从而促使需要回到早期的学习模型检查站以尝试替代方案。可以使用以前的学习模型检查站进行转移学习和微调来更快地调整培训，从而避免昂贵的培训。许多其他用例在科学计算中很重要：悬浮库（例如，要抢占长期运行的工作，支持更高的优先级工作），迁移（一台机器上的检查点，重新启动另一台机器），调试（重现有问题的代码区域以重现错误，而无需从scratch开始），并且在不scratch的情况下进行了重复和重复的数据，并在重复的数据中进行了重复数据）。尽管有广泛的适用性，但当前的最新解决方案缺乏有效解决这些方案所需的灵活性，性能和可伸缩性。可组合状态（VLCC-States）项目版本的谱系驱动的检查点旨在填补这一空白。它将简化用于科学应用的检查点模式的开发和使用，从而简化并改善了不同社区整合工作的可重复性，提高了对多种检查点场景的认识，减少开发工作和成本，并启用灵活的自定义，以提取所需的应用程序场景的最佳性能和可扩展性。首先，它引入了中间状态的可组合提供者，这些提供商隐藏了分布式数据结构的捕获和组装检查点的复杂性及其在不同模块和编程语言上的转换，同时优化其布局以消除冗余，减少尺寸，并提高性能。其次，它提供了多级合作的缓存和预取技术，该技术可以使检查点的生命周期可扩展管理，以使捕获和重复使用操作在同一同时进行异质储存堆栈上进行交织。第三，它为大型人工智能模型开发了专门的检查点工具，重点是与Pytorch和DeepSpeed集成，以使用户能够使用熟悉的API透明地利用高性能和可扩展的检查点。该项目将与行业和国家研究实验室的合作伙伴互动，以共同设计VLCC国家，调整其能力并评估其实施。该项目将开展教育和扩大参与活动，以提高社区的意识和对科学数据管理中挑战的理解。该奖项反映了NSF的法定使命，并被认为是值得通过基金会的知识分子优点和更广泛的影响评估的评估来支持的。