Collaborative Research: OAC Core: Improving Utilization of High-Performance Computing Systems via Intelligent Co-scheduling

合作研究：OAC Core：通过智能协同调度提高高性能计算系统的利用率

• 批准号: 2103511
• 负责人: David Lowenthal
• 金额: $ 25.03万
• 依托单位: University of Arizona
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2103511&HistoricalAwards=false
• 关键词: Collaborative; Research; OAC; Core; Improving

This project is aimed at increasing efficiency of high-performance computing systems by scheduling multiple jobs on the same set of nodes in a system, generally called co-scheduling. This is a break from current practice in which nodes are dedicated to one job at a time, which results in predictable execution time but inefficient use of system resources. To make this practical, the project will develop analyses to determine how to carry out co-scheduling such that overall system efficiency is improved while the performance impact on individual applications is minimized. In particular, the goal is to co-schedule jobs that can co-exist without contending for similar resources on the nodes. The work done in this project will help achieve better efficiency on high-performance systems, which will impact application domains such as climate/weather, renewable energy, and national security. The work will be implemented and validated on systems at Lawrence Livermore and Sandia National Laboratories and then transitioned into software that will be used at these national laboratories. The project will also have an impact on education by integrating the techniques in this research into courses covering parallel and distributed computing at the PIs' institutions. In addition, the project will take place at two Hispanic-serving institutions, and the PIs have a history of advising under-represented students; the project will broaden participation in computing by recruiting Hispanic undergraduates to work on the project and sending them to national laboratories for internships.The long-standing abstraction at high-end computing facilities is one of a submitted job being allocated a set of dedicated nodes. However, this makes systems much less efficient, as there are more per-node resources that will often be used inefficiently. In addition, the demand for high-end systems is increasing and dedicating nodes to jobs can increase job turnaround time and decrease overall system throughput. One way to address this problem is for supercomputer centers to break from the current common practice of assigning each job a private, isolated portion of a supercomputer. The intellectual merit of the project is three-fold. First, novel profile analyses will be developed that will reveal the effects on jobs due to sharing nodes. Second, novel statistical projection techniques will be developed that predict scaling behavior of jobs that are utilizing shared nodes. Third, new job-level scheduling techniques will be designed that use the interference analysis and projections to choose a set of shared nodes that will lead to good job turnaround time and maximize system throughput. The broader impact of the project is multifold. This project will help achieve better efficiency on high-performance systems, which will benefit a broad range of applications that includes climate/weather prediction, nuclear energy, and national security. Through a long-standing collaboration with both Lawrence Livermore and Sandia National Laboratories, the PIs will implement and validate the techniques on LLNL and SNL systems as well as transition the techniques into future resource managers at the national laboratories. In addition, both PIs will broaden participation in computing by recruiting Hispanic undergraduates to work on the project and sending them to national labs for internships.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目旨在通过在系统中的同一组节点上调度多个作业（通常称为协同调度）来提高高性能计算系统的效率。这打破了当前的做法，即节点一次专用于一项作业，从而导致执行时间可预测，但系统资源的使用效率低下。为了使其切实可行，该项目将进行分析，以确定如何进行联合调度，从而提高整体系统效率，同时最大限度地减少对单个应用程序的性能影响。特别是，目标是共同调度可以共存的作业，而无需争用节点上的类似资源。 该项目所做的工作将有助于提高高性能系统的效率，这将影响气候/天气、可再生能源和国家安全等应用领域。这项工作将在劳伦斯利弗莫尔和桑迪亚国家实验室的系统上实施和验证，然后转化为将在这些国家实验室使用的软件。该项目还将通过将本研究中的技术整合到 PI 机构的并行和分布式计算课程中来对教育产生影响。此外，该项目将在两个为西班牙裔服务的机构进行，PI 有着为代表性不足的学生提供建议的历史；该项目将通过招募西班牙裔本科生参与该项目并将他们送到国家实验室实习来扩大对计算的参与。高端计算设施的长期抽象是提交的工作被分配一组专用节点。然而，这使得系统效率大大降低，因为每个节点的资源更多，而这些资源的使用效率往往很低。此外，对高端系统的需求不断增加，将节点专用于作业可能会增加作业周转时间并降低整体系统吞吐量。 解决这个问题的一种方法是超级计算机中心打破当前为每个作业分配超级计算机的私有、隔离部分的常见做法。 该项目的智力价值有三个方面。首先，将开发新颖的概况分析，揭示共享节点对工作的影响。其次，将开发新的统计投影技术来预测利用共享节点的作业的扩展行为。第三，将设计新的作业级调度技术，使用干扰分析和预测来选择一组共享节点，这将导致良好的作业周转时间并最大化系统吞吐量。该项目的更广泛影响是多方面的。 该项目将有助于提高高性能系统的效率，这将有利于包括气候/天气预报、核能和国家安全在内的广泛应用。 通过与劳伦斯利弗莫尔和桑迪亚国家实验室的长期合作，PI 将在 LLNL 和 SNL 系统上实施和验证这些技术，并将这些技术转移到国家实验室未来的资源管理器中。此外，两位 PI 将通过招募西班牙裔本科生参与该项目并将他们送到国家实验室实习来扩大对计算的参与。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优势和更广泛的评估进行评估，被认为值得支持。影响审查标准。

项目成果

Evaluating the Potential of Coscheduling on High-Performance Computing Systems

评估高性能计算系统协同调度的潜力

• 发表时间: 2023-04
• 期刊: Workshop on Job Scheduling Strategies for Parallel Processing
• 作者: Hall, Jason;Lathi, Arjun;Lowenthal, David K;Patki, Tapasya
• 通讯作者: Patki, Tapasya