Collaborative Research: OAC Core: Simulation-driven runtime resource management for distributed workflow applications

协作研究：OAC Core：分布式工作流应用程序的模拟驱动的运行时资源管理

• 批准号: 2106059
• 负责人: Henri Casanova
• 金额: $ 28万
• 依托单位: University of Hawaii
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2106059&HistoricalAwards=false
• 关键词: Collaborative; Research; OAC; Core; Simulation

Many scientific breakthroughs in domains such as health, climate modeling, particle physics, seismology, etc., can only be achieved by performing complex processing of vast amounts of data. This processing is automated by software systems that use the compute, storage, and network hardware provided by the cyberinfrastructure. In addition to automation, a key objective of these systems is the efficient use of the resources as measured by cost and energy usage, while making the processing as fast as possible or as needed. To this end, these systems must make decisions regarding which resources should be used to do what and when. Many such systems are used in production today and make such decisions. Yet making good, let alone best, decisions is still an open research challenge. Theoretical research has proposed solutions that are difficult to put into practice, and practical solutions are known to not make good decisions, or at least not consistently so. However, both theory and practice follow the same basic philosophy: make decisions by reasoning about known information on what needs to be computed and on what hardware resources are available. This philosophy has shown its limits, so this project adopts a radically different approach. The key idea is to repeatedly execute fast, computationally inexpensive simulations of the application execution in order to evaluate large sets of potential resource management decisions and automatically select the most desirable ones. The benefits of this approach will be demonstrated for several software systems used to support scientific applications that are critical for the development and sustainability of society.Software systems are used to run scientific applications on advanced cyberinfrastructure. These systems automate application execution, and make resource management decision along several axes including selecting and provisioning (virtualized) hardware, picking application configuration options, and scheduling application activities in time and space. Their objective is to optimize both application performance and also a set of resource usage efficiency metrics that include monetary and energy costs. Consequently, the resource management decision space is enormous, and making good decisions is a steep challenge that has been the subject of countless efforts, both from theoreticians and practitioners. However, the challenge is far from being solved: theoreticians produce solutions that are rarely used by practitioners, and conversely practitioners implement solutions that may be highly sub-optimal because they not informed by theory. This project resolves this disconnect by obviating the need for developing effective resource management strategies. The key idea is to use online simulations to search the resource management decision space rapidly at runtime. Large numbers of fast simulations of the application's execution are executed throughout that very execution, so as to evaluate many potential resource management options and automatically select desirable ones. This approach thus shifts the overall problem from the design of complex resource management algorithms to the enumeration of many resource management decisions. The transformation of resource management practice in cyberinfrastructure systems not only renders the resource management problem tractable but also unlocks previously out-of-reach resource management decisions. The benefits of this transformation will be demonstrated for a critical class of production systems and applications, specifically Workflow Management Systems and the scientific applications they support.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

健康、气候建模、粒子物理学、地震学等领域的许多科学突破只能通过对大量数据进行复杂处理才能实现。 此处理由使用网络基础设施提供的计算、存储和网络硬件的软件系统自动完成。 除了自动化之外，这些系统的一个关键目标是通过成本和能源使用来衡量资源的有效利用，同时尽可能快地或根据需要进行处理。为此，这些系统必须决定应使用哪些资源以及何时执行哪些操作。 如今，许多此类系统已用于生产并做出此类决策。然而，做出好的决策，更不用说最好的决策，仍然是一个开放的研究挑战。理论研究提出了难以付诸实践的解决方案，而众所周知，实际的解决方案并不能做出好的决策，或者至少不能始终如一地做出好的决策。 然而，理论和实践都遵循相同的基本理念：通过推理有关需要计算的内容和可用的硬件资源的已知信息来做出决策。这种理念已经显示出其局限性，因此该项目采用了完全不同的方法。 关键思想是重复执行快速、计算成本低廉的应用程序执行模拟，以便评估大量潜在资源管理决策并自动选择最理想的决策。这种方法的好处将在用于支持对社会发展和可持续发展至关重要的科学应用程序的多个软件系统中得到证明。软件系统用于在先进的网络基础设施上运行科学应用程序。 这些系统自动执行应用程序，并沿着多个轴做出资源管理决策，包括选择和配置（虚拟化）硬件、选择应用程序配置选项以及在时间和空间上安排应用程序活动。他们的目标是优化应用程序性能以及一组资源使用效率指标，其中包括货币和能源成本。因此，资源管理决策空间巨大，做出正确的决策是一项艰巨的挑战，理论家和实践者都为此付出了无数努力。 然而，挑战还远未得到解决：理论家提出的解决方案很少被实践者使用，相反，实践者实施的解决方案可能非常次优，因为他们没有得到理论的指导。该项目通过消除制定有效资源管理策略的需要来解决这种脱节。 关键思想是使用在线模拟在运行时快速搜索资源管理决策空间。在整个执行过程中执行应用程序执行的大量快速模拟，以便评估许多潜在的资源管理选项并自动选择所需的选项。 因此，这种方法将整体问题从复杂资源管理算法的设计转移到许多资源管理决策的枚举。网络基础设施系统中资源管理实践的转变不仅使资源管理问题变得易于处理，而且还释放了以前无法实现的资源管理决策。 这种转变的好处将在关键类别的生产系统和应用程序中得到证明，特别是工作流程管理系统及其支持的科学应用程序。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优点和技术进行评估，被认为值得支持。更广泛的影响审查标准。

项目成果

WfCommons: Data Collection and Runtime Experiments using Multiple Workflow Systems

WfCommons：使用多个工作流系统的数据收集和运行时实验

• 发表时间: 2023
• 期刊: 1st IEEE International Workshop on Workflows in Distributed Environments (WiDE
• 作者: Casanova, H.;K. Berney, K.;Chastel, S.;R. Ferreira da Silva, Rafael
• 通讯作者: R. Ferreira da Silva, Rafael

On the Feasibility of Simulation-driven Portfolio Scheduling for Cyberinfrastructure Runtime Systems

网络基础设施运行时系统仿真驱动组合调度的可行性

• 发表时间: 2022
• 期刊: 25th Workshop on Job Scheduling Strategies for Parallel Processing
• 作者: Casanova. H.;Wong Y. C.;Pottier, L.;Ferreira da Silva, R.
• 通讯作者: Ferreira da Silva, R.