Collaborative Research: SI2-SSE: WRENCH: A Simulation Workbench for Scientific Worflow Users, Developers, and Researchers

协作研究：SI2-SSE：WRENCH：面向科学 Worflow 用户、开发人员和研究人员的模拟工作台

• 批准号: 1642369
• 负责人: Henri Casanova
• 金额: $ 25.8万
• 依托单位: University of Hawaii
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-01-01 至 2019-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1642369&HistoricalAwards=false
• 关键词: Collaborative; Research; SI2; SSE; WRENCH

Many scientific breakthroughs can only be achieved by performing complex processing of vast amounts of data efficiently. In domains as crucial to our society as climate modeling, oceanography, particle physics, seismology, or computational biology (and in fact in most fields of physics, chemistry, and biology today), scientists nowadays routinely define "scientific workflows". These workflows are complex descriptions of scientific processes as data and inter-dependent computations on these data. When executed, typically with great expenses of computing, storage, and networking hardware, these workflows can produce groundbreaking results. A famous and recent example is the workflow that was used as part of the LIGO project to confirm the first detection of gravitational waves from colliding black holes. Scientific workflows are mainstays in today's science. Their efficient execution (in terms of speed, reliability, and cost) is thus crucial. This project seeks to provide a software framework, called WRENCH (Workflow Simulation Workbench), that will make it possible to simulate large-scale hypothetical scenarios quickly and accurately on a single computer, obviating the need for expensive and time-consuming trial and error experiments. WRENCH potentially enables scientists to make quick and informed choices when executing their workflows, software developers to implement more efficient software infrastructures to support workflows, and researchers to develop novel efficient algorithms to be embedded within these software infrastructures. In addition, WRENCH makes it possible to bring scientific workflow content into undergraduate and graduate computer science curricula. This is because meaningful knowledge can be gained by students using a single computer and the WRENCH software stack, making such learning possible even at institutions without access to high-end computing infrastructures, such as many non-Ph.D.-granting and minority-serving institutions. As a result, this work will contribute to producing computer science graduates better equipped to take an active role in the advancing of science. Due to its potentially transformative impact on scientific workflow usage, development, research, and education, this project promises to promote the progress of science across virtually all its fields, ultimately resulting in broad and numerous benefits to our society.Scientific workflows have become mainstream for conducting large-scale scientific research. As a result, many workflow applications and Workflow Management Systems (WMSs) have been developed as part of the cyberinfrastructure to allow scientists to execute their applications seamlessly on a range of distributed platforms. In spite of many success stories, building large-scale workflows and orchestrating their executions efficiently (in terms of performance, reliability, and cost) remains a challenge given the complexity of the workflows themselves and the complexity of the underlying execution platforms. A fundamental necessary next step is the establishment of a solid "experimental science" approach for future workflow technology development. Such an approach is useful for scientists who need to design workflows and pick execution platforms, for WMS developers who need to compare alternate design and implementation options, and for researchers who need to develop novel decision-making algorithms to be implemented as part of WMSs. The broad objective of this work is to provide foundational software, the Workflow Simulation Workbench (WRENCH), upon which to develop the above experimental science approach. Capitalizing on recent advances in distributed application and platform simulation technology, WRENCH makes it possible to (i) quickly prototype workflow, WMS implementations, and decision-making algorithms; and (ii) evaluate/compare alternative options scalably and accurately for arbitrary, and often hypothetical, experimental scenarios. This project will define a generic and foundational software architecture, that is informed by current state-of-the-art WMS designs and planned future designs. The implementation of the components in this architecture when taken together form a generic "scientific instrument" that can be used by workflow users, developers, and researchers. This scientific instrument will be instantiated for several real-world WMSs and used for a range of real-world workflow applications. In a particular case-study, it will be used with a popular WMS (Pegasus) to revisit published results and scheduling algorithms in the area of workflow planning optimizations. The objective is to demonstrate the benefit of using an experimental science approach for WMS research. Another impact of this project is that it makes it possible to include scientific workflow content pervasively in undergraduate and graduate computer science curricula, even for students without any access to computing infrastructure, by defining meaningful pedagogic activities that only require a computer and the WRENCH software stack. This educational impact will be demonstrated in the classroom in both undergraduate and graduate courses at our institutions.

许多科学的突破只能通过有效地进行大量数据进行复杂的处理来实现。 在对我们社会至关重要的领域中，如气候建模，海洋学，粒子物理学，地震学或计算生物学（实际上在当今的大多数物理，化学和生物学领域），如今的科学家通常定义“科学工作流”。这些工作流是对这些数据的数据和相互依赖的计算的科学过程的复杂描述。当执行时，通常以计算，存储和网络硬件的巨大费用，这些工​​作流可以产生开创性的结果。一个著名且最近的例子是用作Ligo项目的一部分的工作流程，以确认从碰撞黑洞中首次发现引力波。科学工作流程是当今科学的支柱。因此，它们的有效执行（就速度，可靠性和成本而言）至关重要。该项目旨在提供一个称为扳手（工作流仿真工作台）的软件框架，这将使在一台计算机上快速，准确地模拟大规模的假设场景成为可能，从而避免了需要昂贵且耗时的试验和错误实验的需求。扳手有可能使科学家在执行工作流程时能够快速而明智的选择，软件开发人员实施更有效的软件基础架构来支持工作流程，并研究人员开发新的有效算法，以嵌入这些软件基础架构中。 此外，扳手可以将科学工作流程内容带入本科和研究生计算机科学课程。这是因为使用一台计算机和扳手软件堆栈的学生可以获得有意义的知识，即使在机构中也无法访问高端计算基础架构，例如许多非PH.D.授予和少数族裔服务机构，也可以实现此类学习。结果，这项工作将有助于生产计算机科学毕业生更好地在科学发展中发挥积极作用。 由于其对科学工作流的使用，发展，研究和教育的潜在变革影响，该项目有望促进科学在几乎所有领域的进步，最终导致了我们社会的广泛和众多好处。ScientificWorkfrows已成为进行大规模科学研究的主流。 结果，许多工作流程应用程序和工作流程管理系统（WMS）是Cyber​​infradstructure的一部分开发的，允许科学家在一系列分布式平台上无缝执行其应用程序。 尽管有许多成功案例，但考虑到工作流程本身的复杂性以及基本执行平台的复杂性，建立大规模的工作流并有效地策划执行（就性能，可靠性和成本而言）仍然是一个挑战。 下一步的基本必要是为未来的工作流技术开发建立了坚实的“实验科学”方法。这种方法对于需要设计工作流程和选择执行平台的科学家，对于需要比较替代设计和实施选项的WMS开发人员以及需要开发新型决策算法以作为WMS的一部分实现的研究人员。 这项工作的广泛目标是提供基础软件，即工作流仿真工作台（扳手），以开发上述实验科学方法。 利用分布式应用程序和平台模拟技术的最新进展，扳手使（i）可以快速原型工作流程，WMS实现和决策算法； （ii）评估/比较替代选项，以准确，准确地进行任意，通常是假设的实验场景。 该项目将定义一个通用和基础软件体系结构，该架构由当前最新的WMS设计和计划的未来设计所告知。 组件在此架构中的实现形成一种通用的“科学仪器”，可以由工作流的用户，开发人员和研究人员使用。 该科学仪器将用于几个现实世界中的WMS，并用于一系列现实的工作流程应用程序。在特定的案例研究中，它将与流行的WMS（PEGASUS）一起使用，以重新访问工作流计划优化领域中发布的结果和调度算法。目的是证明使用实验科学方法进行WMS研究的好处。 该项目的另一个影响是，它可以通过定义仅需要计算机和扳手软件堆栈的有意义的教学活动来定义有意义的教学活动，从而使本科和计算机科学课程在本科和计算机科学课程中普遍包含科学工作流内容。在我们机构的本科和研究生课程中，将在教室中展示这种教育影响。

项目成果

Checkpointing Workflows for Fail-Stop Errors

• DOI: 10.1109/cluster.2017.14
• 发表时间: 2017-09
• 期刊: 2017 IEEE International Conference on Cluster Computing (CLUSTER)
• 作者: Li Han;Louis-Claude Canon;H. Casanova;Y. Robert;F. Vivien

WRENCH: Workflow Management System Simulation Workbench

WRENCH：工作流程管理系统模拟工作台

• 发表时间: 2018
• 期刊: Workshop on Workflows in Support of Large-Scale Science
• 作者: Casanova, H;Pandey, S;Oeth, J.;Tanaka, R.;Suter, F.;Ferreira da Silva, R.
• 通讯作者: Ferreira da Silva, R.

Computing the expected makespan of task graphs in the presence of silent errors

在存在无提示错误的情况下计算任务图的预期完工时间

• DOI: 10.1016/j.parco.2018.03.004
• 发表时间: 2018
• 期刊: Parallel Computing
• 影响因子: 1.4
• 作者: Casanova, Henri;Herrmann, Julien;Robert, Yves
• 通讯作者: Robert, Yves

SMPI Courseware: Teaching Distributed-Memory Computing with MPI in Simulation

SMPI 课件：在仿真中使用 MPI 教授分布式内存计算

• 发表时间: 2018
• 期刊: Proceedings of EduHPC
• 作者: Casanova, H.;Quinson, M.;Legrand, A.;Suter, F.
• 通讯作者: Suter, F.

Accurately Simulating Energy Consumption of I/O-Intensive Scientific Workflows

• DOI: 10.1007/978-3-030-22734-0_11
• 发表时间: 2019-06
• 作者: Rafael Ferreira da Silva;Anne-Cécile Orgerie;H. Casanova;Ryan Tanaka;E. Deelman;F. Suter