CSR: Small: Accelerating Data Intensive Scientific Workflows with Consistency Contracts

CSR：小：通过一致性合同加速数据密集型科学工作流程

• 批准号: 2317556
• 负责人: Douglas Thain
• 金额: $ 59.97万
• 依托单位: University of Notre Dame
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2317556&HistoricalAwards=false
• 关键词: CSR; Small; Accelerating; Data; Intensive; CSR; Small; Accelerating; Data; Intensive

Advanced discovery in scientific computing increasingly depends upon the successful execution of complex workflows that combine multiple applications together to run in concert on a large high performance cluster. A widespread challenge in this setting is the performance of the shared parallel filesystem. Because each file system interaction has very different needs in terms of performance and consistency, the filesystem is obliged to follow the most conservative approach to handle the worst case. As a result, peak performance is rarely achieved. We propose "consistency contracts" as the solution to this problem. This novel approach requires the workflow as a whole to declare its intended uses of the file system at the start of each execution, allowing the runtime system to perform a variety of optimizations. This project will evaluate the concept of consistency contracts by constructing an experimental system (Pledge) that enables and enforces contracts on existing data intensive workflows, with minimal disruption to current practice. We hypothesize that this approach will significantly improve performance for data intensive scientific applications running on high performance clusters,and has the potential to be more widely applied. Our focus on consistency results from our observation that today's shared parallel filesystem is asked to fill multiple roles: moving large files, delivering complex software trees, providing buffers between tasks, and providing synchronization between tasks. Current filesystems provide the most conservative sequential consistency to handle the worst case. Rather than depend upon the shared filesystem to perform last-minute runtime arbitration of every individual filesystem operation, we argue that the workflow as a whole should declare its intentions for the duration of the execution, indicating the paths, access modes, and consistency requirements needed for the entire workflow run. With a contract in hand, the runtime system can then perform a variety of optimizations that exploit the internal storage and I/O capacity of the cluster as a whole, for example utilizing a streamlined approach for read-only access. We hypothesize that workflow-level consistency management will yield higher effective I/O bandwidth and transaction rates than strict global consistency management for data intensive scientific applications running on high performance clusters. These improved I/O rates will translate into faster end-to-end runtimes and fewer unexpected performance failures for end users and system administrators.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

科学计算中的高级发现越来越多地取决于成功执行复杂的工作流，这些工作流将多个应用程序结合在一起，以在大型高性能集群上共同运行。在这种情况下，广泛的挑战是共享并行文件系统的性能。 由于每个文件系统交互在性能和一致性方面都有不同的需求，因此文件系统必须遵循最保守的方法来处理最坏情况。 结果，很少实现峰值性能。我们建议“一致性合同”作为解决此问题的解决方案。 这种新颖的方法要求整个工作流程在每个执行开始时声明其预期的文件系统用途，从而允许运行时系统执行各种优化。该项目将通过构建实验系统（质疑）来评估一致性合同的概念，该系统能够在现有数据密集型工作流程上实现和执行合同，而对当前实践的中断最小。我们假设这种方法将显着提高在高性能集群上运行的数据密集型科学应用程序的性能，并且有可能更广泛地应用。我们关注的关注一致性结果是，我们的观察结果是，当今的共享并行文件系统被要求填补多个角色：移动大文件，提供复杂的软件树，提供任务之间的缓冲区以及在任务之间提供同步。当前文件系统提供了处理最坏情况的最保守的顺序一致性。 与其依靠共享文件系统执行每个单个文件系统操作的最后一刻运行时仲裁，我们认为整个工作流程应在执行期间声明其意图，表明整个工作流程运行所需的路径，访问模式以及一致性要求。 借助合同，运行时系统可以执行各种优化，以利用整个集群的内部存储和I/O容量，例如使用流线方法来读取仅读取访问。我们假设工作流程级的一致性管理将产生比严格的全球一致性管理的有效I/O带宽和交易率更高的数据密集型科学应用程序，以高性能集群运行。这些提高的I/O利率将转化为更快的端到端运行时间，对于最终用户和系统管理员而言，意外的性能失败较少。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子优点和更广泛影响的评估评估来审查标准的。