CSR: Small: Accelerating Data Intensive Scientific Workflows with Consistency Contracts

CSR：小：通过一致性合同加速数据密集型科学工作流程

• 批准号: 2317556
• 负责人: Douglas Thain
• 金额: $ 59.97万
• 依托单位: University of Notre Dame
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2317556&HistoricalAwards=false
• 关键词: CSR; Small; Accelerating; Data; Intensive

Advanced discovery in scientific computing increasingly depends upon the successful execution of complex workflows that combine multiple applications together to run in concert on a large high performance cluster. A widespread challenge in this setting is the performance of the shared parallel filesystem. Because each file system interaction has very different needs in terms of performance and consistency, the filesystem is obliged to follow the most conservative approach to handle the worst case. As a result, peak performance is rarely achieved. We propose "consistency contracts" as the solution to this problem. This novel approach requires the workflow as a whole to declare its intended uses of the file system at the start of each execution, allowing the runtime system to perform a variety of optimizations. This project will evaluate the concept of consistency contracts by constructing an experimental system (Pledge) that enables and enforces contracts on existing data intensive workflows, with minimal disruption to current practice. We hypothesize that this approach will significantly improve performance for data intensive scientific applications running on high performance clusters,and has the potential to be more widely applied. Our focus on consistency results from our observation that today's shared parallel filesystem is asked to fill multiple roles: moving large files, delivering complex software trees, providing buffers between tasks, and providing synchronization between tasks. Current filesystems provide the most conservative sequential consistency to handle the worst case. Rather than depend upon the shared filesystem to perform last-minute runtime arbitration of every individual filesystem operation, we argue that the workflow as a whole should declare its intentions for the duration of the execution, indicating the paths, access modes, and consistency requirements needed for the entire workflow run. With a contract in hand, the runtime system can then perform a variety of optimizations that exploit the internal storage and I/O capacity of the cluster as a whole, for example utilizing a streamlined approach for read-only access. We hypothesize that workflow-level consistency management will yield higher effective I/O bandwidth and transaction rates than strict global consistency management for data intensive scientific applications running on high performance clusters. These improved I/O rates will translate into faster end-to-end runtimes and fewer unexpected performance failures for end users and system administrators.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

科学计算中的高级发现越来越依赖于复杂工作流程的成功执行，这些工作流程将多个应用程序组合在一起以在大型高性能集群上协同运行。此设置中的一个普遍挑战是共享并行文件系统的性能。 由于每个文件系统交互在性能和一致性方面都有非常不同的需求，因此文件系统必须遵循最保守的方法来处理最坏的情况。 因此，很少能达到峰值性能。我们提出“一致性契约”作为这个问题的解决方案。 这种新颖的方法要求整个工作流程在每次执行开始时声明其对文件系统的预期用途，从而允许运行时系统执行各种优化。该项目将通过构建一个实验系统（Pledge）来评估一致性合同的概念，该系统在现有数据密集型工作流程上启用并执行合同，同时将对当前实践的干扰降至最低。我们假设这种方法将显着提高在高性能集群上运行的数据密集型科学应用程序的性能，并有可能得到更广泛的应用。我们对一致性的关注源于我们的观察，当今的共享并行文件系统被要求扮演多个角色：移动大文件、交付复杂的软件树、在任务之间提供缓冲区以及提供任务之间的同步。当前的文件系统提供最保守的顺序一致性来处理最坏的情况。 我们认为，工作流作为一个整体应该在执行期间声明其意图，指示所需的路径、访问模式和一致性要求，而不是依赖共享文件系统对每个单独的文件系统操作执行最后一刻的运行时仲裁对于整个工作流程运行。 有了合同，运行时系统就可以执行各种优化，充分利用整个集群的内部存储和 I/O 容量，例如利用简化的方法进行只读访问。我们假设，对于在高性能集群上运行的数据密集型科学应用程序，工作流级一致性管理将比严格的全局一致性管理产生更高的有效 I/O 带宽和事务率。这些改进的 I/O 速率将转化为更快的端到端运行时间，并减少最终用户和系统管理员意外的性能故障。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优势和更广泛的评估进行评估，被认为值得支持。影响审查标准。