Collaborative Research: SHF: Small: Scalable and Extensible I/O Runtime and Tools for Next Generation Adaptive Data Layouts

协作研究：SHF：小型：可扩展和可扩展的 I/O 运行时以及下一代自适应数据布局的工具

• 批准号: 2401274
• 负责人: Sidharth kumar
• 金额: $ 30.02万
• 依托单位: University of Illinois at Chicago
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2025-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2401274&HistoricalAwards=false
• 关键词: Collaborative; Research; SHF; Small; Scalable

The ability to perform large scale scientific simulations on supercomputers have fueled a wave of innovation and discoveries across a range of disciplines including energy, cosmology, earth science, medicine, and national security. With the advent of exascale, applications promise to deliver data of ever-increasing size at higher resolution and fidelity. Current technology trends in High Performance Computing (HPC) systems are creating an unprecedented gap between compute and I/O performance, making data movement the slowest component of the simulation-analysis pipeline. Many techniques have been proposed to alleviate this bottleneck including compression and hierarchical data layouts, but current solutions lack scalability and portability, and do not provide a holistic approach to the data-management needs of both parallel I/O and analysis (in situ and post-hoc) workflows. This work will develop a scalable and extensible I/O runtime and tools for the next-generation adaptive data layouts that inherently imbibe compression and progressive data access, advancing the state of art in the field of high-performance data management. The work will lay the foundation for an end-to-end data management solution that will cater to the challenging needs of the entire simulation-analysis pipeline and significantly accelerate science at exascale.The research aims to develop an end-to-end data-management solution for the next generation adaptive data layouts. The proposed data layouts will be hierarchical, compressed, and tunable, making them suitable to deal with the data deluge and the evolving landscape of HPC. A hierarchical layout will allow progressive access to massively large data enabling post-hoc and in situ analysis at any scale. State-of-the-art data compression and reduction techniques will significantly alleviate data-movement bottlenecks, especially while performing parallel I/O. Finally, a tunable layout combined with novel performance analysis and visualization tools will allow data-driven approaches to optimize I/O performance at runtime for different workflows and HPC platforms. This project aims to achieve its goals by developing: a scalable and tunable parallel I/O runtime that will support progressive read/write operations using adaptive data layouts; interfaces to support the adaptive data layouts for in situ workflows; a novel WebGPU-powered visualization system that can take advantage of the progressive nature of the layout enabling interactive exploration of large datasets on web browsers; and performance-mining and -visualization tools to enable data-driven and portable I/O performance prediction and auto-tuning. The solution will be evaluated on leadership supercomputers and mid-scale clusters, and integrated with large-scale simulations, analysis, and I/O frameworks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在超级计算机上进行大规模科学模拟的能力推动了能源、宇宙学、地球科学、医学和国家安全等一系列学科的创新和发现浪潮。随着百亿亿次计算的出现，应用程序有望以更高的分辨率和保真度提供不断增加的数据量。当前高性能计算 (HPC) 系统的技术趋势正在计算和 I/O 性能之间造成前所未有的差距，使数据移动成为仿真分析管道中最慢的组件。人们已经提出了许多技术来缓解这一瓶颈，包括压缩和分层数据布局，但当前的解决方案缺乏可扩展性和可移植性，并且没有提供满足并行 I/O 和分析（现场和事后分析）数据管理需求的整体方法。 -hoc）工作流程。这项工作将为下一代自适应数据布局开发可扩展和可扩展的 I/O 运行时和工具，本质上吸收压缩和渐进式数据访问，从而推进高性能数据管理领域的最新技术。这项工作将为端到端数据管理解决方案奠定基础，该解决方案将满足整个模拟分析管道的挑战性需求，并显着加速百亿亿级科学发展。该研究旨在开发一种端到端数据管理解决方案。下一代自适应数据布局的管理解决方案。所提出的数据布局将是分层的、压缩的和可调的，使其适合应对数据洪流和不断发展的 HPC 格局。分层布局将允许逐步访问海量数据，从而实现任何规模的事后和现场分析。最先进的数据压缩和缩减技术将显着缓解数据移动瓶颈，尤其是在执行并行 I/O 时。最后，可调布局与新颖的性能分析和可视化工具相结合，将允许数据驱动的方法在运行时针对不同的工作流程和 HPC 平台优化 I/O 性能。该项目旨在通过开发以下内容来实现其目标： 一个可扩展且可调的并行 I/O 运行时，它将支持使用自适应数据布局的渐进式读/写操作；支持现场工作流程的自适应数据布局的接口；一种新颖的 WebGPU 支持的可视化系统，可以利用布局的渐进性，从而能够在 Web 浏览器上交互式探索大型数据集；性能挖掘和可视化工具，以实现数据驱动和可移植的 I/O 性能预测和自动调整。该解决方案将在领先的超级计算机和中型集群上进行评估，并与大规模模拟、分析和 I/O 框架集成。该奖项反映了 NSF 的法定使命，并通过利用基金会的智力优势进行评估，被认为值得支持以及更广泛的影响审查标准。