Collaborative Research: Elements: ROCCI: Integrated Cyberinfrastructure for In Situ Lossy Compression Optimization Based on Post Hoc Analysis Requirements

合作研究：要素：ROCCI：基于事后分析要求的原位有损压缩优化的集成网络基础设施

• 批准号: 2247080
• 负责人: Dingwen Tao
• 金额: $ 28万
• 依托单位: Indiana University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2024-10-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2247080&HistoricalAwards=false
• 关键词: Collaborative; Research; Elements; ROCCI; Integrated

Today’s simulations and advanced instruments are producing vast volumes of data, presenting a major storage and I/O burden for scientists. Error-bounded lossy compressors, which can significantly reduce the data volume while controlling data distortion with a constant error bound, have been developed for years. However, a significant gap still remains in practice. On the one hand, the impact of the compression errors on scientific research is not well understood, so how to set an appropriate error bound for lossy compression is very challenging. On the other hand, how to select the best fit compression technology and run it automatically in scientific application codes is non-trivial because of strengths and weaknesses of different compression techniques and diverse characteristics of applications and datasets. This project aims to develop a Requirement-Oriented Compression Cyber-Infrastructure (ROCCI) for data-intensive domains such as astrophysics and materials science, which can select and run the best fit lossy compressor automatically at runtime, in terms of user's requirement on their post hoc analysis.The overarching goal of this project is to offer a complete series of automatic functions and services allowing users to transparently run the best fit compressor at runtime during the scientific simulations or data acquisition. This project advances knowledge and understanding with three key thrusts: (1) it builds an efficient layer to interoperate with different lossy compressors and diverse post hoc analysis requirements on data fidelity by leveraging an existing compression adaptor library (LibPressio) and compression assessment library (Z-checker); (2) it develops an efficient engine to determine the best fit compressor with optimized settings based on user’s post-hoc analysis requirements; and (3) it develops a user-friendly infrastructure that integrates compression optimization and execution via the HDF5 dynamic filter mechanism. This project particularly targets cosmology and materials science applications and their specific requirements of using lossy compressors in practice.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

当今的模拟和先进仪器正在产生大量数据，给科学家带来了巨大的存储和 I/O 负担，人们已经开发出了误差有限有损压缩器，它可以显着减少数据量，同时以恒定误差限度控制数据失真。然而，在实践中仍然存在很大的差距，一方面，压缩误差对科学研究的影响尚不清楚，另一方面，如何为有损压缩设置合适的误差界限是非常具有挑战性的。手，如何选择最适合的压缩方式由于不同压缩技术的优缺点以及应用程序和数据集的不同特征，技术并在科学应用程序代码中自动运行它并非易事。该项目旨在开发面向数据密集型的面向需求的压缩网络基础设施（ROCCI）。天体物理学和材料科学等领域，可以根据用户事后分析的要求，在运行时自动选择和运行最适合的有损压缩器。该项目的总体目标是提供完整的系列自动功能和服务允许用户在科学模拟或数据采集过程中在运行时透明地运行最适合的压缩器。该项目通过三个关键要点推进知识和理解：（1）它构建了一个有效的层来与不同的有损进行互操作。通过利用现有的压缩适配器库 (LibPressio) 和压缩评估库 (Z-checker)，满足压缩器和对数据保真度的各种事后分析要求 (2) 它开发了一个高效的引擎来确定具有优化设置的最佳压缩器；基于用户的事后分析需求；(3)它开发了一个用户友好的基础设施，通过HDF5动态过滤机制集成了压缩优化和执行。该项目特别针对宇宙学和材料科学应用及其使用有损压缩器的特定要求。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

cuZ-Checker: A GPU-Based Ultra-Fast Assessment System for Lossy Compressions

cuZ-Checker：基于 GPU 的有损压缩超快速评估系统

• DOI: 10.1109/cluster48925.2021.00065
• 发表时间: 2021
• 期刊: 2021 IEEE International Conference on Cluster Computing (CLUSTER 2021
• 作者: Yu, Xiaodong;Di, Sheng;Gok, Ali Murat;Tao, Dingwen;Cappello, Franck
• 通讯作者: Cappello, Franck

Ultrafast Error-Bounded Lossy Compression for Scientific Datasets

科学数据集的超快误差限制有损压缩

• DOI: 10.1145/3502181.3531473
• 发表时间: 2022
• 期刊: The 31st ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC 2022
• 作者: Yu, Xiaodong;Di, Sheng;Zhao, Kai;Tian, Jiannan;Tao, Dingwen;Liang, Xin;Cappello, Franck
• 通讯作者: Cappello, Franck

Optimizing Error-Bounded Lossy Compression for Scientific Data With Diverse Constraints

优化具有不同约束的科学数据的误差有限有损压缩

• DOI: 10.1109/tpds.2022.3194695
• 发表时间: 2022
• 期刊: IEEE Transactions on Parallel and Distributed Systems
• 影响因子: 5.3
• 作者: Liu, Yuanjian;Di, Sheng;Zhao, Kai;Jin, Sian;Wang, Cheng;Chard, Kyle;Tao, Dingwen;Foster, Ian;Cappello, Franck
• 通讯作者: Cappello, Franck

Optimizing Error-Bounded Lossy Compression for Scientific Data on GPUs

优化 GPU 上科学数据的误差有限有损压缩

• DOI: 10.1109/cluster48925.2021.00047
• 发表时间: 2021
• 期刊: 2021 IEEE International Conference on Cluster Computing (CLUSTER 2021
• 作者: Tian, Jiannan;Di, Sheng;Yu, Xiaodong;Rivera, Cody;Zhao, Kai;Jin, Sian;Feng, Yunhe;Liang, Xin;Tao, Dingwen;Cappello, Franck
• 通讯作者: Cappello, Franck

FZ-GPU: A Fast and High-Ratio Lossy Compressor for Scientific Computing Applications on GPUs

• DOI: 10.1145/3588195.3592994
• 发表时间: 2023-04
• 期刊: Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing
• 作者: Bo Zhang;Jiannan Tian;S. Di;Xiaodong Yu;Yunhe Feng;Xin Liang;Dingwen Tao;F. Cappello