Elements: Advanced Lossless and Lossy Compression Algorithms for netCDF Datasets in Earth and Engineering Sciences (CANDEE)

元素：地球与工程科学中 netCDF 数据集的高级无损和有损压缩算法 (CANDEE)

• 批准号: 2004993
• 负责人: Charles Zender
• 金额: $ 59.99万
• 依托单位: University of California-Irvine
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-07-01 至 2024-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2004993&HistoricalAwards=false
• 关键词: Elements; Advanced; Lossless; Lossy; Compression

Data compression is used to store and transmit digital data such as music, television, and satellite measurements more efficiently by reducing storage space and download times. The compression software broker that this project provides will facilitate the adoption of modern compression techniques in many branches of science. Compressors come in two flavors: lossless, those that perfectly preserve the original information; and lossy, those that irretrievably discard parts of the "signal" to further improve compression. Modern lossless and lossy compression improvements in efficiency, speed, and fidelity, are striking and will benefit critical research areas by permitting researchers to simulate, store, and analyze phenomena such as stellar evolution, chemical reactions, and hurricane formation at finer detail than before, with no extra storage costs. Since digital storage consumes power, better compression also reduces power consumption and associated greenhouse gas emissions. This project will develop the software infrastructure necessary for scientific researchers to seamlessly shift their applications to produce and use data stored with state-of-the-art lossless techniques, and by new lossy techniques that are more accurate than any others.The two most widely-used self-describing dataset storage formats, HDF5 and netCDF4, support by default only one patent unencumbered lossless compression format, the venerable DEFLATE algorithm standardized in the 1990s. Our project will develop a dynamic and extensible software library of modern COmpressors and DECompressors (codecs) for scientific data called the Community Codec Repository (CCR). We will populate the CCR with cutting-edge open-source compression technology, including the LZ4, Facebook's Zstandard, and Google's Snappy codecs, and will implement default netCDF support for the CCR. Sequential lossy-then-lossless compression improves both the size and speed of compression/decompression yet is currently tedious to perform. We will implement a user-friendly method to "chain" codecs into sequential operations in memory (no intermediate files required) in our widely used netCDF Operators software package. We will also produce a new precision-preserving lossy codec, Granular Bit Grooming, that has unsurpassed compression ratio and statistical accuracy. Technical success will be evaluated by the size and speed improvements of compressing a prototypical geoscience/engineering "big data" project, the Coupled Model Intercomparison Project version 6 (CMIP6).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

数据压缩用于通过减少存储空间和下载时间来更有效地存储和传输数字数据，例如音乐、电视和卫星测量数据。该项目提供的压缩软件代理将促进现代压缩技术在许多科学分支中的采用。压缩器有两种类型：无损，完美保留原始信息；有损，不可挽回地丢弃部分“信号”以进一步提高压缩。现代无损和有损压缩在效率、速度和保真度方面的改进是惊人的，并将使研究人员能够比以前更详细地模拟、存储和分析恒星演化、化学反应和飓风形成等现象，从而使关键研究领域受益。无需额外的存储成本。由于数字存储会消耗电力，因此更好的压缩还可以减少电力消耗和相关的温室气体排放。该项目将开发必要的软件基础设施，使科学研究人员能够无缝地将其应用程序转变为生成和使用通过最先进的无损技术以及比任何其他技术更准确的新有损技术存储的数据。这两个最广泛- 使用自描述数据集存储格式 HDF5 和 netCDF4，默认情况下仅支持一种专利无阻碍无损压缩格式，即 20 世纪 90 年代标准化的古老 DEFLATE 算法。我们的项目将为科学数据开发一个动态且可扩展的现代压缩器和解压缩器（编解码器）软件库，称为社区编解码器存储库（CCR）。我们将为 CCR 填充尖端的开源压缩技术，包括 LZ4、Facebook 的 Zstandard 和 Google 的 Snappy 编解码器，并将为 CCR 实现默认的 netCDF 支持。顺序有损然后无损压缩提高了压缩/解压缩的大小和速度，但目前执行起来很乏味。我们将在广泛使用的 netCDF Operators 软件包中实现一种用户友好的方法，将编解码器“链接”到内存中的顺序操作（不需要中间文件）。我们还将生产一种新的保留精度的有损编解码器，Granular Bit Grooming，它具有无与伦比的压缩比和统计精度。技术成功将通过压缩原型地球科学/工程“大数据”项目（耦合模型比对项目版本 6 (CMIP6)）的大小和速度改进来评估。该奖项反映了 NSF 的法定使命，并被认为值得通过评估获得支持利用基金会的智力优势和更广泛的影响审查标准。