Z-checker: A framework for assessing lossy compression of scientific data

Because of the vast volume of data being produced by today’s scientific simulations and experiments, lossy data compressor allowing user-controlled loss of accuracy during the compression is a relevant solution for significantly reducing the data size. However, lossy compressor developers and users are missing a tool to explore the features of scientific data sets and understand the data alteration after compression in a systematic and reliable way. To address this gap, we have designed and implemented a generic framework called Z-checker. On the one hand, Z-checker combines a battery of data analysis components for data compression. On the other hand, Z-checker is implemented as an open-source community tool to which users and developers can contribute and add new analysis components based on their additional analysis demands. In this article, we present a survey of existing lossy compressors. Then, we describe the design framework of Z-checker, in which we integrated evaluation metrics proposed in prior work as well as other analysis tools. Specifically, for lossy compressor developers, Z-checker can be used to characterize critical properties (such as entropy, distribution, power spectrum, principal component analysis, and autocorrelation) of any data set to improve compression strategies. For lossy compression users, Z-checker can detect the compression quality (compression ratio and bit rate) and provide various global distortion analysis comparing the original data with the decompressed data (peak signal-to-noise ratio, normalized mean squared error, rate–distortion, rate-compression error, spectral, distribution, and derivatives) and statistical analysis of the compression error (maximum, minimum, and average error; autocorrelation; and distribution of errors). Z-checker can perform the analysis with either coarse granularity (throughout the whole data set) or fine granularity (by user-defined blocks), such that the users and developers can select the best fit, adaptive compressors for different parts of the data set. Z-checker features a visualization interface displaying all analysis results in addition to some basic views of the data sets such as time series. To the best of our knowledge, Z-checker is the first tool designed to assess lossy compression comprehensively for scientific data sets.

由于当今科学模拟和实验产生的数据量巨大，允许用户控制压缩过程中精度损失的有损数据压缩器是大幅减少数据大小的相关解决方案。然而，有损压缩器的开发者和用户却缺少一种工具来探索科学数据集的特征，并以系统可靠的方式了解压缩后的数据变化。为了弥补这一不足，我们设计并实现了一个名为 Z-checker 的通用框架。一方面，Z-checker 结合了用于数据压缩的一系列数据分析组件。另一方面，Z-checker 是作为一个开源社区工具实施的，用户和开发人员可以根据自己的额外分析需求为其贡献和添加新的分析组件。在本文中，我们将对现有的有损压缩器进行调查。然后，我们介绍了 Z-checker 的设计框架，其中集成了之前工作中提出的评估指标以及其他分析工具。具体来说，对于有损压缩器开发人员，Z-checker 可用于描述任何数据集的关键属性（如熵、分布、功率谱、主成分分析和自相关性），以改进压缩策略。对于有损压缩用户，Z-checker 可以检测压缩质量（压缩率和比特率），并提供各种全局失真分析，将原始数据与解压缩数据进行比较（峰值信噪比、归一化均方误差、速率失真、速率压缩误差、频谱、分布和导数），以及压缩误差的统计分析（最大、最小和平均误差；自相关性；误差分布）。Z-checker 可以粗粒度（整个数据集）或细粒度（用户定义的块）进行分析，这样用户和开发人员就可以为数据集的不同部分选择最合适的自适应压缩器。Z-checker 具有一个可视化界面，除了显示数据集的一些基本视图（如时间序列）外，还可显示所有分析结果。据我们所知，Z-checker 是第一款用于全面评估科学数据集有损压缩的工具。