Efficient log data compression and analytics system

高效的日志数据压缩和分析系统

• 批准号: 570524-2021
• 负责人: Yuan, DingD
• 金额: $ 17.46万
• 依托单位: University of Toronto
• 依托单位国家: 加拿大
• 项目类别: Alliance Grants
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=751741
• 关键词: Efficient; log; data; compression; analytics

The pandemic has brought a furious growth of computer log data, i.e., the runtime data generated by computer systems. This is catalyzed by two trends. First, during the pandemic many activities are moved to online, hence there is an unprecedented growth of software services, which, in turn, generate an unprecedented amount of log data. In addition, software companies are increasingly relying on analysis of data to make data-driven decisions. They build large scale big data analytics systems that run hundreds of jobs per day. These data analytics systems end up generating even more log data. As a result, it is common for a company to generate Petabytes of log data per day, with 50% - 100% year-over-year growth.This unprecedented amount of log data creates a number of new challenges. First, conventional log analytics systems no longer scale to such large data sizes. In addition, managing such large data is extremely costly at every level, from storage cost, to network bandwidth, and to the cost of compute resources. This research proposes a novel system called CLP (Compressed Log Processor). CLP compresses the logs to unprecedented compression ratio, and more importantly, it allows one to search the compressed logs without decompression. The success of CLP will reduce the cost of log management and storage by over 40x, saving internet companies hundreds of millions of dollars per year. More importantly, it enables users to quickly search through Petabytes of logs efficiently, something that was not possible. The partnership with Uber will make CLP production-ready, and establish Canada as a leader in this emerging area of scalable and efficient log management. It will also result in increased employments of skilled software engineers in Canada, significant enhancements of the competitiveness of Canada's data center technology, and reduced emission of Canadian data centers.

疫情带来了计算机日志数据（即计算机系统生成的运行时数据）的急剧增长。这是由两种趋势推动的。首先，疫情期间许多活动都转移到线上，因此软件服务出现了前所未有的增长，进而产生了前所未有的日志数据量。此外，软件公司越来越依赖数据分析来做出数据驱动的决策。他们构建了每天运行数百个作业的大规模大数据分析系统。这些数据分析系统最终会生成更多的日志数据。因此，公司每天生成 PB 级日志数据是很常见的，并且同比增长 50% - 100%。这种前所未有的日志数据量带来了许多新的挑战。首先，传统的日志分析系统不再能够扩展到如此大的数据量。此外，管理如此大的数据在各个层面上都极其昂贵，从存储成本到网络带宽，再到计算资源成本。这项研究提出了一种称为 CLP（压缩日志处理器）的新颖系统。 CLP将日志压缩到前所未有的压缩率，更重要的是，它允许人们在不解压的情况下搜索压缩日志。 CLP的成功将使日志管理和存储的成本降低40倍以上，每年为互联网公司节省数亿美元。更重要的是，它使用户能够快速有效地搜索 PB 级日志，这是不可能的。与 Uber 的合作将使 CLP 做好生产准备，并使加拿大成为可扩展和高效日志管理这一新兴领域的领导者。它还将增加加拿大熟练软件工程师的就业机会，显着增强加拿大数据中心技术的竞争力，并减少加拿大数据中心的排放。