Frameworks: Collaborative Research: ChronoLog: A High-Performance Storage Infrastructure for Activity and Log Workloads

框架：协作研究：ChronoLog：用于活动和日志工作负载的高性能存储基础架构

• 批准号: 2104013
• 负责人: Xian-He Sun
• 金额: $ 267.65万
• 依托单位: Illinois Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-06-01 至 2025-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2104013&HistoricalAwards=false
• 关键词: Frameworks; Collaborative; Research; ChronoLog; Performance

Modern computing applications generate massive amounts of data at unprecedented rates. Beyond simply storing data, one increasingly common requirement is to store activity data, also known as log data, which describe things that happen rather than things that are. Activity data are generated by computing systems, scientific instruments, electrical devices, etc. as well as by humans. The fast growing of activity data stresses current data management systems beyond their capability and becomes a known killer performance bottleneck of high-performance computing systems. This project develops ChronoLog, a novel system for organizing and storing activity data effectively and efficiently. ChronoLog leverages modern storage hardware and provides user-focused plugins and easy-to-use interface for productivity. It will benefit a diverse range of communities in various ways, such as enabling better fraud detection in financial transactions, faster and more accurate weather predictions and simulations, reduced time-to-insight for medical and bioengineering data, autonomous computing (e.g., driving), and more secure web and mobile services. ChronoLog uses physical time to provide a synchronization-free data distribution and the total ordering on a log. It first leverages multiple storage tiers, such as storage-class memories (e.g., 3D XPoint) and new flash storage (e.g., NVMe SSDs), to transparently scale the log via log auto-tiering. It then adopts a tunable parallel access model, which offers multiple-writers-multiple-readers (MWMR) semantics and highly concurrent I/O, to fully utilize the multi-tiered storage environment. ChronoLog's innovative design supports high-performance data access via I/O isolation between tails and historical operations, efficient resource utilization with newly developed elastic storage capabilities, and scalability using a novel 3D log distribution. It facilitates data processing pipelining by acting as an authoritative source of strong consistency and with the help of fast append and commit semantics. It can be used as an arbitrator offering a plethora of features such as transactional isolation and atomicity, a consensus engine for consistent replication and indexing services, and a scalable data integration and warehousing solution. ChronoLog and its plugins establish a robust, flexible, and high-performance storage ecosystem that promotes the development of scalable applications and services for high performance computing systems. The project includes a diverse group of collaborators who share a common need for a fundamentally new approach to distributed logging to address their use cases. These close partnerships will strengthen the bonds between academic and applied science, ultimately leading to new applications and driving discovery in domains as diverse as geoscience, cosmology, and astrophysics. Forming these collaborations and integrating students and junior IT professionals will create a well-trained workforce in cyberinfrastructure.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代计算应用程序以前所未有的速度生成大量数据。除了简单地存储数据之外，一项日益常见的需求是存储活动数据（也称为日志数据），它描述正在发生的事情而不是实际发生的事情。活动数据由计算系统、科学仪器、电气设备等以及人类生成。活动数据的快速增长给当前的数据管理系统带来了超出其能力的压力，并成为高性能计算系统的已知杀手性能瓶颈。该项目开发了 ChronoLog，这是一种用于有效且高效地组织和存储活动数据的新颖系统。 ChronoLog 利用现代存储硬件，并提​​供以用户为中心的插件和易于使用的界面，以提高生产力。它将以各种方式使各种社区受益，例如更好地检测金融交易中的欺诈、更快、更准确的天气预报和模拟、缩短医疗和生物工程数据的洞察时间、自主计算（例如驾驶） ，以及更安全的网络和移动服务。 ChronoLog 使用物理时间来提供无同步的数据分布和日志上的总排序。它首先利用多个存储层，例如存储级内存（例如 3D XPoint）和新的闪存存储（例如 NVMe SSD），通过日志自动分层透明地扩展日志。然后，它采用可调并行访问模型，提供多写入器多读取器 (MWMR) 语义和高并发 I/O，以充分利用多层存储环境。 ChronoLog的创新设计通过尾部和历史操作之间的I/O隔离支持高性能数据访问，通过新开发的弹性存储功能实现高效的资源利用，并使用新颖的3D日志分布实现可扩展性。它通过充当强一致性的权威来源并借助快速追加和提交语义来促进数据处理流水线。它可以用作仲裁器，提供大量功能，例如事务隔离和原子性、用于一致复制和索引服务的共识引擎以及可扩展的数据集成和仓储解决方案。 ChronoLog 及其插件建立了一个强大、灵活和高性能的存储生态系统，促进高性能计算系统可扩展应用程序和服务的开发。该项目包括一群不同的合作者，他们共同需要一种全新的分布式日志记录方法来解决他们的用例。这些密切的伙伴关系将加强学术和应用科学之间的联系，最终带来新的应用并推动地球科学、宇宙学和天体物理学等不同领域的发现。形成这些合作并整合学生和初级 IT 专业人员将在网络基础设施领域打造一支训练有素的员工队伍。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

An Evaluation of DAOS for Simulation and Deep Learning HPC Workloads

针对模拟和深度学习 HPC 工作负载的 DAOS 评估

• DOI: 10.1145/3578353.3589542
• 发表时间: 2023-05-08
• 期刊: Proceedings of the 3rd Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems
• 作者: Luke Logan;J. Lofstead;Xian;Anthony Kougkas
• 通讯作者: Anthony Kougkas