CSR: Small: IRIS: A unified data access framework for the merging of compute-centric and data-centric storage

CSR：小型：IRIS：用于合并以计算为中心和以数据为中心的存储的统一数据访问框架

• 批准号: 1814872
• 负责人: Xian-He Sun
• 金额: $ 50万
• 依托单位: Illinois Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1814872&HistoricalAwards=false
• 关键词: CSR; Small; IRIS; unified; data

As the boundary between High Performance Computing (HPC) and High-Performance Data Analytics (HPDA) continue to blur, the conventional compute-centric HPC and the newly emerged data-driven Big Data application are converging. HPC becomes more data-intensive. In the meantime, HPDA requires more computing power. MapReduce and Spark software environments are developed and are popular for HPDA. However, these software frameworks are not designed for HPC and not compatible with HPC storage subsystems. In this research, the design and development of a unified data access framework, named IRIS, is proposed for the integration of compute-centric and data-centric storage solutions. The intellectual merit of this research is three-fold. 1) Mapping of incompatible structures: Mapping a file to key-value pairs and vice versa efficiently is a challenging task. 2) Maintaining metadata information: Since IRIS is a unified storage layer, it needs to maintain compatibility with legacy codes. IRIS will address this challenge with tunable consistency, which need to be carefully studied for implementation and design choices. 3) Minimizing overhead and memory footprint of IRIS solutions: Mapping of incompatible structures can cause excessive memory usage; this will be addressed in this research.This project is expected to have significant impact, including bridging data generation and data analysis processes; promoting collaboration between the model simulation and data analysis communities; and building a foundation element for next generation integrated storage systems. This research will create advanced solutions and technologies that will have direct impact on improving the efficiency of data access and management at scale. Since Big Data is a national strategic infrastructure for science, engineering, and industry, this research will advance a broad range of fields. It aims to make significant progress toward a unified storage access system. All data generated from this project will be stored in an electronic format and will be preserved on the server machines at the Illinois Institute of Technology (IIT): http://cs.iit.edu/~scs/. The server machines have hot-copy backup disks for backing up the primary copy of all data. A secondary copy of all data will be kept on the server machines in the computer science department at IIT per semester basis. The data will be transferred to new storage devices every 2 years. All data will be retained within 3 years of the project completion date.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

随着高性能计算（HPC）和高性能数据分析（HPDA）之间的界限不断模糊，传统的以计算为中心的HPC和新出现的数据驱动的大数据应用正在融合。 HPC 变得更加数据密集。同时，HPDA需要更多的计算能力。 MapReduce 和 Spark 软件环境是为 HPDA 开发并流行的。然而，这些软件框架并不是为HPC设计的，并且与HPC存储子系统不兼容。在本研究中，提出了一个名为 IRIS 的统一数据访问框架的设计和开发，用于集成以计算为中心和以数据为中心的存储解决方案。 这项研究的智力价值有三个方面。 1）不兼容结构的映射：有效地将文件映射到键值对，反之亦然是一项具有挑战性的任务。 2）维护元数据信息：由于IRIS是统一的存储层，因此需要保持与遗留代码的兼容性。 IRIS 将通过可调一致性来应对这一挑战，需要仔细研究实施和设计选择。 3）最小化IRIS解决方案的开销和内存占用：不兼容结构的映射可能会导致内存使用过多；本研究将解决这个问题。该项目预计将产生重大影响，包括桥接数据生成和数据分析过程；促进模型模拟和数据分析社区之间的合作；并为下一代集成存储系统构建基础元素。这项研究将创建先进的解决方案和技术，对大规模提高数据访问和管理的效率产生直接影响。由于大数据是科学、工程和工业的国家战略基础设施，这项研究将推动广泛的领域的发展。它的目标是在统一存储访问系统方面取得重大进展。该项目生成的所有数据都将以电子格式存储在伊利诺伊理工学院 (IIT) 的服务器上：http://cs.iit.edu/~scs/。服务器计算机具有热复制备份磁盘，用于备份所有数据的主副本。所有数据的辅助副本将每学期保存在 IIT 计算机科学系的服务器计算机上。数据每两年就会转移到新的存储设备。所有数据将在项目完成之日起 3 年内保留。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

LabStor: A Modular and Extensible Platform for Developing High-Performance, Customized I/O Stacks in Userspace

LabStor：用于在用户空间中开发高性能、定制 I/O 堆栈的模块化和可扩展平台

• DOI: 10.1109/sc41404.2022.00028
• 发表时间: 2022-11-01
• 期刊: SC22: International Conference for High Performance Computing, Networking, Storage and Analysis
• 作者: Luke Logan;Jaime Cernuda Garcia;J. Lofstead;Xian;Anthony Kougkas
• 通讯作者: Anthony Kougkas

DLIO: A Data-Centric Benchmark for Scientific Deep Learning Applications

DLIO：科学深度学习应用程序的以数据为中心的基准

• DOI: 10.1109/ccgrid51090.2021.00018
• 发表时间: 2021-05
• 期刊: Cloud and Internet Computing (CCGrid
• 作者: Devarajan, Hariharan;Zheng, Huihuo;Kougkas, Anthony;Sun, Xian;Vishwanath, Venkatram
• 通讯作者: Vishwanath, Venkatram

HFlow: A Dynamic and Elastic Multi-Layered I/O Forwarder

HFlow：动态、弹性的多层 I/O 转发器

• DOI: 10.1109/cluster48925.2021.00064
• 发表时间: 2021-09-01
• 期刊: 2021 IEEE International Conference on Cluster Computing (CLUSTER)
• 作者: Jaime Cernuda Garcia;H. Devarajan;Luke Logan;Keith Bateman;N. Rajesh;Jie Ye;Anthony Kougkas;Xian
• 通讯作者: Xian

Apollo:: An ML-assisted Real-Time Storage Resource Observer

Apollo:: 机器学习辅助的实时存储资源观察器

• DOI: 10.1145/3431379.3460640
• 发表时间: 2021-06
• 期刊: The 30th International Symposium on High-Performance Parallel and Distributed Computing (HPDC '21
• 作者: Rajesh, Neeraj;Devarajan, Hariharan;Garcia, Jaime Cernuda;Bateman, Keith;Logan, Luke;Ye, Jie;Kougkas, Anthony;Sun, Xian
• 通讯作者: Sun, Xian

NIOBE: An Intelligent I/O Bridging Engine for Complex and Distributed Workflows

NIOBE：适用于复杂分布式工作流程的智能 I/O 桥接引擎

• DOI: 10.1109/bigdata47090.2019.9006363
• 发表时间: 2019-12-01
• 期刊: 2019 IEEE International Conference on Big Data (Big Data)
• 作者: Kun Feng;H. Devarajan;Anthony Kougkas;Xian
• 通讯作者: Xian