CIF21 DIBBs: PD: Building High-Availability Data Capabilities in Data-Centric Cyberinfrastructure

CIF21 DIBB：PD：在以数据为中心的网络基础设施中构建高可用性数据功能

基本信息

批准号：
1724845
负责人：
Haiying Shen
金额：
$ 50万
依托单位：
University of Virginia Main Campus
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2017
资助国家：
美国
起止时间：
2017-08-01 至 2020-07-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1724845&HistoricalAwards=false
关键词：
CIF21 DIBBs PD Building Availability

项目摘要

This project supports data-related analysis in a wide range of science and engineering applications. It will contribute to the development of scalable data-centric cyberinfrastructure capabilities, to accelerate interdisciplinary and collaborative research. The exascale file system will serve as catalyst for research in data storage architectures, and will enable new data-focused services and capabilities that advance scientific discoveries, collaborations, and innovations. The project addresses a major data challenge common to a range of communities such as social science, economics, and bioengineering, and will provide thorough training and collaborative research opportunities for project participants. Both high performance computing (HPC) clusters and Hadoop clusters use file systems. A Hadoop cluster uses the Hadoop Distributed File System (HDFS) that resides on compute nodes, while an HPC cluster usually uses a remote storage system. Despite years of efforts on research and application development on HPC and Hadoop clusters, the file systems in both types of clusters still face a formidable challenge, that of achieving exascale computing capabilities. The centralized data indexing in HDFS and HPC storage architectures cannot provide high scalability and reliability, and both HDFS and HPC storage architectures have shortcomings such as single point of failure and insufficiently efficient data access. This project builds scalable high-availability data capabilities in data-centric cyberinfrastructure to overcome the shortcomings and create a highly scalable file system with new techniques for distributed load balancing, data replication and consistency maintenance.

该项目支持广泛的科学和工程应用中的数据相关分析。它将有助于开发可扩展的以数据为中心的网络基础设施能力，以加速跨学科和协作研究。百亿亿级文件系统将成为数据存储架构研究的催化剂，并将实现以数据为中心的新服务和功能，从而推动科学发现、合作和创新。该项目解决了社会科学、经济学和生物工程等一系列社区常见的主要数据挑战，并将为项目参与者提供全面的培训和合作研究机会。高性能计算 (HPC) 集群和 Hadoop 集群都使用文件系统。 Hadoop 集群使用驻留在计算节点上的 Hadoop 分布式文件系统 (HDFS)，而 HPC 集群通常使用远程存储系统。尽管HPC和Hadoop集群的研究和应用开发经过多年的努力，但这两种集群中的文件系统仍然面临着实现百亿亿次计算能力的艰巨挑战。 HDFS和HPC存储架构中的集中式数据索引无法提供高扩展性和可靠性，并且HDFS和HPC存储架构都存在单点故障、数据访问效率不够高等缺点。该项目在以数据为中心的网络基础设施中构建可扩展的高可用性数据功能，以克服缺点，并利用分布式负载平衡、数据复制和一致性维护的新技术创建高度可扩展的文件系统。