CIF: Small: Distributed Storage Systems from Combinatorial Designs

CIF：小型：组合设计的分布式存储系统

• 批准号: 1320416
• 负责人: Aditya Ramamoorthy
• 金额: $ 40万
• 依托单位: Iowa State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2013
• 资助国家: 美国
• 起止时间: 2013-08-01 至 2018-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1320416&HistoricalAwards=false
• 关键词: CIF; Small; Distributed; Storage; Systems

Large scale content sharing, social networking sites and cloud storage providers require massive amounts of data (on the order of hundreds of petabytes) to be stored in a distributed manner. In these systems, node failures and loss of data are the norm rather than the exception. Thus, it is of critical importance to investigate storage techniques and algorithms that (a) allow for data recovery in the presence of node failures and (b) allow for efficient regeneration of failed nodes. It is, of course, desirable to perform the regeneration in a distributed manner and optimize performance metrics associated with it. For instance, it is desirable that the regeneration process be fast and that only a few nodes be contacted for the repair. The investigator will research storage methods that allow for very simple and efficient repair while simultaneously allowing for data recovery in the presence of a large number of failures. The research involves the investigation of distributed storage systems that are resilient to multiple failures and can be repaired in an exact and uncoded manner; i.e., the new node may produce an exact copy of the failed node by simply downloading packets from the surviving nodes, with no computational overhead. Replication codes are increasingly deployed in large-scale data centers in use by major internet companies. Advances in reconstruction codes are expected to translate into millions of dollars in savings in maintenance costs in such data centers. To ensure accessibility of the results of this research, implementation will be carried out using the open source Hadoop distributed file system, the preferred platform for cutting-edge research and development in distributed cloud storage.

大规模的内容共享，社交网站和云存储提供商需要以分布式方式存储大量数据（按数百pb的顺序）。在这些系统中，节点故障和数据丢失是规范而不是例外。因此，研究存储技术和算法至关重要，这些算法（a）允许在存在节点故障的情况下进行数据恢复，并且（b）允许有效地重新再生失败的节点。当然，希望以分布式方式执行再生并优化与之相关的性能指标。例如，希望再生过程很快，并且只需与几个节点联系以进行维修。研究者将研究存储方法，该方法允许非常简单有效的维修，同时允许在存在大量故障的情况下进行数据恢复。该研究涉及对分布式存储系统的调查，这些存储系统对多个故障有弹性，并且可以以精确且未编码的方式修复；也就是说，新节点可以通过简单地从存活的节点下载数据包来生成失败节点的精确副本，而没有计算开销。重复代码越来越多地部署在大型互联网公司使用的大规模数据中心中。预计重建代码的进步将转化为此类数据中心的维护成本的数百万美元。为了确保本研究结果的可访问性，将使用开源Hadoop分布式文件系统（用于分布式云存储中的尖端研究和开发的首选平台进行实施。