BIGDATA: Small: DCM: Collaborative Research: An efficient, versatile, scalable, and portable storage system for scientific data containers

大数据：小型：DCM：协作研究：用于科学数据容器的高效、多功能、可扩展和便携式存储系统

• 批准号: 1251037
• 负责人: Liuba Shrira
• 金额: $ 15.2万
• 依托单位: Brandeis University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2013
• 资助国家: 美国
• 起止时间: 2013-07-01 至 2017-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1251037&HistoricalAwards=false
• 关键词: BIGDATA; Small; DCM; Collaborative; Research

Scientific Bigdata sets are becoming too large and complex to fit in RAM, forcing scientific applications to perform a lot of slow disk and network I/O. This growth also makes scientific data more vulnerable to corruptions due to crashes and human errors. This project will use recent results from algorithms, database, and storage research to improve the performance and reliability of standard scientific data formats. This will make scientific research cheaper, faster, more reliable, and more reproducible.The Hierarchical Data Format (HDF5) standard is a container format for scientific data. It allows scientists to define and store complex data structures inside HDF5 files. Unfortunately, the current standard forces users to store all data objects and their meta-data properties inside one large physical file; this mix hinders meta-data-specific optimizations. The current storage also uses data-structures that scale poorly for large data. Lastly, the current model lacks snapshot support, important for recovery from errors.A new HDF5 release allows users to create more versatile storage plugins to control storage policies on each object and attribute. This project is developing support for snapshots in HDF5, designing new data structures and algorithms to scale HDF5 data access on modern storage devices to Bigdata. The project is designing several new HDF5 drivers: mapping objects to a Linux file system; storing objects in a database; and accessing data objects on remote Web servers. These improvements are evaluated using large-scale visualization applications with Bigdata, stemming from real-world scientific computations.

科学的BigData集变得太大而复杂，无法适应RAM，迫使科学应用程序执行大量慢速磁盘和网络I/O。 这种增长还使科学数据更容易受到崩溃和人为错误造成的腐败的影响。 该项目将使用算法，数据库和存储研究的最新结果来提高标准科学数据格式的性能和可靠性。 这将使科学研究更便宜，更快，更可靠，更可重复。层次数据格式（HDF5）标准是科学数据的容器格式。 它允许科学家在HDF5文件中定义和存储复杂的数据结构。 不幸的是，当前的标准用户将所有数据对象及其元数据属性存储在一个大物理文件中。这种混合会阻碍元数据特异性优化。 当前的存储还使用数据结构，对于大数据而言，扩展很差。最后，当前模型缺乏快照支持，对于从错误中恢复至关重要。新的HDF5版本允许用户创建更多通用的存储插件来控制每个对象和属性上的存储策略。 该项目正在为HDF5中的快照提供支持，设计了新的数据结构和算法，以扩展现代存储设备上的HDF5数据访问到BigData。该项目正在设计几个新的HDF5驱动程序：将对象映射到Linux文件系统；将对象存储在数据库中；并在远程Web服务器上访问数据对象。 这些改进是使用带有BigData的大规模可视化应用来评估的，这是由现实世界的科学计算引起的。