SI2-SSE: Collaborative Research: Software Elements for Transfer and Analysis of Large-Scale Scientific Data

SI2-SSE：协作研究：用于大规模科学数据传输和分析的软件元素

• 批准号: 1339798
• 负责人: Rajkumar Kettimuthu
• 金额: $ 10万
• 依托单位: University of Chicago
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2013
• 资助国家: 美国
• 起止时间: 2013-09-01 至 2018-04-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1339798&HistoricalAwards=false
• 关键词: SI2; SSE; Collaborative; Research; Software

As science has become increasingly data-driven, and as data volumes and velocity are increasing, scientific advance in many areas will only be feasible if critical `big-data' problems are addressed - and even more importantly, software tools embedding these solutions are readily available to the scientists. Particularly, the major challenge being faced by current data-intensive scientific research efforts is that while the dataset sizes continue to grow rapidly, neither among network bandwidths, memory capacity of parallel machines, memory access speeds, and disk bandwidths are increasing at the same rate.Building on top of recent research at Ohio State University, which includes work on automatic data virtualization, indexing methods for scientific data, and a novel bit-vectors based sampling method, the goal of this project is to fully develop, disseminate, deploy, and support robust software elements addressing challenges in data transfers and analysis. The prototypes that have been already developed at Ohio State are being extended into two robust software elements: an extention of GridFTP (Grid Partial-File Transport Protocol)that allows users to specify a subset of the file to be transferred, avoiding unnecessary transfer of the entire file; and Parallel Readers for NetCDF and HDF5 for Paraview and VTK, data subsetting and sampling tools for NetCDF and HDF5 that perform data selection and sampling at the I/O level, and in parallel.This project impacts a number of scientific areas, i.e., any area that involves big (and growing) dataset sizes and need for data transfers and/or visualization. This project also contributes to computer science research in `big data', including scientific (array-based) databases, and visualization. Another contribution will be towards preparation of the broad science and engineering research community for big data handling and analytics.

随着科学越来越由数据驱动，数据量和速度不断增加，只有解决关键的“大数据”问题，许多领域的科学进步才可能实现——更重要的是，嵌入这些解决方案的软件工具很容易实现可供科学家使用。特别是，当前数据密集型科学研究工作面临的主要挑战是，在数据集规模持续快速增长的同时，网络带宽、并行机内存容量、内存访问速度和磁盘带宽却没有同步增长。该项目以俄亥俄州立大学最近的研究为基础，其中包括自动数据虚拟化、科学数据索引方法以及基于位向量的新颖采样方法，其目标是充分开发、传播、部署、并支持强大的软件元素来应对挑战数据传输和分析。 俄亥俄州立大学已经开发的原型正在扩展到两个强大的软件元素：GridFTP（网格部分文件传输协议）的扩展，允许用户指定要传输的文件的子集，避免不必要的传输整个文件； Paraview 和 VTK 的 NetCDF 和 HDF5 并行读取器、NetCDF 和 HDF5 的数据子集和采样工具，可在 I/O 级别并行执行数据选择和采样。该项目影响许多科学领域，即任何涉及大（且不断增长）的数据集大小以及数据传输和/或可视化需求的领域。 该项目还有助于“大数据”的计算机科学研究，包括科学（基于阵列）数据库和可视化。 另一个贡献将是为广泛的科学和工程研究社区做好大数据处理和分析的准备。