SI2-SSE: Collaborative Research: Software Elements for Transfer and Analysis of Large-Scale Scientific Data

SI2-SSE：协作研究：用于大规模科学数据传输和分析的软件元素

• 批准号: 1339757
• 负责人: Gagan Agrawal
• 金额: $ 40万
• 依托单位: Ohio State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2013
• 资助国家: 美国
• 起止时间: 2013-09-01 至 2017-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1339757&HistoricalAwards=false
• 关键词: SI2; SSE; Collaborative; Research; Software

As science has become increasingly data-driven, and as data volumes and velocity are increasing, scientific advance in many areas will only be feasible if critical `big-data' problems are addressed - and even more importantly, software tools embedding these solutions are readily available to the scientists. Particularly, the major challenge being faced by current data-intensive scientific research efforts is that while the dataset sizes continue to grow rapidly, neither among network bandwidths, memory capacity of parallel machines, memory access speeds, and disk bandwidths are increasing at the same rate.Building on top of recent research at Ohio State University, which includes work on automatic data virtualization, indexing methods for scientific data, and a novel bit-vectors based sampling method, the goal of this project is to fully develop, disseminate, deploy, and support robust software elements addressing challenges in data transfers and analysis. The prototypes that have been already developed at Ohio State are being extended into two robust software elements: Software Element 1: GridPFTP (Grid Partial-File Transport Protocol): An Extention to GridFTP: an extention of GridFTP that allows users to specify a subset of the file to be transferred, avoiding unnecessary transfer of the entire file.Software Element 2: Parallel Readers for NetCDF and HDF5 for Paraview and VTK: Data subsetting and sampling tools for NetCDF and HDF5 that perform data selection and sampling at the I/O level, and in parallel.This project impacts a number of scientific areas, i.e., any area that involves big (and growing) dataset sizes and need for data transfers and/or visualization. This project also contributes to computer science research in `big data', including scientific (array-based) databases, and visualization. Another contribution will be towards preparation of the broad science and engineering research community for big data handling and analytics.

随着科学越来越多地驱动数据，并且随着数据量和速度的提高，只有解决关键的“大数据”问题的问题，科学的进步才能是可行的 - 更重要的是，更重要的是，嵌入这些解决方案的软件工具可以随时可用于科学家。 Particularly, the major challenge being faced by current data-intensive scientific research efforts is that while the dataset sizes continue to grow rapidly, neither among network bandwidths, memory capacity of parallel machines, memory access speeds, and disk bandwidths are increasing at the same rate.Building on top of recent research at Ohio State University, which includes work on automatic data virtualization, indexing methods for scientific data, and a novel bit-vectors based抽样方法，该项目的目标是充分开发，传播，部署和支持可靠的软件元素，以应对数据传输和分析中的挑战。 在俄亥俄州已经开发的原型正在扩展到两个强大的软件元素：软件元素1：gridPFTP（网格局部运输协议）：GridFTP的扩展：GridFTP的扩展：GridFTP的扩展，用户允许用户指定文件的子集，以避免使用不必要的元素转移，以避免使用整个文件。 VTK：在I/O级别以及并行进行数据选择和采样的NETCDF和HDF5的数据子集和采样工具。该项目会影响许多科学领域，即任何涉及大型（并增长的）数据集大小的区域，以及数据传输和/或可视化的大型（以及增长）的数据。 该项目还有助于“大数据”中的计算机科学研究，包括科学（基于阵列的）数据库和可视化。 另一个贡献是为大数据处理和分析的广泛科学和工程研究社区准备。