CRII: III: Declarative array processing for large-scale scientific analyses

CRII：III：用于大规模科学分析的声明性数组处理

• 批准号: 1464381
• 负责人: Spyros Blanas
• 金额: $ 17.5万
• 依托单位: Ohio State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-04-15 至 2018-03-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1464381&HistoricalAwards=false
• 关键词: CRII; III; Declarative; array; processing

Scientists understand complex natural phenomena through data-intensive analyses that run on hundreds of thousands of processing cores. Quickly exploring very large datasets in parallel for insights, however, is challenging. Analyzing larger-than-memory datasets exposes the intricacies of the storage hierarchy and necessitates different implementations based on the anticipated data volume and the system architecture. Domain scientists using large-scale computing facilities are therefore faced with a dilemma: they need to either perpetually maintain and tune their data processing codes to the evolving system infrastructure, or limit their investigation to analyses that can be completed in a reasonable time as datasets continue to grow in size. Declarative data processing techniques can alleviate scientists from the burden of managing how scientific data are accessed or stored. Although many declarative data management systems are actively used by scientists, these systems require time-consuming data transformations, such as loading, chunking and repartitioning, before answering any scientific query. In addition, many data management systems assume complete control of the underlying hardware, and are oblivious to optimizations and scaling opportunities that are offered through the batch execution paradigm of large-scale computing facilities. To address this gap in research, we will investigate techniques in the intersection of data management and scientific computing on how to allocate resources, and how to proactively manage parallel I/O and distributed memory. To impact scientific practice, we will develop a prototype runtime system that will augment an established scientific file format library with declarative querying capabilities for data analysis in leadership computing facilities.For further information see the project web site at: http://go.osu.edu/insitu_analysis

科学家通过对数十万个加工核心进行的数据密集分析来了解复杂的自然现象。但是，快速探索非常大的数据集以寻求见解是具有挑战性的。分析大型内存数据集揭示了存储层次结构的复杂性，因此需要基于预期的数据量和系统体系结构来实现不同的实现。因此，使用大规模计算设施的领域科学家面临困境：他们需要永久维护并调整其数据处理代码到不断发展的系统基础架构，或将其调查限制为可以在合理时间完成的分析，因为数据集的大小不断增长。声明性的数据处理技术可以减轻科学家管理科学数据的访问或存储方式的负担。尽管科学家积极使用许多声明性的数据管理系统，但这些系统需要耗时的数据转换，例如加载，分解和重新分配，然后再回答任何科学查询。此外，许多数据管理系统都可以完全控制基础硬件，并且忽略了通过大规模计算设施的批处理执行范式提供的优化和扩展机会。为了解决研究中的这一差距，我们将研究有关如何分配资源以及如何主动管理并行I/O和分布式内存的数据管理和科学计算交集的技术。为了影响科学实践，我们将开发一个原型运行时系统，该系统将增强既定的科学文件格式库，并具有声明性的查询功能，以便在领导力计算设施中进行数据分析。有关更多信息，请参见项目网站：http：//go.osu.edu/insitu/insitu_analyasis