CRII: III: Declarative array processing for large-scale scientific analyses

CRII：III：用于大规模科学分析的声明性数组处理

• 批准号: 1464381
• 负责人: Spyros Blanas
• 金额: $ 17.5万
• 依托单位: Ohio State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-04-15 至 2018-03-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1464381&HistoricalAwards=false
• 关键词: CRII; III; Declarative; array; processing

Scientists understand complex natural phenomena through data-intensive analyses that run on hundreds of thousands of processing cores. Quickly exploring very large datasets in parallel for insights, however, is challenging. Analyzing larger-than-memory datasets exposes the intricacies of the storage hierarchy and necessitates different implementations based on the anticipated data volume and the system architecture. Domain scientists using large-scale computing facilities are therefore faced with a dilemma: they need to either perpetually maintain and tune their data processing codes to the evolving system infrastructure, or limit their investigation to analyses that can be completed in a reasonable time as datasets continue to grow in size. Declarative data processing techniques can alleviate scientists from the burden of managing how scientific data are accessed or stored. Although many declarative data management systems are actively used by scientists, these systems require time-consuming data transformations, such as loading, chunking and repartitioning, before answering any scientific query. In addition, many data management systems assume complete control of the underlying hardware, and are oblivious to optimizations and scaling opportunities that are offered through the batch execution paradigm of large-scale computing facilities. To address this gap in research, we will investigate techniques in the intersection of data management and scientific computing on how to allocate resources, and how to proactively manage parallel I/O and distributed memory. To impact scientific practice, we will develop a prototype runtime system that will augment an established scientific file format library with declarative querying capabilities for data analysis in leadership computing facilities.For further information see the project web site at: http://go.osu.edu/insitu_analysis

科学家通过在数十万个处理核心上运行的数据密集型分析来了解复杂的自然现象。然而，快速并行地探索非常大的数据集以获得见解是具有挑战性的。分析大于内存的数据集暴露了存储层次结构的复杂性，并且需要根据预期的数据量和系统架构进行不同的实现。因此，使用大规模计算设施的领域科学家面临着两难境地：他们需要永久维护和调整数据处理代码以适应不断发展的系统基础设施，或者将他们的调查限制为随着数据集的不断发展而在合理时间内完成的分析。扩大规模。声明性数据处理技术可以减轻科学家管理科学数据访问或存储方式的负担。尽管科学家们积极使用许多声明性数据管理系统，但这些系统在回答任何科学查询之前需要进行耗时的数据转换，例如加载、分块和重新分区。此外，许多数据管理系统假定对底层硬件的完全控制，并且忽视了通过大规模计算设施的批处理执行范例提供的优化和扩展机会。为了弥补这一研究空白，我们将研究数据管理和科学计算的交叉技术，了解如何分配资源，以及如何主动管理并行 I/O 和分布式内存。为了影响科学实践，我们将开发一个原型运行时系统，该系统将通过声明性查询功能来增强已建立的科学文件格式库，以在领先的计算设施中进行数据分析。有关更多信息，请访问该项目网站：http://go.osu .edu/insitu_analysis