Collaborative Research: An Application Driven I/O Optimization Approach for PetaScale Systems and Scientific Discoveries

协作研究：针对 PetaScale 系统和科学发现的应用驱动 I/O 优化方法

• 批准号: 0938000
• 负责人: Alok Choudhary
• 金额: $ 32万
• 依托单位: Northwestern University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2010
• 资助国家: 美国
• 起止时间: 2010-05-01 至 2014-04-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0938000&HistoricalAwards=false
• 关键词: Collaborative; Research; Application; Driven; Optimization

This research focuses on developing scalable parallel file access methods for multi-scale problem domain decompositions, such as the one presented in Adaptive Mesh Refinement (AMR) based algorithms. Existing parallel I/O methods concentrate on optimizing the process collaboration under a fairly evenly-distributed request pattern. However, they are not suitable for data structures in AMR, because the underlying data distribution is highly irregular and dynamic. Process synchronization in the existing parallel I/O methods can penalize the I/O parallelism if the process collaboration is not carefully coordinated. This research addresses such synchronization issue by developing scalable solutions in the Parallel netCDF library (PnetCDF), particularly to address AMR structured data and its I/O patterns. PnetCDF is a popular I/O library used by many computer simulation communities. A scalable solution for storing and accessing AMR data in parallel is considered a challenging task. This research will design a process-group based parallel I/O approach to eliminate unrelated processes and thus avoid possible I/O serialization. In addition, a new metadata representation will also be developed in pnetCDF for conserving tree-structured AMR data relationship in a portable form.

这项研究重点是开发用于多尺度问题域分解的可扩展并行文件访问方法，例如基于自适应网格细化（AMR）算法中呈现的算法。现有的并行I/O方法集中于在相当均匀分布的请求模式下优化过程协作。但是，它们不适合AMR中的数据结构，因为潜在的数据分布是高度不规则和动态的。如果未仔细协作，则现有的平行I/O方法中的过程同步可以惩罚I/O并行性。这项研究通过在平行NetCDF库（PNETCDF）中开发可扩展解决方案来解决此类同步问题，特别是解决AMR结构化数据及其I/O模式。 PNETCDF是许多计算机模拟社区使用的流行的I/O库。用于并行存储和访问AMR数据的可扩展解决方案被认为是一项具有挑战性的任务。这项研究将设计一种基于过程组的并行I/O方法，以消除无关的过程，从而避免I/O序列化。此外，还将在PNETCDF中开发新的元数据表示，以以便携式形式保存树结构化的AMR数据关系。