CAREER: Data-Intensive HPC Analytics: A Systems Approach Through Extended Interfaces, Data Restructuring and Data-centric Scheduling

职业：数据密集型 HPC 分析：通过扩展接口、数据重组和以数据为中心的调度的系统方法

• 批准号: 0953946
• 负责人: Jun Wang
• 金额: $ 43.89万
• 依托单位: The University of Central Florida Board of Trustees
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2010
• 资助国家: 美国
• 起止时间: 2010-03-15 至 2017-02-28
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0953946&HistoricalAwards=false
• 关键词: CAREER; Data; Intensive; HPC; Analytics

With the advent of emerging e-Science applications, today's scientific research increasingly relies on petascale-and-beyond computing over large data sets with petabyte-and-beyond sizes. Representatives include analytics- and simulation- driven applications such as the human vision simulation, astrophysics data analysis, earthquake modeling, climate modeling using ensemble runs, etc. In many of the above-mentioned fields, scientists are dealing with large amounts of data and analyzing them to explore new concepts and ideas. These applications make up data-intensive HPC analytics, which lies at the intersection of current HPC and Data-Intensive Scalable Computing (DISC).When HPC systems use traditional configurations to support data-intensive HPC analytics, data is copied from a large remote storage system to diskless compute nodes for processing. Copying data back and forth is an expensive and time consuming process. These data-intensive applications do not require compute intensive resources, but rather moderate compute power machines with the capability of local storage so that data can be processed in-place. One such example of this configuration is the Hadoop framework. However, there are currently limitations in this framework which must be overcome in order to make Hadoop an effective HPC tool. The investigator is leveraging the Hadoop framework to process large amount of patterned data in HPC. This research program includes three thrusts. It is developing the MapReduce API to support a wider range of I/O access patterns, various data restructuring schemes to improve I/O performance for these access patterns, and an efficient scheduling scheme considering multiple chunk locations and data transfer latencies over the network. The research is integrated into several educational activities, such as the development of data-intensive HPC curricula.

随着新兴电子科学应用程序的出现，当今的科学研究越来越依赖于对 PB 级及以上规模的大型数据集进行千万级以上的计算。代表包括分析和模拟驱动的应用程序，例如人类视觉模拟、天体物理数据分析、地震建模、使用集合运行的气候建模等。在上述许多领域，科学家正在处理大量数据并进行分析他们探索新的概念和想法。这些应用程序构成了数据密集型 HPC 分析，它位于当前 HPC 和数据密集型可扩展计算 (DISC) 的交叉点。当 HPC 系统使用传统配置来支持数据密集型 HPC 分析时，数据是从大型远程存储复制的系统到无盘计算节点进行处理。来回复制数据是一个昂贵且耗时的过程。这些数据密集型应用程序不需要计算密集型资源，而是需要具有本地存储功能的中等计算能力的机器，以便可以就地处理数据。这种配置的一个例子是 Hadoop 框架。然而，目前该框架存在一些限制，必须克服这些限制才能使 Hadoop 成为有效的 HPC 工具。研究人员正在利用 Hadoop 框架在 HPC 中处理大量模式数据。该研究计划包括三个重点。它正在开发 MapReduce API 以支持更广泛的 I/O 访问模式、各种数据重组方案以提高这些访问模式的 I/O 性能，以及考虑多个块位置和网络上数据传输延迟的高效调度方案。该研究被整合到多项教育活动中，例如数据密集型 HPC 课程的开发。