CAREER: Data-Intensive HPC Analytics: A Systems Approach Through Extended Interfaces, Data Restructuring and Data-centric Scheduling

职业：数据密集型 HPC 分析：通过扩展接口、数据重组和以数据为中心的调度的系统方法

• 批准号: 0953946
• 负责人: Jun Wang
• 金额: $ 43.89万
• 依托单位: The University of Central Florida Board of Trustees
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2010
• 资助国家: 美国
• 起止时间: 2010-03-15 至 2017-02-28
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0953946&HistoricalAwards=false
• 关键词: CAREER; Data; Intensive; HPC; Analytics

With the advent of emerging e-Science applications, today's scientific research increasingly relies on petascale-and-beyond computing over large data sets with petabyte-and-beyond sizes. Representatives include analytics- and simulation- driven applications such as the human vision simulation, astrophysics data analysis, earthquake modeling, climate modeling using ensemble runs, etc. In many of the above-mentioned fields, scientists are dealing with large amounts of data and analyzing them to explore new concepts and ideas. These applications make up data-intensive HPC analytics, which lies at the intersection of current HPC and Data-Intensive Scalable Computing (DISC).When HPC systems use traditional configurations to support data-intensive HPC analytics, data is copied from a large remote storage system to diskless compute nodes for processing. Copying data back and forth is an expensive and time consuming process. These data-intensive applications do not require compute intensive resources, but rather moderate compute power machines with the capability of local storage so that data can be processed in-place. One such example of this configuration is the Hadoop framework. However, there are currently limitations in this framework which must be overcome in order to make Hadoop an effective HPC tool. The investigator is leveraging the Hadoop framework to process large amount of patterned data in HPC. This research program includes three thrusts. It is developing the MapReduce API to support a wider range of I/O access patterns, various data restructuring schemes to improve I/O performance for these access patterns, and an efficient scheduling scheme considering multiple chunk locations and data transfer latencies over the network. The research is integrated into several educational activities, such as the development of data-intensive HPC curricula.

随着新兴的电子科学应用的出现，当今的科学研究越来越多地依赖于Petascale和Beyond计算的计算，而不是具有PBABYTE和BE-BEYOND大小的大型数据集。代表包括分析和模拟驱动的应用，例如人类视觉模拟，天体物理数据分析，地震建模，使用集合运行等的气候建模等。在许多上述领域中，科学家正在处理大量数据并分析它们以探索新的概念和想法。这些应用程序构成了数据密集型HPC分析，这是在当前HPC和数据密集型可扩展计算（DISC）的交集中。当HPC系统使用传统配置来支持数据密集型HPC分析时，数据将从大型远程存储系统中复制到无用的无远程存储系统来处理磁盘用于处理处理。来回复制数据是一个昂贵且耗时的过程。这些数据密集型应用程序不需要计算密集的资源，而是具有本地存储功能的中等计算机机器，因此可以将数据定为就地处理。此配置的一个这样的例子是Hadoop框架。但是，目前在此框架中有一些局限性必须克服，以使Hadoop成为有效的HPC工具。研究人员正在利用Hadoop框架来处理HPC中的大量图案数据。该研究计划包括三个推力。它正在开发MAPREDUCE API，以支持更广泛的I/O访问模式，各种数据重组方案，以改善这些访问模式的I/O性能以及考虑到网络上多个块位置和数据传输潜伏期的高效调度方案。该研究纳入了几项教育活动，例如开发数据密集型HPC课程。