BIGDATA: F: DKM: Collaborative Research: Scalable Middleware for Managing and Processing Big Data on Next Generation HPC Systems

BIGDATA：F：DKM：协作研究：用于在下一代 HPC 系统上管理和处理大数据的可扩展中间件

• 批准号: 1447861
• 负责人: Amitava Majumdar
• 金额: $ 36万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-09-01 至 2017-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1447861&HistoricalAwards=false
• 关键词: BIGDATA; DKM; Collaborative; Research; Scalable

Managing and processing large volumes of data and gaining meaningful insights is a significant challenge facing the Big Data community. Thus, it is critical that data-intensive computing middleware (such as Hadoop, HBase and Spark) to process such data are diligently designed, with high performance and scalability, in order to meet the growing demands of such Big Data applications. While Hadoop, Spark and HBase are gaining popularity for processing Big Data applications, these middleware and the associated Big Data applications are not able to take advantage of the advanced features on modern High Performance Computing (HPC) systems widely deployed all over the world, including many of of the multi-Petaflop systems in the XSEDE environment. Modern HPC systems and the associated middleware (such as MPI and Parallel File systems) have been exploiting the advances in HPC technologies (multi/many-core architectures, RDMA-enabled networking, NVRAMs and SSDs) during the last decade. However, Big Data middleware (such as Hadoop, HBase and Spark) have not embraced such technologies. These disparities are taking HPC and Big Data processing into "divergent trajectories." The proposed research, undertaken by a team of computer and application scientists from OSU and SDSC, aim to bring HPC and Big Data processing into a "convergent trajectory." The investigators will specifically address the following challenges: 1) designing novel communication and I/O runtime for Big Data processing while exploiting the features of modern multi-/many-core, networking and storage technologies; 2) redesigning Big Data middleware (such as Hadoop, HBase and Spark) to deliver performance and scalability on modern and next-generation HPC systems; and 3) demonstrating the benefits of the proposed approach for a set of driving Big Data applications on HPC system. The proposed work targets four major workloads and applications in the Big Data community (namely data analytics, query, interactive, and iterative) using the popular Big Data middleware (Hadoop, HBase and Spark). The proposed framework will be validated on a variety of Big Data benchmarks and applications. The proposed middleware and runtimes will be made publicly available to the community. The research enables curricular advancements via research in pedagogy for key courses in the new data analytics program at Ohio State and SDSC -- among the first of its kind nationwide.

管理和处理大量数据并获得有意义的见解是大数据社区面临的重大挑战。 因此，至关重要的是，数据密集型计算中间件（例如Hadoop，HBase和Spark）以高性能和可扩展性的方式处理此类数据，以满足对此类大数据应用程序不断增长的需求。 尽管Hadoop，Spark和HBase在处理大数据应用程序方面越来越受欢迎，但这些中间件和相关的大数据应用程序无法利用现代高性能计算（HPC）系统的高级功能，包括在世界范围内广泛部署，包括Xsede环境中的许多多petaflop系统。 现代的HPC系统和相关的中间件（例如MPI和并行文件系统）在过去的十次中利用了HPC Technologies（多核架构，启用RDMA的网络，NVRAM和SSD）的进步。但是，大数据中间件（例如Hadoop，HBase和Spark）尚未接受此类技术。这些差异将HPC和大数据处理用于“不同的轨迹”。由OSU和SDSC的计算机和应用科学家团队进行的拟议研究旨在将HPC和大数据处理带入“融合轨迹”。调查人员将具体解决以下挑战：1）设计新颖的通信和I/O运行时进行大数据处理，同时利用现代多核/多核，网络和存储技术的功能； 2）重新设计大数据中间件（例如Hadoop，HBase和Spark），以在现代和下一代HPC系统上提供性能和可伸缩性； 3）展示提出方法在HPC系统上驱动大数据应用程序的好处。 拟议的工作使用流行的大数据中间件（Hadoop，HBase和Spark），针对大数据社区（即数据分析，查询，交互式和迭代）的四个主要工作负载和应用程序。 所提出的框架将在各种大数据基准和应用程序上进行验证。 拟议的中间件和运行时间将公开向社区公开。 该研究可以通过在俄亥俄州立大学和SDSC的新数据分析计划中的教学法研究中进行课程进步，这是全国范围内的第一个。