BIGDATA: F: DKM: Collaborative Research: PXFS: ParalleX Based Transformative I/O System for Big Data

BIGDATA：F：DKM：协作研究：PXFS：基于 ParalleX 的大数据变革性 I/O 系统

• 批准号: 1447771
• 负责人: Walter Ligon
• 金额: $ 30万
• 依托单位: Clemson University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-09-01 至 2017-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1447771&HistoricalAwards=false
• 关键词: BIGDATA; DKM; Collaborative; Research; PXFS

Recent decades have seen the development of computational science where modeling and data analysis are critical to exploration, discovery, and refinement of new innovations in science and engineering. More recently the techniques have been applied to arts, social, political and other fields less traditionally reliant on high performance computing. This innovation has grown out of realization some 20 years ago that I/O (input/output) support for high performance parallel and distributed architectures had lagged behind that of pure computational speed, and further that bring I/O up to speed was both critical, and a rather difficult problem. The core hurdle of contemporary I/O on large HPC machines relates to issues of latency in large parts caused by the deficiencies of the historical I/O model that was relevant when computers were exclusively large, centralized, single processor systems shared by many time-sharing programs. In order to improve I/O on scalability on future hardware architectures novel approaches are required.This project is conducting research on an extension of ParalleX, a new highly innovative parallel execution model. The extension provides a powerful I/O interface that allows researchers to create highly efficient data management, discovery, and analysis codes for Big Data applications. This new extension, known as PXFS, is based on HPX, an implementation of ParalleX based on C++, and OrangeFS, a high performance parallel file system. The research goal driving PXFS is to extend HPX objects into I/O space so that the objects become persistent and storage becomes another class of memory, all accessed as a single virtual address space and managed by an event driven dynamic adaptive computation environment. Critical aspects of this approach include futures-based synchronization, dynamic locality management, dynamic resource management, hierarchical name space, and an active global address space (AGAS). The overall goals of PXFS are to eliminate the division of programming imposed by conventional file system through the unification of name spaces and their management, and to minimize global synchronization in order to support asynchronous concurrency. The research methodology is to implement a Map/Reduce application framework using PXFS and evaluate its effectiveness in both performance and ease of use.This project is conducted at three major research universities involving undergraduate and graduate students, post-docs, and high-school teachers and their students. The project includes a PI from the functional genomics field acting as domain science expert in order to focus the development efforts on real world problems. Graduate students and post-docs involved in the project are trained in these areas to promote scientists who understanding both aspects of Big Data problems. The project engages under represented minorities with the goal to inspire them to pursue a career in computer science or genomics. The software developed by the project is available open-source and archived using an integrated source code revision repository, wiki, and bug tracking software system in addition to code releases with accompanying documentation.

近几十年来，计算科学的发展，其中建模和数据分析对于探索，发现和改进科学和工程学的新创新至关重要。最近，这些技术已应用于艺术，社会，政治和其他领域，传统上不太依赖高性能计算。大约20年前，这种创新已经发展出来，I/O（输入/输出）对高性能平行和分布式体系结构的支持落后于纯计算速度，此外，使I/O达到速度既是至关重要的，又是一个相当困难的问题。当代I/O在大型HPC机器上的核心障碍涉及由历史I/O模型的缺陷引起的延迟问题，而I/O模型的缺陷是相关的，当计算机完全是大型，集中式的，单一的处理器系统，由许多时间共享计划共享。为了改善对未来硬件体系结构的可伸缩性的I/O，需要新的方法。该项目正在对Parallex的扩展进行研究，Parallex是一种新的高度创新的并行执行模型。该扩展名提供了功能强大的I/O接口，使研究人员可以为大数据应用程序创建高效的数据管理，发现和分析代码。 这种新扩展名为PXFS，基于HPX，基于C ++的parallex的实现，而OrangeF则是高性能并行文件系统。 驱动PXFS的研究目标是将HPX对象扩展到I/O空间，以使对象变得持久，并将存储成为另一种内存类，所有这些都作为单个虚拟地址空间访问，并通过事件驱动的动态自适应计算环境进行管理。 这种方法的关键方面包括基于期货的同步，动态局部性管理，动态资源管理，层次名称空间和主动的全球地址空间（AGA）。 PXF的总体目标是消除传统文件系统通过名称空间及其管理施加的编程，并最大程度地减少全局同步以支持异步并发。 该研究方法是使用PXF实施地图/减少应用程序框架，并评估其在绩效和易用性方面的有效性。该项目是在涉及本科生和研究生，毕业后，高中老师及其学生的三个主要研究所进行的。 该项目包括功能基因组学领域的PI，该领域是领域科学专家，以将开发工作集中在现实世界问题上。在这些领域中培训了参与该项目的研究生和培训后，以促进了解大数据问题的两个方面的科学家。 该项目以代表的少数群体为目标，目的是激发他们从事计算机科学或基因组学的职业。该项目开发的软件可提供开源和使用的集成源代码修订存储库Wiki和错误跟踪软件系统的存档，此外还有随附文档的代码发布。