XPS: SDA: Collaborative Research: A Scalable and Distributed System Framework for Compute-Intensive and Data-Parallel Applications

XPS：SDA：协作研究：用于计算密集型和数据并行应用的可扩展分布式系统框架

• 批准号: 1337131
• 负责人: Wuchun Feng
• 金额: $ 37.5万
• 依托单位: Virginia Polytechnic Institute and State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2013
• 资助国家: 美国
• 起止时间: 2013-09-01 至 2017-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1337131&HistoricalAwards=false
• 关键词: XPS; SDA; Collaborative; Research; Scalable

Whereas traditional high-performance computing (HPC) applications are computationally intensive, recent HPC applications require more data-intensive analysis and visualization to extract knowledge. In many cases, these applications execute the same computational algorithm as in the past (e.g., parallel search or parallel rendering) but now must do so for significantly larger data sets. For example, the life sciences, along with the cross-cutting area of scientific visualization, constitute an emerging category of HPC applications that not only perform sophisticated calculations but also ingest a sea of data. Running these new HPC data-parallel applications on today's computing platforms imposes new challenges and demands additional functionality.However, today's HPC platforms still adopt a compute-centric model and do not handle these new challenges well. Such a model often moves a large amount of data to various parallel computational processes. Consequently, long CPU wait times for I/O to complete and enormous data-movement overhead become major stumbling blocks to high performance and scalability. This project encompasses the creation of a scalable cross-layer software framework to enable both computationally intensive and data-intensive parallel HPC applications to run on distributed file systems. This framework consists of two interwoven research tasks: (1) an adaptive, data locality-aware, middleware system that dynamically schedules compute processes to access local data by monitoring physical data locations and (2) a framework that captures the computation and data I/O processing relationship from parallel applications and coordinates the scheduling of the corresponding process and I/O execution for maximum parallel efficiency. The success of this project contributes enhanced productivity and return on investment on HPC resources via the elimination of both CPU wait time and network transfer of frequently accessed data in scientific applications. An open-source, sustainable, and reusable software framework is delivered to speed-up the discovery and innovation process in areas such as bioinformatics, climate, high-energy physics, cosmology, astrophysics, and chromodynamics. The synergy in the two proposing institutions, Virginia Tech and the University of Central Florida, and their collaborating DOE national laboratories, will catalyze new and beneficial perspectives in the graduate education of students and prepare a 21st-century workforce in HPC.

尽管传统的高性能计算（HPC）应用程序在计算密集程度上，但最近的HPC应用程序需要更多的数据密集型分析和可视化才能提取知识。在许多情况下，这些应用程序执行与过去相同的计算算法（例如，并行搜索或并行渲染），但现在必须对更大的数据集进行此操作。例如，生命科学以及科学可视化的交叉切割领域构成了HPC应用程序的新兴类别，不仅执行复杂的计算，而且还会摄取大量数据。在当今的计算平台上运行这些新的HPC数据并行应用程序会引起新的挑战，并需要其他功能。但是，当今的HPC平台仍然采用以计算为中心的模型，并且不能很好地应对这些新挑战。这样的模型通常将大量数据移至各种并行计算过程。因此，长时间的CPU等待时间让I/O完成巨大的数据移动开销成为高性能和可扩展性的主要绊脚石。该项目涵盖了可扩展的跨层软件框架的创建，以实现计算密集型和数据密集的并行HPC应用程序以在分布式文件系统上运行。该框架由两个交织的研究任务组成：（1）自适应，数据位置感知，中间件系统，该系统动态地安排计算过程，通过监视物理数据位置来访问本地数据，以及（2）捕获计算和数据I/O处理的框架，从并行应用程序和对应的程序进行了对相应的过程和I/O的时间表和I/o的执行效率。该项目的成功通过消除了科学应用程序中常见数据的CPU等待时间和网络传输，从而提高了HPC资源的生产率和投资回报率。提供了一个开源，可持续和可重复使用的软件框架，以加快生物信息学，气候，高能物理学，宇宙学，天体物理学和染色体动力学等领域的发现和创新过程。这两个提议的机构，弗吉尼亚理工大学和佛罗里达大学及其合作的DOE国家实验室的协同作用将促进学生研究生教育的新和有益的观点，并在HPC准备21世纪的员工。