CSSI Elements: DataSwarm: A User-Level Framework for Data Intensive Scientific Applications

CSSI 元素：DataSwarm：数据密集型科学应用程序的用户级框架

基本信息

批准号：
1931348
负责人：
Douglas Thain
金额：
$ 56.3万
依托单位：
University of Notre Dame
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2019
资助国家：
美国
起止时间：
2019-09-01 至 2024-08-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1931348&HistoricalAwards=false
关键词：
CSSI Elements DataSwarm User Level

项目摘要

This project creates a capability that will support the construction of large, data intensive scientific applications that must run on top of national cyberinfrastructure, such as large campus clusters, NSF extreme-scale computing facilities, the Open Science Grid, and commercial clouds. The new capability (DataSwarm) brings data requirements and software dependencies to the target cyberinfrastructure systems, and deploys them as and when required, rather than having these requirements pre-installed on the target systems. The motivation comes from applications in high energy physics, molecular dynamics, and quantum chemistry.The main motivation of the work is the challenge of scalable computing frameworks. Based on a prior development by the Principal Investigator (Work Queue), the current project provides technical innovation in three areas: (1) Molecular Task Composition. Molecular task composition is used as an abstraction for the precise construction of tasks that require a custom software environment, large data input, and a scratch data area to capture the outputs. By expressing these aspects explicitly instead of implicitly, the project improves the storage efficiency of large numbers of tasks. (2) In-Situ Data Management. In-situ storage management is performed to offset the increased storage consumption likely to occur under molecular task composition, avoiding unpredictable failures of tasks due to storage exhaustion. (3) Precision Provenance. Precision provenance of both data objects and task components enables the efficient re-use of resources across multiple runs, as well as precise incremental changes to complex workflows.For this project, the three key elements addressed are the software environment, input data, and a scratch data area. These elements are usually independently managed; here, they are bound together to form temporary "molecules" for task execution. The three applications included in this project represent three typical types of complex data and complex software dependencies. They include custom late-stage data analysis codes in high energy physics, complex multidimensional optimization, and ensemble molecular dynamics, respectively.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目创建了一个能力，该功能将支持必须在国家网络基础设施之上运行的大型数据密集的科学应用，例如大型校园集群，NSF极端计算设施，开放科学网格和商业云。新的功能（DataSwarm）将数据需求和软件依赖性带到了目标网络基础结构系统，并在需要时将其部署为和部署，而不是将这些要求预先安装在目标系统上。动机来自高能物理，分子动力学和量子化学的应用。作品的主要动机是可扩展计算框架的挑战。基于首席研究员（工作队列）的先前开发，当前项目在三个领域提供了技术创新：（1）分子任务组成。分子任务组成被用作需要定制软件环境，大数据输入和刮擦数据区域的任务的精确构造来捕获输出的一个抽象。通过明确表示这些方面而不是隐含地表达这些方面，该项目提高了大量任务的存储效率。（2）原位数据管理。执行原位存储管理以抵消在分子任务组成下可能发生的增加的存储消耗，从而避免由于存储耗尽而导致的任务失败。（3）精确出处。数据对象和任务组件的精确出处使得对复杂工作流程的多个运行以及精确的增量更改可以有效地重复资源。对于此项目，解决的三个关键元素是软件环境，输入数据和刮擦数据区域。这些要素通常是独立管理的；在这里，它们被绑定在一起，形成临时的“分子”以执行任务。该项目中包含的三个应用程序代表了三种典型的复杂数据和复杂的软件依赖性。它们分别包括高能物理学中的自定义后期数据分析代码，复杂的多维优化和集合分子动力学。该奖项反映了NSF的法定任务，并被认为是通过基金会的智力优点和更广泛的影响来通过评估来支持的。

项目成果

期刊论文数量（9）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

Lightweight Function Monitors for Fine-Grained Management in Large Scale Python Applications

用于大规模 Python 应用程序中细粒度管理的轻量级函数监视器

DOI：
10.1109/ipdps49936.2021.00088
发表时间：
2021
期刊：
IEEE International Parallel and Distributed Processing Symposium
影响因子：
0
作者：
Shaffer, Tim;Li, Zhuozhao;Tovar, Ben;Babuji, Yadu;Dasso, TJ;Surma, Zoe;Chard, Kyle;Foster, Ian;Thain, Douglas
通讯作者：
Thain, Douglas

Poster: Robust Meta-Workflow Management with Mufasa

海报：使用 Mufasa 进行稳健的元工作流程管理

DOI：
发表时间：
2022
期刊：
IEEE International Conference on eScience
影响因子：
0
作者：
Ben Lyons;Douglas Thain
通讯作者：
Douglas Thain

PONCHO: Dynamic Package Synthesis for Distributed and Serverless Python Applications

PONCHO：分布式和无服务器 Python 应用程序的动态包合成

DOI：
10.1145/3526060.3535459
发表时间：
2022
期刊：
High Performance Serverless Workshop at HPDC
影响因子：
0
作者：
Sly-Delgado, Barry;Locascio, Nick;Simonetti, David;Wiseman, Brett;Tovar, Ben;Thain, Douglas
通讯作者：
Thain, Douglas

Software Environments in Binder Containers

Binder 容器中的软件环境