Elements: Adaptive End-to-End Parallelism for Distributed Science Workflows

要素：分布式科学工作流程的自适应端到端并行性

基本信息

批准号：
2209955
负责人：
Engin Arslan
金额：
$ 45万
依托单位：
Board of Regents, NSHE, obo University of Nevada, Reno
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2022
资助国家：
美国
起止时间：
2022-08-01 至 2024-05-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2209955&HistoricalAwards=false
关键词：
Elements Adaptive End Parallelism Distributed

项目摘要

Technological advancements in sensing and computing technologies have led to an unprecedented increase in the amount of data generated by scientific applications. As science projects are increasingly distributed in nature, the increase in data sizes in turn results in an increased volume of traffic that needs to be moved across geographically distributed locations. Although significant investments have been made to build high-speed networks to facilitate data movements between research and education institutions, it is difficult for domain scientists to efficiently utilize this available capacity mainly due to the lack of scalable data transfer services. This project addresses this need by developing a scalable and reliable data transfer service. It further integrates the data transfer service into elastic workflow management systems to achieve end-to-end optimization for distributed science workflows. This project makes three novel contributions to the field: (i) it innovates scalable integrity verification and encryption for file transfers to ensure the reliability of file transfers without sacrificing performance. It takes advantage of computing resources available at data transfer nodes to scale the performance of integrity verification and channel encryption features. (ii) It innovates end-to-end parallelism for distributed workflows by integrating an online transfer optimization service into elastic workflow management tools. Unlike existing workflow management solutions, which merely focus on the optimization of computing tasks, the proposed integration of online transfer optimization services into elastic workflow schedulers enables true end-to-end parallelism for distributed workflows. (iii) Finally, it demonstrates the performance of the developed service on a real-world bioscience workflow that streams a large volume of sequence read archive data from the NCBI database to extract computation-ready SAM/BAM files.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

传感和计算技术的进步导致科学应用产生的数据量空前增加。随着科学项目在本质上越来越分散，数据大小的增加反过来导致需要在地理分布的位置之间移动的流量增加。尽管已经投入了大量资金来建设高速网络以促进研究和教育机构之间的数据传输，但由于缺乏可扩展的数据传输服务，领域科学家很难有效地利用这种可用容量。该项目通过开发可扩展且可靠的数据传输服务来满足这一需求。它将数据传输服务进一步集成到弹性工作流管理系统中，以实现分布式科学工作流的端到端优化。该项目对该领域做出了三项新颖的贡献：（i）创新了文件传输的可扩展完整性验证和加密，以确保文件传输的可靠性而不牺牲性能。它利用数据传输节点上可用的计算资源来扩展完整性验证和通道加密功能的性能。 (ii) 通过将在线传输优化服务集成到弹性工作流管理工具中，创新了分布式工作流的端到端并行性。与仅关注计算任务优化的现有工作流管理解决方案不同，拟议将在线传输优化服务集成到弹性工作流调度程序中，为分布式工作流实现真正的端到端并行性。 (iii) 最后，它展示了所开发的服务在现实世界生物科学工作流程中的性能，该工作流程从 NCBI 数据库传输大量序列读取存档数据以提取可计算的 SAM/BAM 文件。该奖项反映了 NSF 的法定使命通过使用基金会的智力优点和更广泛的影响审查标准进行评估，并被认为值得支持。