CIF21 DIBBs: PD: OneDataShare: A Universal Data Sharing Building Block for Data-Intensive Applications

CIF21 DIBB：PD：OneDataShare：数据密集型应用程序的通用数据共享构建块

• 批准号: 1724898
• 负责人: Tevfik Kosar
• 金额: $ 49.78万
• 依托单位: SUNY at Buffalo
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-09-01 至 2022-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1724898&HistoricalAwards=false
• 关键词: CIF21; DIBBs; PD; OneDataShare; Universal

Applications in scientific, industrial, and personal spaces now generate more data than ever before. As data become more abundant and data resources become more heterogeneous, the accessing, sharing and disseminating of data sets becomes a bigger challenge. Existing technologies for transferring and sharing data suffer from serious shortcomings, including low transfer performance, inflexibility, restricted protocol support, and poor scalability. This project develops a universal data sharing building block for data-intensive applications, dubbed OneDataShare, with three major goals: (1) optimization of end-to-end data transfers and reduction of the time to delivery of the data; (2) interoperation across heterogeneous and incompatible data resources; and (3) predicting the data delivery time and decreasing the uncertainty in real-time decision-making processes.OneDataShare deliverables include: (1) design and implementation of novel algorithms for application-layer optimization of the data transfer protocol parameters to achieve optimal end-to-end data transfer throughput; (2) development of a universal interface specification for heterogeneous data storage endpoints and a framework for on-the-fly data transfer protocol translation; (3) instrumentation of end-to-end data transfer time prediction capability, and feeding it into real-time scheduling and decision-making processes for advanced provisioning, high-level planning, and co-scheduling of resources; (4) deployment of these capabilities as stand-alone OneDataShare cloud-hosted services to end users; and (5) integration of these capabilities with widely used data scheduling and workflow management tools, and validation in specific applications. OneDataShare services and tools are developed at the application level, and they do not require any changes to the existing infrastructure, nor to the low-level networking stack, although they increase the end-to-end performance of the data movement tasks substantially. These efficient and high-performance data transfer techniques will help the scientific community, industry, and end-users to save significant time and effort in transferring and sharing data.

科学、工业和个人空间中的应用现在产生比以往更多的数据。随着数据变得更加丰富、数据资源更加异构，数据集的访问、共享和传播成为更大的挑战。 现有的数据传输和共享技术存在严重的缺点，包括传输性能低、灵活性差、协议支持受限和可扩展性差。 该项目为数据密集型应用程序开发了一个通用数据共享构建块，称为 OneDataShare，具有三个主要目标：（1）优化端到端数据传输并减少数据交付时间； (2) 跨异构、不兼容的数据资源的互操作； (3) 预测数据传输时间并减少实时决策过程中的不确定性。OneDataShare 可交付成果包括： (1) 设计和实现新算法，用于数据传输协议参数的应用层优化，以实现最佳结果- 到终端的数据传输吞吐量； (2) 开发异构数据存储端点的通用接口规范和动态数据传输协议转换框架； (3) 检测端到端数据传输时间预测能力，并将其反馈到实时调度和决策过程中，以实现资源的高级配置、高层规划和联合调度； (4) 将这些功能作为独立的 OneDataShare 云托管服务部署给最终用户； (5) 将这些功能与广泛使用的数据调度和工作流管理工具集成，并在特定应用中进行验证。 OneDataShare 服务和工具是在应用程序级别开发的，不需要对现有基础设施或低级网络堆栈进行任何更改，尽管它们大大提高了数据移动任务的端到端性能。 这些高效、高性能的数据传输技术将帮助科学界、行业和最终用户在传输和共享数据时节省大量时间和精力。

项目成果

GreenDataFlow: Minimizing the Energy Footprint of Global Data Movement

• DOI: 10.1109/bigdata.2018.8622570
• 发表时间: 2018-10
• 期刊: 2018 IEEE International Conference on Big Data (Big Data)
• 作者: M. S. Q. Z. Nine;Luigi Di Tacchio;A. Imran;T. Kosar;Muhammed Fatih Bulut;Jinho Hwang

Energy-saving Cross-layer Optimization of Big Data Transfer Based on Historical Log Analysis

基于历史日志分析的大数据传输节能跨层优化

• DOI: 10.1109/icc42927.2021.9500693
• 发表时间: 2021
• 期刊: ICC 2021 - IEEE International Conference on Communications
• 作者: Rodolph, Lavone;Zulkar Nine, MD S;Di Tacchio, Luigi;Kosar, Tevfik
• 通讯作者: Kosar, Tevfik

High-Speed Transfer Optimization Based on Historical Analysis and Real-Time Tuning

基于历史分析和实时调优的高速传输优化

• DOI: 10.1109/tpds.2018.2790948
• 发表时间: 2018
• 期刊: IEEE Transactions on Parallel and Distributed Systems
• 影响因子: 5.3
• 作者: Arslan, Engin;Kosar, Tevfik
• 通讯作者: Kosar, Tevfik

Big data transfer optimization based on offline knowledge discovery and adaptive sampling

• DOI: 10.1109/bigdata.2017.8257959
• 发表时间: 2017-12
• 期刊: 2017 IEEE International Conference on Big Data (Big Data)
• 作者: M. S. Q. Z. Nine;Kemal Guner;Ziyun Huang;Xiangyu Wang;Jinhui Xu;T. Kosar

Cross-Layer Optimization of Big Data Transfer Throughput and Energy Consumption

• DOI: 10.1109/cloud.2019.00017
• 发表时间: 2019-07
• 期刊: 2019 IEEE 12th International Conference on Cloud Computing (CLOUD)
• 作者: Luigi Di Tacchio;M. S. Q. Z. Nine;T. Kosar;Muhammed Fatih Bulut;Jinho Hwang