CIF21 DIBBs: PD: OneDataShare: A Universal Data Sharing Building Block for Data-Intensive Applications

CIF21 DIBB：PD：OneDataShare：数据密集型应用程序的通用数据共享构建块

• 批准号: 1724898
• 负责人: Tevfik Kosar
• 金额: $ 49.78万
• 依托单位: SUNY at Buffalo
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-09-01 至 2022-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1724898&HistoricalAwards=false
• 关键词: CIF21; DIBBs; PD; OneDataShare; Universal

Applications in scientific, industrial, and personal spaces now generate more data than ever before. As data become more abundant and data resources become more heterogeneous, the accessing, sharing and disseminating of data sets becomes a bigger challenge. Existing technologies for transferring and sharing data suffer from serious shortcomings, including low transfer performance, inflexibility, restricted protocol support, and poor scalability. This project develops a universal data sharing building block for data-intensive applications, dubbed OneDataShare, with three major goals: (1) optimization of end-to-end data transfers and reduction of the time to delivery of the data; (2) interoperation across heterogeneous and incompatible data resources; and (3) predicting the data delivery time and decreasing the uncertainty in real-time decision-making processes.OneDataShare deliverables include: (1) design and implementation of novel algorithms for application-layer optimization of the data transfer protocol parameters to achieve optimal end-to-end data transfer throughput; (2) development of a universal interface specification for heterogeneous data storage endpoints and a framework for on-the-fly data transfer protocol translation; (3) instrumentation of end-to-end data transfer time prediction capability, and feeding it into real-time scheduling and decision-making processes for advanced provisioning, high-level planning, and co-scheduling of resources; (4) deployment of these capabilities as stand-alone OneDataShare cloud-hosted services to end users; and (5) integration of these capabilities with widely used data scheduling and workflow management tools, and validation in specific applications. OneDataShare services and tools are developed at the application level, and they do not require any changes to the existing infrastructure, nor to the low-level networking stack, although they increase the end-to-end performance of the data movement tasks substantially. These efficient and high-performance data transfer techniques will help the scientific community, industry, and end-users to save significant time and effort in transferring and sharing data.

现在，科学，工业和个人空间的应用程序比以往任何时候都产生更多的数据。随着数据变得越来越丰富，数据资源变得更加异质，数据集的访问，共享和传播将成为一个更大的挑战。 转移和共享数据的现有技术遭受了严重的缺点，包括转移性能低，僵化性，限制协议支持和可扩展性差。 该项目为数据密集型应用程序开发了一个通用数据共享构建块，称为OnEdatashare，具有三个主要目标：（1）优化端到端数据传输和减少数据传递的时间； （2）跨异构和不兼容的数据资源之间的互操作； （3）预测数据传递时间并减少实时决策过程中的不确定性。启动交付物包括：（1）设计和实施新型算法，用于应用数据传输协议参数的应用程序层优化，以实现最佳的端端端口数据传输吞吐量； （2）开发用于异质数据存储终点的通用接口规范和直接数据传输协议翻译的框架； （3）端到端数据传输时间预测能力的仪器，并将其归为实时计划和决策过程，以进行高级供应，高级计划和资源共同安排； （4）将这些功能部署为最终用户的独立OnEdatashare云托管服务； （5）将这些功能与广泛使用的数据调度和工作流管理工具以及在特定应用程序中进行验证的集成。 OneDatashare服务和工具是在应用程序级别开发的，尽管它们大大增加了数据移动任务的端到端性能，但它们不需要对现有基础架构的任何更改，也不需要对低级网络堆栈进行任何更改。 这些高效且高性能的数据传输技术将有助于科学界，行业和最终用户节省大量时间和精力在传输和共享数据方面。

项目成果

GreenDataFlow: Minimizing the Energy Footprint of Global Data Movement

• DOI: 10.1109/bigdata.2018.8622570
• 发表时间: 2018-10
• 期刊: 2018 IEEE International Conference on Big Data (Big Data)
• 作者: M. S. Q. Z. Nine;Luigi Di Tacchio;A. Imran;T. Kosar;Muhammed Fatih Bulut;Jinho Hwang

Energy-saving Cross-layer Optimization of Big Data Transfer Based on Historical Log Analysis

基于历史日志分析的大数据传输节能跨层优化

• DOI: 10.1109/icc42927.2021.9500693
• 发表时间: 2021
• 期刊: ICC 2021 - IEEE International Conference on Communications
• 作者: Rodolph, Lavone;Zulkar Nine, MD S;Di Tacchio, Luigi;Kosar, Tevfik
• 通讯作者: Kosar, Tevfik

High-Speed Transfer Optimization Based on Historical Analysis and Real-Time Tuning

基于历史分析和实时调优的高速传输优化

• DOI: 10.1109/tpds.2018.2790948
• 发表时间: 2018
• 期刊: IEEE Transactions on Parallel and Distributed Systems
• 影响因子: 5.3
• 作者: Arslan, Engin;Kosar, Tevfik
• 通讯作者: Kosar, Tevfik

Cross-Layer Optimization of Big Data Transfer Throughput and Energy Consumption

• DOI: 10.1109/cloud.2019.00017
• 发表时间: 2019-07
• 期刊: 2019 IEEE 12th International Conference on Cloud Computing (CLOUD)
• 作者: Luigi Di Tacchio;M. S. Q. Z. Nine;T. Kosar;Muhammed Fatih Bulut;Jinho Hwang

Big data transfer optimization based on offline knowledge discovery and adaptive sampling

• DOI: 10.1109/bigdata.2017.8257959
• 发表时间: 2017-12
• 期刊: 2017 IEEE International Conference on Big Data (Big Data)
• 作者: M. S. Q. Z. Nine;Kemal Guner;Ziyun Huang;Xiangyu Wang;Jinhui Xu;T. Kosar