EAGER: Harnessing Dependency to Achieve Efficient Resource Management in Scalable Multi-stage Data Processing Systems

EAGER：利用依赖性在可扩展的多级数据处理系统中实现高效的资源管理

• 批准号: 1552525
• 负责人: Bo Sheng
• 金额: $ 12.5万
• 依托单位: University of Massachusetts Boston
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-09-01 至 2017-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1552525&HistoricalAwards=false
• 关键词: EAGER; Harnessing; Dependency; Achieve; Efficient

This EArly-concept Grants for Exploratory Research (EAGER) award will develop an efficient resource management framework for a large-scale system to serve a large volume of multi-stage data processing jobs. The major research goals include enabling cross-job data flows, and exploiting the dependency between stages to efficiently allocate system resources. The current data processing framework encapsulates each job during its execution which disables possible data sharing between jobs (stages) and query optimization. The proposed work for this research will allow users to specify enriched job meta-data that identifies the data input of each stage. Processing servers will recognize the cross-job data flows according to the job meta-data. In addition, a significant contribution will be a dependency-driven resource management scheme that carefully examines the unique overlapping transition and dependency between consecutive stages. The proposed work will improve the existing platforms of multi-stage data processing. In addition, the collaborations with industrial contacts have the potential for rapid technology transfer.

这项早期概念授予探索性研究（急切）奖将为大规模系统提供有效的资源管理框架，以服务大量的多阶段数据处理工作。主要的研究目标包括启用跨乔布数据流，并利用阶段之间的依赖性以有效分配系统资源。当前的数据处理框架在执行过程中封装了每个作业，该框架可以在作业（阶段）和查询优化之间禁用可能的数据共享。这项研究的拟议工作将使用户可以指定识别每个阶段数据输入的丰富工作元数据。处理服务器将根据作业元数据识别跨工作数据流。此外，重大贡献将是依赖性驱动的资源管理方案，该方案仔细研究了连续阶段之间独特的重叠过渡和依赖性。拟议的工作将改善多阶段数据处理的现有平台。此外，与工业接触的合作有可能进行快速技术转移。