CSR: Small: Collaborative Research: An Integrated Approach to Performance Modeling and Optimization of Big-data Scientific Workflows

CSR：小型：协作研究：大数据科学工作流程性能建模和优化的综合方法

• 批准号: 1525537
• 负责人: Yi Gu
• 金额: $ 12.5万
• 依托单位: Middle Tennessee State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-10-01 至 2018-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1525537&HistoricalAwards=false
• 关键词: CSR; Small; Collaborative; Research; Integrated

Next-generation e-science is producing colossal amounts of data, commonly known as Big Data, on the order of terabyte at present and petabyte or even exabyte in the predictable future. These scientific applications typically feature data- and network-intensive workflows comprised of computing modules with intricate inter-module dependencies. Application users oftentimes need to manually configure their computing workflows in distributed environments in an ad-hoc manner, which significantly limits the productivity of scientists and constrains the utilization of resources.The end-to-end performance of big data scientific workflows depends on both the mapping scheme that determines module assignment and the scheduling policy that determines resource allocation. These two aspects of a workflow-based research process are traditionally treated as two separate topics, and the interactions between them have not been fully explored. As the scale and complexity of scientific workflows and network environments rapidly increase, each individual aspect of performance optimization has limited success. This research is an in-depth investigation into workflow execution dynamics in resource sharing environments to explore the interactions between workflow mapping and node scheduling on a unified application-support platform. The idea is to build a three-layer workflow optimization architecture that seamlessly integrates three interrelated components based on rigorous algorithmic design, theoretical dynamics analysis, and real network implementation, deployment, and evaluation. The successful completion of this project will provide a solid mathematical foundation for the analysis and control of system dynamics of big data scientific workflows, produce a suite of cooperative mapping and scheduling optimization solutions to facilitate scientific collaborations, and add an additional level of intelligence to existing workflow engines widely adopted in the current grid and cloud computing middleware. The resulting workflow optimization solutions will benefit a broad spectrum of workflow-based scientific applications

下一代电子科学正在产生大量数据，通常称为大数据，目前约为 TB 量级，在可预测的未来将达到 PB 甚至 EB 量级。这些科学应用程序通常具有数据和网络密集型工作流程，由具有复杂模块间依赖性的计算模块组成。应用程序用户经常需要在分布式环境中以临时方式手动配置计算工作流程，这极大地限制了科学家的生产力并限制了资源的利用。大数据科学工作流程的端到端性能取决于以下两个方面：映射方案决定模块分配，调度策略决定资源分配。基于工作流的研究过程的这两个方面传统上被视为两个独立的主题，并且它们之间的相互作用尚未得到充分探讨。随着科学工作流程和网络环境的规模和复杂性迅速增加，性能优化的每个单独方面都取得了有限的成功。本研究深入研究资源共享环境中的工作流执行动态，以探索统一应用支持平台上工作流映射和节点调度之间的交互。其想法是建立一个三层工作流优化架构，基于严格的算法设计、理论动力学分析和真实的网络实现、部署和评估，无缝集成三个相互关联的组件。该项目的成功完成将为大数据科学工作流程的系统动力学分析和控制提供坚实的数学基础，产生一套协作制图和调度优化解决方案以促进科学协作，并为现有的智能系统增添额外的智能水平。工作流引擎在当前的网格和云计算中间件中广泛采用。 由此产生的工作流程优化解决方案将有利于广泛的基于工作流程的科学应用