Eager: Collaborative Research: DiRecMR: Reconciling the Dichotomy of MapReduce for Efficient Speculation and Resilience

Eager：协作研究：DiRecMR：调和 MapReduce 的二分法以实现高效推测和弹性

• 批准号: 1744317
• 负责人: Xian-He Sun
• 金额: $ 8万
• 依托单位: Illinois Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-08-01 至 2018-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1744317&HistoricalAwards=false
• 关键词: Eager; Collaborative; Research; DiRecMR; Reconciling; Eager; Collaborative; Research; DiRecMR; Reconciling

MapReduce systems have great capabilities in processing large amounts of data and have become a research target for governmental, academic and industrial organizations. However, their task management and fault handling policies do not recognize a tacit dichotomy that exists between its inherent two phases (map and reduce). This results in a number of critical issues, such as resource underutilization, prolonged task execution, myopic speculation, and failure amplifications. This project adopts a transformative combination of theoretical analysis, simulation and modeling, and systems design and implementation approaches in order to reconcile the dichotomy of MapReduce. The techniques from this project are potentially impactful to all organizations that deploy MapReduce systems and support Big Data applications from business analytics, social networks, and scientific computing research.Instead of empirical analysis of system behaviors to pinpoint resource management and task scheduling abnormalities, this project takes a different perspective on MapReduce efficiency and resilience, and formulates a Markov chain for the transition of Hadoop MapReduce containers, and a fork-join model for the queueing of map and reduce tasks. These formulations facilitate a theoretical analysis of the dichotomy of MapReduce and help shed light on its impact to asymptotic behaviors of large-scale workloads. This project aims to blend simulation and real system development together, and addresses the myopic speculation caused by dichotomy, liberates the scope of task speculation, and ensures task resilience without failure amplifications. These techniques are developed to enhance MapReduce platforms such as YARN and Spark. Besides the target on MapReduce systems, the research from this project addresses a general issue in distributed analytics environments.

MapReduce系统在处理大量数据方面具有很大的功能，并已成为政府，学术和工业组织的研究目标。但是，他们的任务管理和故障处理策略并未认识到其固有的两个阶段（地图和减少）之间存在的默认二分法。 这导致许多关键问题，例如资源不足，任务执行延长，近视投机和失败放大。该项目采用了理论分析，模拟和建模以及系统设计和实施方法的变革性组合，以调和MapReduce的二分法。该项目的技术对所有部署MAPREDUCE系统并支持大数据应用程序的组织有可能影响用于排队地图并减少任务的叉-Join模型。这些配方有助于对MapReduce的二分法进行理论分析，并有助于阐明其对大规模工作量的渐近行为的影响。该项目旨在将仿真和实际系统开发融合在一起，并解决由二分法引起的近视猜测，解放任务猜测的范围，并确保任务弹性而不会放大。这些技术的开发是为了增强MapReduce平台，例如纱线和Spark。除了对MapReduce系统的目标外，该项目的研究还解决了分布式分析环境中的一般问题。