Optimizing compute task scheduling at Shopify

优化 Shopify 的计算任务调度

• 批准号: 514614-2017
• 负责人: Beschastnikh, Ivan
• 金额: $ 1.82万
• 依托单位: University of British Columbia
• 依托单位国家: 加拿大
• 项目类别: Engage Grants Program
• 财政年份: 2017
• 资助国家: 加拿大
• 起止时间: 2017-01-01 至 2018-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=640686
• 关键词: Optimizing; compute; task; scheduling; Shopify

Big Data processing frameworks allow enterprises to drive their business based on key insights extracted fromnear real-time data. Apache Spark is a framework that Shopify leverages to collect and analyze customerinformation and expose data insights to the shop owners who are users of the Shopify platform. For example,the enterprise which built its online store using the Shopify platform can send pre-set queries to the platform tolearn about the impact of their recent marketing campaign (e.g., with an improved Google ad keywords) bygeographical region or by age group, and see if the marketing campaign resulted in actual product sales.Shopify uses Spark to analyze data by passing the data through a processing pipeline. The pipeline consists ofmultiple jobs, each applying some computation to the data. A set of connected jobs form a flow which mightcorrespond to the query exposed to the shop owner. In this project we will explore algorithmic solutions tomaintain a useful flow execution frequency while absorbing the variance caused by job failures, and ensuringco-execution of pre-scheduled and ad-hoc jobs without resource starvation. We intend to apply approachesbased on heuristics and constraint satisfaction, study their tradeoffs, and develop a solution that works best inpractice.

大数据处理框架使企业能够根据从Near实时数据提取的关键见解来推动其业务。 Apache Spark是一个框架，该框架将杠杆利用收集和分析客户信息，并将数据见解传播给Shopify平台用户的商店所有者。例如，使用Shopify平台构建其在线商店的企业可以将预设的查询发送到Tolarearn，以了解其最近的营销活动的影响（例如，具有改进的Google Ad关键字）副网站或按年龄组群体，并查看营销活动是否导致了实际产品销售。管道组成了数量工作，每个作业都对数据应用了一些计算。一组连接的作业形成了一个流动，这可能与面向商店所有者的查询相对应。在这个项目中，我们将探索算法解决方案，同时吸收由工作失败引起的差异，并确保在没有资源饥饿的情况下进行预先安排和临时工作的工作。我们打算将基于启发式和约束满意度的方法应用，研究他们的权衡，并开发一种最有效的解决方案。