NeTS: Small: Collaborative Research: Rethinking Erasure Codes for Cloud Storage: A Quantitative Framework for Latency, Reliability, and Cost Optimization

NeTS：小型：协作研究：重新思考云存储纠删码：延迟、可靠性和成本优化的定量框架

• 批准号: 1618335
• 负责人: Vaneet Aggarwal
• 金额: $ 25万
• 依托单位: Purdue University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-10-01 至 2022-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1618335&HistoricalAwards=false
• 关键词: NeTS; Small; Collaborative; Research; Rethinking

This project aims to develop an analytical framework that quantifies tail latency and reliability of erasure-coded storage through investigation of novel scheduling and repair strategies, mandating rethinking of erasure codes for online storage. As erasure coding is increasingly adopted by large-scale storage systems such as Microsoft Azure and Facebook, conventional approaches that primarily rely on system design heuristics has become inadequate in pushing the performance boundaries in terms of latency and reliability optimization. Quantifying tail latency and reliability of an erasure-coded storage that employs dynamic workload scheduling and online repair is an open problem. There exists little work illuminating the design space via mathematical crystallization of key tradeoffs and associated engineering "control knobs". This project will enable a joint optimization of latency, reliability and storage cost, which mandates rethinking of erasure-coded storage optimization and service pricing models. This proposal will concentrate on the following specific aspects: (1) Investigating a family of new probabilistic scheduling policies and extend order statistic analysis to derive closed-form bounds on tail latency for erasure-coded storage with arbitrary configurations, general service-time distributions, and potentially differentiated service classes. (2) Through a novel reliability metric, Time to Data Loss, The research will investigate online repair strategies that significantly improve reliability using a class of bandwidth-efficient codes, enabling a tradeoff between repair timeliness and reliability optimization. (3) Employ the theoretical analysis in this research to pursue a holistic solution that jointly optimizes reliability, latency, and storage costs over seven key control dimensions: choice of erasure codes, chunk placement, network resource allocation, cache management, dynamic scheduling, pricing, and online repair strategy. (4) Integrate the proposed framework with current cloud systems to bridge the gap between the theoretical results and practical systems. By developing new analytical models and algorithms for joint optimization of latency, reliability, and storage cost, the project will mandate rethinking of erasure-coded storage design and service pricing models. It will produces novel, interdisciplinary curriculum modules for teaching both these theories and systems.

该项目旨在开发一个分析框架，通过研究新颖的调度和修复策略来量化纠删码存储的尾部延迟和可靠性，要求重新考虑在线存储的纠删码。随着纠删码越来越多地被 Microsoft Azure 和 Facebook 等大型存储系统采用，主要依赖于系统设计启发式的传统方法已经不足以突破延迟和可靠性优化方面的性能界限。量化采用动态工作负载调度和在线修复的纠删码存储的尾部延迟和可靠性是一个悬而未决的问题。通过关键权衡和相关工程“控制旋钮”的数学结晶来阐明设计空间的工作很少。该项目将实现延迟、可靠性和存储成本的联合优化，这要求重新思考纠删码存储优化和服务定价模型。该提案将集中在以下具体方面：（1）研究一系列新的概率调度策略并扩展顺序统计分析，以导出具有任意配置、一般服务时间分布的纠删码存储的尾部延迟的封闭形式界限，以及潜在的差异化服务类别。 (2) 通过一种新颖的可靠性指标“数据丢失时间”，该研究将研究在线修复策略，使用一类带宽高效的代码显着提高可靠性，从而在修复及时性和可靠性优化之间进行权衡。 (3) 利用本研究中的理论分析来寻求整体解决方案，在七个关键控制维度上联合优化可靠性、延迟和存储成本：纠删码的选择、块放置、网络资源分配、缓存管理、动态调度、定价，以及在线修复策略。 （4）将所提出的框架与当前的云系统相集成，以弥合理论结果与实际系统之间的差距。通过开发新的分析模型和算法来联合优化延迟、可靠性和存储成本，该项目将要求重新思考纠删码存储设计和服务定价模型。它将产生新颖的跨学科课程模块来教授这些理论和系统。