CIF: Medium: Collaborative Research: Coded Computing for Large-Scale Machine Learning

CIF：媒介：协作研究：大规模机器学习的编码计算

基本信息

批准号：
1763702
负责人：
Georgios-Alex Dimakis
金额：
$ 29.95万
依托单位：
University of Texas at Austin
依托单位国家：
美国
项目类别：
Continuing Grant
财政年份：
2018
资助国家：
美国
起止时间：
2018-09-01 至 2022-08-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1763702&HistoricalAwards=false
关键词：
CIF Medium Collaborative Research Coded

项目摘要

Deep learning models are breaking new ground in data science tasks including image recognition, automatic translation and autonomous driving. This is achieved by neural networks that can be hundreds of layers deep and involve hundreds of millions of parameters. Training such large models requires distributed computations, very long training times and expensive hardware. This project studies coding theoretic techniques that can accelerate distributed machine learning and allow training with cheaper commodity hardware. Beyond the development of theoretical foundations, this project develops new algorithms for providing fault tolerance over unreliable cloud infrastructure that can significantly reduce the cost of large-scale machine learning. The research outcomes of the project will be broadly disseminated and integrated into education. The specific focus of this research program is on mitigating the bottlenecks of distributed machine learning. Currently, scaling benefits are limited because of two reasons: first, communication is typically the bottleneck and second, straggler effects limit performance. Both problems can be mitigated using coding theoretic methods. This work proposes "coded computing", a transformative framework that combines coding theory with distributed computing to inject computational redundancy in a novel coded form. This framework is then used to develop three research thrusts: a) Coding for Linear Algebraic Computations b) Coding for Iterative Computations and c) Coding for General Distributed Computations. Each of the thrusts operates on a different layer of a machine learning pipeline but all rely on coding theoretic tools and distributed information processing.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习模型正在打破数据科学任务中的新基础，包括图像识别，自动翻译和自动驾驶。这是通过神经网络实现的，这些神经网络可能是数百万层，并涉及数亿个参数。培训这样的大型模型需要分布式计算，培训时间很长和昂贵的硬件。该项目研究了可以加速分布式机器学习并允许使用更便宜的商品硬件进行培训的理论技术编码。除了发展理论基础之外，该项目还开发了新的算法，以提供对不可靠的云基础架构的容忍度，从而可以大大降低大型机器学习的成本。该项目的研究成果将被广泛传播并纳入教育。该研究计划的具体重点是减轻分布式机器学习的瓶颈。目前，由于两个原因，扩展好处是限制的：首先，沟通通常是瓶颈，其次是Straggler效果限制性能。可以使用编码理论方法来减轻这两个问题。这项工作提出了“编码计算”，这是一个将编码理论与分布式计算结合到新颖编码形式注入计算冗余的变革框架。然后，该框架用于开发三个研究推力：a）编码线性代数计算的编码b）编码迭代计算，c）编码一般分布式计算。每个推力都在机器学习管道的不同层上运行，但所有推力都依赖于编码理论工具和分布式信息处理。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子和更广泛影响的评估评估来审查标准的。