CIF: Small: Coding Techniques for Distributed Machine Learning

CIF：小型：分布式机器学习的编码技术

• 批准号: 1910447
• 负责人: Jun Li
• 金额: $ 50万
• 依托单位: Florida International University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2020-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1910447&HistoricalAwards=false
• 关键词: CIF; Small; Coding; Techniques; Distributed

Modern machine learning models have achieved great success and have been widely deployed across many sectors. As the size of data used to train machine learning models keeps growing, it is now routine to use distributed computing infrastructures such as the cloud. This strategy allows the computation of training to be distributed among a large number of nodes hosted in the cloud, where each node processes a partition of the whole data set. However, the performance of nodes in the cloud is often unreliable, due to system failures, resource contention, load imbalance, etc., and that unreliability can significantly delay the training process. This project pursues a coding-based framework that not only tolerates the effects of faulty nodes, but also further enhances the performance of machine learning training by dynamically taking advantage of the resources available on all nodes, whether they are faulty or not. The outcomes of this project should lead to a significant performance boost for distributed training of machine learning models.To enable the efficient use of distributed computing across unreliable infrastructure for training machine learning models from big data sets, the technical objectives of this project are divided into three levels. This project will first study coding theory for distributed matrix multiplication, a universal operation in various machine learning algorithms, and propose a coding framework with both fault tolerance and a significant performance boost. This framework will then be applied into parameter servers at the architecture level and deep neural networks at the model level, respectively. Combining these three parts, this work will lead to a practical coding framework that can efficiently scale out computation on heterogeneous unreliable nodes, where the coding schemes will be applied to distributed machine learning at different levels including fundamental arithmetic, architectures, and models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代机器学习模型取得了巨大的成功，并已广泛部署在许多领域。随着用于训练机器学习模型的数据的大小不断增长，现在使用分布式计算基础架构（例如云）是常规的。该策略允许对训练的计算分布在云中托管的大量节点之间，其中每个节点都会处理整个数据集的分区。但是，由于系统故障，资源争夺，负载不平衡等，云中节点的性能通常是不可靠的，并且不可靠会大大延迟培训过程。该项目追求一个基于编码的框架，该框架不仅可以容忍故障节点的效果，而且还通过动态利用所有节点上可用的资源（无论它们是否有故障）而进一步增强了机器学习培训的性能。该项目的结果应为机器学习模型的分布式培训带来重大的性能提升。为了有效利用来自不可靠的基础架构的分布式计算，用于从大数据集中培训机器学习模型，该项目的技术目标被分为三个级别。该项目将首先研究用于分布式矩阵乘法的编码理论，这是各种机器学习算法中的通用操作，并提出了具有故障耐受性和显着性能提升的编码框架。然后，该框架将分别在体系结构级别和模型级别的深神经网络上应用于参数服务器。结合这三个部分，这项工作将导致一个实用的编码框架，可以有效地扩展对异质不可靠的节点进行计算，在该节点中，编码方案将应用于不同级别的分布式机器学习，包括基本算术，建筑，建筑和模型，这些奖项通过NSF的法规及其构成的范围来表现出来，这反映了NSF的范围，并具有宽广的影响。 标准。

项目成果

Leveraging Stragglers in Coded Computing with Heterogeneous Servers

• DOI: 10.1109/iwqos49365.2020.9213028
• 发表时间: 2020-01-01
• 期刊: 2020 IEEE/ACM 28TH INTERNATIONAL SYMPOSIUM ON QUALITY OF SERVICE (IWQOS)
• 作者: Fan, Xiaodi;Soto, Pedro;Li, Jun
• 通讯作者: Li, Jun

Straggler-free Coding for Concurrent Matrix Multiplications

• DOI: 10.1109/isit44484.2020.9174239
• 发表时间: 2020-06
• 期刊: 2020 IEEE International Symposium on Information Theory (ISIT)
• 作者: Pedro Soto;Jun Li

Local Re-encoding for Coded Matrix Multiplication

编码矩阵乘法的本地重新编码

• DOI: 10.1109/isit44484.2020.9174041
• 发表时间: 2020
• 期刊: 2020 IEEE International Symposium on Information Theory (ISIT
• 作者: Su, Xian;Zhong, Xiaomei;Fan, Xiaodi;Li, Jun
• 通讯作者: Li, Jun