CIF: Small: Coding Techniques for Distributed Machine Learning

CIF：小型：分布式机器学习的编码技术

• 批准号: 2101388
• 负责人: Jun Li
• 金额: $ 37.23万
• 依托单位: CUNY Queens College
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-30 至 2022-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2101388&HistoricalAwards=false
• 关键词: CIF; Small; Coding; Techniques; Distributed

Modern machine learning models have achieved great success and have been widely deployed across many sectors. As the size of data used to train machine learning models keeps growing, it is now routine to use distributed computing infrastructures such as the cloud. This strategy allows the computation of training to be distributed among a large number of nodes hosted in the cloud, where each node processes a partition of the whole data set. However, the performance of nodes in the cloud is often unreliable, due to system failures, resource contention, load imbalance, etc., and that unreliability can significantly delay the training process. This project pursues a coding-based framework that not only tolerates the effects of faulty nodes, but also further enhances the performance of machine learning training by dynamically taking advantage of the resources available on all nodes, whether they are faulty or not. The outcomes of this project should lead to a significant performance boost for distributed training of machine learning models.To enable the efficient use of distributed computing across unreliable infrastructure for training machine learning models from big data sets, the technical objectives of this project are divided into three levels. This project will first study coding theory for distributed matrix multiplication, a universal operation in various machine learning algorithms, and propose a coding framework with both fault tolerance and a significant performance boost. This framework will then be applied into parameter servers at the architecture level and deep neural networks at the model level, respectively. Combining these three parts, this work will lead to a practical coding framework that can efficiently scale out computation on heterogeneous unreliable nodes, where the coding schemes will be applied to distributed machine learning at different levels including fundamental arithmetic, architectures, and models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代机器学习模型取得了巨大成功，并已广泛应用于许多领域。随着用于训练机器学习模型的数据规模不断增长，使用云等分布式计算基础设施现在已成为惯例。该策略允许训练计算分布在云中托管的大量节点中，其中每个节点处理整个数据集的一部分。然而，由于系统故障、资源争用、负载不平衡等原因，云中节点的性能往往不可靠，并且这种不可靠性会显着延迟训练过程。该项目追求一种基于编码的框架，该框架不仅可以容忍故障节点的影响，而且还可以通过动态利用所有节点上的可用资源（无论故障与否）来进一步增强机器学习训练的性能。该项目的成果应该会显着提高机器学习模型的分布式训练的性能。为了能够跨不可靠的基础设施有效地使用分布式计算来训练来自大数据集的机器学习模型，该项目的技术目标分为三个级别。该项目将首先研究分布式矩阵乘法的编码理论，这是各种机器学习算法中的通用运算，并提出一种兼具容错性和显着性能提升的编码框架。然后，该框架将分别应用于架构级别的参数服务器和模型级别的深度神经网络。结合这三个部分，这项工作将产生一个实用的编码框架，可以有效地扩展异构不可靠节点上的计算，其中编码方案将应用于不同级别的分布式机器学习，包括基础算法、架构和模型。反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Coded Matrix Chain Multiplication

• DOI: 10.1109/iwqos52092.2021.9521282
• 发表时间: 2021-06
• 期刊: 2021 IEEE/ACM 29th International Symposium on Quality of Service (IWQOS)
• 作者: Xiaodi Fan;Angel Saldivia;Pedro Soto;Jun Li

Local Re-encoding for Coded Matrix Multiplication

编码矩阵乘法的本地重新编码

• DOI: 10.1109/isit44484.2020.9174041
• 发表时间: 2020
• 期刊: 2020 IEEE International Symposium on Information Theory (ISIT
• 作者: Su, Xian;Zhong, Xiaomei;Fan, Xiaodi;Li, Jun
• 通讯作者: Li, Jun

Rook Coding for Batch Matrix Multiplication

• DOI: 10.1109/tcomm.2022.3165201
• 发表时间: 2022-06
• 期刊: IEEE Transactions on Communications
• 影响因子: 8.3
• 作者: Pedro Soto;Xiaodi Fan;Angel Saldivia;Jun Li

Lightweight Projective Derivative Codes for Compressed Asynchronous Gradient Descent

• 发表时间: 2022-01
• 作者: Pedro Soto;Ilia Ilmer;Haibin Guan;Jun Li