CRII: CIF: Unifying Scheduling and Optimization Techniques to Speed-up Distributed Stochastic Gradient Descent

CRII：CIF：统一调度和优化技术来加速分布式随机梯度下降

基本信息

批准号：
1850029
负责人：
Gauri Joshi
金额：
$ 17.5万
依托单位：
Carnegie-Mellon University
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2019
资助国家：
美国
起止时间：
2019-03-01 至 2022-02-28
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1850029&HistoricalAwards=false
关键词：
CRII CIF Unifying Scheduling Optimization

项目摘要

Stochastic gradient descent (SGD) is at the core of state-of-the-art supervised learning, which is revolutionizing inference and decision-making in many diverse applications such as self-driving cars, robotics, personalized search and recommendations, and medical diagnosis. Thus, improving the speed of stochastic gradient descent is a timely and important research problem. Due to the massive scale of neural network models and training data sets used today, it has become advantageous to parallelize SGD across multiple computing nodes. Although parallelizing SGD boosts the amount of data processed per iteration, it exposes the algorithm to unpredictable node slowdown and communication delays stemming from variability in the computing infrastructure. The goal of this project is to design provably fast SGD algorithms that easily lend themselves to distributed implementations, and are robust to fluctuations in computation and network delays as well as unpredictable node failures. This project can assist in making machine learning universally accessible, without requiring access to expensive high-performance computing infrastructure. An open-source implementation of the resulting adaptive distributed SGD algorithms will be released. The research outcomes will also be incorporated into two new machine learning classes at Carnegie Mellon University, and into curriculum development and research sampler workshops for K-12 teachers and students.The speed of single-node SGD is typically measured in terms of the convergence of training error with respect to the number of iterations. In distributed SGD, the runtime per iteration depends on system-level factors such as the computation delays at worker nodes and the gradient aggregation mechanism. Thus, there is a critical need to understand the error convergence with respect to the wall-clock time rather than the number of iterations. This project will improve the true convergence of distributed SGD with respect to wall-clock time by jointly optimizing the runtime-per-iteration and error-versus-iterations. It will consider two popular distributed SGD frameworks, the parameter server model and the communication-efficient SGD model. The research is expected to provide novel runtime and error analyses of distributed SGD in these frameworks and design the first adaptive distributed SGD algorithms that strike the best error-runtime trade-off.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

随机梯度下降（SGD）是最先进的监督学习的核心，这在许多不同的应用中彻底改变了推理和决策，例如自动驾驶汽车，机器人技术，个性化搜索和建议以及医疗诊断。因此，提高随机梯度下降的速度是一个及时且重要的研究问题。由于当今使用的大量神经网络模型和训练数据集大规模，因此在多个计算节点上并行化SGD已成为有利的。尽管平行化的SGD增加了通过迭代处理的数据的数量，但它将算法暴露于无法预测的节点放缓和通信延迟，并源于计算基础结构中的可变性。该项目的目的是设计可证明快速的SGD算法，这些算法可以轻松地借助分布式实现，并且对计算和网络延迟的波动以及无法预测的节点失败是强大的。该项目可以帮助使机器学习普遍访问，而无需访问昂贵的高性能计算基础架构。将发布由此产生的自适应分布式SGD算法的开源实现。研究成果还将在卡内基·梅隆大学（Carnegie Mellon University）的两个新机器学习课程中纳入，并为K-12教师和学生的课程开发和研究采样器讲习班。单节点SGD的速度通常是根据训练错误相对于迭代次数的培训误差来衡量的。在分布式SGD中，每次迭代的运行时间取决于系统级因子，例如工人节点处的计算延迟和梯度聚合机制。因此，迫切需要了解误差收敛相对于壁锁定时间而不是迭代次数。该项目将通过共同优化每识别运行时和错误对话 - 材料来改善分布式SGD相对于墙锁定时间的真正收敛性。它将考虑两个流行的分布式SGD框架，即参数服务器模型和通信有效的SGD模型。预计该研究将在这些框架中提供分布式SGD的新型运行时和错误分析，并设计了第一个自适应分布式SGD算法，从而实现了最佳的错误暴跌权衡。该奖项反映了NSF的法定任务，并通过使用该基金会的智力功能和广泛的影响来评估Criteria criteria criteria criteria criteria criteria。

项目成果

期刊论文数量（6）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

ADAPTIVE QUANTIZATION OF MODEL UPDATES FOR COMMUNICATION-EFFICIENT FEDERATED LEARNING

DOI：
10.1109/icassp39728.2021.9413697
发表时间：
2021-01-01
期刊：
2021 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP 2021)
影响因子：
0
作者：
Jhunjhunwala, Divyansh;Gadhikar, Advait;Eldar, Yonina C.
通讯作者：
Eldar, Yonina C.

Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms

DOI：
发表时间：
2018-08
期刊：
ArXiv
影响因子：
0
作者：
Jianyu Wang;Gauri Joshi
通讯作者：
Jianyu Wang;Gauri Joshi

Cooperative SGD: A Unified Framework for the Design and Analysis of Local-Update SGD Algorithms

DOI：
发表时间：
2021
期刊：
J. Mach. Learn. Res.
影响因子：
0
作者：
Jianyu Wang;Gauri Joshi
通讯作者：
Jianyu Wang;Gauri Joshi

Slow and Stale Gradients Can Win the Race

DOI：
10.1109/jsait.2021.3103770
发表时间：
2018-03
期刊：
IEEE Journal on Selected Areas in Information Theory
影响因子：
0
作者：
Sanghamitra Dutta;Gauri Joshi;Soumyadip Ghosh;Parijat Dube;P. Nagpurkar
通讯作者：
Sanghamitra Dutta;Gauri Joshi;Soumyadip Ghosh;Parijat Dube;P. Nagpurkar

Overlap Local-SGD: An Algorithmic Approach to Hide Communication Delays in Distributed SGD

DOI：
10.1109/icassp40776.2020.9053834
发表时间：
2020-02
期刊：
ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
影响因子：
0
作者：
Jianyu Wang;Hao Liang;Gauri Joshi
通讯作者：
Jianyu Wang;Hao Liang;Gauri Joshi