Accelerated distributed stochastic optimization methods and applications in machine learning

加速分布式随机优化方法及其在机器学习中的应用

• 批准号: 2208394
• 负责人: Yangyang Xu
• 金额: $ 25万
• 依托单位: Rensselaer Polytechnic Institute
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-07-01 至 2025-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2208394&HistoricalAwards=false
• 关键词: Accelerated; distributed; stochastic; optimization; methods

Machine learning, and in particular, deep learning, has become increasingly impactful in a wide range of applications, including face recognition, digital image classification, natural language processing, self-driving vehicles, and scientific computing. The success of deep learning largely depends on the availability of a huge amount of data. This "big" data, on one hand, enables successful learning of the underlying distributions of the data, and thus the learned model can yield high prediction accuracy on new data points that follow similar distributions. On the other hand, the huge amount of data raises great challenges when designing efficient numerical approaches. This project focuses on addressing the challenges that are caused by distributed "big" data that can contain private information, from the computational and mathematical perspectives. Research findings from this project will be included in graduate-level topics courses, undergraduate and graduate students will be trained in this field and will participate in this research, and a weekly seminar will be organized to exchange ideas relevant to this project. New computational approaches will be developed for training machine learning models on a cluster of computing nodes as well as solving decentralized multi-agent optimization problems that have the capacity to handle coupling constraints. The main goal is to design fast-convergent and communication-efficient optimization methods with theoretical guarantees for solving large-scale distributed machine learning problems. Accelerated compressed proximal stochastic gradient methods will be designed for distributed composite smooth stochastic problems, accelerated compressed stochastic subgradient methods will be designed for distributed nonconvex nonsmooth problems, optimal decentralized stochastic gradient methods will be designed for solving multi-agent optimization with nonlinear coupling constraints, and asynchronous implementations will be performed in the proposed methods in order to have high parallelization speed up.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

机器学习，尤其是深度学习，在广泛的应用中越来越有影响力，包括面部识别，数字图像分类，自然语言处理，自动驾驶汽车和科学计算。深度学习的成功很大程度上取决于大量数据的可用性。一方面，这个“大”数据可以成功地学习数据的基础分布，因此学习模型可以在遵循类似分布的新数据点上产生高预测准确性。另一方面，在设计有效的数值方法时，大量数据引起了巨大的挑战。该项目着重于解决可以从计算和数学角度来包含私人信息的分布式“大”数据引起的挑战。该项目的研究结果将包括在研究生级主题课程中，本科生和研究生将在该领域接受培训，并将参加这项研究，并将组织一次研讨会，以交换与该项目相关的想法。 将开发新的计算方法，用于在计算节点群体上培训机器学习模型，并解决具有处理耦合约束能力的分散多代理优化问题。主要目标是设计快速且沟通高效的优化方法，并通过理论保证解决大规模的分布式机器学习问题。将设计加速的压缩近端随机梯度方法，用于分布的分布式复合的平滑随机问题，加速压缩的随机亚加速度方法将设计用于分布式非凸出非刻度非平滑梯度问题，最佳分散分散化的随机梯度将被设计用于与非线化型求解的求解，以实现多个型号的范围，并实现了构成量的构成量，并实现了conterron的范围。为了使其高于并行的速度提高。该奖项反映了NSF的法定任务，并被认为是使用基金会的知识分子优点和更广泛影响的评论标准的评估值得支持的。

项目成果

Proximal Stochastic Recursive Momentum Methods for Nonconvex Composite Decentralized Optimization

• DOI: 10.1609/aaai.v37i7.26087
• 发表时间: 2022-11
• 作者: Gabriel Mancino-Ball;Shengnan Miao;Yangyang Xu;Jiewei Chen

Parallel and distributed asynchronous adaptive stochastic gradient methods

• DOI: 10.1007/s12532-023-00237-5
• 发表时间: 2020-02
• 期刊: Mathematical Programming Computation
• 影响因子: 6.3
• 作者: Yangyang Xu;Yibo Xu;Yonggui Yan;Colin Sutcher-Shepard;Leopold Grinberg;Jiewei Chen

A Decentralized Primal-Dual Framework for Non-Convex Smooth Consensus Optimization

• DOI: 10.1109/tsp.2023.3239799
• 发表时间: 2021-07
• 期刊: IEEE Transactions on Signal Processing
• 影响因子: 5.4
• 作者: Gabriel Mancino-Ball;Yangyang Xu;Jiewei Chen