CCF-BSF: AF: Small: Convex and Non-Convex Distributed Learning

CCF-BSF：AF：小：凸和非凸分布式学习

基本信息

批准号：
1718970
负责人：
Nathan Srebro
金额：
$ 25万
依托单位：
Toyota Technological Institute at Chicago
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2018
资助国家：
美国
起止时间：
2018-01-01 至 2021-12-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1718970&HistoricalAwards=false
关键词：
CCF BSF AF Small Convex

项目摘要

Machine learning is an increasingly important approach in tackling many difficult scientific, engineering and artificial intelligence tasks, ranging from machine translation and speech recognition, through control of self driving cars, to protein structure prediction and drug design. The core idea of machine learning is to use examples and data to automatically train a system to perform some task. Accordingly, the success of machine learning is tied to availability of large amounts of training data and our ability to process it. Much of the recent success of machine learning is fueled by the large amounts of data (text, images, videos, etc) that can now be collected. But all this data also needs to be processed and learned from---indeed this data flood has shifted the bottleneck, to a large extent, from availability of data to our ability to process it. In particular, the amounts of data involved can no longer be stored and handled on single computers. Consequently, distributed machine learning, where data is processed and learned from on many computers that communicate with each other, is a crucial element of modern large scale machine learning.The goal of this project is to provide a rigorous framework for studying distributed machine learning, and through it develop efficient methods for distributed learning and a theoretical understanding of the benefits of these methods, as well as the inherent limitations of distributed learning. A central component in the PIs' approach is to model distributed learning as a stochastic optimization problem, where different machines receive samples drawn from the same source distribution, thus allowing methods and analysis that specifically leverage the relatedness between data on different machines. This is crucial for studying how availability of multiple computers can aid in reducing the computational cost of learning. Furthermore, the project also encompasses the more challenging case where there are significant differences between the nature of the data on different machines (for instance, when different machines serve different geographical regions, or when each machine is a personal device, collecting data from a single user). In such a situation, the proposed approach to be studied is to integrate distributed learning with personalization or adaptation, which the PIs argue can not only improve learning performance, but also better leverage distributed computation.This is an international collaboration, made possible through joint funding with the US-Israel Binational Science Foundation (BSF). The project brings together two PIs that have worked together extensively on related topics in machine learning and optimization.

机器学习是一种越来越重要的方法，可以解决许多困难的科学，工程和人工智能任务，从机器翻译和语音识别到控制自动驾驶汽车到蛋白质结构的预测和药物设计等等。机器学习的核心思想是使用示例和数据自动训练系统以执行某些任务。根据机器学习的成功，与大量培训数据的可用性及其处理能力有关。现在，机器学习的最新成功都取决于现在可以收集的大量数据（文本，图像，视频等）。但是所有这些数据还需要从处理和学习 - 确实，这些数据洪水在很大程度上使瓶颈从数据的可用性转移到了我们处理数据的能力。特别是，所涉及的数据量不再可以存储和处理在单个计算机上。因此，分布式机器学习是在许多相互通信的计算机上处理和了解的数据，是现代大型机器学习的关键要素。该项目的目的是为研究分布式机器学习提供严格的框架，并通过其开发有效的方法来开发分布式学习的有效方法，并对分布式学习的益处以及分布式学习的效果以及分发性的限制。 PIS方法中的一个核心组成部分是将分布式学习作为随机优化问题建模，其中不同的机器接收从相同的源分布中绘制的样品，从而允许使用方法和分析，以专门利用不同机器上数据之间的相关性。这对于研究多台计算机的可用性如何有助于降低学习的计算成本至关重要。此外，该项目还包括更多的挑战案例，即不同机器上数据的性质之间存在显着差异（例如，当不同的机器服务于不同的地理区域时，或者每台机器是个人设备时，从单个用户那里收集数据）。在这种情况下，提议的研究方法是将分布式学习与个性化或适应性整合在一起，PIS认为这不仅可以提高学习绩效，而且可以更好地杠杆分布式计算。这是一项国际合作，通过与美国 - 以色列双性科学基金会（BSF）通过共同资助而成为可能。该项目汇集了两个PI，这些PI在机器学习和优化中广泛合作。

项目成果

期刊论文数量（19）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

Minibatch vs Local SGD for Heterogeneous Distributed Learning

DOI：
发表时间：
2020-06
期刊：
ArXiv
影响因子：
0
作者：
Blake E. Woodworth;Kumar Kshitij Patel;N. Srebro
通讯作者：
Blake E. Woodworth;Kumar Kshitij Patel;N. Srebro

The Everlasting Database: Statistical Validity at a Fair Price

DOI：
发表时间：
2018-03
期刊：
影响因子：
0
作者：
Blake E. Woodworth;V. Feldman;Saharon Rosset;N. Srebro
通讯作者：
Blake E. Woodworth;V. Feldman;Saharon Rosset;N. Srebro

A Stochastic Newton Algorithm for Distributed Convex Optimization

DOI：
发表时间：
2021-10
期刊：
ArXiv
影响因子：
0
作者：
Brian Bullins;Kumar Kshitij Patel;Ohad Shamir;N. Srebro;Blake E. Woodworth
通讯作者：
Brian Bullins;Kumar Kshitij Patel;Ohad Shamir;N. Srebro;Blake E. Woodworth

Lower bounds for non-convex stochastic optimization

DOI：
10.1007/s10107-022-01822-7
发表时间：
2019-12
期刊：
Mathematical Programming
影响因子：
2.7
作者：
Yossi Arjevani;Y. Carmon;John C. Duchi;Dylan J. Foster;N. Srebro;Blake E. Woodworth
通讯作者：
Yossi Arjevani;Y. Carmon;John C. Duchi;Dylan J. Foster;N. Srebro;Blake E. Woodworth

The Complexity of Making the Gradient Small in Stochastic Convex Optimization

DOI：
发表时间：
2019-02
期刊：
影响因子：
0
作者：
Dylan J. Foster;Ayush Sekhari;Ohad Shamir;N. Srebro;Karthik Sridharan;Blake E. Woodworth
通讯作者：
Dylan J. Foster;Ayush Sekhari;Ohad Shamir;N. Srebro;Karthik Sridharan;Blake E. Woodworth