BIGDATA: F: Optimization in Federated Networks of Devices

BIGDATA：F：设备联合网络的优化

• 批准号: 1838017
• 负责人: Ameet Talwalkar
• 金额: $ 99.94万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1838017&HistoricalAwards=false
• 关键词: BIGDATA; Optimization; Federated; Networks; Devices

Modern networks of remote devices, such as mobile phones, wearable devices, and autonomous vehicles, generate massive amounts of data each day. This rich data has the potential to power a wide range of statistical machine learning-based applications, such as learning the activities of mobile phone users, adapting to pedestrian behavior in autonomous vehicles, predicting health events like low blood sugar from wearable devices, or detecting burglaries within smart homes. Due to the growing storage and computational power of remote devices, as well as privacy concerns associated with personal data, it is increasingly attractive to store and process data directly on each device. In the burgeoning field of "federated learning," the aim is to use a central server to learn statistical models from data stored across these remote devices, while relying on substantial computation from each device. Federated learning can be naturally cast through the lens of mathematical optimization, a key component in formulating and training most machine learning models. This project focuses on tackling several of the unique statistical and systems challenges associated with federated optimization. As part of this project, a novel open-source benchmarking framework is also being developed to concretely define the research challenges in federated learning and promote reproducibility in empirical evaluations. This project involves participation from students from underrepresented populations. The focus of this project is to develop a novel suite of optimization methods to tackle the unique challenges of learning on remote devices, including (a) expensive communication between remote devices and a central server; (b) high variability in data, computational resources, and communication bandwidth across devices; and (c) a very small fraction of remote devices participating in the training process at any one time. While numerous optimization methods in the data center setting have been proposed to tackle (a), none allow significant flexibility in terms of (b) and (c). Further, the limited number of recently introduced federated methods either lack theoretical convergence guarantees or do not adequately address these three challenges. This project aims to develop a suite of federated optimization methods to tackle these issues, specifically developing and understanding techniques for: convex optimization, non-convex optimization, and network-aware optimization. These methods will unleash the computational power of federated networks to train highly-accurate predictive models while adhering to strict systems, network, and privacy constraints. This project leverages ideas from optimization, statistics, machine learning, distributed computing, and sensor networks. In addition to developing foundational federated optimization methods, the broader impact of this project includes the creation of a novel open-source benchmarking framework.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代远程设备网络（例如手机、可穿戴设备和自动驾驶汽车）每天都会生成大量数据。这些丰富的数据有潜力为各种基于统计机器学习的应用提供支持，例如学习手机用户的活动、适应自动驾驶汽车中的行人行为、通过可穿戴设备预测低血糖等健康事件，或检测智能家居内的入室盗窃。由于远程设备的存储和计算能力不断增强，以及与个人数据相关的隐私问题，直接在每个设备上存储和处理数据越来越有吸引力。在新兴的“联邦学习”领域，目标是使用中央服务器从这些远程设备上存储的数据中学习统计模型，同时依赖每个设备的大量计算。联邦学习可以自然地通过数学优化的视角来体现，数学优化是制定和训练大多数机器学习模型的关键组成部分。该项目专注于解决与联合优化相关的几个独特的统计和系统挑战。作为该项目的一部分，我们还正在开发一种新颖的开源基准测试框架，以具体定义联邦学习中的研究挑战并提高实证评估的可重复性。该项目涉及来自代表性不足群体的学生的参与。 该项目的重点是开发一套新颖的优化方法，以应对远程设备学习的独特挑战，包括（a）远程设备和中央服务器之间昂贵的通信； (b) 设备间数据、计算资源和通信带宽的高度可变性； (c) 任何时候参与训练过程的远程设备的一小部分。虽然已经提出了数据中心环境中的许多优化方法来解决（a）问题，但没有一种方法在（b）和（c）方面具有显着的灵活性。此外，最近引入的联邦方法数量有限，要么缺乏理论收敛保证，要么不能充分解决这三个挑战。该项目旨在开发一套联合优化方法来解决这些问题，特别是开发和理解以下技术：凸优化、非凸优化和网络感知优化。这些方法将释放联合网络的计算能力来训练高精度的预测模型，同时遵守严格的系统、网络和隐私约束。该项目利用了优化、统计、机器学习、分布式计算和传感器网络的想法。 除了开发基础联合优化方法外，该项目更广泛的影响还包括创建新颖的开源基准测试框架。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优势和更广泛的影响进行评估，被认为值得支持审查标准。

项目成果

Ditto: Fair and Robust Federated Learning Through Personalization

• 发表时间: 2020-12
• 作者: Tian Li;Shengyuan Hu;Ahmad Beirami;Virginia Smith

Adaptive Gradient-Based Meta-Learning Methods

• 发表时间: 2019-06
• 作者: M. Khodak;Maria-Florina Balcan;Ameet Talwalkar

Provable Guarantees for Gradient-Based Meta-Learning

• 发表时间: 2019-02
• 期刊: ArXiv
• 作者: M. Khodak;Maria-Florina Balcan;Ameet Talwalkar

On Noisy Evaluation in Federated Hyperparameter Tuning

• DOI: 10.48550/arxiv.2212.08930
• 发表时间: 2022-12
• 期刊: ArXiv
• 作者: Kevin Kuo;Pratiksha Thaker;M. Khodak;John Nguyen;Daniel Jiang;Ameet Talwalkar;Virginia Smith

On Large-Cohort Training for Federated Learning

• 发表时间: 2021-06
• 期刊: ArXiv
• 作者: Zachary B. Charles;Zachary Garrett;Zhouyuan Huo;Sergei Shmulyian;Virginia Smith