Distributed Optimization for Machine Learning on Decentralized Data and Features

基于分散数据和特征的机器学习分布式优化

• 批准号: RGPIN-2019-04998
• 负责人: Niu, Di
• 金额: $ 2.99万
• 依托单位: University of Alberta
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2020
• 资助国家: 加拿大
• 起止时间: 2020-01-01 至 2021-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=714919
• 关键词: Distributed; Optimization; Machine; Learning; Decentralized

The need to scale up machine learning, in the presence of a rapid growth in both data volumes and model complexity, has sparked broad interests in developing distributed machine learning systems as well as parallel optimization algorithms. Most existing studies focus on partitioning computation among a tightly coupled cluster of machines either in a data-parallel fashion, to deal with large amounts of training samples, or a model-parallel fashion, to deal with large models, such as deep neural networks. In contrast, the goal of this research program is to glean insights and build models when the dataset (features, samples, labels or a combination of them) used for machine learning is inherently decentralized and owned by multiple participants/domains. Our long-term vision is to design reliable distributed algorithms and systems that can build models from decentralized data, without letting participants share original data with each other or to a central site. By effectively leveraging data from other domains, each participant is expected to enhance its predicting power over a model built only based on its local data, while the joint model built in a decentralized way is expected to approach and approximate the global model if all data were collected centrally. In the meantime, the sharing of model parameters among parties should also be minimized to preserve privacy and reduce communication overhead. Toward these objectives, we will introduce generic composite model structures that can jointly reap insights from data in different decentralization scenarios, including decentralization by features, by samples, by labels or decentralization by a combination of them. We will design theoretically inspired distributed optimization algorithms to solve these problems, develop effective communication compression techniques to reduce overhead, and also study implementation issues for specific applications. Specifically, our algorithms will be inspired by the recent advancements in the convergence of ADMM, stochastic gradient descent (SGD) and proximal SGD in an asynchronous and blockwise setting. Our communication compression techniques will be inspired by the opportunity to suppress model parameter transfers in flat regions of the optimization objective function, whereas existing literature mainly considers significance filters for gradients. Finally, we will use the proposed model architectures and algorithms to solve various real-world applications in cross-domain recommender systems, multitasked natural language understanding and collaborative mobile edge computing. We will design specific model composition and decomposition structures as well as distributed algorithms based on the data decentralization pattern inherent in each problem.

随着数据量和模型复杂性的快速增长，扩大机器学习的需求引发了人们对开发分布式机器学习系统和并行优化算法的广泛兴趣。大多数现有研究侧重于在紧密耦合的机器集群之间进行计算分区，要么以数据并行方式处理大量训练样本，要么以模型并行方式处理大型模型，例如深度神经网络。相比之下，该研究计划的目标是当用于机器学习的数据集（特征、样本、标签或它们的组合）本质上是分散的并由多个参与者/域拥有时，收集见解并构建模型。我们的长期愿景是设计可靠的分布式算法和系统，可以根据分散的数据构建模型，而无需让参与者彼此共享原始数据或将原始数据共享到中央站点。通过有效地利用来自其他领域的数据，每个参与者都有望增强其对仅基于其本地数据构建的模型的预测能力，而如果所有数据都被充分利用，则以去中心化方式构建的联合模型有望接近并近似全局模型。集中收集。同时，各方之间的模型参数共享也应尽量减少，以保护隐私并减少通信开销。 为了实现这些目标，我们将引入通用的复合模型结构，这些模型结构可以共同从不同去中心化场景中的数据中获取洞察，包括按特征、按样本、按标签进行去中心化或通过它们的组合进行去中心化。我们将设计受理论启发的分布式优化算法来解决这些问题，开发有效的通信压缩技术以减少开销，并研究特定应用的实现问题。具体来说，我们的算法将受到异步和分块设置中 ADMM、随机梯度下降 (SGD) 和近端 SGD 收敛的最新进展的启发。我们的通信压缩技术将受到抑制优化目标函数平坦区域中模型参数传输的机会的启发，而现有文献主要考虑梯度的重要性过滤器。 最后，我们将使用所提出的模型架构和算法来解决跨领域推荐系统、多任务自然语言理解和协作移动边缘计算中的各种实际应用。我们将根据每个问题固有的数据去中心化模式，设计具体的模型组成和分解结构以及分布式算法。