MLWiNS: Wireless On-the-Edge Training of Deep Networks Using Independent Subnets

MLWiNS：使用独立子网的深度网络无线边缘训练

• 批准号: 2003137
• 负责人: Christopher Jermaine
• 金额: $ 30万
• 依托单位: William Marsh Rice University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-06-01 至 2023-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2003137&HistoricalAwards=false
• 关键词: MLWiNS; Wireless; Edge; Training; Deep

Neural networks (NN) have led to many recent successes in machine learning (ML). However, this success comes at a prohibitive cost: to obtain better ML models, larger and larger NNs need to be trained and deployed. This is a problem for mobile ML applications, where model training and inference need to be carried out in a timely fashion on a computation-/communication-light, and energy-limited platform. Such applications must run on handheld devices or drones and edge infrastructure and introduce new challenges: the heterogeneity of edge networks, the unreliability of the mobile devices, the computational and energy restrictions on such devices, and the communication bottlenecks in wireless networks. This project will address these challenges by investigating a new paradigm for computation- and communication-light, energy-limited distributed NN learning. Success in this project will produce fundamental ideas and tools that will make mobile distributed learning practical. Further, the project will generate courses and open-education resources that can attract diverse groups of students. The specific idea investigated is a new class of distributed NN training algorithms, called independent subnetwork training (IST). IST decomposes a NN into a set of independent subnetworks. Each of those subnetworks is trained at a different device, for one or more backpropagation steps, before a synchronization step. Updated subnetworks are sent from edge-devices to the parameter server for reassembly into the original NN, before the next round of decomposition and local training. Because the subnetworks share no parameters, synchronization requires no aggregation—it is just an exchange of parameters. Moreover, each of the subnetworks is a fully operational classifier by itself; no synchronization pipelines between subnetworks are required. Key benefits of the proposed IST are that: i) IST assigns fewer training parameters to each mobile node per iteration, significantly reducing the communication overhead, and ii) each device trains a much smaller model, resulting in less computational costs and better energy consumption. Thus, there is good reason to expect that IST will scale much better than classic training algorithms for mobile applications. The project will investigate how to incorporate/extend IST to various NN architectures, develop new theories that explain the efficiency of IST, and unify theory with practice by proposing hardware-level system implementations that scale up and out for mobile applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

神经网络 (NN) 最近在机器学习 (ML) 领域取得了许多成功，但这种成功的代价却高昂：为了获得更好的 ML 模型，需要训练和部署越来越大的神经网络，这是一个问题。移动机器学习应用程序，需要在计算/通信轻且能源有限的平台上及时进行模型训练和推理，此类应用程序必须在手持设备或无人机和边缘基础设施上运行，并带来新的挑战：异质性该项目将通过研究计算和通信轻、能量有限的新范式来解决这些挑战。分布式神经网络学习。该项目的成功将产生使移动分布式学习变得实用的基本思想和工具。此外，该项目还将产生可以吸引不同学生群体的课程和开放教育资源。所研究的具体想法是一门新课程。分布式神经网络训练算法，称为独立子网络训练 (IST) 将神经网络分解为一组独立的子网络，然后在同步步骤将更新的子网络发送到不同的设备上进行一个或多个反向传播步骤。参数服务器用于在下一轮分解和本地训练之前重新组装成原始神经网络，因为子网络不共享参数，所以同步不需要聚合——它只是一个。此外，每个子网络本身都是一个完全可操作的分类器；所提出的 IST 的主要优点是：i）每次迭代为每个移动节点分配更少的训练参数。减少通信开销，ii) 每个设备训练一个更小的模型，从而减少计算成本和更好的能源消耗。因此，有充分的理由预期 IST 将比移动应用程序的经典训练算法更好地扩展。将调查如何将 IST 合并/扩展到各种神经网络架构，开发解释 IST 效率的新理论，并通过提出针对移动应用程序进行扩展和扩展的硬件级系统实现来将理论与实践结合起来。该奖项反映了 NSF 的法定使命，并被视为值得通过使用基金会的智力优点和更广泛的影响审查标准进行评估来支持。

项目成果

Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?

• DOI: 10.48550/arxiv.2203.08392
• 发表时间: 2022-03
• 期刊: ArXiv
• 作者: Y. Fu;Shunyao Zhang;Shan-Hung Wu;Cheng Wan;Yingyan Lin

BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Sampling

• DOI: 10.48550/arxiv.2203.10983
• 发表时间: 2022-03
• 期刊: ArXiv
• 作者: Cheng Wan;Youjie Li;Ang Li;Namjae Kim;Yingyan Lin

LDP: Learnable Dynamic Precision for Efficient Deep Neural Network Training and Inference

• DOI: 10.48550/arxiv.2203.07713
• 发表时间: 2022-03
• 期刊: ArXiv
• 作者: Zhongzhi Yu;Y. Fu;Shang Wu;Mengquan Li;Haoran You;Yingyan Lin

PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communication

• DOI: 10.48550/arxiv.2203.10428
• 发表时间: 2022-03
• 期刊: ArXiv
• 作者: Cheng Wan;Youjie Li;Cameron R. Wolfe;Anastasios Kyrillidis;Namjae Kim;Yingyan Lin