Collaborative Research: SHF: Medium: TensorNN: An Algorithm and Hardware Co-design Framework for On-device Deep Neural Network Learning using Low-rank Tensors

合作研究：SHF：Medium：TensorNN：使用低秩张量进行设备上深度神经网络学习的算法和硬件协同设计框架

• 批准号: 1954549
• 负责人: Xiaodong Wang
• 金额: $ 40万
• 依托单位: Columbia University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-07-01 至 2024-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1954549&HistoricalAwards=false
• 关键词: Collaborative; Research; SHF; Medium; TensorNN

Deep neural network (DNN) is an important Artificial Intelligence (AI) technique and it has recently gained widespread applications in numerous fields such as image recognition, machine translation, autonomous vehicles and healthcare diagnosis. Conventional DNNs are implemented using cloud computing, where a large amount of computing resource is available in a centrally-pooled manner. In order to achieve stronger data privacy, less response time and relaxed data transmission burden, deploying DNN functionality in a distributed manner at the edges of the network has become a very attractive proposition. However, DNN-learning on mobile devices that are at the edge of the network is very challenging due to conflicting requirements of large time and energy consumption, and limited on-device resources. In order to address this challenge, this project leverages low-rank tensors as a powerful mathematical tool for representing and compressing tensor-format data, to form a new family of ultra-low cost deep neural networks. This brings an order-of-magnitude reduction in time and energy consumption for deep neural network learning. Investigations in many areas of BigData research will benefit as well. This project involves graduate and undergraduate students, especially from underrepresented groups, through summer research experiences, and senior design projects to broaden the participation of computing. The outcomes of this project will be disseminated to the community in the format of technical publications, talks and tutorials in both academic institutions and industry.In order to remove the barriers of realizing real-time energy-efficient DNN-learning on the resource and energy-constrained embedded devices, this project considers innovations at three levels: 1) at theory level, it develops a novel redundancy-free matrix-vector multiplication scheme to reduce computational cost, including a new online update scheme for low-rank tensors to enable fast compressed data update; 2) at algorithm level, it develops low-rank tensor-based forward and backward propagation schemes to support low-cost accelerated inference and training, including catastrophic forgetting-resilient training scheme and training-aware compression scheme to improve the learning robustness and memory efficiency; and 3) at hardware design level, it proposes efficient hardware architecture that fully utilize the benefits provided by low-rank tensors to achieve improved hardware performance for on-device DNN inference and learning. Finally, the efficacy of the proposed research will be validated and evaluated, via software implementations on different DNN models in different target applications. A field-programmable gate array (FPGA)-based hardware prototype will also be developed.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深神经网络（DNN）是一种重要的人工智能（AI）技术，最近在诸如图像识别，机器翻译，自动驾驶汽车和医疗保健诊断等众多领域中获得了广泛的应用。传统的DNN是使用云计算实现的，其中大量计算资源以中央式的方式获得。为了获得更强的数据隐私，较少的响应时间和放松的数据传输负担，以分布式方式在网络边缘部署DNN功能已成为一个非常有吸引力的主张。但是，由于大量时间和能源消耗的要求相互矛盾，并且在设备上的资源有限，因此在网络边缘的移动设备上进行的DNN学习非常具有挑战性。为了应对这一挑战，该项目利用低级张量作为代表和压缩张量 - 格式数据的强大数学工具，形成了一个新的超低成本成本的新家族。这为深度神经网络学习的时间和能源消耗降低了降低。在Bigdata研究的许多领域进行的调查也将受益。该项目涉及毕业生和本科生，尤其是来自代表性不足的团体，通过夏季研究经验以及高级设计项目，以扩大计算的参与。 The outcomes of this project will be disseminated to the community in the format of technical publications, talks and tutorials in both academic institutions and industry.In order to remove the barriers of realizing real-time energy-efficient DNN-learning on the resource and energy-constrained embedded devices, this project considers innovations at three levels: 1) at theory level, it develops a novel redundancy-free matrix-vector multiplication scheme to reduce计算成本，包括针对低级数张量的新的在线更新方案，以启用快速压缩数据更新； 2）在算法水平下，它开发了基于低量的前向和向后传播方案，以支持低成本加速的推理和训练，包括灾难性的遗忘遗忘训练方案和训练感知的压缩方案，以提高学习的鲁棒性和记忆效率； 3）在硬件设计级别上，它提出了有效的硬件体系结构，该体系结构充分利用了低级张量提供的好处，以实现改进的硬件性能，以实现DNN推理和学习。最后，将通过对不同目标应用中不同DNN模型的软件实现进行验证和评估拟议研究的功效。还将开发一个基于现场编程的门阵列（FPGA）的硬件原型。该奖项反映了NSF的法定任务，并且使用基金会的知识分子优点和更广泛的影响评估标准，认为值得通过评估来获得支持。

项目成果

Integrated Actor-Critic for Deep Reinforcement Learning

用于深度强化学习的综合 Actor-Critic

• 发表时间: 2021
• 期刊: ICANN 2021
• 作者: Jiaohao Zheng, Mehmet Necip

ADMM-Net for Communication Interference Removal in Stepped-Frequency Radar

• DOI: 10.1109/tsp.2021.3076900
• 发表时间: 2020-09
• 期刊: IEEE Transactions on Signal Processing
• 影响因子: 5.4
• 作者: Jeremy Johnston;Yinchuan Li;M. Lops;Xiaodong Wang

Homomorphic Matrix Completion

• 发表时间: 2022
• 作者: Xiao-Yang Liu;Zechu Li;Xiaodong Wang

Structured Alternating Minimization for Union of Nested Low-Rank Subspaces Data Completion

• DOI: 10.1109/jsait.2020.3039170
• 发表时间: 2020-11
• 期刊: IEEE Journal on Selected Areas in Information Theory
• 作者: Morteza Ashraphijuo;Xiaodong Wang

Fundamental sampling patterns for low-rank multi-view data completion

低秩多视图数据完成的基本采样模式

• DOI: 10.1016/j.patcog.2020.107307
• 发表时间: 2020
• 期刊: Pattern Recognition
• 影响因子: 8
• 作者: Ashraphijuo, Morteza;Wang, Xiaodong;Aggarwal, Vaneet
• 通讯作者: Aggarwal, Vaneet