Collaborative Research: Enabling Intelligent Cameras in Internet-of-Things via a Holistic Platform, Algorithm, and Hardware Co-design

协作研究：通过整体平台、算法和硬件协同设计实现物联网中的智能相机

• 批准号: 2053272
• 负责人: Zhangyang Wang
• 金额: $ 21.94万
• 依托单位: University of Texas at Austin
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-08-01 至 2022-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2053272&HistoricalAwards=false
• 关键词: Collaborative; Research; Enabling; Intelligent; Cameras

There has been a tremendous demand for bringing Deep Neural Network (DNN) powered functionality into Internet of Thing (IoT) devices to enable ubiquitous intelligent "IoT cameras". However, state-of-the-art DNNs have a prohibitive energy cost, making them impractical to be deployed in resource-constrained IoT platforms. This project will develop a novel energy-efficient DNN framework, via a systematic integration of platform, hardware, and algorithm co-design innovations. Despite a growing interest in energy-efficient DNNs, existing techniques lack a systematic optimization across the full stack of design abstraction, from systems through algorithms to hardware implementation. The proposed research advocates an innovative, holistic effort towards energy-efficient and adaptive DNN-powered "IoT cameras" by jointly optimizing the platform-, hardware-, and algorithm-level co-design efforts. On the system level, we will address how to automatically generate and adapt DNN models and implementation, to meet a variety of "IoT devices" application-specific performance needs and device-specific resource constraints. On the hardware level, we will leverage the observed high sparsity in DNN activations for energy-efficient hardware implementations of both DNN training and inference by using low-cost zero predictors and hence bypass unnecessary computations. On the algorithm level, we will develop innovative factorized sparsity regularization in DNN training as well as efficient, controllable adaptive inference mechanisms, fully complementing and closely integrating with our hardware innovations. The proposed research will advance the scientific domain of each level, from system and algorithm, to hardware and a holistic, systematic cross-level methodology for designing energy-efficient intelligent systems. Progress on this project will enable ubiquitous DNN-powered intelligent functions in a significantly increased number of resource-constrained daily-life devices, across numerous camera-based Internet-of-Things (IoT) applications such as traffic monitoring, self-driving and smart cars, personal digital assistants, surveillance and security, and augmented reality. As camera-based IoT devices penetrate all walks of life, by enabling DNN-powered intelligence to be pervasive in these devices, the proposed research can have a tremendous impact on global societies and economies. The research will be integrated with education on energy efficient deep learning. Educational activities include curriculum development, undergraduate research, and outreach to K-12 students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

人们迫切需要将深度神经网络 (DNN) 驱动的功能引入物联网 (IoT) 设备，以实现无处不在的智能“物联网相机”。然而，最先进的 DNN 的能源成本过高，使其部署在资源有限的物联网平台中不切实际。该项目将通过平台、硬件和算法协同设计创新的系统集成，开发一种新颖的节能 DNN 框架。尽管人们对节能 DNN 的兴趣日益浓厚，但现有技术缺乏对整个设计抽象堆栈（从系统到算法到硬件实现）的系统优化。拟议的研究主张通过联合优化平台、硬件和算法级协同设计工作，对节能和自适应 DNN 驱动的“物联网相机”进行创新、全面的努力。在系统层面，我们将解决如何自动生成和调整DNN模型和实现，以满足各种“物联网设备”特定应用的性能需求和特定设备的资源限制。在硬件层面，我们将利用在 DNN 激活中观察到的高稀疏性，通过使用低成本的零预测器来实现 DNN 训练和推理的节能硬件实现，从而绕过不必要的计算。在算法层面，我们将创新DNN训练中的因子式稀疏正则化以及高效可控的自适应推理机制，与我们的硬件创新充分互补并紧密结合。 拟议的研究将推进各个层面的科学领域，从系统和算法到硬件以及设计节能智能系统的整体、系统的跨层面方法。该项目的进展将使无处不在的 DNN 支持的智能功能在数量显着增加的资源有限的日常生活设备中实现，跨越众多基于摄像头的物联网 (IoT) 应用，例如交通监控、自动驾驶和智能汽车汽车、个人数字助理、监控和安全以及增强现实。随着基于摄像头的物联网设备渗透到各行各业，通过使 DNN 驱动的智能在这些设备中普遍存在，拟议的研究可以对全球社会和经济产生巨大影响。 该研究将与节能深度学习教育相结合。 教育活动包括课程开发、本科生研究以及对 K-12 学生的推广。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，认为值得支持。

项目成果

Audio Lottery: Speech Recognition Made Ultra-Lightweight, Noise-Robust, and Transferable

音频彩票：语音识别变得超轻量、抗噪且可转移

• 发表时间: 2024-09-13
• 作者: Shaojin Ding;Tianlong Chen;Zhangyang Wang
• 通讯作者: Zhangyang Wang

Sanity Checks for Lottery Tickets: Does Your Winning Ticket ReallyWin the Jackpot?

彩票的健全性检查：您的中奖彩票真的赢得大奖吗？

• 发表时间: 2021-12
• 期刊: Advances in Neural Information Processing Systems (NeurIPS
• 作者: Ma, Xiaolong;Yuan, Geng;Shen, Xuan;Chen, Tianlong;Chen, Xuxi;Chen, Xiaohan;Liu, Ning;Qin, Minghai;Liu, Sijia;Wang, Zhangyang;et al
• 通讯作者: et al

Data-Efficient Double-Win Lottery Tickets from Robust Pre-training

通过强大的预训练实现数据高效的双赢彩票

• DOI: 10.48550/arxiv.2206.04762
• 发表时间: 2022-06-09
• 作者: Tianlong Chen;Zhenyu (Allen) Zhang;Sijia Liu;Yang Zhang;Shiyu Chang;Zhangyang Wang
• 通讯作者: Zhangyang Wang

The Lottery Tickets Hypothesis for Supervised and Self-Supervised Pre-Training in Computer Vision Models

计算机视觉模型中监督和自监督预训练的彩票假设

• 发表时间: 2021-06
• 期刊: IEEE Computer Society Conference on Computer Vision and Pattern Recognition
• 作者: Chen, Tianlong;Frankle, Jonathan;Chang, Shiyu;Liu, Sijia;Zhang, Yang;Carbin, Michael;Wang, Zhangyang
• 通讯作者: Wang, Zhangyang

The lottery ticket hypothesis for pre-trained BERT networks

预训练 BERT 网络的彩票假设

• 发表时间: 2020-12
• 期刊: Advances in neural information processing systems
• 作者: Chen, Tianlong;Frankle, Jonathan;Chang, Shiyu;Liu, Sijia;Zhang, Yang;Wang, Zhangyang;Carbin, Michael
• 通讯作者: Carbin, Michael