Collaborative Research: Enabling Intelligent Cameras in Internet-of-Things via a Holistic Platform, Algorithm, and Hardware Co-design

协作研究：通过整体平台、算法和硬件协同设计实现物联网中的智能相机

• 批准号: 2053272
• 负责人: Zhangyang Wang
• 金额: $ 21.94万
• 依托单位: University of Texas at Austin
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-08-01 至 2022-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2053272&HistoricalAwards=false
• 关键词: Collaborative; Research; Enabling; Intelligent; Cameras

There has been a tremendous demand for bringing Deep Neural Network (DNN) powered functionality into Internet of Thing (IoT) devices to enable ubiquitous intelligent "IoT cameras". However, state-of-the-art DNNs have a prohibitive energy cost, making them impractical to be deployed in resource-constrained IoT platforms. This project will develop a novel energy-efficient DNN framework, via a systematic integration of platform, hardware, and algorithm co-design innovations. Despite a growing interest in energy-efficient DNNs, existing techniques lack a systematic optimization across the full stack of design abstraction, from systems through algorithms to hardware implementation. The proposed research advocates an innovative, holistic effort towards energy-efficient and adaptive DNN-powered "IoT cameras" by jointly optimizing the platform-, hardware-, and algorithm-level co-design efforts. On the system level, we will address how to automatically generate and adapt DNN models and implementation, to meet a variety of "IoT devices" application-specific performance needs and device-specific resource constraints. On the hardware level, we will leverage the observed high sparsity in DNN activations for energy-efficient hardware implementations of both DNN training and inference by using low-cost zero predictors and hence bypass unnecessary computations. On the algorithm level, we will develop innovative factorized sparsity regularization in DNN training as well as efficient, controllable adaptive inference mechanisms, fully complementing and closely integrating with our hardware innovations. The proposed research will advance the scientific domain of each level, from system and algorithm, to hardware and a holistic, systematic cross-level methodology for designing energy-efficient intelligent systems. Progress on this project will enable ubiquitous DNN-powered intelligent functions in a significantly increased number of resource-constrained daily-life devices, across numerous camera-based Internet-of-Things (IoT) applications such as traffic monitoring, self-driving and smart cars, personal digital assistants, surveillance and security, and augmented reality. As camera-based IoT devices penetrate all walks of life, by enabling DNN-powered intelligence to be pervasive in these devices, the proposed research can have a tremendous impact on global societies and economies. The research will be integrated with education on energy efficient deep learning. Educational activities include curriculum development, undergraduate research, and outreach to K-12 students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

将深度神经网络（DNN）功能功能带入物联网（IoT）设备以实现无处不在的智能“ IoT相机”的需求巨大。但是，最先进的DNN的能源成本过高，使其不切实际地部署在资源受限的物联网平台中。该项目将通过平台，硬件和算法共同设计创新的系统集成来开发一种新型的节能DNN框架。尽管对节能DNN的兴趣越来越大，但现有技术仍缺乏整个设计抽象堆栈的系统优化，从系统到算法到硬件实现。拟议的研究主张通过共同优化平台，硬件和算法级的共同设计工作来提倡为节能和适应性DNN驱动的“物联网相机”进行创新的整体努力。在系统级别上，我们将讨论如何自动生成和调整DNN模型和实现，以满足各种“ IoT设备”特定于应用程序的性能需求和特定于设备的资源约束。在硬件级别上，我们将利用DNN激活中观察到的高稀疏性来通过使用低成本零预测变量，从而绕过不必要的计算，从而为DNN培训和推理提供节能硬件实现。在算法级别上，我们将在DNN培训以及高效，可控制的自适应推理机制中开发创新的分解稀疏性，并与我们的硬件创新完全补充并密切相结合。 拟议的研究将使每个级别的科学领域从系统和算法中推进到硬件和整体，系统的跨层次方法，用于设计节能智能系统。该项目的进展将在许多基于摄像机的电池Inforet（IoT）应用程序（例如交通监视，自动驾驶和智能汽车，个人数字助理，监视和安全性以及增强现实现实）等众多基于摄像机的电池互联网应用程序（IOT）应用程序中，使无处不在的DNN驱动智能功能能够显着增加。由于基于摄像机的物联网设备可以通过使DNN驱动的智能在这些设备中具有广泛性来渗透到各行各业时，拟议的研究可能会对全球社会和经济产生巨大影响。 该研究将与能源有效的深度学习有关。 教育活动包括课程开发，本科研究和向K-12学生推广。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子优点和更广泛的影响评估标准通过评估来支持的。

项目成果

The Lottery Ticket Hypothesis for Pre-trained BERT Networks

• 发表时间: 2020-07
• 期刊: ArXiv
• 作者: Tianlong Chen;Jonathan Frankle;Shiyu Chang;Sijia Liu;Yang Zhang;Zhangyang Wang;Michael Carbin

Data-Efficient Double-Win Lottery Tickets from Robust Pre-training

• DOI: 10.48550/arxiv.2206.04762
• 发表时间: 2022-06
• 作者: Tianlong Chen;Zhenyu (Allen) Zhang;Sijia Liu;Yang Zhang;Shiyu Chang;Zhangyang Wang

Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets

• 发表时间: 2022-02
• 作者: Tianlong Chen;Xuxi Chen;Xiaolong Ma;Yanzhi Wang;Zhangyang Wang

The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models

• DOI: 10.1109/cvpr46437.2021.01604
• 发表时间: 2021-01-01
• 期刊: 2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, CVPR 2021
• 作者: Chen, Tianlong;Frankle, Jonathan;Wang, Zhangyang
• 通讯作者: Wang, Zhangyang

Audio Lottery: Speech Recognition Made Ultra-Lightweight, Noise-Robust, and Transferable

• 发表时间: 2022
• 作者: Shaojin Ding;Tianlong Chen;Zhangyang Wang