Algorithms and Theory for Compressing Deep Neural Networks

压缩深度神经网络的算法和理论

基本信息

批准号：
2208126
负责人：
Penghang Yin
金额：
$ 27.66万
依托单位：
SUNY at Albany
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2022
资助国家：
美国
起止时间：
2022-09-01 至 2025-08-31
项目状态：
未结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2208126&HistoricalAwards=false
关键词：
Algorithms Theory Compressing Deep Neural

项目摘要

Deep neural networks (DNNs) have been the main driving force for recent advancements in artificial intelligence (AI) technology, profoundly impacting society in the areas of transportation, public safety, entertainment, health care, and other areas of public life. One of the biggest obstacles to AI's even broader impact on our daily lives is the typically enormous power consumption of DNNs upon deployment. The aim of this project is to develop mathematical and computational approaches for DNN compression to realize the fast and efficient deployment of AI systems on mobile platforms with low-power budgets such as smartphones. Results of this work will have a variety of applications which include video security systems, autopilot, smart robots, and face identification. The project will involve training of graduate students, development of data science courses, as well as collaboration with industry. The PI plans to (1) develop and analyze coarse gradient algorithms, featuring a biased first-order oracle, for the discretization of various neural architectures including transformer-based networks; (2) develop and analyze efficient thresholding-based algorithms for compressing networks via structured sparsity on both balanced and unbalanced data; (3) investigate the model capacity of compressed DNNs and establish universal finite-sample expressivity theory. The proposed research will also explore the applications of coarse gradient algorithms to other machine learning problems with discrete-valued loss functions and advance knowledge in discrete optimization.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度神经网络（DNN）一直是人工智能（AI）技术近期进步的主要驱动力，深刻影响着社会的交通、公共安全、娱乐、医疗保健和其他公共生活领域。人工智能对我们日常生活产生更广泛影响的最大障碍之一是 DNN 在部署时通常会产生巨大的功耗。该项目的目标是开发 DNN 压缩的数学和计算方法，以实现人工智能系统在智能手机等低功耗移动平台上的快速高效部署。这项工作的成果将具有多种应用，包括视频安全系统、自动驾驶仪、智能机器人和人脸识别。该项目将涉及研究生培训、数据科学课程开发以及与行业的合作。 PI 计划 (1) 开发和分析粗梯度算法，以有偏差的一阶预言机为特色，用于各种神经架构的离散化，包括基于 Transformer 的网络； (2) 开发和分析有效的基于阈值的算法，通过平衡和不平衡数据的结构化稀疏性来压缩网络； (3) 研究压缩 DNN 的模型容量并建立通用的有限样本表达理论。拟议的研究还将探索粗梯度算法在具有离散值损失函数的其他机器学习问题中的应用，以及离散优化方面的先进知识。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值进行评估，被认为值得支持以及更广泛的影响审查标准。