Collaborative Reserach: SHF:Medium: Analog EDA-Inspired Methods for Efficient and Robust Neural Network Designs

协作研究：SHF:Medium：用于高效、鲁棒神经网络设计的模拟 EDA 启发方法

• 批准号: 2107373
• 负责人: Luca Daniel
• 金额: $ 30.31万
• 依托单位: Massachusetts Institute of Technology
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-07-15 至 2025-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2107373&HistoricalAwards=false
• 关键词: Collaborative; Reserach; SHF; Medium; Analog

Deep neural networks have achieved great success in many engineering fields including, but not limited to, image classification, speech recognition, recommendation systems and autonomous driving. However, they suffer from two major challenges. Firstly, many neural network models are not robust, i.e. a neural network could produce inaccurate results when the input data experiences a very small amount of perturbation. Secondly, the huge cost of generating and deploying large-size neural networks limits their applications in resource-constrained platforms (e.g. mobile devices and robots). The research team notices that there is a strong mathematical connection between certain types of neural networks and analog integrated circuits. It is also known that the EDA (electronic design automation) field has 50 years of successful history of modeling, simulating, verifying and optimizing analog integrated circuits. Therefore, this project aims to substantially enrich the algorithms and theoretical understanding of neural networks by leveraging the principled approaches in the EDA community. This research will support the cross-disciplinary development of a diverse cohort of graduate and undergraduate students at the University of California at Santa Barbara, the University of California at San Diego, and the Massachusetts Institute of Technology. Several graduate-level courses on computational methods, data science and artificial intelligence are being created or enriched. The research team willis also collaborating with industry to ensure effective technology transfers.This project focuses on certain types of deep neural networks (e.g., residual neural networks, recurrent neural networks and normalizing flows) that can be described as ordinary differential equations. The technical aims of the project are divided into three thrusts. The first thrust investigates the training and compression algorithms of deep neural networks from circuit simulation and modeling perspectives. Specifically, parallel training algorithms are being developed for neural networks by borrowing the idea from parallel circuit simulation. Hardware-friendly neural-network compression algorithms are being developed from the perspective of circuit model order reduction, thereby enabling energy-efficient and real-time inference of deep neural networks. The second thrust investigates probabilistic and accurate verification techniques for the robustness of deep neural networks from circuit uncertainty quantification perspectives. Specifically, high-confidence and tighter verification bounds are being developed to describe the reachable set of a deep neural network by leveraging the hierarchical and non-Monte-Carlo techniques in analog circuit uncertainty quantification. The third thrust aims to improve the robustness of a deep neural network from the perspective of analog circuit yield optimization. In this final thrust, two ideas are being explored: (1) pre-silicon yield optimization techniques for robust neural network training, and (2) post-silicon self-healing techniques for robustness improvement of a trained neural network.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度神经网络在许多工程领域取得了巨大的成功，包括但不限于图像分类，语音识别，推荐系统和自动驾驶。但是，他们面临两个主要挑战。首先，许多神经网络模型不健壮，即，当输入数据经历少量扰动时，神经网络可能会产生不准确的结果。其次，生成和部署大型神经网络的巨大成本限制了其在资源受限平台（例如移动设备和机器人）中的应用程序。研究小组注意到某些类型的神经网络与模拟整合电路之间存在牢固的数学联系。众所周知，EDA（电子设计自动化）领域在建模，模拟，验证和优化模拟集成电路方面拥有50年的成功历史。因此，该项目旨在通过利用EDA社区的原则方法来实质性地丰富对神经网络的算法和理论理解。这项研究将支持加利福尼亚大学圣塔芭芭拉分校，加利福尼亚大学圣地亚哥分校和马萨诸塞州技术学院的多元化研究生和本科生的跨学科发展。正在创建或丰富一些有关计算方法，数据科学和人工智能的研究生级课程。研究团队Willis还与行业合作，以确保有效的技术转移。该项目重点介绍了某些类型的深神经网络（例如，残留的神经网络，经常性神经网络和正常化的流动），可以描述为普通微分方程。该项目的技术目标分为三个推力。第一个推力从电路模拟和建模的角度研究了深神经网络的训练和压缩算法。具体而言，通过从并行电路模拟借用该想法来为神经网络开发并行训练算法。从电路模型降低的角度开发了对硬件友好的神经网络压缩算法，从而实现了深层神经网络的节能和实时推理。第二个推力研究了从电路不确定性定量的角度研究深神经网络的稳健性，研究了概率和准确的验证技术。具体而言，正在开发高信心和更严格的验证范围，以通过在模拟电路不确定性量化中利用层次结构和非蒙特 - 卡洛技术来描述深度神经网络的可达集。第三个推力旨在从模拟回路优化的角度提高深神网络的鲁棒性。在这个最后的解决方案中，正在探索两个想法：（1）用于强大的神经网络培训的固体产量优化技术，以及（2） - 塞利康后自我修复技术，以提高训练有素的神经网络的鲁棒性。该奖项反映了NSF的法定任务，并通过使用基金会的智力效果来评估NSF的法定任务，并获得了基础的评估。

项目成果

Fast Convergence for Unstable Reinforcement Learning Problems by Logarithmic Mapping

通过对数映射快速收敛不稳定强化学习问题

• 发表时间: 2022
• 期刊: 2022.
• 作者: Zhang, Wang;Nguyen, Lam M.;Das, Subhro;Megretski, Alexandre;Daniel, Luca;Weng, Tsui-Wei
• 通讯作者: Weng, Tsui-Wei

Robust Deep Reinforcement Learning through Adversarial Loss

• 发表时间: 2020-08
• 期刊: ArXiv
• 作者: Tuomas P. Oikarinen;Tsui-Wei Weng;L. Daniel

Hidden Cost of Randomized Smoothing

• 发表时间: 2021
• 期刊: Journal of Industrial and Management Optimization
• 影响因子: 1.3
• 作者: Jeet Mohapatra;Ching-Yun Ko;Lily Weng;Pin-Yu Chen;Sijia Liu;L. Daniel

Revisiting Contrastive Learning through the Lens of Neighborhood Component Analysis: an Integrated Framework

• 发表时间: 2021-12
• 期刊: ArXiv
• 作者: Ching-Yun Ko;Jeet Mohapatra;Sijia Liu;Pin-Yu Chen;Lucani E. Daniel;Lily Weng

SynBench: Task-Agnostic Benchmarking of Pretrained Representations using Synthetic Data

• DOI: 10.48550/arxiv.2210.02989
• 发表时间: 2022-10
• 期刊: ArXiv
• 作者: Ching-Yun Ko;Pin-Yu Chen;Jeet Mohapatra;Payel Das;Lucani E. Daniel