Collaborative Research: CDS&E-MSS: Deep Network Compression and Continual Learning: Theory and Application

合作研究：CDS

• 批准号: 2053448
• 负责人: Paul Hand
• 金额: $ 15万
• 依托单位: Northeastern University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-07-01 至 2024-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2053448&HistoricalAwards=false
• 关键词: Collaborative; Research; CDS&E; MSS; Deep

Deep neural networks (DNNs) have led to transformative developments in a wide number of tasks, such as identifying people or objects in photographs, translating and synthesizing natural language, and generating scientific and medical images. These advances were possible because neural networks are trained to find patterns in large datasets. Balancing the trade-off between the size and performance of a deep network is a vital aspect of designing deep neural networks that can easily be translated to hardware. Although deep learning yields remarkable performance in real-world problems, they consume a large amount of memory and computational resources, which limits their large-scale deployment. Once neural networks are trained, they can also be brittle in the sense that when a network is trained on a new task, it typically forgets previously learned tasks. This brittleness presents challenges in building artificial intelligence systems that are intended to learn continually throughout their lifespan, and it leads to even more computational resources spent to retrain networks on tasks they have already learned. This results in large demands for electrical power, which leads to an increased carbon footprint and adverse environmental impacts. In this project the investigators propose novel algorithms and theoretical analysis for reducing the power consumption of neural networks by compressing their learned parameters and using these compressed parameters for continual learning. Graduate students will be involved in the research and receive interdisciplinary training.The overall goal of this project is to develop a novel probabilistic framework for neural network compression. Using this framework, the investigators will develop network compression algorithms based on the connectivity between filters and layers, which provides a sparsification criterion that is efficient in both training and testing processes. After network compression identifies which parameters of a neural network are more important than others, this feature can be used to develop algorithms for continual learning which are efficient because the important parameters are prioritized for the learning of subsequent tasks. The investigators will develop compression-inspired algorithms for continual learning based on statistics of individual layers and the connectivity between layers. The investigators will also provide a sparsity analysis and theoretical explanations in the form of mathematical theorems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度神经网络 (DNN) 带来了许多任务的变革性发展，例如识别照片中的人或物体、翻译和合成自然语言以及生成科学和医学图像。 这些进步之所以成为可能，是因为神经网络经过训练可以在大型数据集中查找模式。平衡深度网络的大小和性能之间的权衡是设计可以轻松转换为硬件的深度神经网络的一个重要方面。尽管深度学习在解决现实问题中产生了显着的性能，但它们消耗大量的内存和计算资源，这限制了它们的大规模部署。一旦神经网络经过训练，它们也可能很脆弱，因为当网络接受新任务的训练时，它通常会忘记以前学过的任务。 这种脆弱性给构建旨在在整个生命周期中不断学习的人工智能系统带来了挑战，并且导致需要花费更多的计算资源来重新训练网络来执行已经学会的任务。 这导致对电力的大量需求，从而导致碳足迹增加和对环境的不利影响。在这个项目中，研究人员提出了新颖的算法和理论分析，通过压缩学习的参数并使用这些压缩的参数进行持续学习来降低神经网络的功耗。 研究生将参与研究并接受跨学科培训。该项目的总体目标是开发一种新颖的神经网络压缩概率框架。 使用该框架，研究人员将开发基于过滤器和层之间的连接性的网络压缩算法，该算法提供了在训练和测试过程中都有效的稀疏化标准。 在网络压缩识别出神经网络的哪些参数比其他参数更重要之后，此功能可用于开发高效的持续学习算法，因为重要参数会优先用于后续任务的学习。研究人员将开发受压缩启发的算法，以基于各个层的统计数据和层之间的连接进行持续学习。 研究人员还将以数学定理的形式提供稀疏性分析和理论解释。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。