RI:Small:Dynamic Networks for Efficient, Adaptive, and Multimodal Vision

RI:Small：用于高效、自适应和多模态视觉的动态网络

• 批准号: 2303153
• 负责人: Nuno Vasconcelos
• 金额: $ 60万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-07-01 至 2026-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2303153&HistoricalAwards=false
• 关键词: RI; Small; Dynamic; Networks; Efficient

The recent introduction of a class of artificial intelligence (AI) methods known as deep learning has enabled transformational advances in computer vision. It is now possible to imagine AI systems capable of understanding and responding to a visual scene in a manner similar to people. However, much research is still needed to enable effective solutions for perceptual tasks combining multiple modalities (vision, audio, language), with great robustness to domain variability, and high efficiency in terms of computation and energy consumption. While unified solutions to these problems are critically important for applications such as robotics or the deployment of AI on edge devices, such as cellphones, the problems are usually addressed independently, leading to disconnected solutions. Hence, there is a need for basic research in neural network architectures that not only advance each of the problems per se but also contribute to the integrated solution of all problems. Drawing inspiration from the plasticity of biological brains, the project will explore the use of dynamic networks, which vary their computations depending on their input, to achieve this goal. The challenge is that, because it is impossible to dynamically predict the massive number of parameters of a modern network, this requires careful selection of the parameters to endow with dynamics. The project aims to develop algorithms for such parameter selection and investigate their benefits for a range of applications in computer vision.To achieve its objectives, the project will leverage the well-known importance of feature mixing in deep learning, implementing network dynamics efficiently via feature-based attention mechanisms that perform mixing through dynamic matrix factorizations. These are factorizations of layer weight matrices that enable the restriction of dynamic parameters to a small latent kernel per layer. Two classes of factorization are proposed, enabling the implementation of dynamic attention by parameter synthesis or non-linear transformations. These factorizations are then proposed as a unified substrate for the design of perception algorithms with state-of-the-art performance for multimodal perceptual tasks, such as visual grounding or audio-visual spatial localization. These algorithms will be implementable with extremely light weight deep learning networks to enable high computation and energy efficiency, and are applicable across architectures that range from convolutional networks (CNNs) to transformers. Finally, they will have strong robustness to variability of operating environments and so can support challenging applications such as home robotics or perception on edge devices. The research has applicability in areas of societal relevance, such as manufacturing, self-driving vehicles, intelligent systems, assisted living, homeland security, etc. Educationally, the project will provide exciting opportunities for both graduate and undergraduate research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

最近引入的一类人工智能（AI）方法被称为深度学习，使计算机视觉的变革性进步。现在可以想象AI系统能够以类似于人的方式理解和响应视觉场景。但是，仍然需要进行大量研究，以实现有效的解决方案，以将多种方式（视觉，音频，语言）和对域变异性的强大鲁棒性以及在计算和能源消耗方面的高效率结合起来。尽管这些问题的统一解决方案对于机器人技术或在边缘设备（例如手机）上部署等应用非常重要，但这些问题通常是独立解决的，从而导致解决方案断开的解决方案。因此，需要在神经网络体系结构中进行基础研究，这些研究不仅可以推进每个问题本身，而且还有助于所有问题的综合解决方案。该项目从生物学大脑的可塑性中汲取灵感，将探索动态网络的使用，这些网络根据其输入而改变其计算，以实现这一目标。面临的挑战是，由于不可能动态预测现代网络的大量参数数量，因此需要仔细选择参数才能赋予动力学。该项目旨在开发用于选择参数的算法，并调查其对计算机视觉中一系列应用的好处。为了实现其目标，该项目将利用深度学习中功能混合的众所周知的重要性，通过基于功能的注意力机制有效地实施网络动态，这些机制通过动态Matrix分解执行混合。这些是层权重矩阵的因素化，可以使动态参数限制为每层潜在的小内核。提出了两类分解，从而可以通过参数合成或非线性转换来实现动态注意力。然后将这些因素化作为用于设计具有最先进性能的感知算法的统一底物，以进行多模式感知任务，例如视觉接地或视听空间定位。这些算法将具有极大的重量深度学习网络，以实现高度计算和能源效率，并且适用于从卷积网络（CNN）到变形金刚的范围内的架构。最后，它们将对操作环境的可变性具有强大的稳健性，因此可以支持诸如Edge设备上的家庭机器人技术或感知等具有挑战性的应用程序。这项研究适用于社会相关性领域，例如制造业，自动驾驶汽车，智能系统，辅助生活，国土安全等。从教育上讲，该项目将为研究生和本科研究提供令人兴奋的机会。该奖项反映了NSF的法定任务，并认为通过基金会的知识优点和广泛的criperia criperia criperia criperia criperia criperia rection the Action the Awardia奖。