RI: Small: An Optimization Framework for Understanding Deep Networks

RI：小型：理解深度网络的优化框架

• 批准号: 1618485
• 负责人: Rene Vidal
• 金额: $ 45万
• 依托单位: Johns Hopkins University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-07-01 至 2020-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1618485&HistoricalAwards=false
• 关键词: RI; Small; Optimization; Framework; Understanding

The past few years have seen a dramatic increase in the performance of pattern recognition systems due to the introduction of deep neural networks. However, the mathematical reasons for this success remain elusive. A key challenge is that the problem of learning the parameters of a neural network is a non-convex optimization problem, which makes finding the globally optimal parameters extremely difficult. Another challenge is that there is currently very limited theory about how the network architecture should be constructed (i.e., number of layers, number of neurons per layer, connectivity patterns, etc.). The goal of this project is to develop an optimization framework that provides theoretical insights for the success of current network architectures and guides the design of novel architectures with guarantees of global optimality. This project will develop a mathematical framework for the analysis of a broad class of non-convex optimization problems, including matrix factorization, tensor factorization, and deep learning. In particular, this project will study the problem of minimizing the sum of a loss function and a regularization function, both of which can be non-convex, but should satisfy a certain "positive homogeneity" property. By properly designing positively homogeneous regularizers that constrain the "network size," this project aims to show that, under certain conditions, all local minima are globally optimal, and one can find a global minimum from any initialization using a local descent strategy. A deeper understanding of the mathematical properties of deep networks will impact not only machine learning and optimization, where our understanding of non-convex problems continues to be very limited, but also application areas such as computer vision, speech and natural language processing, where deep networks currently give state-of-the-art results.

过去几年，由于深度神经网络的引入，模式识别系统的性能得到了显着提高。然而，这一成功的数学原因仍然难以捉摸。一个关键的挑战是学习神经网络参数的问题是一个非凸优化问题，这使得寻找全局最优参数变得极其困难。另一个挑战是，目前关于如何构建网络架构（即层数、每层神经元数量、连接模式等）的理论非常有限。该项目的目标是开发一个优化框架，为当前网络架构的成功提供理论见解，并在保证全局最优性的情况下指导新型架构的设计。该项目将开发一个数学框架，用于分析广泛的非凸优化问题，包括矩阵分解、张量分解和深度学习。特别是，该项目将研究最小化损失函数和正则化函数之和的问题，这两个函数都可以是非凸的，但应满足一定的“正同质性”属性。通过正确设计限制“网络大小”的正同质正则化器，该项目旨在表明，在某些条件下，所有局部最小值都是全局最优的，并且可以使用局部下降策略从任何初始化中找到全局最小值。对深度网络数学特性的更深入理解不仅会影响机器学习和优化（我们对非凸问题的理解仍然非常有限），还会影响计算机视觉、语音和自然语言处理等应用领域（其中深度网络）。网络目前给出了最先进的结果。