RI: Small: An Optimization Framework for Understanding Deep Networks

RI：小型：理解深度网络的优化框架

• 批准号: 1618485
• 负责人: Rene Vidal
• 金额: $ 45万
• 依托单位: Johns Hopkins University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-07-01 至 2020-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1618485&HistoricalAwards=false
• 关键词: RI; Small; Optimization; Framework; Understanding

The past few years have seen a dramatic increase in the performance of pattern recognition systems due to the introduction of deep neural networks. However, the mathematical reasons for this success remain elusive. A key challenge is that the problem of learning the parameters of a neural network is a non-convex optimization problem, which makes finding the globally optimal parameters extremely difficult. Another challenge is that there is currently very limited theory about how the network architecture should be constructed (i.e., number of layers, number of neurons per layer, connectivity patterns, etc.). The goal of this project is to develop an optimization framework that provides theoretical insights for the success of current network architectures and guides the design of novel architectures with guarantees of global optimality. This project will develop a mathematical framework for the analysis of a broad class of non-convex optimization problems, including matrix factorization, tensor factorization, and deep learning. In particular, this project will study the problem of minimizing the sum of a loss function and a regularization function, both of which can be non-convex, but should satisfy a certain "positive homogeneity" property. By properly designing positively homogeneous regularizers that constrain the "network size," this project aims to show that, under certain conditions, all local minima are globally optimal, and one can find a global minimum from any initialization using a local descent strategy. A deeper understanding of the mathematical properties of deep networks will impact not only machine learning and optimization, where our understanding of non-convex problems continues to be very limited, but also application areas such as computer vision, speech and natural language processing, where deep networks currently give state-of-the-art results.

在过去的几年中，由于引入了深层神经网络，模式识别系统的性能急剧提高。但是，这一成功的数学原因仍然难以捉摸。一个关键的挑战是，学习神经网络参数的问题是一个非凸优化问题，这使得找到全球最佳参数极为困难。另一个挑战是，当前关于如何构建网络体系结构的理论非常有限（即，层数，每层神经元的数量，连接模式等）。该项目的目的是开发一个优化框架，该框架为当前网络体系结构的成功提供了理论见解，并通过保证全球最优性来指导新型体系结构的设计。该项目将开发一个数学框架，用于分析一类广泛的非凸优化问题，包括矩阵分解，张量分解和深度学习。特别是，该项目将研究最小化损失函数和正则化函数总和的问题，这两种函数都可以是非凸的，但应满足某些“正同质性”特性。通过正确设计限制“网络大小”的正质量正规化器，该项目旨在表明，在某些条件下，所有本地最小值都是全球最佳的，并且可以使用本地下降策略从任何初始化中找到全球最小值。对深网的数学特性的更深入的了解不仅会影响机器学习和优化，在这种情况下，我们对非凸问题的理解仍然非常有限，而且还会影响应用领域，例如计算机视觉，语音和自然语言处理，而深层网络目前为最先进的结果提供了最新的结果。