CAREER: Random Neural Nets and Random Matrix Products

职业：随机神经网络和随机矩阵产品

• 批准号: 2143754
• 负责人: Boris Hanin
• 金额: $ 57.72万
• 依托单位: Princeton University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-06-01 至 2027-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2143754&HistoricalAwards=false
• 关键词: CAREER; Random; Neural; Nets; Matrix

We live in an era of big data and inexpensive computation. Vast stores of information can efficiently be analyzed for underlying patterns by machine learning algorithms, leading to remarkable progress in applications ranging from self-driving cars to automatic drug discovery and machine translation. Underpinning many of these exciting practical developments is a class of computational models called neural networks. Originally developed in the 1940's and 1950's, the neural nets used today are as complex as they are powerful. The purpose of this project is to develop a range of principled techniques for understanding key aspects of how neural networks work in practice and how to make them better. The approach taken by this project is probabilistic and statistical in nature. Just as the ideal gas law accurately describes the large-scale properties of a gas directly through pressure, volume, and temperature without the need specify the state of each individual gas molecule, this project will explore and identify emergent statistical behaviors of large neural networks that provably explain many of their key properties observed in practice. The project will also provide research training and educational opportunities through organization of summer schools in machine learning for graduate students. At a high level, a neural network is a family of functions given by composing affine transformations with elementary non-linear operations. The simplest important kind of neural networks are roughly described by two parameters called depth and width. The former is the dimension of the spaces on which the affine transformations act and the latter is the number of compositions. The technical heart of this project is to understand the statistical behavior of such networks when the affine transformations are chosen at random. The starting point is an analytically tractable regime in which the network width is sent to infinity at fixed depth. In this infinite width limit, random networks converge to Gaussian processes and optimization of network parameters from their randomly chosen starting points reduces to a kernel method. Unfortunately, this concise description cannot capture what is perhaps the most important empirical property of neural networks, namely their ability to learn data-dependent features. Understanding how feature learning occurs is at the core of this project and requires new probabilistic and analytic tools for studying random neural networks at finite width. The basic idea is to perform perturbation theory around the infinite width limit, treating the reciprocal of the network width as a small parameter. The goal is then to obtain, to all orders in this reciprocal, the expressions for joint distribution of the values and derivatives (with respect to both model inputs and model parameters) of a random neural network. Such formulas have practical consequences for understanding the numerical stability of neural network training, suggesting principled settings for optimization hyper-parameters, and quantifying feature learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

我们生活在一个大数据和廉价计算的时代。机器学习算法可以有效地分析大量信息的潜在模式，从而在从自动驾驶汽车到自动药物发现和机器翻译等应用领域取得显着进展。许多这些令人兴奋的实际发展的基础是一类称为神经网络的计算模型。最初开发于 20 世纪 40 年代和 1950 年代，如今使用的神经网络既复杂又强大。该项目的目的是开发一系列原则性技术，以了解神经网络在实践中如何工作的关键方面以及如何使其变得更好。该项目采用的方法本质上是概率和统计的。正如理想气体定律直接通过压力、体积和温度准确描述气体的大尺度特性，而无需指定每个气体分子的状态一样，该项目将探索和识别大型神经网络的紧急统计行为，可证明地解释了在实践中观察到的许多关键特性。该项目还将通过为研究生组织机器学习暑期学校来提供研究培训和教育机会。在较高层次上，神经网络是通过将仿射变换与基本非线性运算组合而给出的一系列函数。最简单的重要类型的神经网络大致由称为深度和宽度的两个参数来描述。前者是仿射变换作用的空间的维度，后者是组合的数量。该项目的技术核心是了解随机选择仿射变换时此类网络的统计行为。起点是一个分析上易于处理的机制，其中网络宽度在固定深度处发送到无穷大。在这种无限宽度限制下，随机网络收敛于高斯过程，并且从随机选择的起点对网络参数的优化简化为核方法。不幸的是，这种简洁的描述无法捕捉神经网络可能最重要的经验特性，即它们学习数据相关特征的能力。了解特征学习如何发生是该项目的核心，需要新的概率和分析工具来研究有限宽度的随机神经网络。基本思想是围绕无限宽度极限进行扰动理论，将网络宽度的倒数视为一个小参数。然后，目标是获得该倒数中所有阶数的随机神经网络的值和导数（相对于模型输入和模型参数）的联合分布表达式。这些公式对于理解神经网络训练的数值稳定性、提出优化超参数的原则性设置以及量化特征学习具有实际意义。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优势和评估进行评估，被认为值得支持。更广泛的影响审查标准。

项目成果

Deep ReLU Networks Preserve Expected Length

深度 ReLU 网络保留预期长度

• 发表时间: 2022-01
• 期刊: International Conference on Learning Representations
• 作者: Hanin, Boris;Rolnick, David;Jeong, Ryan
• 通讯作者: Jeong, Ryan

Deep ReLU Networks Preserve Expected Length

深度 ReLU 网络保留预期长度

• 发表时间: 2021-01
• 期刊: Advances in neural information processing systems
• 作者: Hanin, B;Jeong, R.;Rolnick, D
• 通讯作者: Rolnick, D

Random neural networks in the infinite width limit as Gaussian processes

作为高斯过程的无限宽度限制中的随机神经网络

• DOI: 10.1214/23-aap1933
• 发表时间: 2023-12
• 期刊: The Annals of Applied Probability
• 作者: Hanin; Boris
• 通讯作者: Boris

Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit

残差网络中的深度超参数传递：动态和缩放限制

• DOI: 10.48550/arxiv.2309.16620
• 发表时间: 2023-09-28
• 期刊: ArXiv
• 作者: Blake Bordelon;Lorenzo Noci;Mufan Bill Li;Boris Hanin;C. Pehlevan
• 通讯作者: C. Pehlevan

Maximal Initial Learning Rates in Deep ReLU Networks

深度 ReLU 网络中的最大初始学习率

• 发表时间: 2023-07
• 期刊: Proceedings of Machine Learning Research
• 作者: Iyer, Gaurav;Hanin, Boris;Rolnick, David
• 通讯作者: Rolnick, David