Random Matrix Limit Theorems for Deep Neural Networks

深度神经网络的随机矩阵极限定理

• 批准号: RGPIN-2021-02533
• 负责人: Nica, Mihai
• 金额: $ 1.89万
• 依托单位: University of Guelph
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=759077
• 关键词: Random; Matrix; Limit; Theorems; Deep

Recent advances in deep neural networks (DNNs) have had a tremendous impact on the modern world. However, the theoretical understanding of these systems is still in its infancy. As a matter of course, the research in this area has been empirically driven and computationally focused rather than emphasizing mathematically rigorous results. There are many open theoretical questions which have been uncovered by empirical work that are now ripe for mathematical analysis. I propose a research program that will develop and apply tools from theoretical probability, specifically random matrix theory, to gain a better understanding of the theory of DNNs and other machine learning systems. I will focus on developing new limit theorems which describe behavior when the number of parameters and/or data becomes very large. These results will help us understand how DNNs work and help us design more effective systems in the future. My objectives of the research program include: 1. The neural tangent kernel: A random matrix that explains the behavior of large networks The neural tangent kernel (NTK) is a recently discovered non-random asymptotic object that explains the behavior of DNNs of fixed depth in the infinite width limit, when the number of neurons in each hidden layer tends to infinity. When applied to random data, the NTK gives a random matrix whose dimensions are the number of given data points. Analysis of this random matrix can explain how DNNs behave during training and can be used to understand the generalization error in deep neural networks. I propose to study this model using random matrix theory. 2. Applied free probability: Advanced tools for random matrix analysis The theory of free probability was originally developed in connection to pure problems in the field of operator algebras. More recently however, methods from free probability and its extensions have emerged as powerful tools for computing asymptotic features of complicated random matrix models. One application is to use free probability to compute the limiting spectrum of large random matrix models connected to DNNs. I also plan to investigate the use of operator valued free probability, a powerful extension of free probability, to study block random matrices related to DNNs. 3. Kardar-Parisi-Zhang (KPZ) universality: Fluctuations of random matrix eigenvalues The KPZ universality class is a collection of stochastic systems, including examples from stochastic PDEs and interacting particle systems, which all share the same type of universal asymptotic random behavior. An important application is the behaviour of the largest eigenvalues in many random matrix models. (As opposed to the bulk behavior of the spectrum captured by other random matrix tools). I plan to apply ideas from KPZ to random matrix problems coming from DNNs and other statistical learning models to analyze the evolution of the largest eigenvalues in these problems.

深度神经网络 (DNN) 的最新进展对现代世界产生了巨大影响，然而，对这些系统的理论理解仍处于起步阶段，这一领域的研究是基于经验和计算的。我提出了一个研究计划，该计划将开发和应用理论概率（特别是随机矩阵理论）的工具来进行数学分析。更好地了解我将专注于开发新的极限定理，这些定理描述参数和/或数据数量变得非常大时的行为，这些结果将帮助我们理解 DNN 的工作原理，并帮助我们设计更有效的系统。我的研究计划的未来目标包括： 1. 神经正切核：解释大型网络行为的随机矩阵 神经正切核 (NTK) 是最近发现的一种非随机渐近对象，它解释了 DNN 的行为。的在无限宽度限制下的固定深度，当每个隐藏层中的神经元数量趋于无穷大时，NTK给出一个随机矩阵，其维度是给定数据点的数量，分析这个随机矩阵可以解释。 DNN 在训练过程中的行为方式以及可用于理解深度神经网络中的泛化误差 我建议使用随机矩阵理论来研究该模型 2. 应用自由概率：随机矩阵分析的高级工具 自由概率理论最初是开发出来的。关于操作员领域的纯粹问题然而，最近，自由概率方法及其扩展已成为计算复杂随机矩阵模型渐近特征的强大工具，其中一个应用是使用自由概率来计算连接到 DNN 的大型随机矩阵模型的极限谱。还计划研究使用算子值自由概率（自由概率的强大扩展）来研究与 DNN 相关的块随机矩阵。 3. Kardar-Parisi-Zhang (KPZ) 普适性：随机矩阵的波动。特征值 KPZ 普适性类是随机系统的集合，包括随机偏微分方程和相互作用粒子系统的示例，它们都具有相同类型的普适渐近随机行为，一个重要的应用是许多系统中最大特征值的行为。 （与其他随机矩阵工具捕获的频谱的大量行为相反），我计划将 KPZ 的思想应用于来自 DNN 和其他统计学习模型的随机矩阵问题，以分析最大的演化。这些问题中的特征值。