Collaborative Research: CIF: Medium: Analysis and Geometry of Neural Dynamical Systems

合作研究：CIF：媒介：神经动力系统的分析和几何

基本信息

批准号：
2106377
负责人：
Philippe Rigollet
金额：
$ 52.95万
依托单位：
Massachusetts Institute of Technology
依托单位国家：
美国
项目类别：
Continuing Grant
财政年份：
2021
资助国家：
美国
起止时间：
2021-06-01 至 2024-05-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2106377&HistoricalAwards=false
关键词：
Collaborative Research CIF Medium Analysis

项目摘要

The complexity of modern neural nets, with their millions of parameters and unprecedented computational demands, has been a major hurdle for the conventional approaches which had been successfully applied in machine learning over the past decades. This project aims to develop new mathematical and computational foundations for the analysis and design of these systems through a radically new conceptualization of their architectures as continuous dynamical systems. The key pillar of this framework is the idealization of depth as a continuum of layers and width as a continuum of neurons. Infinitesimal abstractions of this type have successfully unlocked many disciplines throughout the twentieth century, including probability, optimization, control, and many more. This collaborative project involving UIUC and MIT will push the boundaries of the theory and practice of deep learning, while sparking sustained interactions between the communities of electrical engineering, mathematics, statistics, and theoretical computer science. The project will also have broad impacts through a deliberate approach to education and training. The education and outreach activities will include research opportunities for undergraduate students at both institutions, as well as an exchange program to foster the collaboration and exchange of ideas. This project on Analysis and Geometry of Neural Dynamical Systems is developing the mathematical foundations of deep learning by synthesizing tools from probability, statistics, dynamical systems, geometric analysis, partial differential equations, and optimal transport. The research program is articulated around three major directions: (1) continuous models of neural dynamical systems; (2) discretization schemes; and (3) algorithms. The first direction is focusing on characterizing the tradeoffs between the expressive power and complexity of idealized infinitely wide and deep neural nets. The second direction builds on these continuous abstractions to develop, from first principles, mathematically rigorous and practically implementable techniques for analyzing large but finite neural nets. The third direction emphasizes algorithmic and computational aspects, such as the computational complexity of numerical methods, stability, and implicit regularization, using a novel synthesis of analytic and geometric methods developed as part of the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代神经网的复杂性及其数百万参数和前所未有的计算需求，是过去几十年来成功应用于机器学习的常规方法的一个重大障碍。该项目旨在通过将其架构作为连续动力学系统的彻底新概念化来开发新的数学和计算基础，以分析和设计这些系统。该框架的关键支柱是深度的理想化，作为层和宽度的连续性，作为神经元的连续体。这种类型的无限抽象在整个20世纪成功解锁了许多学科，包括概率，优化，控制等。这个涉及UIUC和MIT的协作项目将突破深度学习理论和实践的界限，同时激发电气工程，数学，统计和理论计算机科学社区之间的持续互动。该项目还将通过故意的教育和培训方法产生广泛的影响。教育和外展活动将包括各机构的本科生的研究机会，以及促进思想的协作和交流的交流计划。关于神经动力学系统分析和几何形状的项目正在通过从概率，统计，动力学系统，几何分析，部分微分方程和最佳传输中综合工具来开发深度学习的数学基础。该研究计划围绕三个主要方向阐明：（1）神经动力学系统的连续模型；（2）离散方案；（3）算法。第一个方向的重点是表征理想化的无限宽和深神经网的表达力与复杂性之间的权衡。第二个方向建立在这些连续的抽象基础上，从第一原则，在数学上严格且实际上可以实现的技术来分析大型但有限的神经网络。第三个方向强调算法和计算方面，例如使用新颖的分析和几何方法的合成数字方法的计算复杂性，稳定性和隐式正则化，作为项目的一部分。本奖反映了NSF的法定任务，并通过对基础的智力效果进行评估，以评估依据，并以评估的评估值得评估。