Collaboration Research: Probabilistic, Geometric, and Topological Analysis of Neural Networks, From Theory to Applications

合作研究：神经网络的概率、几何和拓扑分析，从理论到应用

• 批准号: 2133851
• 负责人: Gilbert Strang
• 金额: $ 10.8万
• 依托单位: Massachusetts Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-01-01 至 2024-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2133851&HistoricalAwards=false
• 关键词: Collaboration; Research; Probabilistic; Geometric; Topological

One of the most exciting technical developments of the last decade is the widespread adoption of a family of algorithms called neural networks, used in cutting-edge industrial applications ranging from self-driving cars to predicting the three-dimensional shapes of proteins from their amino acid sequences. The goals of this project are twofold. First, the investigators seek to use tools from mathematics (specifically probability and combinatorics) to better understand how neural networks behave and then to fashion this understanding into new, more efficient, and safer algorithms. This involves a collaborative effort between mathematicians, computer scientists, and electrical engineers. The project team seeks to unravel a fundamental mystery: why is it that neural networks appear to be incredibly complex, yet despite their seeing intricacy, still learn parsimonious and useful ways of making predictions? Put another way, the investigators aim to define and analyze different mathematical notions of neural network complexity and then to use them as theoretically grounded guides in the search for ever more efficient and interpretable algorithms related to neural networks. The second goal is to create a series of educational resources, ranging from videos to course notes, that will enable various segments of society at large (e.g. students, policy makers, scientists, and so on) to engage with and get a usable appreciation for the ideas, challenges, and opportunities surrounding modern neural networks. The research in this project consists of three interconnected parts. The first is a probabilistic analysis of a variety of neural network complexity measures before, during, and after training. Relevant tools come from probability, functional analysis, information theory, and geometry. Key theoretical questions include quantifying implicit bias and bounding generalization error for learning structured functions. The second is a topological and geometric analysis of both individual ReLU network functions and spaces of ReLU networks. Relevant tools come from Morse Theory and low-dimensional topology. Key theoretical questions hinge on understanding topological implicit bias and topological depth separation. Finally, the investigators seek theory-guided insights for applied deep learning via (i) principled, efficient neural architecture search using average case complexity measures as surrogates for practical expressivity, trainability, and generalization and (ii) novel approaches to model compression and scaling via topological expressivity of ReLU networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

过去十年中，最令人兴奋的技术发展之一是广泛采用了一种称为神经网络的算法，用于从自动驾驶汽车到预测其氨基酸序列的蛋白质的三维形状，用于尖端的工业应用。该项目的目标是双重的。首先，研究人员试图使用数学（特别是概率和组合学）的工具更好地了解神经网络的行为方式，然后将这种理解塑造成新的，更高效，更安全的算法。这涉及数学家，计算机科学家和电气工程师之间的合作努力。项目团队试图揭开一个基本的谜团：为什么神经网络似乎非常复杂，但是尽管他们看到了复杂性，但仍然学习了简约且有用的预测方法？换句话说，研究人员的目的是定义和分析神经网络复杂性的不同数学概念，然后将它们用作理论上扎根的指南，以搜索与神经网络相关的更有效和可解释的算法。第二个目标是创建一系列的教育资源，从视频到课程笔记，这将使整个社会的各个部分（例如，学生，政策制定者，科学家等）能够与周围现代神经网络周围的想法，挑战和机会相互动并获得可用的欣赏。该项目的研究由三个相互联系的部分组成。第一个是对训练之前，期间和之后的各种神经网络复杂性度量的概率分析。相关工具来自概率，功能分析，信息理论和几何形状。关键理论问题包括量化学习结构化功能的隐性偏见和界限概括误差。第二个是对Relu网络的各个Relu网络函数和空间的拓扑和几何分析。相关工具来自Morse理论和低维拓扑。关键的理论问题取决于理解拓扑隐式偏见和拓扑深度分离。 Finally, the investigators seek theory-guided insights for applied deep learning via (i) principled, efficient neural architecture search using average case complexity measures as surrogates for practical expressivity, trainability, and generalization and (ii) novel approaches to model compression and scaling via topological expressivity of ReLU networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts审查标准。