Collaborative Research: New Perspectives on Deep Learning: Bridging Approximation, Statistical, and Algorithmic Theories

合作研究：深度学习的新视角：桥接近似、统计和算法理论

• 批准号: 2134133
• 负责人: Aarti Singh
• 金额: $ 45万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-11-01 至 2024-10-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2134133&HistoricalAwards=false
• 关键词: Collaborative; Research; New; Perspectives; Deep

Deep Learning (DL) has led to a renaissance in neural network methods in data-driven science and engineering. The development of DL systems and applications, including computer vision and natural language understanding, has been led primarily by experiments and engineering practice. Mathematical analysis has only begun to provide insights into these complex machine learning systems. The lack of basic understanding has contributed to serious challenges and shortcomings ranging from the fragility and susceptibility to corrupted data to their uninterpretable behaviors. These problems can be traced to fundamental gaps in the mathematical understanding of DL. This project tackles this challenge by bringing approximation, statistical, and algorithmic theories together to develop new mathematical foundations for DL. The goals of the project are to mathematically characterize the strengths and limitations of DL models, and to understand the properties of DL models trained using examples of desired behavior (training data) as well as the tradeoffs between the performance of DL systems and the training dataset size. While DL is already in widespread use, the continued success of DL requires far more complete mathematical understandings and principled approaches to guide its use and reliable application. The project will provide practitioners with clearer guidance on the strengths, limitations, and best approaches to using DL. Broader impacts of the project also include education and mentoring, including the training of graduate students in mathematical fields such as approximation theory, signal processing, statistics, and machine learning and, most importantly, how these fields collectively inform the theory and practice of DL.DL seeks to learn an unknown function from data using compositions (layers) of linear combinations of simple functions (neurons). The shortcomings of DL can be traced to fundamental gaps in its mathematical theory including the following issues. The function spaces that capture the salient properties of DL applications are poorly understood. The characteristics of functions learned through neural network training are mysterious. The ability of DL models to discriminate between data distributions has not yet been quantified satisfactorily. Understanding of the tradeoffs between accuracy and training set size is lacking. This project tackles these challenges by bringing approximation, statistical, and algorithmic theories together to develop new theoretical foundations for DL. This project builds innovative bridges between approximation theory, nonparametric statistics, learning theory and algorithms to develop new mathematical foundations for DL. This includes the development of new model classes of functions that are naturally suited to characterize the properties, strengths, and limitations of deep neural network architectures and applications; novel approaches to understand the roles of regularization and sparsity in DL; fundamental frameworks to quantify the discrimination power of DL and generalized adversarial networks; and innovative theory to make DL algorithms more data efficient through the use of side-information, partial differential equations, and richer forms of data than the conventional function evaluations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习（DL）导致了数据驱动科学和工程学的神经网络方法的复兴。 DL系统和应用的开发，包括计算机视觉和自然语言的理解，主要由实验和工程实践领导。数学分析才刚刚开始为这些复杂的机器学习系统提供见解。缺乏基本的理解导致了严重的挑战和缺点，从损坏的数据的脆弱性和易感性到其无法解释的行为。 这些问题可以追溯到DL的数学理解中的基本差距。该项目通过将近似，统计和算法理论融合在一起，以开发DL的新数学基础来应对这一挑战。该项目的目标是数学表征DL模型的优势和局限性，并了解使用所需行为示例（培训数据）训练的DL模型的属性，以及DL Systems的性能与培训数据集大小之间的权衡。尽管DL已经广泛使用，但DL的持续成功需要更完整的数学理解和原则性方法来指导其使用和可靠的应用。该项目将为从业人员提供有关使用DL的优势，局限性和最佳方法的更清晰的指导。该项目的更广泛影响还包括教育和指导，包括在数学领域的研究生培训，例如近似理论，信号处理，统计和机器学习，最重要的是，这些领域如何共同为DL.DL的理论和实践提供信息，以寻求使用简单功能的线性组合（Neurons（Neurons of Neurons of Simeal of Simeal of Simple））的理论和实践。 DL的缺点可以追溯到其数学理论中的基本差距，包括以下问题。捕获DL应用的显着特性的功能空间知之甚少。通过神经网络培训学到的功能的特征是神秘的。 DL模型区分数据分布的能力尚未令人满意地量化。缺乏了解准确性和训练设定规模之间的权衡。该项目通过将近似，统计和算法理论融合在一起以开发DL的新理论基础来应对这些挑战。该项目在近似理论，非参数统计，学习理论和算法之间建立了创新的桥梁，以开发DL的新数学基础。这包括开发新的模型类别的功能类别，这些功能自然适合表征深神经网络架构和应用的属性，优势和局限性；了解正规化和稀疏性在DL中的作用的新颖方法；量化DL和广义对抗网络的歧视能力的基本框架；与传统功能评估相比，通过使用侧信息，部分微分方程和更丰富的数据形式，使DL算法更有效地数据有效的创新理论。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子优点和更广泛影响的评估来通过评估来支持的。