EAGER: CDS&E: Applied geometry and harmonic analysis in deep learning regularization: theory and applications

渴望：CDS

基本信息

批准号：
2140982
负责人：
Wei Zhu
金额：
$ 10.34万
依托单位：
University of Massachusetts Amherst
依托单位国家：
美国
项目类别：
Continuing Grant
财政年份：
2021
资助国家：
美国
起止时间：
2021-09-01 至 2024-08-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2140982&HistoricalAwards=false
关键词：
EAGER CDS&E Applied geometry harmonic

项目摘要

In this era of Big Data, deep learning has become a burgeoning domain with immense potential to advance science, technology, and human life. Despite the tremendous practical success of deep neural networks (DNNs) in various data-intensive machine learning applications, there remain many open problems to be addressed: (1) DNNs tend to suffer from overfitting when the available training data are scarce, which renders them less effective in the small data regime. (2) DNNs have been shown to have the capability of perfectly “memorizing” random training samples, making them less trustworthy when the training data are noisy and corrupted. (3) While symmetry is ubiquitous in machine learning (e.g., in image classification, the class label of an image remains the same if the image is spatially rescaled and translated,) generic DNN architectures typically destroy such symmetry in the representation, which leads to significant redundancy in the model to “memorize” such information from the data. The goal of this project is to address these challenges in deep learning by exploiting the low-dimensional geometry and symmetry within the data and their network representations, aiming at developing new theories and methodologies for deep learning regularization that can lead to tangible advances in machine learning and artificial intelligence, especially in the small/corrupted data regime. In addition, the project also provides research training opportunities for postdocs. The overarching theme of this project is to leverage recent progress in mathematical methods from differential geometry and applied harmonic analysis to improve the stability, reliability, data efficiency, and interpretability of deep learning. This will involve developing both foundational theories and efficient algorithms to achieve the following three objectives: (1) developing manifold-based DNN regularizations with significantly improved generalization performance by focusing on the topology and geometry of both the input data and their representations. This will unlock the potential of deep learning in the small data regime. (2) Establishing and analyzing an innovative framework of imposing geometric constraints in deep learning that has immense potential to limit the memorizing capacity of DNN. The mathematical analysis of the training dynamics of such a model will shed light on the understanding of the fundamental difference between “memorization” and generalization in deep learning. (3) The construction of deformation robust symmetry-preserving DNN architectures for various symmetry transformations on different data domains. By "hardwiring" the symmetry information into the deformation robust representations, the regularized DNN models will have improved performance and interpretability with reduced redundancy and model size. In terms of application, the project will demonstrate and deploy the proposed theories in real-world machine learning tasks, such as object recognition, localization, and segmentation. The techniques developed in this project will be widely applicable across different disciplines, providing fundamental building blocks for the next generation of mathematical tools for the computational modeling of Big Data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在这个大数据的时代，深度学习已成为一个富有科学，技术和人类生活的潜力的水堡领域。尽管深度神经网络（DNN）在各种数据密集型机器学习应用中取得了巨大的实际成功，但仍有许多开放问题要解决：（1）当可用的培训数据稀缺时，DNN往往会遭受过度拟合的苦难，这使得它们在小型数据方面的有效性较低。（2）已显示DNN具有完美“记忆”随机训练样本的能力，在训练数据噪声且损坏时，它们的信任程度不值得。（3）虽然对称性在机器学习中无处不在（例如，在图像分类中，如果图像的类标签在空间恢复和翻译上，则图像的类标签保持不变，）通用DNN架构通常会在表示形式中破坏这种对称性，这会导致模型中“记住”此类信息的显着冗余，从而从数据中获得了此类信息。该项目的目的是通过利用数据及其网络表示中的低维几何形状和对称性来应对深度学习的这些挑战，旨在开发新的理论和方法来进行深度学习调节，从而在机器学习和人工智能方面有形进步，尤其是在小型/损坏的数据制度中。此外，该项目还为博士后提供了研究培训机会。该项目的总体主题是利用差异几何形状和应用谐波分析的数学方法的最新进展，以提高深度学习的稳定性，可靠性，数据效率和解释性。这将涉及开发基本理论和有效算法以实现以下三个目标：（1）通过专注于输入数据及其表示形式的拓扑和几何形状，开发基于歧管的DNN调节具有显着改善的概括性能。这将释放小型数据制度中深度学习的潜力。（2）建立和分析深度学习中施加几何约束的创新框架，具有限制DNN的记忆能力的巨大潜力。对这种模型的训练动力学的数学分析将阐明对“纪念”和深度学习中的概括之间的基本差异的理解。（3）构建变形鲁棒对称性的DNN体系结构，用于不同数据域上的各种对称性转换。通过将对称信息“硬化”到变形鲁棒表示形式中，正则化DNN模型将具有提高的性能和可解释性，并减少冗余和模型大小。就应用程序而言，该项目将在现实世界的机器学习任务中演示和部署所提出的理论，例如对象识别，本地化和细分。该项目中开发的技术将在不同的学科中广泛适用，为下一代数学工具提供基本的构建块，用于大数据的计算建模。该奖项反映了NSF的法定任务，并被认为是通过基金会的知识分子优点和更广泛影响的审查标准来通过评估来获得的支持。