Collaborative Research: RI: Medium: Principles for Optimization, Generalization, and Transferability via Deep Neural Collapse

合作研究：RI：中：通过深度神经崩溃实现优化、泛化和可迁移性的原理

• 批准号: 2312841
• 负责人: Jeremias Sulam
• 金额: $ 40万
• 依托单位: Johns Hopkins University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-07-01 至 2026-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2312841&HistoricalAwards=false
• 关键词: Collaborative; Research; RI; Medium; Principles

Deep learning has demonstrated unprecedented performance across various domains in engineering and science. However, the theoretical understanding of their success has remained elusive. Very recently, researchers discovered and characterized an elegant mathematical structure within the learned features and classifiers called Neural Collapse. This phenomenon persists across a variety of different network architectures, datasets, and data domains. This project will leverage the symmetry of Neural Collapse to develop a rigorous mathematical theory to explain when and why it happens and how it can be used to quantify generalization performance and provide guidelines to understand and improve transferability. By advancing the mathematical foundations of deep learning, this project is expected to influence not only the machine learning community, but also related areas such as optimization, signal and image processing, and natural language processing. The project also involves an integrated outreach and education plan, including promoting accessibility and awareness of computing and STEM concepts for K-12 students.This project will expand our understanding of the principles behind non-convex optimization of training deep learning models, and provide new mathematical insights on their generalization and transferability properties, leading to practical implications. In particular, the project is focused on the following three overarching research thrusts: (i) provide a unified framework to analyze convergence guarantees for training deep and overparametrized models through general loss functions to states of neural collapse, first for simplified cases and then for more general deep models that exhibit progressive neural collapse, with multi-labels and data imbalance; (ii) harness the structure of neural collapse to provide tighter generalization bounds for deep models, by characterizing the structure of the resulting classifiers and their mild dependence on the training data, as well as by making natural distributional assumptions; (iii) leverage the generalization of progressive neural collapse to new environments to understand transferability of deep models to new domains and tasks, and develop principled approaches for improving transferability and efficient fine-tuning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习在工程和科学的各个领域都表现出了前所未有的性能。然而，对其成功的理论理解仍然难以捉摸。最近，研究人员在学习的特征和分类器中发现并描述了一种优雅的数学结构，称为“神经崩溃”。这种现象在各种不同的网络架构、数据集和数据域中持续存在。该项目将利用神经崩溃的对称性来开发严格的数学理论，以解释其发生的时间和原因，以及如何使用它来量化泛化性能，并为理解和提高可迁移性提供指导。通过推进深度学习的数学基础，该项目预计不仅会影响机器学习社区，还会影响优化、信号和图像处理以及自然语言处理等相关领域。该项目还涉及综合外展和教育计划，包括促进 K-12 学生对计算和 STEM 概念的可及性和认识。该项目将扩大我们对训练深度学习模型的非凸优化背后原理的理解，并提供新的方法。对它们的泛化性和可转移性特性的数学见解，产生实际影响。特别是，该项目侧重于以下三个总体研究主旨：（i）提供一个统一的框架来分析通过神经崩溃状态的一般损失函数训练深度和过参数化模型的收敛保证，首先针对简化情况，然后针对更多情况表现出渐进性神经崩溃、多标签和数据不平衡的一般深度模型； (ii) 通过表征所得分类器的结构及其对训练数据的轻微依赖性以及做出自然分布假设，利用神经崩溃的结构为深度模型提供更严格的泛化界限； (iii) 利用渐进式神经崩溃对新环境的泛化，了解深度模型向新领域和任务的可迁移性，并开发提高可迁移性和高效微调的原则性方法。该奖项反映了 NSF 的法定使命，并被认为是值得的通过使用基金会的智力优势和更广泛的影响审查标准进行评估来提供支持。