III: Small: Towards the Foundations of Training Deep Neural Networks: New Theory and Algorithms

III：小：迈向训练深度神经网络的基础：新理论和算法

• 批准号: 2008981
• 负责人: Quanquan Gu
• 金额: $ 50万
• 依托单位: University of California-Los Angeles
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2008981&HistoricalAwards=false
• 关键词: III; Small; Towards; Foundations; Training

Deep learning has achieved tremendous successes in the past decade. Despite these empirical successes, the theoretical understanding of deep learning is still largely falling behind. There exists a huge gap between the empirical successes of deep learning and conventional optimization and machine learning theories. This project aims to bridge this gap by establishing the theoretical foundations of deep learning to understand why and how it works, and use this theory to develop new models and algorithms. The expected outcome of this project includes new theories and the state-of-the-art approaches for deep learning. The project will push the frontier of deep learning and train next-generation researchers and practitioners in artificial intelligence. Research demonstrations and lab tours will be given to K-12 school students by showing the wide range of applications of AI and their connection to society, to motivate them to pursue a STEM discipline.This project consists of two synergistic research thrusts: (1) understanding the optimization dynamics of training algorithms such as stochastic gradient descent for deep learning models, and deriving algorithm-dependent generalization error bounds to assess their generalization performance; and (2) developing a new suite of faster training algorithms for deep learning, as well as principled neural architecture search algorithms guided by the generalization error bounds to design better neural network models. To evaluate the developed approaches, both theoretical analyses and extensive experimental evaluations will be performed on real-world benchmarks including but not limited to image classification and natural language processing. The open source software and course materials developed in this project will be made publicly available to the broader community, to help engineers and scientists better understand and apply deep learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在过去的十年中，深度学习取得了巨大的成功。尽管取得了这些经验的成功，但对深度学习的理论理解仍在很大程度上落后。深度学习的经验成功与传统优化的经验成功与机器学习理论之间存在巨大差距。该项目旨在通过建立深度学习的理论基础来了解其原因和方式，并使用该理论来开发新的模型和算法来弥合这一差距。该项目的预期结果包括新理论和深度学习的最新方法。该项目将推动人工智能领域的深度学习和培训下一代研究人员和从业人员的前沿。 Research demonstrations and lab tours will be given to K-12 school students by showing the wide range of applications of AI and their connection to society, to motivate them to pursue a STEM discipline.This project consists of two synergistic research thrusts: (1) understanding the optimization dynamics of training algorithms such as stochastic gradient descent for deep learning models, and deriving algorithm-dependent generalization error bounds to assess their generalization performance; （2）为深度学习开发了一套新的更快的培训算法，以及以概括误差界限为指导的原则性神经体系结构搜索算法，以设计更好的神经网络模型。为了评估开发的方法，将对包括但不限于图像分类和自然语言处理在内的实际基准进行理论分析和广泛的实验评估。该项目中开发的开源软件和课程材料将公开向更广泛的社区公开使用，以帮助工程师和科学家更好地理解和应用深入学习。该奖项反映了NSF的法定任务，并被认为是通过基金会的知识分子的智力优点和更广泛影响的审查标准来通过评估来获得支持的。

项目成果

Provable Robustness of Adversarial Training for Learning Halfspaces with Noise

• 发表时间: 2021-04
• 期刊: 2023 IEEE International Conference on Quantum Computing and Engineering (QCE)
• 作者: Difan Zou;Spencer Frei;Quanquan Gu

The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shift

• DOI: 10.48550/arxiv.2208.01857
• 发表时间: 2022-08
• 期刊: ArXiv
• 作者: Jingfeng Wu;Difan Zou;V. Braverman;Quanquan Gu;S. Kakade

How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?

• 发表时间: 2019-11
• 期刊: ArXiv
• 作者: Zixiang Chen;Yuan Cao;Difan Zou;Quanquan Gu

Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descent

• 发表时间: 2021-06
• 期刊: National Science Review
• 影响因子: 20.6
• 作者: Spencer Frei;Quanquan Gu

Towards Understanding the Mixture-of-Experts Layer in Deep Learning

理解深度学习中的专家混合层

• 发表时间: 2022
• 期刊: Advances in neural information processing systems
• 作者: Chen, Zixiang;Deng, Yihe;Wu, Yue;Gu, Quanquan;Li, Yuanzhi
• 通讯作者: Li, Yuanzhi