III: Small: Towards the Foundations of Training Deep Neural Networks: New Theory and Algorithms

III：小：迈向训练深度神经网络的基础：新理论和算法

基本信息

批准号：
2008981
负责人：
Quanquan Gu
金额：
$ 50万
依托单位：
University of California-Los Angeles
依托单位国家：
美国
项目类别：
Continuing Grant
财政年份：
2020
资助国家：
美国
起止时间：
2020-10-01 至 2024-09-30
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2008981&HistoricalAwards=false
关键词：
III Small Towards Foundations Training

项目摘要

Deep learning has achieved tremendous successes in the past decade. Despite these empirical successes, the theoretical understanding of deep learning is still largely falling behind. There exists a huge gap between the empirical successes of deep learning and conventional optimization and machine learning theories. This project aims to bridge this gap by establishing the theoretical foundations of deep learning to understand why and how it works, and use this theory to develop new models and algorithms. The expected outcome of this project includes new theories and the state-of-the-art approaches for deep learning. The project will push the frontier of deep learning and train next-generation researchers and practitioners in artificial intelligence. Research demonstrations and lab tours will be given to K-12 school students by showing the wide range of applications of AI and their connection to society, to motivate them to pursue a STEM discipline.This project consists of two synergistic research thrusts: (1) understanding the optimization dynamics of training algorithms such as stochastic gradient descent for deep learning models, and deriving algorithm-dependent generalization error bounds to assess their generalization performance; and (2) developing a new suite of faster training algorithms for deep learning, as well as principled neural architecture search algorithms guided by the generalization error bounds to design better neural network models. To evaluate the developed approaches, both theoretical analyses and extensive experimental evaluations will be performed on real-world benchmarks including but not limited to image classification and natural language processing. The open source software and course materials developed in this project will be made publicly available to the broader community, to help engineers and scientists better understand and apply deep learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习在过去十年中取得了巨大的成功。尽管取得了这些实证上的成功，但对深度学习的理论理解仍然在很大程度上落后。深度学习的实证成功与传统优化和机器学习理论之间存在巨大差距。该项目旨在通过建立深度学习的理论基础来理解其工作原理和方式，并利用该理论开发新的模型和算法，从而弥补这一差距。该项目的预期成果包括深度学习的新理论和最先进的方法。该项目将推动深度学习的前沿发展，并培训下一代人工智能研究人员和从业者。将为 K-12 学校的学生提供研究演示和实验室参观，展示人工智能的广泛应用及其与社会的联系，以激励他们学习 STEM 学科。该项目由两个协同研究主旨组成：(1)了解训练算法的优化动态，例如深度学习模型的随机梯度下降，并导出算法相关的泛化误差界限以评估其泛化性能； (2) 开发一套新的更快的深度学习训练算法，以及以泛化误差界限为指导的原则性神经架构搜索算法，以设计更好的神经网络模型。为了评估所开发的方法，将在现实世界基准上进行理论分析和广泛的实验评估，包括但不限于图像分类和自然语言处理。该项目开发的开源软件和课程材料将向更广泛的社区公开，以帮助工程师和科学家更好地理解和应用深度学习。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准。