CIF: Medium: Collaborative Research: Theory of Optimization Geometry and Algorithms for Neural Networks

CIF：媒介：协作研究：神经网络优化几何理论和算法

基本信息

批准号：
1900145
负责人：
Yingbin Liang
金额：
$ 40万
依托单位：
Ohio State University
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2019
资助国家：
美国
起止时间：
2019-10-01 至 2024-09-30
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1900145&HistoricalAwards=false
关键词：
CIF Medium Collaborative Research Theory

项目摘要

Deep learning has attracted a significant amount of interest in recent years due to its widespread applicability in computer vision, artificial intelligence and natural language processing, alongside recent strides in autonomous driving. The theoretical underpinnings behind such success, however, remain elusive to a large extent, hindering its further adoption in other applications. This project aims to advance the theoretical foundations of training neural networks in terms of optimization landscape and algorithmic efficacy, which in turn should have a measurable impact on the practice of deep learning by providing guiding principles for network design, algorithm selection, hyperparameter tuning, and adversarial training. This project adopts an interdisciplinary approach fusing ideas from machine learning, optimization, statistical signal processing, high-dimensional statistics, nonparametric statistics, and information theory. This project will likewise develop courses and tutorials on theoretical foundations of large-scale machine learning and provide extensive training opportunities for students at all levels.This project aims to develop a comprehensive theory to characterize the optimization landscape and geometry of loss functions and algorithmic regularizations of major neural network training problems, and explore how the network architecture---including depth, width, and activation functions---affect these properties, thus providing guidelines for the design of algorithms to train these networks more efficiently with theoretical performance guarantees. The project will explore the geometric properties and their impact on the optimization performance in training multi-layer neural networks, auto-encoders, generative adversarial networks, and adversarial training involving non-convex and saddle-point problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

近年来，深度学习因其在计算机视觉、人工智能和自然语言处理方面的广泛适用性以及自动驾驶方面的最新进展而引起了人们的极大兴趣。然而，这种成功背后的理论基础在很大程度上仍然难以捉摸，阻碍了其在其他应用中的进一步采用。该项目旨在在优化景观和算法功效方面推进神经网络训练的理论基础，从而通过为网络设计、算法选择、超参数调整和优化提供指导原则，从而对深度学习实践产生可衡量的影响。对抗性训练。该项目采用跨学科方法，融合了机器学习、优化、统计信号处理、高维统计、非参数统计和信息论的思想。该项目还将开发有关大规模机器学习理论基础的课程和教程，并为各个级别的学生提供广泛的培训机会。该项目旨在开发一种全面的理论来描述损失函数的优化景观和几何以及算法正则化主要的神经网络训练问题，并探索网络架构（包括深度、宽度和激活函数）如何影响这些特性，从而为算法设计提供指导，以在理论性能保证的情况下更有效地训练这些网络。该项目将探索几何特性及其对训练多层神经网络、自动编码器、生成对抗网络以及涉及非凸和鞍点问题的对抗训练中优化性能的影响。该奖项反映了 NSF 的法定使命和通过使用基金会的智力价值和更广泛的影响审查标准进行评估，该项目被认为值得支持。

项目成果

期刊论文数量（26）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

Convergence of meta-learning with task-specific adaptation over partial parameters

元学习与部分参数的特定任务适应的融合

DOI：
发表时间：
2020-01
期刊：
Proc. Advances in Neural Information Processing Systems (NeurIPS
影响因子：
0
作者：
Ji, K;Lee, J D;Liang, Y;Poor, H V.
通讯作者：
Poor, H V.

Provably efficient UCB-type algorithms for learning predictive state representation

用于学习预测状态表示的可证明有效的 UCB 型算法

DOI：
发表时间：
2024-05
期刊：
Proc. International Conference on Learning Representations (ICLR
影响因子：
0
作者：
Huang, Ruiquan;Liang, Yingbin;Yang, Jing.
通讯作者：
Yang, Jing.

Doubly robust off-policy actor-critic: Convergence and optimality

双稳健的离策略演员批评家：收敛性和最优性

DOI：
发表时间：
2021-01
期刊：
Proc. International Conference on Machine Learning (ICML
影响因子：
0
作者：
Xu, T;Yang, Z;Wang, Z;Liang, Y.
通讯作者：
Liang, Y.

Proximal gradient descent-ascent: Variable convergence under KL geometry

近端梯度下降-上升：KL 几何下的可变收敛

DOI：
发表时间：
2021-01
期刊：
Proc. International Conference on Learning Representations (ICLR
影响因子：
0
作者：
Chen, Ziyi;Zhou, Yi;Xu, Tengyu;Liang, Yingbin.
通讯作者：
Liang, Yingbin.

When will generative adversarial imitation learning algorithms attain global convergence

生成对抗性模仿学习算法何时能实现全局收敛

DOI：
发表时间：
2021-01
期刊：
Proc. International Conference on Artificial Intelligence and Statistics (AISTATS
影响因子：
0
作者：
Guan, Ziwei;Xu, Tengyu;Liang, Yingbin.
通讯作者：
Liang, Yingbin.