CIF: Small: Self-Adaptive Optimization Algorithms with Fast Convergence via Geometry-Adapted Hyper-Parameter Scheduling

CIF：小型：通过几何自适应超参数调度实现快速收敛的自适应优化算法

• 批准号: 2106216
• 负责人: Yi Zhou
• 金额: $ 41.12万
• 依托单位: University of Utah
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-07-01 至 2024-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2106216&HistoricalAwards=false
• 关键词: CIF; Small; Self; Adaptive; Optimization

Machine-learning and artificial-intelligence techniques have been widely applied in modern society to enhance quality of lifr. In these applications, machine-learning models such as neural networks are trained on a large dataset using various optimization algorithms, which iteratively adjust the model parameters and converge to a good model. In particular, the convergence of these optimization algorithms often relies on choosing a good set of hyper-parameters. For example, one important algorithm hyper-parameter is the step size, which controls the scale of the update applied to the model parameters in every iteration, and it must be carefully chosen to avoid slow convergence and possible divergence. In practice, these algorithm hyper-parameters either are guided by optimization theory or are set through manual fine-tuning. While theory-guided algorithm hyper-parameters often rely on certain unknown geometrical information of the model and are often too conservative, resulting in result in slow convergence, manually fine-tuned algorithm hyper-parameters critically depend on the specific application and algorithm, and often introduce much computation overhead. This project aims to address these issues by developing a principled, computation-light and effective hyper-parameter scheduling scheme for different types of optimization algorithms to achieve fast and stable convergence. The developed adapted hyper-parameter scheduling scheme is intended to facilitate machine-learning practitioners tuning the algorithm hyper-parameters and dynamically adapt them to the ongoing optimization process. This has further positive impact on implementation of large-scale machine learning applications such as autonomous driving, training adversary-robust models, robust decision making in finance and control, etc. In this project, the researchers are developing a principled and efficient algorithm hyper-parameter scheduling framework that jointly adapts different algorithm hyper-parameters to the local geometry of the nonconvex objective function for a variety of popular optimization algorithms, and corroborate them with strong theoretical convergence guarantees in nonconvex machine learning. Specifically, the researchers are developing such geometry-adapted hyper-parameter scheduling scheme for deterministic optimization algorithms, including first-order gradient-based algorithms, accelerated gradient algorithms and second-order Newton-type algorithms. The researchers are developing new analysis tools that advance the understanding of the relation between hyper-parameters and the dynamic optimization process. Iteration and computation complexities of these algorithms is being established in nonconvex optimization. Based on this development, the researchers are extending the adapted hyper-parameter scheduling scheme to stochastic optimization algorithms, which use mini-batch random sampling and therefore necessitate a joint scheduling of step-size and batch size. Analysis of sample complexity and high probability convergence guarantee is being established for these algorithms. Furthermore, these developments are guiding the design of adapted hyper-parameter scheduling scheme for gradient-based minimax optimization algorithms.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

机器学习和人工智能技术已广泛应用于现代社会，以提高生活质量。在这些应用中，神经网络等机器学习模型使用各种优化算法在大型数据集上进行训练，迭代地调整模型参数并收敛到良好的模型。特别是，这些优化算法的收敛通常依赖于选择一组好的超参数。例如，一个重要的算法超参数是步长，它控制每次迭代中应用于模型参数的更新规模，必须仔细选择它以避免收敛缓慢和可能的发散。 在实践中，这些算法超参数要么以优化理论为指导，要么通过手动微调来设置。理论指导的算法超参数往往依赖于模型某些未知的几何信息，往往过于保守，导致收敛速度慢，而手动微调的算法超参数则严重依赖于具体的应用和算法，并且往往会导致收敛速度变慢。引入大量计算开销。本项目旨在解决这些问题，为不同类型的优化算法开发一种原则性强、计算量小、有效的超参数调度方案，以实现快速稳定的收敛。开发的自适应超参数调度方案旨在促进机器学习从业者调整算法超参数并动态地使其适应正在进行的优化过程。这对大规模机器学习应用的实施产生了进一步的积极影响，例如自动驾驶、训练对手鲁棒模型、金融和控制领域的鲁棒决策等。在这个项目中，研究人员正在开发一种有原则且高效的超算法。参数调度框架，联合使不同算法超参数适应各种流行优化算法的非凸目标函数的局部几何，并在非凸机器学习中用强大的理论收敛保证来证实它们。具体来说，研究人员正在开发这种用于确定性优化算法的几何自适应超参数调度方案，包括基于一阶梯度的算法、加速梯度算法和二阶牛顿型算法。研究人员正在开发新的分析工具，以增进对超参数与动态优化过程之间关系的理解。这些算法的迭代和计算复杂性是在非凸优化中建立的。基于这一进展，研究人员正在将适应的超参数调度方案扩展到随机优化算法，该算法使用小批量随机采样，因此需要步长和批量大小的联合调度。正在为这些算法建立样本复杂性分析和高概率收敛保证。此外，这些进展正在指导基于梯度的极小极大优化算法的自适应超参数调度方案的设计。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优点和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Sample Efficient Stochastic Policy Extragradient Algorithm for Zero-Sum Markov Game

零和马尔可夫博弈的高效随机策略超梯度算法示例

• 发表时间: 2024-09-13
• 作者: Ziyi Chen;Shaocong Ma;Yi Zhou
• 通讯作者: Yi Zhou

Proximal Gradient Descent-Ascent: Variable Convergence under KŁ Geometry

近端梯度下降-上升：KÅ 几何下的可变收敛

• 发表时间: 2021-02-09
• 期刊: ArXiv
• 作者: Ziyi Chen;Yi Zhou;Tengyu Xu;Yingbin Liang
• 通讯作者: Yingbin Liang

Greedy-GQ with Variance Reduction: Finite-time Analysis and Improved Complexity

具有方差减少的 Greedy-GQ：有限时间分析和改进的复杂性

• 发表时间: 2021-03-30
• 期刊: ArXiv
• 作者: Shaocong Ma;Ziyi Chen;Yi Zhou;Shaofeng Zou
• 通讯作者: Shaofeng Zou

A Cubic Regularization Approach for Finding Local Minimax Points in Nonconvex Minimax Optimization

非凸极小极大优化中寻找局部极小极大点的三次正则化方法

• 发表时间: 2021-10-14
• 期刊: Trans. Mach. Learn. Res.
• 作者: Ziyi Chen;Zhengyang Hu;Qunwei Li;Zhe Wang;Yi Zhou
• 通讯作者: Yi Zhou

An Accelerated Proximal Algorithm for Regularized Nonconvex and Nonsmooth Bi-level Optimization

正则化非凸非光滑双层优化的加速近似算法

• 发表时间: 2023-01
• 期刊: Machine learning
• 影响因子: 7.5
• 作者: Chen, Ziyi;Kailkhura, Bhavya;Zhou, Yi
• 通讯作者: Zhou, Yi