CAREER: Parameter-free Optimization Algorithms for Machine Learning

职业：机器学习的无参数优化算法

基本信息

批准号：
2046096
负责人：
Francesco Orabona
金额：
$ 58.39万
依托单位：
Trustees of Boston University
依托单位国家：
美国
项目类别：
Continuing Grant
财政年份：
2021
资助国家：
美国
起止时间：
2021-02-15 至 2026-01-31
项目状态：
未结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2046096&HistoricalAwards=false
关键词：
CAREER Parameter free Optimization Algorithms

项目摘要

Machine Learning (ML) has been described as the fuel of the next industrial revolution. Yet, current state-of-the-art ML algorithms still heavily rely on having a human in the loop in order to work properly. Indeed, the training process of a ML algorithm requires significant human intervention through twisting and tuning of the many knobs of the algorithm. Moreover, most of the time this tweaking process is carried out without any theoretical guidelines. The most common options for the ML practitioners are to follow their intuitions or to exhaustively evaluate all the possibilities, making the overall time needed to train these algorithms difficult to predict. In this project, the investigator aims at designing truly parameter-free training algorithms for ML, to remove the burden of human parameter-tuning and make ML algorithms more scalable. The investigator also proposes an Education Plan that is designed to fill the gap between theoretical and applied ML in the minds of most students. It will involve a collaboration with the PROgram in Mathematics for Young Scientists (PROMYS), to introduce the study of very basic concepts of the theory of machine learning to both high school students and teachers, and with the women’s Smith College.The project stems from new methods the investigator has recently introduced to design optimal parameter-free optimization algorithms, e.g., Stochastic Gradient Descent without learning rates, for the particular case of convex functions. In this project, the investigator proposes to go beyond convex functions. In particular, the investigator plans to pursue the following inter-related aims:1. Computational complexity of learning without prior knowledge. The objective is to fully characterize the computational complexity of learning, without assuming knowledge of unknown quantities, in worst and "easy" cases. In other words, if a learning algorithm needs parameters to achieve optimal performance, the goal is to fully characterize the overall computational complexity, including the search for the optimal parameters.2. Non-convex stochastic optimization without parameters. The project's goal is to generalize recent results on parameter-free optimization to deal with the non-convex problems in modern ML algorithms. The investigator plans to use a novel optimization framework that mixes elements of Online Mirror Descent with Dual Averaging, the two frameworks used to design optimization algorithms.3. Reducing parameters in optimization for Deep Learning. Deep Learning models are the most parameter-heavy in ML. The project considers three applications of the project's research to widely used deep-learning algorithms: adaptive scaling heuristics, saddle-point optimization, and Bayesian Optimization with Gaussian Processes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

机器学习（ML）被描述为下一次工业革命的燃料。然而，当前的最新ML算法仍然很大程度上依赖于循环中的人来正常工作。实际上，ML算法的训练过程需要通过扭曲和调整算法的许多旋钮来进行大量的人类干预。此外，在大多数情况下，此调整过程是在没有任何理论指南的情况下进行的。 ML从业者最常见的选择是遵循其直觉或详尽评估所有可能性，使训练这些算法所需的总时间难以预测。在该项目中，研究人员旨在设计ML的真正无参数培训算法，以删除人类参数调整的伯宁，并使ML算法更可扩展。研究人员还提出了一项旨在填补大多数学生心目中理论和应用ML之间差距的教育计划。它将涉及与年轻科学家的数学课程合作（PROMYS），以对机器学习理论的非常基本的概念介绍给高中生和老师，以及女子史密斯学院的基本概念。研究人员最近介绍了新方法的项目步骤，用于设计最佳的无参数优化算法，例如，不需要设计，例如，不需要学习效率，以实现效率，以实现效率，以实现效率，以实现效率。在这个项目中，调查员的建议超出了凸功能。特别是，调查人员计划追求以下相关目的：1。没有先验知识的学习的计算复杂性。目的是充分表征学习的计算复杂性，而不假设最坏和“简单”情况下的未知数知识。换句话说，如果学习算法需要参数才能实现最佳性能，则目标是完全表征整体计算复杂性，包括搜索最佳参数。2。无参数的非凸随机优化。该项目的目标是概括有关无参数优化的最新结果，以处理现代ML算法中的非凸问题。研究人员计划使用一种新颖的优化框架，该框架将在线镜像的元素与双重平均元素混合在一起，即用于设计优化算法的两个框架3。减少对深度学习优化的参数。深度学习模型是ML中最重的参数。该项目认为该项目研究的三种应用是广泛使用深度学习算法：适应性缩放式启发式方法，鞍点优化和对高斯流程的贝叶斯优化。该奖项反映了NSF的法定任务，并通过该基金会的知识绩效和广泛的影响来评估NSF的法定任务，并被认为是值得的支持。

项目成果

期刊论文数量（12）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

On the Last Iterate Convergence of Momentum Methods

DOI：
发表时间：
2021-02
期刊：
影响因子：
0
作者：
Xiaoyun Li;Mingrui Liu;Francesco Orabona
通讯作者：
Xiaoyun Li;Mingrui Liu;Francesco Orabona

Better Parameter-free Stochastic Optimization with ODE Updates for Coin-Betting

通过 ODE 更新实现更好的无参数随机优化以进行硬币投注

DOI：
发表时间：
2022
期刊：
Proceedings of the AAAI Conference on Artificial Intelligence
影响因子：
0
作者：
Chen, Keyi;Langford, John;Orabona, Francesco
通讯作者：
Orabona, Francesco

Robustness to Unbounded Smoothness of Generalized SignSGD

DOI：
10.48550/arxiv.2208.11195
发表时间：
2022-08
期刊：
ArXiv
影响因子：
0
作者：
M. Crawshaw;Mingrui Liu;Francesco Orabona;W. Zhang;Zhenxun Zhuang
通讯作者：
M. Crawshaw;Mingrui Liu;Francesco Orabona;W. Zhang;Zhenxun Zhuang

Minimax Optimal Quantile and Semi-Adversarial Regret via Root-Logarithmic Regularizers

通过根对数正则化器的最小最大最优分位数和半对抗性遗憾

DOI：
发表时间：
2021
期刊：
Advances in neural information processing systems
影响因子：
0
作者：
Negrea, Jeffrey;Bilodeau, Blair;Campolongo, Nicolò;Orabona, Francesco;Roy, Dan
通讯作者：
Roy, Dan

Understanding AdamW through Proximal Methods and Scale-Freeness