Collaborative Research: TRIPODS Institute for Optimization and Learning

合作研究：TRIPODS 优化与学习研究所

• 批准号: 1925930
• 负责人: Francesco Orabona
• 金额: $ 25.32万
• 依托单位: Trustees of Boston University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-01-01 至 2021-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1925930&HistoricalAwards=false
• 关键词: Collaborative; Research; TRIPODS; Institute; Optimization

This Phase I project forms an NSF TRIPODS Institute, based at Lehigh University and in collaboration with Stony Brook and Northwestern Universities, with a focus on new advances in tools for machine learning applications. A critical component for machine learning is mathematical optimization, where one uses historical data to train tools for making future predictions and decisions. Traditionally, optimization techniques for machine learning have focused on simplified models and algorithms. However, recent revolutionary leaps in the successes of machine learning tools---e.g., for image and speech recognition---have in many cases been made possible by a shift toward using more complicated techniques, often involving deep neural networks. Continued advances in the use of such techniques require combined efforts between statisticians, computer scientists, and applied mathematicians to develop more sophisticated models and algorithms along with more comprehensive theoretical guarantees that support their use. In addition to its research goals, the institute trains Ph.D. students and postdoctoral fellows in statistics, computer science, and applied mathematics, and hosts interdisciplinary workshops and Winter/Summer schools. The research efforts in Phase I are on the analysis of nonconvex machine learning models, the design of optimization algorithms for training them, and on the development of nonparametric models and associated algorithms. The focus is on deep neural networks (DNNs), mostly in general, but also with respect to specific architectures of interest. The institute's research efforts emphasize the need to develop connections between state-of-the-art approaches for training DNNs and statistical performance guarantees (e.g., on generalization errors), which are currently not well understood. Optimization algorithms development centers on second-order-derivative-type techniques, including (Hessian-free) Newton, quasi-Newton, Gauss-Newton, and their limited memory variants. Recent advances have been made in the design of such methods; the PIs' work builds upon these efforts with their broad expertise in the design and implementation (including in parallel and distributed computing environments) of such methods. The development of nonparametric models promises to free machine learning approaches from restrictions imposed by large numbers of user-defined parameters (e.g., defining a network structure or learning rate of an optimization algorithm). Such models could lead to great advances in machine learning, and the institute's work in this area also draws on the PIs expertise in derivative-free optimization methods, which are needed for training in nonparametric settings.In this TRIPODS institute, the PIs approach all of these research directions with a unified perspective in the three disciplines of statistics, computer science, and applied mathematics. Indeed, as machine learning draws so heavily from these areas, future progress requires close collaborations between optimization experts, learning theorists, and statisticians---communities of researchers that, as yet, have tended to operate separately with differing terminology and publication venues. With an emphasis on deep learning, this institute aims to foster intercollegiate and interdisciplinary collaborations that overcome these hindrances.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该第一阶段项目成立了 NSF TRIPODS 研究所，总部设在理海大学，与石溪大学和西北大学合作，重点关注机器学习应用工具的新进展。 机器学习的一个关键组成部分是数学优化，即使用历史数据来训练用于未来预测和决策的工具。 传统上，机器学习的优化技术主要集中在简化的模型和算法上。 然而，机器学习工具（例如图像和语音识别）最近取得的革命性飞跃在许多情况下是通过转向使用更复杂的技术（通常涉及深度神经网络）而实现的。 此类技术的使用不断取得进展，需要统计学家、计算机科学家和应用数学家共同努力，开发更复杂的模型和算法，以及支持其使用的更全面的理论保证。除了其研究目标外，该研究所还培养博士。统计、计算机科学和应用数学领域的学生和博士后研究员，并举办跨学科研讨会和冬季/夏季学校。 第一阶段的研究工作是非凸机器学习模型的分析、训练它们的优化算法的设计以及非参数模型和相关算法的开发。 重点是深度神经网络 (DNN)，主要是一般性的，但也涉及感兴趣的特定架构。 该研究所的研究工作强调需要在最先进的 DNN 训练方法和统计性能保证（例如，泛化误差）之间建立联系，而目前人们对此还没有很好地理解。优化算法开发以二阶导数类型技术为中心，包括（无 Hessian）牛顿、拟牛顿、高斯牛顿及其有限的内存变体。 此类方法的设计最近取得了进展； PI 的工作建立在这些努力的基础上，他们在这些方法的设计和实现（包括并行和分布式计算环境）方面拥有广泛的专业知识。 非参数模型的发展有望使机器学习方法摆脱大量用户定义参数（例如，定义网络结构或优化算法的学习率）所施加的限制。 此类模型可能会带来机器学习的巨大进步，并且该研究所在该领域的工作还利用了 PI 在无导数优化方法方面的专业知识，这些知识是在非参数环境中进行培训所必需的。在该 TRIPODS 研究所中，PI 处理了所有这些研究方向以统一的视角涵盖了统计学、计算机科学和应用数学三个学科。 事实上，由于机器学习在这些领域的应用如此之多，未来的进步需要优化专家、学习理论家和统计学家之间的密切合作——迄今为止，这些研究人员群体倾向于使用不同的术语和出版场所分开运作。 该研究所以深度学习为重点，旨在促进校际和跨学科合作，克服这些障碍。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Online Learning Algorithms

在线学习算法

• DOI: 10.1146/annurev-statistics-040620-035329
• 发表时间: 2021-01
• 期刊: Annual review of statistics and its application
• 作者: Cesa;Orabona, Francesco
• 通讯作者: Orabona, Francesco

On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes

自适应步长随机梯度下降的收敛性

• 发表时间: 2019-04
• 期刊: Proceedings of Machine Learning Research
• 作者: Li, Xiaoyu;Orabona, Francesco
• 通讯作者: Orabona, Francesco

Parameter-Free Online Convex Optimization with Sub-Exponential Noise

具有次指数噪声的无参数在线凸优化

• 发表时间: 2019-06
• 期刊: Proceedings of the Thirty-Second Conference on Learning Theory
• 作者: Jun, Kwang;Orabona, Francesco
• 通讯作者: Orabona, Francesco

On the Last Iterate Convergence of Momentum Methods

关于动量方法的最后迭代收敛

• 发表时间: 2021-02-13
• 作者: Xiaoyun Li;Mingrui Liu;Francesco Orabona
• 通讯作者: Francesco Orabona

On the Last Iterate Convergence of Momentum Methods

关于动量方法的最后迭代收敛

• 发表时间: 2022-01
• 期刊: Proceedings of Machine Learning Research
• 作者: Li, Xiaoyu;Liu, Mingrui;Orabona, Francesco
• 通讯作者: Orabona, Francesco