Collaborative Research: TRIPODS Institute for Optimization and Learning

合作研究：TRIPODS 优化与学习研究所

基本信息

批准号：
1925930
负责人：
Francesco Orabona
金额：
$ 25.32万
依托单位：
Trustees of Boston University
依托单位国家：
美国
项目类别：
Continuing Grant
财政年份：
2019
资助国家：
美国
起止时间：
2019-01-01 至 2021-12-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1925930&HistoricalAwards=false
关键词：
Collaborative Research TRIPODS Institute Optimization

项目摘要

This Phase I project forms an NSF TRIPODS Institute, based at Lehigh University and in collaboration with Stony Brook and Northwestern Universities, with a focus on new advances in tools for machine learning applications. A critical component for machine learning is mathematical optimization, where one uses historical data to train tools for making future predictions and decisions. Traditionally, optimization techniques for machine learning have focused on simplified models and algorithms. However, recent revolutionary leaps in the successes of machine learning tools---e.g., for image and speech recognition---have in many cases been made possible by a shift toward using more complicated techniques, often involving deep neural networks. Continued advances in the use of such techniques require combined efforts between statisticians, computer scientists, and applied mathematicians to develop more sophisticated models and algorithms along with more comprehensive theoretical guarantees that support their use. In addition to its research goals, the institute trains Ph.D. students and postdoctoral fellows in statistics, computer science, and applied mathematics, and hosts interdisciplinary workshops and Winter/Summer schools. The research efforts in Phase I are on the analysis of nonconvex machine learning models, the design of optimization algorithms for training them, and on the development of nonparametric models and associated algorithms. The focus is on deep neural networks (DNNs), mostly in general, but also with respect to specific architectures of interest. The institute's research efforts emphasize the need to develop connections between state-of-the-art approaches for training DNNs and statistical performance guarantees (e.g., on generalization errors), which are currently not well understood. Optimization algorithms development centers on second-order-derivative-type techniques, including (Hessian-free) Newton, quasi-Newton, Gauss-Newton, and their limited memory variants. Recent advances have been made in the design of such methods; the PIs' work builds upon these efforts with their broad expertise in the design and implementation (including in parallel and distributed computing environments) of such methods. The development of nonparametric models promises to free machine learning approaches from restrictions imposed by large numbers of user-defined parameters (e.g., defining a network structure or learning rate of an optimization algorithm). Such models could lead to great advances in machine learning, and the institute's work in this area also draws on the PIs expertise in derivative-free optimization methods, which are needed for training in nonparametric settings.In this TRIPODS institute, the PIs approach all of these research directions with a unified perspective in the three disciplines of statistics, computer science, and applied mathematics. Indeed, as machine learning draws so heavily from these areas, future progress requires close collaborations between optimization experts, learning theorists, and statisticians---communities of researchers that, as yet, have tended to operate separately with differing terminology and publication venues. With an emphasis on deep learning, this institute aims to foster intercollegiate and interdisciplinary collaborations that overcome these hindrances.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该阶段I项目构成了位于Lehigh University的NSF三脚架学院，并与Stony Brook和Northwestern大学合作，重点是用于机器学习应用工具的新进展。机器学习的关键组成部分是数学优化，其中人们使用历史数据来训练工具来做出未来的预测和决策。传统上，机器学习的优化技术集中在简化的模型和算法上。但是，最近的革命性在机器学习工具的成功中跳跃 - 例如，用于图像和语音识别 - 在许多情况下，通过转向使用更复杂的技术，通常涉及深层神经网络。统计学家，计算机科学家和应用数学家之间的综合努力在使用此类技术方面的持续进展需要开发更复杂的模型和算法，并提供更全面的理论保证，以支持其使用。除了研究目标外，该研究所还培训博士学位。统计，计算机科学和应用数学的学生和博士后研究员，并举办跨学科研讨会和冬季/暑期学校。第一阶段的研究工作正在分析非凸机机器学习模型，用于培训它们的优化算法的设计以及非参数模型和相关算法的开发。重点是深度神经网络（DNN），大多是一般而言，也是关于特定感兴趣的特定体系结构的重点。该研究所的研究工作强调了需要在培训DNN的最先进方法和统计绩效保证（例如，在概括错误上）之间建立联系，这些方法目前尚不清楚。优化算法的开发集中在二阶衍生型技术上，包括（无黑西西士）牛顿，Quasi-Newton，Gauss-Newton及其有限的内存变体。在这种方法的设计中，最近取得了进步。 PIS的工作以这些方法的广泛专业知识（包括在平行和分布式计算环境中）的广泛专业知识为基础。非参数模型的开发有望从大量用户定义的参数（例如，定义网络结构或优化算法的学习率）中施加的限制中释放机器学习方法。这样的模型可能会导致机器学习方面的巨大进步，该研究所在该领域的工作还借鉴了无衍生化的优化方法的PIS专业知识，这是在非参数环境中进行培训所需的。在该三脚架研究所中，PIS在所有这些研究方向上都以统一的统计数据统一的统计数据来处理所有这些研究指导。的确，随着机器学习从这些领域大大吸引，未来的进步需要优化专家，学习理论家和统计学家之间的密切合作 - 研究人员的社区至今仍倾向于在不同的术语和出版物场所中分开运作。该研究所的重点是深入学习，旨在培养克服这些障碍的跨学科和跨学科合作。该奖项反映了NSF的法定使命，并被认为是值得通过基金会的知识分子优点和更广泛的影响审查标准通过评估来获得支持的。