Collaborative Research: AF: Small: Adaptive Optimization of Stochastic and Noisy Function

合作研究：AF：小：随机和噪声函数的自适应优化

基本信息

批准号：
2008484
负责人：
Frank Curtis
金额：
$ 8.5万
依托单位：
Lehigh University
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2020
资助国家：
美国
起止时间：
2020-10-01 至 2023-05-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2008484&HistoricalAwards=false
关键词：
Collaborative Research AF Small Adaptive

项目摘要

The science of artificial intelligence, and the technology of machine learning (ML) in particular, has had a huge impact on modern society. This impact is only expected to grow in the future. At the heart of ML is the process of training the parameters of an intelligent (computer) system, which requires applied-mathematics techniques in the area known as mathematical optimization. The many recent successes of ML, such as in computer vision and natural-language processing, have been made possible with the use of a certain mathematical-optimization algorithm. This algorithm allows the intelligent system to learn through the iterative random selection of data points from within a large-scale dataset. This random sampling is absolutely essential, since otherwise the learning process of any intelligent system would be slowed as the amount of available data increases. However, despite these recent successes, optimization techniques such as this one have fundamental shortcomings that impede them from being effective for next-generation ML tasks. For example, each application of the algorithm requires a careful data-dependent tuning process, which may cause the training of an intelligent system for a single task to require weeks or months of computation on a supercomputer. One avenue for avoiding such computational expense is through the design of optimization techniques that "adaptively" tune themselves. The goals of this project are to design and provide theoretical guarantees for such adaptive algorithms.There have been various previously proposed enhancements and extensions to the aforementioned algorithm, known as the stochastic gradient (SG) algorithm. However, many of these algorithms also possess the shortcoming of being nonadaptive, meaning that their successful application in practice requires expensive "hyperparameter" tuning efforts. The adaptive algorithms considered in this project for the "stochastic optimization" setting of ML are based on the various successful methodologies in the "deterministic optimization" literature. These include so-called "line search" and "trust region" methodologies. However, since neither of these methodologies result in optimal worst-case complexity guarantees, the focus of the project is on the design of adaptive optimal-complexity methods, such as so-called "cubic-regularization" algorithms. The design of adaptive cubic-regularization algorithms for the stochastic setting will be achieved by building on a theoretical framework that views adaptive minimization as a "renewal-reward" stochastic process. This work will combine analytical techniques from the mathematical-optimization and stochastic-process literatures, and will provide a solid theoretical and practical foundation for researchers working in applied mathematics, computer science, statistics, and various engineering fields.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

人工智能科学尤其是机器学习技术（ML），对现代社会产生了巨大影响。这种影响只有未来才会增长。 ML的核心是训练智能（计算机）系统参数的过程，该系统需要在称为数学优化的区域中应用的数学技术。通过使用某种数学优化算法，ML最近取得了许多成功，例如计算机视觉和自然语言处理。该算法使智能系统可以通过大规模数据集中的数据点进行迭代随机选择。这种随机抽样绝对是必不可少的，因为否则，随着可用数据的增加，任何智能系统的学习过程都会减慢。但是，尽管最近取得了这些成功，但诸如此类的优化技术具有根本的缺点，这阻碍了它们对下一代ML任务有效。例如，该算法的每种应用都需要仔细的数据依赖数据的调整过程，这可能会导致对单个任务进行智能系统的训练，以需要在超级计算机上进行数周或数月的计算。避免这种计算费用的一种途径是通过“适应”调整自己的优化技术的设计。该项目的目标是设计和为这种适应性算法提供理论保证。在上述算法（称为随机梯度（SG）算法）上，已经有各种以前提出的增强和扩展。但是，这些算法中的许多也具有非适应性的缺点，这意味着它们在实践中的成功应用需要昂贵的“超参数”调整工作。 ML的“随机优化”设置中考虑的自适应算法基于“确定性优化”文献中的各种成功方法。这些包括所谓的“线搜索”和“信任区域”方法。但是，由于这些方法都没有产生最佳的最坏情况复杂性的保证，因此该项目的重点是设计自适应最佳复杂性方法，例如所谓的“立方体调节”算法。通过在理论框架上构建自适应最小化作为“更新回报”随机过程，可以实现随机设置的自适应立方体化算法的设计。这项工作将结合来自数学优化和随机过程文献的分析技术，并将为从事应用数学，计算机科学，统计学和各种工程领域的研究人员提供扎实的理论和实用基础。该奖项反映了NSF的法定任务，并通过评估了基金会的范围，并通过评估了基金会的范围，并已被评估和宽广的范围。