A Non-Asymptotic Analysis of Stochastic Mirror Descent for Non-Convex Learning

非凸学习的随机镜像下降的非渐近分析

• 批准号: 2444063
• 金额: --
• 依托单位: University of Oxford
• 依托单位国家: 英国
• 项目类别: Studentship
• 财政年份: 2020
• 资助国家: 英国
• 起止时间: 2020 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=studentship-2444063
• 关键词: Non; Asymptotic; Analysis; Stochastic; Mirror

Many state of the art machine learning techniques depend heavily on the optimisation of some non-convex objective. Often, this procedure is computationally expensive and requires techniques that employ stochastic approximations or regularisation using artificial noise. For instance, there are a variety of techniques in deep learning that use artificial noise applied to the data, parameters or update direction, all of which have been shown to encourage faster convergence and improved generalisation. Though many of these techniques have demonstrated their validity through repeated experimentation and testing, the theoretical framework to validate and understand them is still in its infancy.Another technique which is used to overcome the aforementioned computational difficulties is the mirror descent framework, which does so by taking advantage of the known geometric properties specific to the problem in consideration. It is highly popular in convex optimisation as in this setting there are many provable advantages to this algorithm. Its stochastic counterpart, stochastic mirror descent, is rapidly gaining popularity in non-convex learning. But again, the theoretical framework in this setting is under-developed.Through our project, we seek to develop a better understanding of the statistical and computational properties of these techniques in the large-scale learning setting. Specifically, we are interested in quantitively comparing the consistency and generalisation performance of these techniques and seeing how they change as the problem grows more challenging.Guiding much of our methodology will be the focus on non-asymptotic analysis, that is compared to analyses in which quantities are taken to their infinite limits. We believe this is essential for applications in large-scale learning where only a limited number of iterations can take place and the effect of the number of parameters on performance is of key interest.Fundamentally, our approach will be based on approximating the iterative optimisation procedure with stochastic processes that evolve continuously in time, specifically diffusion processes. With this, we gain access to the broad suite of powerful techniques for analysing diffusions. This approach has seen a growing popularity in recent years and has already yielded interesting results in the learning setting. The project will also draw heavily from the study of stochastic differential equations as well as high dimensional statistics.This project falls within the EPSRC Statistics and Applied probability area.

许多最先进的机器学习技术在很大程度上取决于某些非凸目标的优化。通常，此过程在计算上很昂贵，需要使用人工噪声采用随机近似或正则化的技术。例如，深度学习中有多种技术，它们使用用于数据，参数或更新方向的人造噪声，所有这些噪声已显示为鼓励更快的收敛和改善的概括。尽管这些技术中的许多通过重复实验和测试证明了它们的有效性，但验证和理解它们的理论框架仍处于起步阶段。其他技术用于克服上述计算困难的是镜像下降框架，它是通过利用已知的特定属性来考虑该问题来考虑的。它在凸优化中非常受欢迎，因为在这种情况下，该算法具有许多可证明的优势。它的随机对应物是随机镜下降，在非凸学习中迅速越来越受欢迎。但同样，这种设置中的理论框架是不发达的。通过我们的项目，我们寻求更好地了解大规模学习环境中这些技术的统计和计算属性。具体而言，我们有兴趣定量比较这些技术的一致性和泛化性能，并随着问题的越来越具有挑战性而看到它们的变化。指导我们的大部分方法将重点放在非催化分析上，这与分析的分析相比，在这些分析中，这些分析将数量限制为无限限制。我们认为，这对于在大规模学习中的应用至关重要，在大规模学习中，只能进行有限数量的迭代，并且参数数对性能的影响是关键的兴趣。从根本上讲，我们的方法将基于近似迭代的优化过程，其随机过程在时间上连续发展，特别是扩散过程。因此，我们可以访问广泛的强大技术来分析扩散。近年来，这种方法越来越受欢迎，并且在学习环境中已经产生了有趣的结果。该项目还将大量从随机微分方程的研究以及高维统计数据中获取。该项目属于EPSRC统计和应用概率领域。