DMS-EPSRC: Asymptotic Analysis of Online Training Algorithms in Machine Learning: Recurrent, Graphical, and Deep Neural Networks

DMS-EPSRC：机器学习中在线训练算法的渐近分析：循环、图形和深度神经网络

• 批准号: EP/Y029089/1
• 负责人: Justin Sirignano
• 金额: $ 52.17万
• 依托单位: University of Oxford
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2024
• 资助国家: 英国
• 起止时间: 2024 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FY029089%2F1
• 关键词: DMS; EPSRC; Asymptotic; Analysis; Online

Neural network models in machine learning have achieved immense practical success over the past decade, revolutionizing fields such as image, text, and speech recognition. Neural networks have also become widely-used in science, engineering, medicine, and finance. In particular, deep learning, which uses multi-layer neural networks, has transformed the field of machine learning. The training algorithms used for these complex machine learning problems -- although successful in practice -- are often ad hoc. Mathematical theory is yet to be established in many cases, and there is the potential to improve training algorithms and models via rigorous mathematical analysis. The primary purpose of this research is to develop new mathematical theory for the training algorithms and neural network models used in several key areas of machine learning. The problems in this project are motivated by both fundamental mathematical questions and questions highly relevant to applications. Developing and testing mathematical theory for widely-used training algorithms is crucial for ensuring their reliability and guaranteeing their performance in practice. The successful conclusion of this proposal's research program will contribute to the machine learning community by developing new mathematical methods which can be broadly used for the analysis of neural networks and machine learning algorithms. This research project is integrated with an educational programme which will contribute to the training of students in the mathematical foundations of machine learning and deep learning.Our proposed research will develop a rigorous mathematical analysis for the training algorithms and neural network models used in several important areas of machine learning, including: recurrent neural networks, reinforcement learning, graph neural networks, and deep neural networks. Our analysis will characterize the asymptotic behavior of these algorithms and neural network models as the number of training steps and the number of hidden units in the neural network go to infinity. The researchers will develop new mathematical methods designed for the analysis of neural networks by leveraging methods from stochastic analysis and weak convergence theory to study the asymptotics of online, stochastic training algorithms and neural network models as the number of hidden units becomes large. Applications of the mathematical results to parameter initialization, hyperparameter selection, design of optimization/training algorithms, and the selection of model architectures will be investigated. The research project is highly interdisciplinary, leveraging methods from probability, partial differential equations, large deviations theory, stochastic analysis, optimization, and machine learning. In addition to proving convergence theory for important neural network training algorithms, the research will be of broader interest outside of machine learning as it will study a new set of mean-field problems with novel and mathematically challenging features.

机器学习中的神经网络模型在过去十年中取得了巨大的实际成功，彻底改变了图像、文本和语音识别等领域。神经网络也广泛应用于科学、工程、医学和金融领域。特别是，使用多层神经网络的深度学习已经改变了机器学习领域。用于这些复杂机器学习问题的训练算法虽然在实践中很成功，但通常是临时的。在许多情况下，数学理论尚未建立，有潜力通过严格的数学分析来改进训练算法和模型。这项研究的主要目的是为机器学习的几个关键领域中使用的训练算法和神经网络模型开发新的数学理论。该项目中的问题是由基本数学问题和与应用高度相关的问题引起的。开发和测试广泛使用的训练算法的数学理论对于确保其可靠性并保证其在实践中的性能至关重要。该提案研究计划的成功完成将通过开发可广泛用于神经网络和机器学习算法分析的新数学方法为机器学习社区做出贡献。该研究项目与教育计划相结合，有助于培养学生机器学习和深度学习的数学基础。我们提出的研究将为几个重要领域中使用的训练算法和神经网络模型进行严格的数学分析机器学习，包括：循环神经网络、强化学习、图神经网络和深度神经网络。我们的分析将描述这些算法和神经网络模型的渐近行为，因为训练步骤的数量和神经网络中隐藏单元的数量趋于无穷大。研究人员将开发用于分析神经网络的新数学方法，利用随机分析和弱收敛理论的方法来研究在线随机训练算法和神经网络模型随着隐藏单元数量变大的渐近性。将研究数学结果在参数初始化、超参数选择、优化/训练算法设计以及模型架构选择中的应用。该研究项目是高度跨学科的，利用概率、偏微分方程、大偏差理论、随机分析、优化和机器学习等方法。除了证明重要的神经网络训练算法的收敛理论之外，该研究还将在机器学习之外引起更广泛的兴趣，因为它将研究一组具有新颖且具有数学挑战性特征的新平均场问题。