Multilevel Architectures and Algorithms in Deep Learning

深度学习中的多级架构和算法

• 批准号: 464103607
• 负责人: Professor Dr. Roland Herzog
• 金额: --
• 依托单位: Interdisziplinäres Zentrum für Wissenschaftliches Rechnen (IWR)
• 依托单位国家: 德国
• 项目类别: Priority Programmes
• 资助国家: 德国
• 项目状态: 未结题
• 来源: https://gepris.dfg.de/gepris/projekt/464103607?language=en
• 关键词: Multilevel; Architectures; Algorithms; Deep; Learning

The design of deep neural networks (DNNs) and their training is a central issue in machine learning. Progress in these areas is one of the driving forces for the success of these technologies. Nevertheless, tedious experimentation and human interaction is often still needed during the learning process to find an appropriate network structure and corresponding hyperparameters to obtain the desired behavior of a DNN. The strategic goal of the proposed project is to provide algorithmic means to improve this situation. Our methodical approach relies on well established mathematical techniques: identify fundamental algorithmic quantities and construct a-posteriori estimates for them, identify and consistently exploit an appropriate topological framework for the given problem class, establish a multilevel structure for DNNs to account for the fact that DNNs only realize a discrete approximation of a continuous nonlinear mapping relating input to output data. Combining this idea with novel algorithmic control strategies and preconditioning, we will establish the new class of adaptive multilevel algorithms for deep learning, which not only optimize a fixed DNN, but also adaptively refine and extend the DNN architecture during the optimization loop. This concept is not restricted to a particular network architecture, and we will study feedforward neural networks, ResNets, and PINNs as relevant examples. Our integrated approach will thus be able to replace many of the current manual tuning techniques by algorithmic strategies, based on a-posteriori estimates. Moreover, our algorithm will reduce the computational effort for training and also the size of the resulting DNN, compared to a manually designed counterpart, making the use of deep learning more efficient in many aspects. Finally, in the long run our algorithmic approach has the potential to enhance the reliability and interpretability of the resulting trained DNN.

深度神经网络（DNN）的设计及其训练是机器学习的核心问题。这些领域的进步是这些技术成功的驱动力之一。然而，在学习过程中通常仍然需要繁琐的实验和人机交互来找到合适的网络结构和相应的超参数，以获得 DNN 的期望行为。该项目的战略目标是提供算法手段来改善这种情况。我们的系统方法依赖于完善的数学技术：识别基本算法量并为其构建后验估计，识别并持续利用给定问题类的适当拓扑框架，建立 DNN 的多级结构以解释 DNN 的事实仅实现输入到输出数据的连续非线性映射的离散近似。将此思想与新颖的算法控制策略和预处理相结合，我们将建立新型的深度学习自适应多级算法，它不仅优化固定的 DNN，而且在优化循环期间自适应地细化和扩展 DNN 架构。这个概念并不局限于特定的网络架构，我们将研究前馈神经网络、ResNets 和 PINNs 作为相关示例。因此，我们的集成方法将能够通过基于后验估计的算法策略取代许多当前的手动调整技术。此外，与手动设计的算法相比，我们的算法将减少训练的计算量以及生成的 DNN 的大小，从而使深度学习在许多方面的使用更加高效。最后，从长远来看，我们的算法方法有可能提高训练后的 DNN 的可靠性和可解释性。