Never look back - A modified EnKF method and its application to the training of neural networks without back propagation

textabstractIn this work, we present a new derivative-free optimization method and investigate its use for training neural networks. Our method is motivated by the Ensemble Kalman Filter (EnKF), which has been used successfully for solving optimization problems that involve large-scale, highly nonlinear dynamical systems. A key benefit of the EnKF method is that it requires only the evaluation of the forward propagation but not its derivatives. Hence, in the context of neural networks, it alleviates the need for back propagation and reduces the memory consumption dramatically. However, the method is not a pure "black-box" global optimization heuristic as it efficiently utilizes the structure of typical learning problems. Promising first results of the EnKF for training deep neural networks have been presented recently by Kovachki and Stuart. We propose an important modification of the EnKF that enables us to prove convergence of our method to the minimizer of a strongly convex function. Our method also bears similarity with implicit filtering and we demonstrate its potential for minimizing highly oscillatory functions using a simple example. Further, we provide numerical examples that demonstrate the potential of our method for training deep neural networks.

TextAbsTractin这项工作，我们提出了一种新的无衍生化优化方法，并研究了其在训练神经网络中的使用。我们的方法是由集合卡尔曼滤波器（ENKF）的动机，该滤波器已成功用于解决涉及大规模，高度非线性动力学系统的优化问题。 ENKF方法的一个关键好处是，它仅需要评估远期传播，而不需要其衍生物。因此，在神经网络的背景下，它减轻了对背部传播的需求，并大大减少了记忆消耗。但是，该方法并不是纯粹的“黑盒”全局优化启发式方法，因为它有效地利用了典型的学习问题的结构。 Kovachki和Stuart最近提出了ENKF培训深度神经网络的有希望的首先结果。我们提出了对ENKF的重要修改，使我们能够证明我们的方法与强凸功能的最小化器的收敛性。我们的方法还具有与隐式过滤相似的相似性，我们证明了它使用一个简单的示例最大程度地减少高度振荡函数的潜力。此外，我们提供了数值示例，这些示例证明了我们方法训练深神经网络的潜力。