On the Performance of Temporal Difference Learning With Neural Networks

Neural Temporal Difference (TD) Learning is an approximate temporal difference method for policy evaluation that uses a neural network for function approximation. Analysis of Neural TD Learning has proven to be challenging. In this paper we provide a convergence analysis of Neural TD Learning with a projection onto $B(\theta_0, \omega)$, a ball of fixed radius $\omega$ around the initial point $\theta_0$. We show an approximation bound of $O(\epsilon) + \tilde{O} (1/\sqrt{m})$ where $\epsilon$ is the approximation quality of the best neural network in $B(\theta_0, \omega)$ and $m$ is the width of all hidden layers in the network.

神经时序差分（TD）学习是一种用于策略评估的近似时序差分方法，它使用神经网络进行函数逼近。对神经TD学习的分析已被证明具有挑战性。在本文中，我们对神经TD学习进行了收敛性分析，其中涉及到在$B(\theta_0, \omega)$上的投影，$B(\theta_0, \omega)$是围绕初始点$\theta_0$、半径为固定值$\omega$的球。我们展示了一个$O(\epsilon) + \tilde{O} (1/\sqrt{m})$的逼近界，其中$\epsilon$是$B(\theta_0, \omega)$中最佳神经网络的逼近质量，$m$是网络中所有隐藏层的宽度。