Optimization of Random Feature Method in the High-Precision Regime

Machine learning has been widely used for solving partial differential equations (PDEs) in recent years, among which the random feature method (RFM) exhibits spectral accuracy and can compete with traditional solvers in terms of both accuracy and efficiency. Potentially, the optimization problem in the RFM is more difficult to solve than those that arise in traditional methods. Unlike the broader machine-learning research, which frequently targets tasks within the low-precision regime, our study focuses on the high-precision regime crucial for solving PDEs. In this work, we study this problem from the following aspects: (i) we analyze the coefficient matrix that arises in the RFM by studying the distribution of singular values; (ii) we investigate whether the continuous training causes the overfitting issue; (iii) we test direct and iterative methods as well as randomized methods for solving the optimization problem. Based on these results, we find that direct methods are superior to other methods if memory is not an issue, while iterative methods typically have low accuracy and can be improved by preconditioning to some extent.

近年来，机器学习已被广泛用于求解偏微分方程（PDEs），其中随机特征方法（RFM）具有谱精度，并且在精度和效率方面可与传统求解器相媲美。潜在地，RFM中的优化问题比传统方法中出现的问题更难解决。与更广泛的机器学习研究不同（其通常针对低精度领域内的任务），我们的研究侧重于对求解PDEs至关重要的高精度领域。在这项工作中，我们从以下几个方面研究这个问题：（i）通过研究奇异值的分布来分析RFM中出现的系数矩阵；（ii）研究连续训练是否会导致过拟合问题；（iii）测试用于解决优化问题的直接方法、迭代方法以及随机方法。基于这些结果，我们发现如果内存不是问题，直接方法优于其他方法，而迭代方法通常精度较低，并且在一定程度上可以通过预处理来改进。