SHF: Small: Algorithms and Software for Scalable Kernel Methods

SHF：小型：可扩展核方法的算法和软件

• 批准号: 1817048
• 负责人: George Biros
• 金额: $ 47.62万
• 依托单位: University of Texas at Austin
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-07-01 至 2021-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1817048&HistoricalAwards=false
• 关键词: SHF; Small; Algorithms; Software; Scalable

Scientists and engineers are increasingly interested in using machine learning methods on huge datasets that cannot be processed on a single workstation. At the same time public and private institutions are making significant investments on high-performance computing (HPC) clusters equipped with thousands of leading edge processors and network connectivity. However, despite the availability of such HPC systems, data analysis tasks are mostly restricted to a single or a few workstations. The reason is that, with few exceptions, existing machine learning software does not scale efficiently on HPC systems. The need to process in-situ large scientific and engineering datasets is not met with current software and significant downsampling is required in order to use existing tools. A serious bottleneck in current artificial intelligence (AI) workflows is the significant cost of training for large scale problems. The slow convergence of existing methods and the large number of calibration hyper-parameters (learning rate, batch size, and other knobs that control the performance of the AI system) make training extremely expensive. Design and analysis of scalable optimization algorithms for faster training, that is the fitting of the machine learning (ML) model parameters to the data, are needed for analytics in real time and at scale, which is the goal of this project.The proposed research will introduce novel numerical methods and parallel algorithms for second-order/Newton methods that will be tailored to machine learning (ML) models and will be many orders of magnitude faster than the existing state of-the-art (first-order methods like steepest descent). The researchers plan to design, analyze, and implement robust approximations for covariance matrices, a class of matrices in AI and computational statistics, used in statistical analysis (e.g., sampling, risk assessment, and uncertainty quantification). The investigators plan to design, analyze, and implement scalable fast algorithms in the context of high-performance computing for the so called nearest-neighbor problem, a particular method in ML, data analysis, and information retrieval. The resulting software library will provide a means for end-to-end tools for discovery and innovation and provide new capabilities in the NSF XSEDE infrastructure project. Along with research activities, an educational and dissemination program is designed to communicate the results of this work to both students and researchers, as well as a more general audience of computational and application scientists.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

科学家和工程师越来越有兴趣在无法在单个工作站上处理的庞大数据集上使用机器学习方法。 与此同时，公共和私人机构正在对配备数千个领先处理器和网络连接的高性能计算 (HPC) 集群进行大量投资。然而，尽管存在此类 HPC 系统，但数据分析任务大多仅限于单个或几个工作站。原因是，除了少数例外，现有的机器学习软件无法在 HPC 系统上有效扩展。当前的软件无法满足现场处理大型科学和工程数据集的需求，并且需要大量下采样才能使用现有工具。当前人工智能 (AI) 工作流程的一个严重瓶颈是大规模问题的训练成本高昂。现有方法的收敛速度缓慢以及大量的校准超参数（学习率、批量大小和其他控制人工智能系统性能的旋钮）使得训练成本极其昂贵。设计和分析可扩展的优化算法以实现更快的训练，即机器学习 (ML) 模型参数与数据的拟合，需要进行实时和大规模的分析，这是该项目的目标。拟议的研究将为二阶/牛顿方法引入新颖的数值方法和并行算法，这些方法将针对机器学习（ML）模型进行定制，并且比现有最先进的方法（一阶方法，如最陡方法）快多个数量级血统）。研究人员计划设计、分析和实现协方差矩阵的稳健近似，协方差矩阵是人工智能和计算统计学中的一类矩阵，用于统计分析（例如抽样、风险评估和不确定性量化）。研究人员计划在高性能计算的背景下设计、分析和实现可扩展的快速算法，解决所谓的最近邻问题，这是机器学习、数据分析和信息检索中的一种特殊方法。由此产生的软件库将为发现和创新的端到端工具提供一种手段，并在 NSF XSEDE 基础设施项目中提供新功能。除了研究活动之外，还设计了一项教育和传播计划，旨在向学生和研究人员以及更广泛的计算和应用科学家受众传达这项工作的结果。该奖项反映了 NSF 的法定使命，并被认为是值得的通过使用基金会的智力优势和更广泛的影响审查标准进行评估来提供支持。

项目成果

Fast Approximation of the Gauss--Newton Hessian Matrix for the Multilayer Perceptron

高斯的快速逼近--多层感知器的牛顿Hessian矩阵

• DOI: 10.1137/19m129961x
• 发表时间: 2021-01
• 期刊: SIAM Journal on Matrix Analysis and Applications
• 影响因子: 1.5
• 作者: Chen, Chao;Reiz, Severin;Yu, Chenhan D.;Bungartz, Hans;Biros, George
• 通讯作者: Biros, George

RCHOL: Randomized Cholesky Factorization for Solving SDD Linear Systems

RCHOL：用于求解 SDD 线性系统的随机 Cholesky 分解

• DOI: 10.1137/20m1380624
• 发表时间: 2021-01
• 期刊: SIAM Journal on Scientific Computing
• 影响因子: 3.1
• 作者: Chen, Chao;Liang, Tianyu;Biros, George
• 通讯作者: Biros, George

Distributed O(N) Linear Solver for Dense Symmetric Hierarchical Semi-Separable Matrices

密集对称分层半可分离矩阵的分布式 O(N) 线性求解器

• DOI: 10.1109/mcsoc.2019.00008
• 发表时间: 2019-10
• 期刊: 2019 IEEE 13th International Symposium on Embedded Multicore/Many-core Systems-on-Chip (MCSoC
• 作者: Yu, Chenhan D.;Reiz, Severin;Biros, George
• 通讯作者: Biros, George

ANODEv2: A Coupled Neural ODE Framework

ANODEv2：耦合神经常微分方程框架

• 发表时间: 2022-04
• 期刊: Advances in neural information processing systems
• 作者: Zhang; Tianjun and
• 通讯作者: Tianjun and

Hardware Accelerator Integration Tradeoffs for High-Performance Computing: A Case Study of GEMM Acceleration in N-Body Methods

高性能计算的硬件加速器集成权衡：N 体方法中 GEMM 加速的案例研究

• DOI: 10.1109/tpds.2021.3056045
• 发表时间: 2021-01
• 期刊: IEEE Transactions on Parallel and Distributed Systems
• 影响因子: 5.3
• 作者: Asri, Mochamad;Malhotra, Dhairya;Wang, Jiajun;Biros, George;John, Lizy K;Gerstlauer, Andreas
• 通讯作者: Gerstlauer, Andreas