CIF: Small: Exploring and Exploiting the Universality Phenomenon in High-Dimensional Estimation

CIF：小：探索和利用高维估计中的普遍性现象

• 批准号: 1910410
• 负责人: Yue Lu
• 金额: $ 49.97万
• 依托单位: Harvard University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2022-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1910410&HistoricalAwards=false
• 关键词: CIF; Small; Exploring; Exploiting; Universality

Understanding the performance of various algorithms used in practice is a central question in information processing and machine learning. Such performance guarantees are very important to practitioners. For example, data analysts need to know how many data samples to collect for a given inference algorithm to reach a prediction with sufficient statistical accuracy and confidence. Although significant progress has been made in precisely characterizing the performance of various estimation and inference algorithms, a big gap exists between theory and practice. Most of existing theoretical work on performance analysis relies upon strong and often unrealistic assumptions on the underlying statistical models. Such idealistic models, while useful and convenient for mathematical proofs, often do not fit the situations encountered in practice. This project aims to narrow the gap between theory and practice in performance analysis by leveraging the universality phenomenon. In short, universality is the observation that universal laws govern the macroscopic behavior of many high-dimensional systems, irrespectively of how different they might be in their microscopic constructions. By exploiting the universality phenomenon, this project contributes to an understanding of the fundamental limits of various estimation and inference methods under more realistic models. In addition, this project makes broad impacts through the dissemination of datasets, the organization of workshops/tutorials, and the mentoring and supporting of students from diverse backgrounds.The specific goals of this project are organized into three main thrusts. In the first thrust, the investigator analyzes the exact asymptotic performance of a class of spectral methods that have been widely used in recent work on nonconvex optimization approaches for signal estimation. In particular, the project extends the current analysis from independent ensembles to more general ensembles, and explores new applications including multiplexed imaging and the training of multilayer neural networks. In the second thrust, the project investigates the performance bounds for regularized M-estimators when the sensing matrices are drawn from general non-independent ensembles. The third thrust of the project uses extensive numerical simulations to explore the strength, robustness, as well as the limitations of the universality phenomenon in high-dimensional estimation. The numerical experiments are guided by the theory and insights developed in the first two thrusts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

了解实践中使用的各种算法的性能是信息处理和机器学习的核心问题。这样的履约保证对于从业者来说非常重要。例如，数据分析师需要知道为给定的推理算法收集多少数据样本才能以足够的统计准确性和置信度做出预测。尽管在精确表征各种估计和推理算法的性能方面已经取得了重大进展，但理论与实践之间仍存在很大差距。大多数现有的绩效分析理论工作都依赖于对基础统计模型的强有力且往往不切实际的假设。这种理想主义模型虽然对于数学证明有用且方便，但通常不适合实践中遇到的情况。该项目旨在利用普遍性现象缩小绩效分析理论与实践之间的差距。简而言之，普遍性是这样一种观察：普遍法则支配着许多高维系统的宏观行为，无论它们的微观结构有多么不同。通过利用普遍性现象，该项目有助于理解更现实的模型下各种估计和推理方法的基本局限性。此外，该项目通过传播数据集、组织研讨会/教程以及对来自不同背景的学生的指导和支持，产生了广泛的影响。该项目的具体目标分为三个主要目标。在第一个主旨中，研究人员分析了一类谱方法的精确渐近性能，这些方法在最近的信号估计非凸优化方法工作中得到了广泛使用。特别是，该项目将当前的分析从独立的集成扩展到更通用的集成，并探索了包括多重成像和多层神经网络训练在内的新应用。在第二个主旨中，该项目研究了当传感矩阵取自一般非独立系综时正则化 M 估计器的性能界限。该项目的第三个重点是使用广泛的数值模拟来探索高维估计中普遍性现象的强度、鲁棒性以及局限性。数值实验以前两个主旨中发展的理论和见解为指导。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression

点积核回归的精确学习曲线和高阶缩放限制

• 发表时间: 2022-10
• 期刊: Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS
• 作者: Xiao, L.;Hu, H;Misiakiewicz, T.;Lu, Y.;Pennington, J.
• 通讯作者: Pennington, J.

On the Inherent Regularization Effects of Noise Injection During Training

关于训练期间噪声注入的固有正则化效果

• 发表时间: 2021-10
• 期刊: Proceedings of the 38th International Conference on Machine Learning
• 作者: Dhifallah, Oussama;Lu, Yue
• 通讯作者: Lu, Yue

Universality Laws for High-Dimensional Learning With Random Features

具有随机特征的高维学习的普遍性定律

• DOI: 10.1109/tit.2022.3217698
• 发表时间: 2023-03
• 期刊: IEEE Transactions on Information Theory
• 影响因子: 2.5
• 作者: Hu, Hong;Lu, Yue M.
• 通讯作者: Lu, Yue M.

Householder Dice: A Matrix-Free Algorithm for Simulating Dynamics on Gaussian and Random Orthogonal Ensembles

Householder Dice：一种用于模拟高斯和随机正交系综动力学的无矩阵算法

• DOI: 10.1109/tit.2021.3114351
• 发表时间: 2021-12
• 期刊: IEEE Transactions on Information Theory
• 影响因子: 2.5
• 作者: Lu; Yue M.
• 通讯作者: Yue M.

SLOPE for Sparse Linear Regression: Asymptotics and Optimal Regularization

稀疏线性回归的 SLOPE：渐近和最优正则化

• DOI: 10.1109/tit.2022.3188753
• 发表时间: 2022-11
• 期刊: IEEE Transactions on Information Theory
• 影响因子: 2.5
• 作者: Hu, Hong;Lu, Yue M.
• 通讯作者: Lu, Yue M.