CAREER: Statistical Learning from a Modern Perspective: Over-parameterization, Regularization, and Generalization

职业：现代视角下的统计学习：过度参数化、正则化和泛化

• 批准号: 2143215
• 负责人: Yuting Wei
• 金额: $ 40万
• 依托单位: University of Pennsylvania
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-09-01 至 2027-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2143215&HistoricalAwards=false
• 关键词: CAREER; Statistical; Learning; Modern; Perspective

Statistical methods have been a major driving force towards interpretable, actionable, and trustworthy machine learning. However, the existing statistical theory remains highly inadequate in explaining many new phenomena that emerge, and become pervasive in modern machine learning applications. For instance, the prevalence of over-parameterized models (i.e., the ones that have more model parameters than samples) challenges our classical statistical insights about the bias-variance tradeoff; the fact that many learning algorithms exhibit favorable algorithmic regularization to alleviate overfitting is largely beyond the reach of previous statistical literature, and the unconventional shapes of the risk curves in modern applications puzzle many statisticians. Compared to the rich theory developed for classical settings, however, the statistical underpinnings for these curious yet mysterious phenomena remain far from sufficient. Motivated by this, the overarching goal of the project is to enrich the statistical foundation of machine learning by adapting it to contemporary settings, thereby bridging classical statistics and cutting-edge machine learning. In addition, the project will provide valuable opportunities for training students (particularly underrepresented groups) at all levels across multiple disciplines in the STEM field, and will exert scientific and societal impacts on several domains beyond the tasks described herein, including but not limited to neuroscience, online education, and equitable machine learning.Striving for interpretability and actionable insights, this project plans to revisit multiple classical statistical problems---ranging from minimum-norm interpolation, risk estimation, cross validation, kernel boosting, data-imbalanced classification, to transfer learning---with an emphasis on unveiling new insights for modern yet under-explored regimes. Several recurring themes include: (i) characterizing precise risk behavior in the face of large model complexity; (ii) reconciling the seemingly conflicting goals of over-parameterization and regularization; (iii) developing algorithm-specific statistical reasoning tools; and (iv) exploring the interplay between regularization and generalization. The project comprises three distinct yet related thrusts: (1) statistical insights for over-parameterization: which explores the prolific interplay between model complexity and out-of-sample performance; (2) algorithmic regularization via early stopping: which aims to develop statistical principles that underlie early stopping; (3) risk (non)-monotonicity with imbalanced data: which is motivated by the non-monotonicity of generalization errors in the sample size and pursues principled debiasing methods to rectify it. The project will develop a suite of statistical insights that can inform cutting-edge machine learning practice, as well as an array of statistical methodologies that will be practically appealing for modern data-driven applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

统计方法一直是可解释、可操作和值得信赖的机器学习的主要推动力。然而，现有的统计理论仍然不足以解释现代机器学习应用中出现并普遍存在的许多新现象。例如，过度参数化模型（即模型参数多于样本的模型）的盛行挑战了我们关于偏差-方差权衡的经典统计见解；事实上，许多学习算法表现出有利的算法正则化来减轻过度拟合，这在很大程度上超出了以前的统计文献的范围，并且现代应用中风险曲线的非常规形状让许多统计学家感到困惑。然而，与针对经典环境开发的丰富理论相比，这些奇怪而神秘的现象的统计基础仍然远远不够。受此启发，该项目的总体目标是通过适应当代环境来丰富机器学习的统计基础，从而架起经典统计学和前沿机器学习的桥梁。此外，该项目将为培训 STEM 领域多个学科各个级别的学生（特别是代表性不足的群体）提供宝贵的机会，并将对本文所述任务之外的多个领域产生科学和社会影响，包括但不限于神经科学、在线教育和公平的机器学习。为了寻求可解释性和可操作的见解，该项目计划重新审视多个经典统计问题——包括最小范数插值、风险估计、交叉验证、核提升、数据不平衡分类，到迁移学习——重点是为现代但尚未充分探索的制度揭示新的见解。几个反复出现的主题包括：（i）在面对复杂的模型时精确描述风险行为； (ii) 调和过度参数化和正则化看似相互冲突的目标； (iii) 开发特定算法的统计推理工具； (iv) 探索正则化和泛化之间的相互作用。该项目包括三个不同但相关的主旨：（1）过度参数化的统计见解：探索模型复杂性和样本外性能之间的丰富相互作用； (2) 通过提前停止进行算法正则化：旨在开发支持提前停止的统计原理； （3）不平衡数据的风险（非）单调性：其动机是样本量中泛化误差的非单调性，并追求有原则的去偏方法来纠正它。该项目将开发一套可为尖端机器学习实践提供信息的统计见解，以及一系列对现代数据驱动应用程序具有实际吸引力的统计方法。该奖项反映了 NSF 的法定使命，并被认为是值得的通过使用基金会的智力优势和更广泛的影响审查标准进行评估来获得支持。

项目成果

The Lasso with general Gaussian designs with applications to hypothesis testing

• DOI: 10.1214/23-aos2327
• 发表时间: 2020-07
• 期刊: ArXiv
• 作者: Michael Celentano;A. Montanari;Yuting Wei

Softmax policy gradient methods can take exponential time to converge

• DOI: 10.1007/s10107-022-01920-6
• 发表时间: 2021-02
• 期刊: Mathematical Programming
• 影响因子: 2.7
• 作者: Gen Li;Yuting Wei;Yuejie Chi;Yuantao Gu;Yuxin Chen

Derandomizing Knockoffs

• DOI: 10.1080/01621459.2021.1962720
• 发表时间: 2020-12
• 期刊: Journal of the American Statistical Association
• 影响因子: 3.7
• 作者: Zhimei Ren;Yuting Wei;E. Candès