CRII: III: Robust Machine Learning Methods for Messy Data

CRII：III：针对杂乱数据的鲁棒机器学习方法

• 批准号: 1657155
• 负责人: James Zou
• 金额: $ 17.5万
• 依托单位: Stanford University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-05-01 至 2019-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1657155&HistoricalAwards=false
• 关键词: CRII; III; Robust; Machine; Learning

Messy data is ubiquitous in modern science. Data come from heterogeneous sources; there are many latent confounding factors; and it is often unclear what are the relevant questions to ask and models to use. This reality is in sharp contrast with the usual modeling assumptions of machine learning and statistics, where data are assumed to come from well-specified models and the hypotheses to test are clearly laid out. The glaring gap between standard theory and the actual practice of messy data is a major contributor to the reproducibility crises across science and prevents researchers from harnessing the full insights from data. This project will develop rigorous mathematical foundations and robust machine learning algorithms to address the core challenges of messy data. The PI will explore novel techniques to quantify and reduce different types of selection biases that arise from exploratory data analysis. The PI will also investigate algorithms to perform statistical inference when the model is mis- or under-specified. The project will apply these new methods to tackle challenging problems in human population genomics. The PI recently initialized a framework based on information usage to quantify the magnitude of over-fitting and bias arising from data exploration. This project will significantly expand this framework. In particular the PI will apply this information usage approach to quantify and reduce bias in data generated from adaptive experimentation, such as online A/B testing and more general multi-arm bandits. Related to over-fitting is the problem of mis- and under-specified statistical models. The PI has recently developed method-of-cumulant approaches to learn probabilistic models when the observations are perturbed by unknown and arbitrary interference. A promising direction of research is to extend this approach to more general settings that allow for nonlinear interference and to develop software tools for the broad data science community. Genomics exemplify many of the challenges of messy data-genomic data typically requires substantial exploratory analysis and faces modeling uncertainty. This makes genomics a high impact domain to apply the new messy data algorithms developed here. Bio-medical databases are interactively analyzed by many researchers and thus are particularly prone to exploration bias and overfitting. The PI will explore piloting the information usage framework on the bio-medical data hubs being created at Stanford in order to quantify and reduce exploration bias. As a part of the project, PI is also developing courses, workshops and tutorials to bring together researchers and practitioners across machine learning, statistics, information theory and bio-medical data science to address the ubiquitous challenge of messy data.

混乱的数据在现代科学中无处不在。数据来自异构源；存在许多潜在的混杂因素；而且通常不清楚要提出哪些相关问题和要使用哪些模型。这一现实与机器学习和统计学的通常建模假设形成鲜明对比，其中数据被假设来自明确指定的模型，并且清楚地列出了要测试的假设。标准理论与混乱数据的实际实践之间的明显差距是整个科学领域再现性危机的主要原因，并阻碍研究人员充分利用数据的见解。该项目将开发严格的数学基础和强大的机器学习算法，以解决混乱数据的核心挑战。 PI 将探索新技术来量化和减少探索性数据分析中产生的不同类型的选择偏差。当模型指定错误或指定不足时，PI 还将研究执行统计推断的算法。该项目将应用这些新方法来解决人类基因组学中的挑战性问题。 PI 最近初始化了一个基于信息使用的框架，以量化数据探索中产生的过度拟合和偏差的程度。该项目将显着扩展该框架。特别是，PI 将应用这种信息使用方法来量化和减少自适应实验（例如在线 A/B 测试和更常见的多臂老虎机）生成的数据中的偏差。与过度拟合相关的是统计模型指定错误和指定不足的问题。 PI 最近开发了累积方法，用于在观测受到未知和任意干扰干扰时学习概率模型。一个有前途的研究方向是将这种方法扩展到允许非线性干扰的更一般的设置，并为广泛的数据科学界开发软件工具。基因组学体现了杂乱数据的许多挑战——基因组数据通常需要大量的探索性分析，并面临建模的不确定性。这使得基因组学成为应用此处开发的新混乱数据算法的高影响力领域。生物医学数据库由许多研究人员进行交互式分析，因此特别容易出现探索偏差和过度拟合。 PI 将探索在斯坦福大学创建的生物医学数据中心上试行信息使用框架，以量化和减少探索偏差。作为该项目的一部分，PI 还在开发课程、研讨会和教程，将机器学习、统计学、信息论和生物医学数据科学领域的研究人员和从业者聚集在一起，以应对无处不在的混乱数据挑战。

项目成果

Exploring patterns enriched in a dataset with contrastive principal component analysis.

• DOI: 10.1038/s41467-018-04608-8
• 发表时间: 2018-05-30
• 期刊: Nature communications
• 影响因子: 16.6
• 作者: Abid A;Zhang MJ;Bagaria VK;Zou J
• 通讯作者: Zou J

Word embeddings quantify 100 years of gender and ethnic stereotypes

• DOI: 10.1073/pnas.1720347115
• 发表时间: 2018-04-17
• 期刊: PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA
• 影响因子: 11.1
• 作者: Garg, Nikhil;Schiebinger, Londa;Zou, James
• 通讯作者: Zou, James

NeuralFDR: learning decision threshold from hypothesis features.

NeuralFDR：从假设特征中学习决策阈值。

• 发表时间: 2017
• 期刊: Proceedings of Neural Information Processing Conference
• 作者: Xia, Fei;Zhang, Martin;Zou, James;Tse, David
• 通讯作者: Tse, David