CIF: Small: Statistically Optimal Subsampling for Big Data and Rare Events Data

CIF：小：大数据和稀有事件数据的统计最佳子采样

基本信息

批准号：
2105571
负责人：
HaiYing Wang
金额：
$ 39.95万
依托单位：
University of Connecticut
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2021
资助国家：
美国
起止时间：
2021-06-15 至 2025-05-31
项目状态：
未结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2105571&HistoricalAwards=false
关键词：
CIF Small Statistically Optimal Subsampling

项目摘要

The ever-increasing amounts of big data offer unprecedented opportunities for advancing knowledge across scientific fields. However, traditional analyses of big data involve high computational costs and often require supercomputers. This project aims to develop computational tools that empower practitioners to analyze big data without dependency on supercomputers. It produces optimal algorithms that extract the maximum amount of information from massive data with limited computing resources. Rare-events data are common in big data where the numbers of interested events are relatively small although available full data are massive. This project is identifying conditions when the majority data can be discarded without any information loss, and developing methods for valid analysis and appropriate decision-making with rare events data. Education is another key component of the project, with a significant focus on classroom integration and next-generation workforce training, aiming to attract and equip a broader range of participants, especially underrepresented groups, to the field of computational data science.Subsampling has demonstrated a pervasive potential to enable better use of a fixed amount of computing resources. However, existing investigations focus on calculations of the collected data, and available results are not suitable for statistical inference on the underlying model. This project develops and expands the subsampling technique in the following directions: 1) It establishes a framework to determine statistically optimal subsampling probabilities by examining statistical distributional properties of subsample estimators; 2) it derives the maximum subsampled conditional likelihood estimator that has the smallest asymptotic variance among a large class of asymptotically unbiased estimators; and 3) it obtains new theoretical insights on rare-events data and challenges a long-standing view of underestimated probabilities for rare events. The research is a significant addition to the field of big data subsampling and provides tools that are widely applicable to facilitate practical inference and decision-making. It also answers important questions that are essential for extracting valid information from rare-events data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

不断增加的大数据为推进科学领域的知识提供了前所未有的机会。然而，传统的大数据分析涉及较高的计算成本，并且通常需要超级计算机。该项目旨在开发计算工具，使从业者能够在不依赖超级计算机的情况下分析大数据。它产生最优算法，利用有限的计算资源从海量数据中提取最大量的信息。稀有事件数据在大数据中很常见，尽管可用的完整数据很大，但感兴趣事件的数量相对较少。该项目正在确定可以丢弃大多数数据而不丢失任何信息的条件，并开发利用罕见事件数据进行有效分析和适当决策的方法。教育是该项目的另一个关键组成部分，重点关注课堂整合和下一代劳动力培训，旨在吸引和装备更广泛的参与者，特别是代表性不足的群体，进入计算数据科学领域。二次抽样已经证明了能够更好地利用固定数量的计算资源的普遍潜力。然而，现有的研究重点是对收集的数据进行计算，现有的结果不适合对基础模型进行统计推断。该项目在以下方向发展和扩展了子采样技术： 1）建立了一个框架，通过检查子样本估计量的统计分布特性来确定统计上最优的子采样概率； 2) 导出一大类渐近无偏估计量中渐近方差最小的最大子采样条件似然估计量； 3）它获得了关于罕见事件数据的新理论见解，并挑战了长期以来低估罕见事件概率的观点。该研究是大数据子采样领域的重要补充，并提供了广泛适用的工具来促进实际推理和决策。它还回答了从罕见事件数据中提取有效信息所必需的重要问题。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。