Collaborative Research: Statistical Inference for High Dimensional and High Frequency Data

合作研究：高维高频数据的统计推断

• 批准号: 2015530
• 负责人: Lan Zhang
• 金额: $ 15万
• 依托单位: University of Illinois at Chicago
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-07-01 至 2024-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2015530&HistoricalAwards=false
• 关键词: Collaborative; Research; Statistical; Inference; Dimensional

To pursue the promise of the big data revolution, the current project will focus on a common form of data, high dimensional high frequency data (HDHFD), where a snapshot of the data involves a large number of variables, and at the same time new data streams in every fraction of milliseconds. With technological advances in data collection, HDHFD occurs in medical applications from neuroscience to patient care; finance and economics; geosciences such as earthquake data; marine science including fishing and shipping; turbulence; internet data; and other areas where data streaming is available. The Principal Investigators' (PIs') research focuses on how to extract information from complex big data and how to turn data into knowledge. In particular, the project seeks to develop cutting-edge mathematics and statistical methodology to uncover the structure governing HDHFD systems. This structure is characterized by a web of dependence across both time and dimension, and the role of analysis is to provide guidance on how to reduce the complexity while retaining the important features of the data architecture. An integral part of this research is also about how to quantify the uncertainty in estimates and forecasts in HDHFD systems. In addition to developing a general theory, the project is concerned with applications to financial data, including risk management, forecasting, and portfolio management. More precise estimators, with improved margins of error, will be useful in all these areas of finance. The results are of interest to main-street investors, regulators and policymakers, and the results are entirely in the public domain. The purpose of this project is to explore high dimensional high frequency data (HDHFD) from several angles. A fundamental approach is to extend the PIs’ contiguity theory. Under a contiguous probability, the structure of the observations is often more accessible (frequently Gaussian) in local neighborhoods, facilitating statistical analysis. This is achieved without altering current models. In a contribution to factor modeling of the HDHFD data, the PIs will explore time-varying matrix decompositions, including the development of a singular value decomposition (SVD) for high frequency data, as a more direct path to a factor model. We plan to compare the new SVD with PCA based methods, as well as L1 type methods such as nonnegative matrix factorization. The PIs have discovered a new way to look at time and cross-dimension dependence, originally developed by the PIs in connection with their observed asymptotic variance (observed AVAR). They will now look into the possibility to "borrow" information across time and dimension. This tool will be used for matrix decompositions, as well as to develop volatility matrices for the drift part of a financial process, which will interface with their planned work on matrix decompositions. The PIs will explore a path to an observed AVAR that takes place in continuous time, thereby improving accuracy and simplifying both implementation and theoretical analysis.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

为了实现大数据革命的承诺，当前的项目将重点关注一种常见的数据形式，即高维高频数据（HDHFD），其中数据快照涉及大量变量，同时又包含新的数据。随着数据收集技术的进步，HDHFD 出现在从神经科学到患者护理的医学应用中；诸如地震数据的海洋科学；以及地区首席研究员（PI）的研究重点是如何从复杂的大数据中提取信息以及如何将数据转化为知识，该项目旨在开发尖端的数学和统计方法来揭示问题。该结构的特点是跨时间和维度的依赖网络，分析的作用是为如何降低复杂性提供指导，同时保留数据架构的重要组成部分。研究还涉及如何量化估计和预测中的不确定性除了开发一般理论外，该项目还涉及金融数据的应用，包括风险管理、预测和投资组合管理，更精确的估算器以及改进的误差范围将在所有这些领域发挥作用。研究结果引起了主流投资者、监管机构和政策制定者的兴趣，并且该结果完全属于公共领域，该项目的目的是从多个角度探索高维高频数据（HDHFD）。方法是扩展 PI 的邻接理论。在邻域中，观测结果的结构通常更容易理解（通常是高斯分布），这有助于在不改变当前模型的情况下实现统计分析。数据方面，PI 将探索时变矩阵分解，包括开发高频数据的奇异值分解 (SVD)，作为通往因子模型的更直接途径。我们计划比较新的模型。基于 PCA 的方法以及 L1 类型方法（例如非负矩阵分解）的 SVD 发现了一种查看时间和跨维度依赖性的新方法，最初是由 PI 结合观察到的渐近方差（观察到的 AVAR）开发的。 ）。他们现在将研究跨时间和维度“借用”信息的可能性，该工具将用于矩阵分解，以及为金融的漂移部分开发波动率矩阵。过程，该过程将与他们计划的矩阵分解工作相结合，探索一条观察连续时间发生的 AVAR 的路径，从而提高准确性并简化实施和理论分析。该奖项反映了 NSF 的法定使命，并已得到批准。通过使用基金会的智力优点和更广泛的影响审查标准进行评估，认为值得支持。

项目成果

The Observed Asymptotic Variance: Hard edges, and a regression approach

观察到的渐近方差：硬边和回归方法

• DOI: 10.1016/j.jeconom.2020.07.008
• 发表时间: 2021
• 期刊: Journal of Econometrics
• 影响因子: 6.3
• 作者: Mykland, Per A.;Zhang, Lan
• 通讯作者: Zhang, Lan

A CLT for second difference estimators with an application to volatility and intensity

• DOI: 10.1214/22-aos2176
• 发表时间: 2020-11
• 期刊: The Annals of Statistics
• 作者: E. A. Stoltenberg;P. Mykland;Lan Zhang

The Five Trolls Under the Bridge: Principal Component Analysis With Asynchronous and Noisy High Frequency Data

桥下的五个巨魔：异步和噪声高频数据的主成分分析

• DOI: 10.1080/01621459.2019.1672555
• 发表时间: 2020
• 期刊: Journal of the American Statistical Association
• 影响因子: 3.7
• 作者: Chen, Dachuan;Mykland, Per A.;Zhang, Lan
• 通讯作者: Zhang, Lan