CIF: SMALL: kNN methods for functional estimation and machine learning

CIF：SMALL：用于功能估计和机器学习的 kNN 方法

• 批准号: 2112504
• 负责人: Lifeng Lai
• 金额: $ 50万
• 依托单位: University of California-Davis
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2112504&HistoricalAwards=false
• 关键词: CIF; SMALL; kNN; methods; functional

K Nearest Neighbor (kNN) methods are a class of nonparametric statistical methods. Compared with other methods, kNN methods have several advantages. In particular, kNN methods can automatically adapt to any continuous underlying functions without relying on any specific models. In addition, kNN methods are simple to use and do not require too much parameter tuning. Furthermore, kNN methods have achieved excellent empirical results. Due to these advantages, kNN methods are widely used in a large variety of statistical problems, including functional-estimation and machine-learning problems. However, the understanding of theoretical properties of kNN methods for these applications is incomplete. As the result, there is a pressing need to investigate the theoretical properties of kNN methods. By addressing the main sources of estimation errors identified by these investigations, one can design improved kNN methods that have better performance.In this project, the investigator is investigating: 1) theoretical properties of kNN methods in functional estimation and machine learning problems; and 2) the design of improved kNN algorithms with better performance for these applications. Despite many existing studies, several theoretical problems still need further investigation. In particular: 1) The theoretical convergence rates of kNN methods for functional estimations, classification and regression, etc., are still not fully established; 2) For many applications of practical interests, it is not clear under what conditions this type of methods is optimal; 3) Most of the existing analysis of kNN methods rely on availability of independent and identically distributed training data, while in certain applications (such as those involving Markov chains) the available data are dependent; 4) While there are many applications and analysis of kNN methods for supervised learning, the applications and analysis of kNN methods for reinforcement learning etc. are limited. To address these challenges, this project is focusing on two interconnected thrusts. In the first thrust, the project is investigating the application of kNN methods in the estimation of information-theoretic functionals, including entropy, mutual information, Kullback-Leibler divergence, directed information, etc. In the second thrust, the project is designing and analyzing kNN based algorithms for machine learning problems, including supervised learning, nonconvex optimization and reinforcement learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

K 最近邻 (kNN) 方法是一类非参数统计方法。与其他方法相比，kNN 方法有几个优点。特别是，kNN 方法可以自动适应任何连续的底层函数，而不依赖于任何特定模型。此外，kNN方法使用简单，不需要太多的参数调整。此外，kNN 方法取得了出色的实证结果。由于这些优点，kNN 方法广泛应用于各种统计问题，包括函数估计和机器学习问题。然而，对这些应用的 kNN 方法的理论特性的理解并不完整。因此，迫切需要研究 kNN 方法的理论特性。通过解决这些研究中发现的估计误差的主要来源，我们可以设计出具有更好性能的改进的 kNN 方法。在这个项目中，研究人员正在研究：1）kNN 方法在函数估计和机器学习问题中的理论特性； 2) 为这些应用设计具有更好性能的改进 kNN 算法。尽管已有许多研究，但仍有一些理论问题需要进一步研究。特别是： 1）用于函数估计、分类和回归等的kNN方法的理论收敛速度尚未完全建立； 2）对于许多实际利益的应用，尚不清楚此类方法在什么条件下是最佳的； 3）现有的kNN方法分析大多依赖于独立且同分布的训练数据的可用性，而在某些应用中（例如涉及马尔可夫链的应用）可用数据是相关的； 4）虽然kNN方法在监督学习方面有很多应用和分析，但kNN方法在强化学习等方面的应用和分析却很有限。为了应对这些挑战，该项目重点关注两个相互关联的目标。在第一个主旨中，该项目正在研究 kNN 方法在信息论泛函估计中的应用，包括熵、互信息、Kullback-Leibler 散度、有向信息等。在第二个主旨中，该项目正在设计和分析基于 kNN 的机器学习问题算法，包括监督学习、非凸优化和强化学习。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Bayesian Two-Stage Sequential Change Diagnosis via Sensor Arrays

通过传感器阵列进行贝叶斯两阶段顺序变化诊断

• DOI: 10.1109/tit.2023.3298231
• 发表时间: 2023-11
• 期刊: IEEE Transactions on Information Theory
• 影响因子: 2.5
• 作者: Ma, Xiaochuan;Lai, Lifeng;Cui, Shuguang
• 通讯作者: Cui, Shuguang

Analysis of KNN Density Estimation

KNN密度估计分析

• DOI: 10.1109/tit.2022.3195870
• 发表时间: 2022-12
• 期刊: IEEE Transactions on Information Theory
• 影响因子: 2.5
• 作者: Zhao, Puning;Lai, Lifeng
• 通讯作者: Lai, Lifeng