Randomized Feature Engineering as a Fast and Accurate Alternative to Kernel Methods

Feature engineering has found increasing interest in recent years because of its ability to improve the effectiveness of various machine learning models. Although tailored feature engineering methods have been designed for various domains, there are few that simulate the consistent effectiveness of kernel methods. At the core, the success of kernel methods is achieved by using similarity functions that emphasize local variations in similarity. Unfortunately, this ability comes at the price of the high level of computational resources required and the inflexibility of the representation as it only provides the similarity of two data points instead of vector representations of each data point; while the vector representations can be readily used as input to facilitate various models for different tasks. Furthermore, kernel methods are also highly susceptible to overfitting and noise and it cannot capture the variety of data locality. In this paper, we first analyze the inner working and weaknesses of kernel method, which serves as guidance for designing feature engineering. With the guidance, we explore the use of randomized methods for feature engineering by capturing multi-granular locality of data. This approach has the merit of being time and space efficient for feature construction. Furthermore, the approach is resistant to overfitting and noise because the randomized approach naturally enables fast and robust ensemble methods. Extensive experiments on a number of real world datasets are conducted to show the effectiveness of the approach for various tasks such as clustering, classification and outlier detection.

近年来，特征工程因其能够提高各种机器学习模型的有效性而受到越来越多的关注。尽管针对不同领域设计了特定的特征工程方法，但很少有方法能模拟核方法的持续有效性。从本质上讲，核方法的成功是通过使用强调相似性局部变化的相似性函数实现的。不幸的是，这种能力是以所需计算资源高以及表示缺乏灵活性为代价的，因为它只提供两个数据点的相似性，而不是每个数据点的向量表示；而向量表示可以很容易地用作输入，以便为不同任务的各种模型提供便利。此外，核方法也极易受到过拟合和噪声的影响，并且无法捕捉数据局部性的多样性。在本文中，我们首先分析核方法的内部工作原理和弱点，这为特征工程设计提供了指导。在该指导下，我们通过捕捉数据的多粒度局部性来探索随机方法在特征工程中的应用。这种方法在特征构建方面具有时间和空间高效的优点。此外，该方法对过拟合和噪声具有抵抗力，因为随机方法自然能够实现快速且稳健的集成方法。在多个真实世界数据集上进行了大量实验，以证明该方法对聚类、分类和异常检测等各种任务的有效性。