III: Small: Fast Subset Scan for Anomalous Pattern Detection

III：小：用于异常模式检测的快速子集扫描

• 批准号: 0916345
• 负责人: Daniel Neill
• 金额: $ 50万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2009
• 资助国家: 美国
• 起止时间: 2009-08-01 至 2013-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0916345&HistoricalAwards=false
• 关键词: III; Small; Fast; Subset; Scan

This work will develop new methods for fast and scalable detection of anomalous patterns (subsets of the data that are interesting or unexpected) in massive, multivariate datasets. There will be a focus on real-world applications such as an emerging disease outbreak or a pattern of smuggling activity with complex, subtle, and probabilistic patterns that are difficult to spot with existing techniques. The research is based on two key insights. First, the pattern detection problem can be framed as a search over all subsets of the data, in which can be defined a measure of the "anomalousness" of a subset and then maximize this measure over all potentially relevant subsets. Second, it has been discovered that, for many spatial detection methods (including Kulldor's spatial scan statistic and many recently proposed variants), one can perform an exact search which efficiently maximizes the measure of anomalousness over all subsets of the data. The research team will explore this new combinatorial optimization method, investigate how it can be extended to constrained subset scans and to more general multivariate pattern detection problems, and examine how it can be incorporated into a subset scan framework, enabling the creation a variety of fast, scalable, and useful methods for anomalous pattern detection. Intellectual MeritThe research team will develop, implement, and evaluate a general probabilistic framework for efficient detection of anomalous patterns in both spatial and non-spatial datasets. The proposed work will address these challenging and important research questions:1)How can one define a useful measure of the "anomalousness" of a subset of the data, and efficiently optimize this measure over all subsets to find the most anomalous patterns?2) What are the necessary and sufficient conditions for a set function F (S ) to satisfy the "linear- time subset scanning" (LTSS) property, enabling exact unconstrained optimization of F (S ) over all 2 N subsets of N records while only requiring O(N ) subsets to be evaluated?3) How can one extend fast subset scanning methods to general multivariate datasets, and incorporate search constraints such as proximity, connectivity, and self-similarity?4) How can one deal with uncertainty about the effects of an anomalous pattern by searching over subsets of "input" and "output" attributes as well as subsets of records? Broader ImpactDevelopment and testing will be prioritized in three areas: 1) early detection of disease outbreaks, 2) detecting illicit container shipments, and 3) identifying anomalous trends in social networks. These applications will allow the demonstration the value of these methods across a wide spectrum of domains. Through existing collaborations, the algorithms will be incorporated into deployed systems for health and crime surveillance that contribute directly to the public good. The Principle Investigator's lab has over 5 years of history offering free machine learning software, and the software implementations of all algorithms developed through this grant will be made publicly available. The bulk of the funding will go to training graduate students who will become the next generation of researchers to explore new methods for anomalous pattern detection. Key Words: anomalous patterns; pattern detection; fast subset scan; scan statistics; optimization.

这项工作将开发新的方法，用于在大型多元数据集中快速，可扩展的检测异常模式（数据的子集（有趣或意外））。将重点关注现实世界中的应用，例如新兴疾病爆发或具有复杂，微妙和概率模式的走私活动模式，这些模式很难通过现有技术发现。该研究基于两个关键见解。首先，可以将模式检测问题作为对数据的所有子集进行搜索，在其中可以定义子集的“异常性”度量，然后在所有潜在相关的子集上最大化该度量。其次，已经发现，对于许多空间检测方法（包括Kulldor的空间扫描统计量和许多最近提出的变体），人们可以执行精确的搜索，从而有效地最大程度地提高了所有数据子集的异常性度量。研究团队将探索这种新的组合优化方法，研究如何将其扩展到受约束的子集扫描和更一般的多元模式检测问题，并研究如何将其纳入子集扫描框架中，从而使创建各种快速，可扩展性和有用的方法用于检测异常模式检测。知识分子的研究团队将开发，实施和评估一个普遍的概率框架，以有效检测空间和非空间数据集中的异常模式。拟议的工作将解决这些具有挑战性和重要的研究问题：1）如何定义数据子集的“异常”的有用度量，并有效地优化了该措施在所有子集上优化该措施以找到最异常的模式，以找到最必要的和足够的条件，以满足“线性 - 时间为sefss secss secans secans secans secants scan sects sects sects secants secants”（s）（s）（s）（s）（s）（s）（s）（s）（s）的范围（s）（s）（s）（s）（s）（s s）。 ) over all 2 N subsets of N records while only requiring O(N ) subsets to be evaluated?3) How can one extend fast subset scanning methods to general multivariate datasets, and incorporate search constraints such as proximity, connectivity, and self-similarity?4) How can one deal with uncertainty about the effects of an anomalous pattern by searching over subsets of "input" and "output" attributes as well as subsets of记录？在三个领域将优先考虑更广泛的影响开发和测试：1）早期发现疾病爆发，2）检测非法集装箱的运输； 3）确定社交网络中的异常趋势。这些应用将允许演示这些方法在各种域中的值。通过现有的合作，该算法将被纳入部署的健康和犯罪监视系统中，直接对公共利益做出贡献。主要调查员的实验室拥有5年的历史，可提供免费的机器学习软件，并且将公开提供通过该赠款开发的所有算法的软件实施。资金的大部分将用于培训研究生，他们将成为下一代研究人员，以探索新的模式检测方法。关键词：异常模式；模式检测；快速子集扫描；扫描统计；优化。