III: Medium: Collaborative Research: Algorithms and Cyberinfrastructure for High-Precision Automated Quality Control of Hydro-Meteo Sensor Networks

III：媒介：合作研究：Hydro-Meteo 传感器网络高精度自动化质量控制的算法和网络基础设施

• 批准号: 1514550
• 负责人: Thomas Dietterich
• 金额: $ 63.55万
• 依托单位: Oregon State University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-09-01 至 2019-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1514550&HistoricalAwards=false
• 关键词: III; Medium; Collaborative; Research; Algorithms

Advances in sensor technology are greatly expanding the range of quantities that can be measured while simultaneously reducing the cost. However, deployed sensors drift out of calibration and fail, so every sensor network requires quality control (QC) procedures to promptly detect these failures. Existing QC methods rely on human experts to carefully examine the data, which means that when the number of sensors in a network doubles, the number of experts must double too. This project will develop algorithms and software to increase the level of automation in sensor QC so that a smaller number of experts can manage a much larger network of sensors. The methods will be tested on weather data from Oklahoma (the Oklahoma Mesonet), Oregon (the Andrews Long-Term Ecological Network site), the US (the Earth Networks "WeatherBug" network), and sub-Saharan Africa (the TAHMO project), and if the methods are found to work well, they will be deployed in these networks at at the CUAHSI Water Data Center. Accurate weather data could significantly increase the productivity of farms and improve food security, particularly in Africa.The project will develop an open-source standards-compliant system, SENSOR-DX, that implements automated data QC. Existing probabilistic QC methods assume that correct sensor readings are jointly Gaussian and readings from broken sensors obey a uniform distribution. These assumptions lead to many QC mistakes. This project will develop a new approach in which novel nonparametric anomaly detection algorithms analyze the sensor data. Correct sensor readings have low anomaly scores, while broken sensor readings have high scores; both follow parametric distributions. Probabilistic methods can therefore model the distribution of the resulting anomaly scores instead of the joint distribution of the original sensor readings and infer (probabilistically) whether each sensor is working correctly. To enhance the fault-detection capability of the anomaly detection algorithms, the raw sensor data will be detrended and assembled into multiple views that highlight various correlations among sensor values. The project will develop a novel View-Anomaly-Diagnosis (VAD) framework in which anomaly detection algorithms are applied to the tuples in each view, and then the anomaly scores are combined via a probabilistic diagnostic model to infer which sensors are broken and which are functioning correctly. The project will study how good the detrending models need to be in order to enhance the accuracy of anomaly detection. The new anomaly detection algorithms are based on a new anomaly detection principle: "anomaly detection by overfitting". Existing methods fit a statistical model to "normal" behavior and then identify data points that do not fit well ("are underfit") and mark them as anomalies. The new principle measures how easy it is to "overfit" a statistical model that separates candidate anomalies from the rest of the data. The project will develop new algorithms based on this principle and understand how they relate to existing methods of anomaly detection by underfitting. The VAD framework will be implemented in the SENSOR-DX system: a series of Kepler workflows that provide support for connecting a new sensor network, training the detrending and anomaly detection models, performing real-time anomaly detection, and repairing bad sensor readings using predictive models. SENSOR-DX will also support semantic matching of new sensor data streams by extending the EnvThs controlled vocabulary thesaurus.For further information see the project web site at http://tahmo.org/sensor-dx

传感器技术的进步极大地扩大了可测量的数量范围，同时降低了成本。然而，部署的传感器会偏离校准并发生故障，因此每个传感器网络都需要质量控制 (QC) 程序来及时检测这些故障。现有的质量控制方法依赖人类专家仔细检查数据，这意味着当网络中的传感器数量加倍时，专家的数量也必须加倍。该项目将开发算法和软件，以提高传感器质量控制的自动化水平，以便更少的专家可以管理更大的传感器网络。这些方法将在俄克拉荷马州（俄克拉荷马州 Mesonet）、俄勒冈州（安德鲁斯长期生态网络站点）、美国（地球网络“WeatherBug”网络）和撒哈拉以南非洲地区（TAHMO 项目）的天气数据上进行测试，如果发现这些方法效果良好，它们将部署在 CUAHSI 水数据中心的这些网络中。准确的天气数据可以显着提高农场的生产力并改善粮食安全，特别是在非洲。该项目将开发一个符合开源标准的系统 SENSOR-DX，该系统可实现自动化数据质量控制。现有的概率质量控制方法假设正确的传感器读数是联合高斯分布的，并且来自损坏传感器的读数服从均匀分布。这些假设会导致许多质量控制错误。该项目将开发一种新方法，利用新颖的非参数异常检测算法分析传感器数据。正确的传感器读数具有较低的异常分数，而损坏的传感器读数具有较高的分数；两者都遵循参数分布。因此，概率方法可以对所得异常分数的分布（而不是原始传感器读数的联合分布）进行建模，并（概率地）推断每个传感器是否正常工作。 为了增强异常检测算法的故障检测能力，原始传感器数据将被去趋势化并组装成多个视图，突出传感器值之间的各种相关性。该项目将开发一种新颖的视图异常诊断（VAD）框架，其中异常检测算法应用于每个视图中的元组，然后通过概率诊断模型组合异常分数，以推断哪些传感器损坏了，哪些传感器损坏了。正常运行。该项目将研究去趋势模型需要有多好才能提高异常检测的准确性。新的异常检测算法基于新的异常检测原理：“通过过度拟合进行异常检测”。现有方法将统计模型拟合为“正常”行为，然后识别不太拟合（“拟合不足”）的数据点并将其标记为异常。新原理衡量了将候选异常与其余数据分开的统计模型“过度拟合”的容易程度。该项目将基于这一原理开发新算法，并了解它们与现有的欠拟合异常检测方法的关系。 VAD 框架将在 SENSOR-DX 系统中实现：一系列 Kepler 工作流程，为连接新传感器网络、训练去趋势和异常检测模型、执行实时异常检测以及使用预测修复不良传感器读数提供支持。模型。 SENSOR-DX 还将通过扩展 EnvThs 受控词汇同义词库来支持新传感器数据流的语义匹配。有关更多信息，请参阅该项目网站：http://tahmo.org/sensor-dx