BIGDATA: F: DKM: Addressing the two V's of Veracity and Variety in Big Data

BIGDATA：F：DKM：解决大数据中的准确性和多样性这两个 V

• 批准号: 1447795
• 负责人: Nitesh Chawla
• 金额: $ 100万
• 依托单位: University of Notre Dame
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-09-01 至 2020-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1447795&HistoricalAwards=false
• 关键词: BIGDATA; DKM; Addressing; two; Veracity

Data of questionable quality have led to significantly negative economic and social impacts on organizations, leading to overrun in costs, lost revenue, and decreased efficiencies. The issues on data reliability, credibility, and provenance have become even more daunting when dealing with the variety of data, especially data that are not directly collected by an organization, but from the third-party sources such as social media, data brokers, and crowdsourcing. To address such issues, this project aims to develop a Data Valuation Engine (DVE) that solves the critical problem of data reliability, credibility and provenance, and provides accountability and quality processes right from data acquisition. The DVE leverages and innovates techniques in estimation theory, data fusion and machine learning to fill a critical gap in data accountability and quality, thereby providing a transformative step in countering the ubiquitous data quality issues found in almost every application domain from business to environment to health to national security. The DVE will be integrated in the Hadoop ecosystem and will be agnostic to the data source, application or analytics, and provided as a hosted solution to the community. The user will interact with DVE by providing the data sources and relevant data necessary to solve a problem. The DVE in this project will be developed in a largely application-independent manner. The key challenges to develop this engine include: (i) How to generate the data quality indication labels to score data sources and the content of data based on various factors such as reliability, credibility, uncertainty and confidence? (ii) How to integrate data from various sources with different labeled scores? (iii) How to robustly evaluate the proposed engine in a broad spectrum of applications that serve as a proxy of a variety of real-world scenarios? The research plan has been designed to synergistically address the above challenges with a robust evaluation plan. Given the generality of the proposed methods, models and system, the project will potentially impact variety of applications of science, engineering, and social science and have broad environmental, economic, and health benefits. The PIs will release open source software and applicable data. The PIs will also provide a hosted DVE platform for a broad user and participant base. This project is also providing students with greater exposure to the areas of big data analytics, cloud computing, data fusion and data mining, both in courses and research experiences.

质量可疑的数据给组织带来了严重的负面经济和社会影响，导致成本超支、收入损失和效率下降。在处理各种数据时，数据的可靠性、可信度和来源问题变得更加令人畏惧，尤其是那些不是由组织直接收集、而是来自社交媒体、数据经纪人和其他第三方来源的数据。众包。为了解决这些问题，该项目旨在开发一个数据评估引擎（DVE），解决数据可靠性、可信度和来源的关键问题，并从数据采集开始提供问责制和质量流程。 DVE 利用估计理论、数据融合和机器学习方面的技术并进行创新，以填补数据责任和质量方面的关键空白，从而为解决从业务到环境到健康的几乎每个应用领域中普遍存在的数据质量问题提供了变革性的一步为了国家安全。 DVE 将集成到 Hadoop 生态系统中，并且与数据源、应用程序或分析无关，并作为托管解决方案提供给社区。 用户将通过提供解决问题所需的数据源和相关数据来与 DVE 进行交互。该项目中的 DVE 将以很大程度上独立于应用程序的方式进行开发。开发该引擎的关键挑战包括：（i）如何生成数据质量指示标签，根据可靠性、可信度、不确定性和置信度等各种因素对数据源和数据内容进行评分？ (ii) 如何整合具有不同标记分数的不同来源的数据？ (iii) 如何在广泛的应用程序中稳健地评估所提出的引擎，作为各种现实场景的代理？该研究计划旨在通过强有力的评估计划协同应对上述挑战。鉴于所提出的方法、模型和系统的通用性，该项目将潜在地影响科学、工程和社会科学的各种应用，并具有广泛的环境、经济和健康效益。 PI 将发布开源软件和适用数据。 PI 还将为广泛的用户和参与者群体提供托管的 DVE 平台。该项目还让学生在课程和研究经验中更多地接触大数据分析、云计算、数据融合和数据挖掘领域。