SBIR Phase I: Real-Time Data Analytics Over The Deep Web

SBIR 第一阶段：深网实时数据分析

• 批准号: 1248486
• 负责人: Nan Zhang
• 金额: $ 14.86万
• 依托单位: WiseAgg
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2013
• 资助国家: 美国
• 起止时间: 2013-01-01 至 2013-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1248486&HistoricalAwards=false
• 关键词: SBIR; Phase; Real; Time; Data

This Small Business Innovation Research (SBIR) Phase I project addresses the problem of real-time data analytics over deep web repositories. A major part of the deep web consists of online data repositories that are hidden behind restrictive web search interfaces and therefore cannot be effectively crawled by existing search engines such as Google. The proposed technology uses a sampling-based framework to quickly generate visible depictions of deep web analytics by issuing a small number of search requests though the existing web interfaces of deep web repositories. The specific technical objectives include the ability to 'drill into' a small subarea of interest and download the desired data with minimal query cost, as well as the ability to extract metadata information automatically from a deep web repository. The anticipated technical results are algorithms that discover the data of interest and/or metadata information after issuing a predetermined number of requests through the search interfaces of deep web repositories.The broader impact/commercial potential of this project is understood by recognizing that it democratizes the market sectors of financial, political, and market analysis which would otherwise require heavy human effort and/or computing resources such as highly paid subject matter experts or servers and storage for web crawling/indexing. Instead of incurring such high costs, the proposed technology provides customers an affordable solution for the real-time aggregate analysis of multiple deep web data repositories. More broadly, real-time data analytics over the deep web is needed by knowledge workers in a wide variety of corporations, governments, and intelligence agencies. The prospects of empowering the general public with the ability to pose high-level analytical queries over the deep web, using the enhanced scientific and technological understanding achieved through this project, are tantalizing and beneficial to the entire society at large.

这个小型企业创新研究 (SBIR) 第一阶段项目解决了深层网络存储库的实时数据分析问题。深层网络的主要部分由隐藏在限制性网络搜索界面后面的在线数据存储库组成，因此无法被现有的搜索引擎（例如 Google）有效地抓取。所提出的技术使用基于采样的框架，通过通过深网存储库的现有 Web 界面发出少量搜索请求，快速生成深网分析的可见描述。具体的技术目标包括能够“深入”感兴趣的小分区并以最小的查询成本下载所需的数据，以及从深层网络存储库自动提取元数据信息的能力。预期的技术结果是通过深网存储库的搜索界面发出预定数量的请求后发现感兴趣的数据和/或元数据信息的算法。通过认识到该项目使信息民主化，可以理解该项目更广泛的影响/商业潜力。金融、政治和市场分析等市场领域，否则需要大量的人力和/或计算资源，例如高薪主题专家或服务器以及用于网络爬行/索引的存储。所提出的技术不会产生如此高的成本，而是为客户提供了一种经济实惠的解决方案，用于对多个深层网络数据存储库进行实时聚合分析。更广泛地说，各种公司、政府和情报机构的知识工作者都需要通过深层网络进行实时数据分析。利用通过该项目获得的增强的科学和技术理解，使公众能够在深层网络上提出高级分析查询的能力，这一前景是诱人的，并且对整个社会有益。