Large-Scale Web Research Testbed

大规模网络研究测试平台

• 批准号: 0322975
• 负责人: Hector Garcia-Molina
• 金额: $ 43.98万
• 依托单位: Stanford University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2003
• 资助国家: 美国
• 起止时间: 2003-09-15 至 2006-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0322975&HistoricalAwards=false
• 关键词: Large; Scale; Web; Research; Testbed

This project, establishing a very large repository of current and historical Web content, supports two group research efforts: Management of the Web content and Analysis and mining of the content.The current facilities have been instrumental in examining many aspects of the World Wide Web (WWW). These aspects include experimentation toward understanding, optimally utilizing, and improving the Web. The facility has enabled researchers to try out various hypothesis and techniques for indexing and modeling WWW information. The system's highly configurable crawlers collect a large number of Web pages, storing them locally for testing novel algorithms, such as ranking, filtering, or Web linkage mapping on the collection. The current WebBase is underpowered; for example, the crawling speeds are limited by CPU performance (retrieved pages are compressed before being stored), and often by virtual memory space. Removing these two bottlenecks will enable sustaining a higher Web sample rate and covering larger areas of the Web. An upgraded testbed, developed by scaling up in size and processing speed of the current hardware facilities of an existing system called WebBase, will be used to study and evaluate different Web crawling, archive refreshing, data compression, and storage and indexing techniques. Moreover, the project investigates problems related to data extraction, semantic search, searching for non-text objects, access control, cross-temporal analysis, and mining patterns or relationships between entities. Problems to be addressed include: How to Collect ever-growing amount of Web data, and keep it up to date, Provide improved search capabilities over such data, better exploiting the semantics of data and user requests, Efficiently process high-volume real-time data streams, Organize a Web archive that captures the "history" of the Web, and Deal with new types of sources (e.g., the hidden web or chat rooms) and new types of data (e.g., images).In addition, the new WebBase facility will support teaching at various universities by providing a testbed where the students can develop new searching, indexing, and user presentation ideas. WebBase draws together faculty in the areas of data mining, security, natural language processing, and database systems; consequently, the areas enhance each other. Thus, the infrastructure will support: Experimental research in a critical area: management and exploration of Web information; Researchers at institutions that do not have sufficient facilities for large-scale Web crawling; and Teaching of courses on information retrieval and data mining.

该项目建立了当前和历史网络内容的非常大的存储库，它支持了两个小组研究工作：网络内容和分析以及内容的挖掘的管理。当前的设施在研究万维网（www）的许多方面都起着重要作用。这些方面包括实验，以理解，最佳利用和改进网络。该设施使研究人员能够尝试索引和建模www信息的各种假设和技术。该系统的高度可配置的爬网器收集了大量的网页，将它们存储在本地测试新算法，例如排名，过滤或网络链接映射集合。当前的Webbase的功率不足；例如，爬行速度受到CPU性能的限制（在存储之前已检索的页面被压缩），通常是通过虚拟内存空间。去除这两个瓶颈将使维持更高的网络样本率并涵盖网络的较大区域。升级的测试台，通过扩大名为Webbase的当前硬件设施的大小和处理速度来开发，将用于研究和评估不同的网络爬行，档案清新，数据压缩以及存储以及索引技术。此外，该项目还研究了与数据提取，语义搜索，搜索非文本对象，访问控制，跨颞分析以及实体之间的采矿模式或关系有关的问题。 Problems to be addressed include: How to Collect ever-growing amount of Web data, and keep it up to date, Provide improved search capabilities over such data, better exploiting the semantics of data and user requests, Efficiently process high-volume real-time data streams, Organize a Web archive that captures the "history" of the Web, and Deal with new types of sources (e.g., the hidden web or chat rooms) and new types of data (e.g., images).In此外，新的Webbase设施将通过提供一个测试床来支持各大学的教学，学生可以在其中开发新的搜索，索引和用户演示想法。 Webbase在数据挖掘，安全性，自然语言处理和数据库系统的领域中汇集了教师；因此，这些区域相互增强。因此，基础架构将支持：关键领域的实验研究：网络信息的管理和探索；机构的研究人员没有足够的设施来进行大规模的网络爬行；以及有关信息检索和数据挖掘课程的教学。