CSR: Small: Large-Scale Web Crawling and Spam Avoidance in Search-Engine Applications

CSR：小：搜索引擎应用中的大规模网络爬行和垃圾邮件避免

• 批准号: 1017766
• 负责人: Dmitri Loguinov
• 金额: $ 40.01万
• 依托单位: Texas A&M Engineering Experiment Station
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2010
• 资助国家: 美国
• 起止时间: 2010-09-01 至 2015-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1017766&HistoricalAwards=false
• 关键词: CSR; Small; Large; Scale; Web

Search engines and various data-mining applications commonly rely on web crawlers to navigate the web, discover valuable content, and keep it fresh. However, the enormous volume of available information and sophisticated spam techniques commonly used to deceive search engines present significant challenges in web crawling, especially in non-commercial applications such as research. The first part of this project designs efficient real-time graph-manipulation algorithms and builds a high-performance distributed web-crawler architecture that seamlessly couples the various components of Internet-scale networking, information retrieval, and graph theory. The second part creates probabilistic techniques for quick estimation of domain reputation and explores various ranking techniques to achieve better robustness against spam. The third part designs advanced budgeting mechanisms to control the crawl rate of different parts of the web at multiple levels of granularity. The project is expected to engage students at Texas A&M in research-intensive education in cross-disciplinary fields (such as data-intensive computing, networking, graph theory, distributed systems, parallel architectures, and modeling), broaden integration of web research into classroom teaching, attract undergraduate students to REU, extend participation of minority groups in engineering, stimulate collaboration among students and sharing of ideas, and permit web-related research at other institutions through publicly shared outcomes of our work.

搜索引擎和各种数据挖掘应用程序通常依赖网络爬网来浏览网络，发现有价值的内容并保持新鲜。但是，通常用于欺骗搜索引擎的大量可用信息和复杂的垃圾邮件技术在网络爬行中遇到了重大挑战，尤其是在非商业应用程序（例如研究）中。该项目的第一部分设计有效的实时图形操作算法，并构建了高性能分布式的Web-Crawler架构，该体系结构无缝地构造了Internet级网络，信息检索和图形论的各个组件。第二部分创建了概率技术，以快速估算领域的声誉，并探索各种排名技术，以实现对垃圾邮件的更好鲁棒性。第三部分设计了先进的预算机制，以控制粒度多个层次的网络不同部分的爬网速率。预计该项目将吸引得克萨斯州A＆M的学生在跨学科领域的研究密集型教育（例如数据密集型计算，网络，网络，图形理论，分布式系统，平行体系结构和建模），将网络研究扩展到课堂教学中，吸引本科生在reu中扩展课堂教学，扩展少数群体的参与，并促进其他群体的参与，并在少数群体中进行了研究，并在少数派中进行了研究，并在研究过程中进行了研究，并促进了少数派的参与，并促进了少数派的参与，并在研究过程中进行了促进，并在研究中进行了研究，并在少数群体中进行了研究，并将其参与研究，并促进了少数派的参与，并将其参与培训，并将其与众不同。我们工作的共同成果。