DC: Large: Collaborative Research: Mining a Million Scanned Books: Linguistic and Structure Analysis, Fast Expanded Search, and Improved OCR

DC：大型：协作研究：挖掘一百万本扫描书籍：语言和结构分析、快速扩展搜索和改进的 OCR

• 批准号: 0911018
• 负责人: Brewster Kahle
• 金额: $ 30.52万
• 依托单位: Internet Archive
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2009
• 资助国家: 美国
• 起止时间: 2009-10-01 至 2013-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0911018&HistoricalAwards=false
• 关键词: DC; Large; Collaborative; Research; Mining

The Center for Intelligent Information Retrieval at UMass Amherst, the Perseus Digital Library Project at Tufts, and the Internet Archive are investigating large-scale information extraction and retrieval technologies for digitized book collections. To provide effective analysis and search for scholars and the general public, and to handle the diversity and scale of these collections, this project focuses on improvements in seven interlocking technologies: improved OCR accuracy through word spotting, creating probabilistic models using joint distributions of features, and building topic-specific language models across documents; structural metadata extraction, to mine headers, chapters, tables of contents, and indices; linguistic analysis and information extraction, to perform syntactic analysis and entity extraction on noisy OCR output; inferred document relational structure, to mine citations, quotations, translations, and paraphrases; latent topic modeling through time, to improve language modeling for OCR and retrieval, and to track the spread of ideas across periods and genres; query expansion for relevance models, to improve relevance in information retrieval by offline pre-processing of document comparisons; and interfaces for exploratory data analysis, to provide users of the document collection with efficient tools to update complex models of important entities, events, topics, and linguistic features. When applied across large corpora, these technologies reinforce each other: improved topic modeling enables more targeted language models for OCR; extracting structural metadata improves citation analysis; and entity extraction improves topic modeling and query expansion.The testbed for this project is the growing corpus of over one million open-access books from the Internet Archive.

麻省大学阿默斯特分校的智能信息检索中心、塔夫茨大学的珀尔修斯数字图书馆项目和互联网档案馆正在研究数字化图书收藏的大规模信息提取和检索技术。为了为学者和公众提供有效的分析和搜索，并处理这些馆藏的多样性和规模，该项目重点改进七种连锁技术：通过单词识别提高 OCR 准确性、使用特征联合分布创建概率模型、跨文档构建特定于主题的语言模型；结构元数据提取，挖掘标题、章节、目录和索引；语言分析和信息提取，对噪声 OCR 输出进行句法分析和实体提取；推断文档关系结构，挖掘引文、引用、翻译和释义；随着时间的推移进行潜在主题建模，以改进 OCR 和检索的语言建模，并跟踪跨时期和流派的思想传播；相关性模型的查询扩展，通过文档比较的离线预处理来提高信息检索的相关性；和探索性数据分析的接口，为文档集合的用户提供有效的工具来更新重要实体、事件、主题和语言特征的复杂模型。当应用于大型语料库时，这些技术会相互加强：改进的主题建模为 OCR 提供了更有针对性的语言模型；提取结构元数据改进引文分析；实体提取改进了主题建模和查询扩展。该项目的测试平台是来自互联网档案馆的超过一百万个开放获取书籍的不断增长的语料库。