A Document Processing System

文档处理系统

• 批准号: 8344939
• 负责人: Willy Wilbur
• 金额: $ 7.99万
• 依托单位: NATIONAL LIBRARY OF MEDICINE
• 依托单位国家: 美国
• 资助国家: 美国
• 起止时间: 至
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/8344939
• 关键词: Algorithms; Code; Data; Databases; Literature; MEDLINE; Machine Learning; Maps; Memory; Methods; Online Systems; Process; Programming Languages; PubMed; Records; Research; Retrieval; Speed; Structure; System; Testing; Text; Work; base; computerized data processing; repository; software systems

A system of C++ language programs has been developed for the purpose of finding the closely related documents in Medline and for the purpose of performing machine learning on sets of documents. The system has a number of unique features: 1) It is based on a number of C++ classes and highly modular so that alterations in the system are relatively simple to perform. 2) The system currently processes PubMed data by extracting from the Sybase repositories using a C++ interface to Sybase. However, a change in the interface portion of the system would allow it to be applied to any large database consisting of discrete textual records. 3) Data processed by the system is stored as compressed file structures, etc. These structures are updatable so that new data may be continually added to the system as it becomes available. 4) Documents are compared with each other using a Bayesian form of analysis. 5) The latest work on this system has involved adding the ability to generate themes using an EM algorithm approach. Also recently code has been multithreaded and memory mapping capabilities added to speed up processing. The system described here is now not only being used to process all of MEDLINE for our research purposes, but also to produce the related documents for arbitrary pieces of text by other groups here in the NLM and outside of the NLM. The system is currently proving useful in testing different retrieval parameters and methods on the PubMedHealth records.

已经开发了C ++语言程序的系统，目的是在MEDLINE中查找密切相关的文档，并为了在一组文档上执行机器学习。该系统具有许多独特的功能：1）它基于许多C ++类，并且高度模块化，因此系统中的更改相对较简单。 2）该系统当前通过使用SYBASE的C ++接口从SYBASE存储库中提取PubMed数据。但是，系统的接口部分的更改将允许将其应用于由离散文本记录组成的任何大型数据库。 3）系统处理的数据被存储为压缩文件结构等。这些结构可更新，因此可以在系统中不断添加新数据。 4）使用贝叶斯的分析形式将文档相互比较。 5）该系统的最新工作涉及使用EM算法方法添加生成主题的能力。最近，代码也是多线程，并添加了内存映射功能以加快处理。 现在，此处描述的系统不仅用于为我们的研究目的处理所有MEDLINE，而且还用于制作NLM和NLM外部其他组的任意文本的相关文档。目前，该系统在PubMedHealth记录上测试不同的检索参数和方法有用。