Integrating image and text information for biomedical information retrieval

整合图像和文本信息进行生物医学信息检索

• 批准号: 8344956
• 负责人: Dina Demner-Fushman
• 金额: $ 62.2万
• 依托单位: NATIONAL LIBRARY OF MEDICINE
• 依托单位国家: 美国
• 资助国家: 美国
• 起止时间: 至
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/8344956
• 关键词: Classification; Clinical; Clinical Research; Collection; Color; Databases; Decision Support Systems; Differential Diagnosis; Electronic Health Record; Elements; Evaluation; Eye; Goals; Graph; Hybrids; Image; Image Analysis; Industry; Information Retrieval; International; Journals; Letters; Link; Literature; MEDLINE; Machine Learning; MeSH Thesaurus; Medical; Metadata; Methods; Modality; Natural Language Processing; Ontology; Outcome; Patients; Physicians; Probability; Process; Retrieval; Roentgen Rays; Semantics; Shapes; Source; Structure; System; Techniques; Text; Training; Ultrasonography; Unified Medical Language System; United States National Library of Medicine; Visual; Work; abstracting; base; bioimaging; health record; image processing; imaging modality; improved; indexing; interest; journal article; meetings; monomethoxypolyethylene glycol; patient oriented; phrases; text searching; tool; visual search

The search for relevant and actionable information is key to achieving clinical and research goals in biomedicine. Biomedical information exists in different forms: as text and illustrations in journal articles and other documents, in images stored in databases, and as patients cases in electronic health records. In the context of this work, an image includes not only biomedical images, but also illustrations, charts, graphs, and other visual material appearing in biomedical journals, electronic health records, and other relevant databases. The project objectives may be formulated as seeking better ways to retrieve information from these entities, by moving beyond conventional text-based searching to combining both text and visual features in search queries. The approaches to meeting these objectives use a combination of techniques and tools from the fields of Information Retrieval (IR), Content-Based Image Retrieval (CBIR), and Natural Language Processing (NLP). The first objective is to improve the retrieval of biomedical literature by targeting the visual content in articles, a rich source of information not typically exploited by conventional bibliographic or full-text databases. We index these figures (including illustrations and images) using (i) text in captions and where they are mentioned in the body of the article (mentions), (ii) image features, such as color, shape, size, etc., and, if available, (iii) annotation markers within figures such as arrows, letters or symbols that are extracted from the image and correlated with concepts in the caption. These annotation markers can help isolate regions of interest (ROI) in images, the ROI being useful for improving the relevance of the figures retrieved. It is hypothesized that augmenting conventional search results with relevant images offers a richer search. Taking the retrieval of biomedical literature a step further, within the first objective our goal is to find information relevant to a patients case from the literature and then link it to the patients health record. The case is first represented in structured form using both text and image features, and then literature and EHR databases are searched for similar cases. A second objective is to find semantically similar images in image databases, an important step in differential diagnosis. We explore approaches that automatically combine image and text features in contrast to visual decision support systems (for example, VisualDx) that use only text driven menus. Such menu driven systems guide a physician to describe a patient and then present a set of images from which a clinician can select the ones most similar to the patients, and access relevant information manually linked to the images. Our methods use text and image features extracted from relevant components in a document, database, or case description to achieve our objectives. For the document retrieval task, we rely on the U.S. National Library of Medicine (NLM) developed search engine. This is a phrase-based search engine with NLMs Unified Medical Language System (UMLS) based term and concept query expansion and probabilistic relevancy ranking that exploits document structure. Optimizing these features, we create structured representations of every full-text document and all its figures. These structured documents presented to the user as search results include typical fields found in MEDLINE citations (e.g., titles, abstracts and MeSH terms), the figures in the original documents, and image-specific fields extracted from the original documents (such as captions segmented into parts pertaining to each pane in a multi-panel image, ROI described in each caption, and modality of the image). In addition, patient-oriented outcomes extracted from the abstracts are provided to the user. Automatic image annotation and retrieval objectives can be achieved in the following ways: (i) using image analysis alone; (ii) by indexing the text assigned to images; and (iii) using a combination of image and text analysis. One approach is to compute image similarity, the traditional CBIR task of finding images that are overall visually similar to a query image, using machine learning classifiers (e.g., Support Vector Machine) and fusion of class probabilities. These classifiers are trained on a variety of image features such as wavelets, edge histograms and those recommended by the MPEG-7 committee. Additional steps include describing an image by automatically detecting its modality (for example, CT, MR, X-ray, ultrasound, etc.) and generating a visual ontology, i.e., concepts assigned to image patches. Elements from the visual ontology are called visual keywords and are used to find images with similar concepts. To evaluate and demonstrate our techniques, we have developed OpenI (Open "eye"), a hybrid system combining text-based searching with an image similarity engine. Using this framework we explore alternative approaches to the problem of searching for information using a combination of visual and text features: (i) starting a text-based search of an image database, and refining the search using image features; (ii) starting a visual search using the (clinical) image of a given patient, and then linking the image to relevant information found by using visual and text features; (iii) merging the results of independent text and image searches; and (iv) starting a multimodal search that combines text and image features. In an international evaluation our approaches were shown to be among the best in image type classification, image retrieval using only visual features, ad hoc retrieval, and medical case retrieval among over a dozen teams from around the world, including several from the industry.

寻找相关和可行的信息是实现生物医学临床和研究目标的关键。生物医学信息以不同的形式存在：作为期刊文章和其他文档中的文本和插图，存储在数据库中的图像以及电子健康记录中的患者中。在这项工作的背景下，图像不仅包括生物医学图像，还包括生物医学期刊，电子健康记录和其他相关数据库中出现的插图，图表，图形和其他视觉材料。通过超越传统的基于文本的搜索，可以将项目目标从这些实体中检索信息，以寻求更好的方法来检索这些实体的信息。实现这些目标的方法结合了信息检索（IR），基于内容的图像检索（CBIR）和自然语言处理（NLP）的技术和工具。 第一个目的是通过针对文章中的视觉内容来改善生物医学文献的检索，这是常规书目或全文数据库通常不利用的丰富信息来源。我们使用（i）字幕中的文本以及文章（提及），（ii）图像特征（例如颜色，形状，大小等）中提到的这些数字（包括插图和图像）索引这些数字，以及（iii）的图像特征，（iii）注释标记，例如箭头，字母或符号，这些图形，字母或符号从图像和构想中提取的字母或符号。这些注释标记可以帮助孤立图像中感兴趣的区域（ROI），ROI对于改善检索到的数字的相关性很有用。假设使用相关图像增强传统搜索结果提供了更丰富的搜索。 将生物医学文献的检索取回进一步，在第一个目标中，我们的目标是从文献中找到与患者病例相关的信息，然后将其与患者的健康记录联系起来。该案例首先使用文本和图像特征以结构化形式表示，然后将文献和EHR数据库搜索相似的情况。 第二个目标是在图像数据库中找到语义上相似的图像，这是鉴别诊断的重要一步。我们探讨了与仅使用文本驱动菜单的视觉决策支持系统（例如VisualDx）相比，自动组合图像和文本功能的方法。这种菜单驱动的系统指导医生描述患者，然后介绍一组临床医生可以从中选择与患者最相似的图像，并手动访问与图像的相关信息。 我们的方法使用文本和图像功能从文档，数据库或案例描述中相关组件提取的文本和图像功能来实现我们的目标。对于文件检索任务，我们依靠美国国家医学图书馆（NLM）开发的搜索引擎。这是一个基于短语的搜索引擎，具有NLMS统一医学语言系统（UMLS）的术语和概念查询扩展以及利用文档结构的概率相关性排名。为了优化这些功能，我们创建了每个全文文档及其所有数字的结构化表示。这些结构化的文档作为搜索结果包括在Medline引用中发现的典型字段（例如，标题，摘要和网格术语），原始文档中的数字以及从原始文档中提取的特定图像特定字段（例如，在每个窗格中分为每个窗格中的captions caption caption caption和mod atie caption和mod of Modality caption captions tos captions captions captions captions captions captions captions captions caption tos caption tos caption caption tos caption s。此外，从摘要中提取的面向患者的结果还提供给用户。 可以通过以下方式实现自动图像注释和检索目标：（i）单独使用图像分析； （ii）用索引分配给图像的文本； （iii）结合图像和文本分析。一种方法是计算图像相似性，这是使用机器学习分类器（例如支持向量机）和类概率融合的传统CBIR任务，该任务是查找与查询图像总体相似的图像的传统任务。这些分类器经过各种图像特征的培训，例如小波，边缘直方图和MPEG-7委员会建议的分类器。其他步骤包括通过自动检测其模态来描述图像（例如CT，MR，X射线，超声等）并生成视觉本体论，即分配给图像贴片的概念。视觉本体学的元素称为视觉关键字，用于查找具有相似概念的图像。 为了评估和演示我们的技术，我们开发了Openi（开放式“眼睛”），这是一种将基于文本的搜索与图像相似性引擎相结合的混合系统。使用此框架，我们探讨了使用视觉和文本功能组合搜索信息问题的替代方法：（i）启动基于文本的图像数据库搜索，并使用图像功能来完善搜索； （ii）使用给定患者的（临床）图像开始视觉搜索，然后将图像链接到使用视觉和文本特征发现的相关信息； （iii）合并独立文本和图像搜索的结果； （iv）启动组合文本和图像特征的多模式搜索。 在国际评估中，我们的方法被证明是图像类型分类中最好的方法之一，仅使用视觉功能，临时检索和医疗案例检索来自世界各地的十几个团队，包括该行业的几个。