SGER: Multimodal Information Retrieval from Large Multimedia Databases

SGER：大型多媒体数据库的多模式信息检索

• 批准号: 0634695
• 负责人: Avinash Kak
• 金额: --
• 依托单位: Purdue University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2006
• 资助国家: 美国
• 起止时间: 2006-09-01 至 2008-02-29
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0634695&HistoricalAwards=false
• 关键词: SGER; Multimodal; Information; Retrieval; Large

This project explores a novel approach to multimodal information retrieval from multimedia databases. Self-diagnosis and personal health monitoring for skin diseases serves as the motivation driving this research as well as a setting for the evaluation. The multimedia database consists of both images (visual symptoms) and textual information (such as patient history). An individual interacts with the system via a multimodal interface. He/she uploads digital photographs of his/her visual symptoms and enters additional textual information pertaining to their symptoms and general health information. The retrieval consists of (anonymized) cases similar to his/hers (images along with the associated disease identifications/description), their possible prognoses, the need for immediate physician consult, etc. The key to the success of the retrieval engine lies in the novel indexing of the cases on the basis of both image content and text, along with the definition of the similarity metric used for retrieval. This metric must optimize over both image and text content. It needs to be noted that the system that will be explored in this project does not aim to replace the physician. Its aim is to better inform the individuals and assist them in monitoring the symptoms exhibited by their medical condition. The broader impacts of this work include a novel approach to patient participation in the diagnosis process, as well as an individual's life expectancy when early diagnosis is critical for successful treatment. The project's Web site (http://rvl1.ecn.purdue.edu/~cbirdev/multimodal.html) will be used for disseminating fundamental results in multimodal retrieval of multimedia information as well as provide access to the experimental system.

该项目探索了一种从多媒体数据库中检索多模式信息的新方法。对皮肤疾病的自我诊断和个人健康监测是推动这项研究的动机，也是评估的设置。多媒体数据库包括图像（视觉症状）和文本信息（例如患者历史记录）。一个人通过多模式接口与系统交互。他/她上传了其视觉症状的数字照片，并输入了与其症状和一般健康信息有关的其他文本信息。该检索由（匿名的）案例（匿名）案件（图像以及相关的疾病识别/描述），可能的预后，对立即医师咨询的需求等。检索引擎成功的关键在于根据图像内容和文本的基础进行案例的新索引，以及用于检索的相似性的定义。该指标必须在图像和文本内容上优化。需要注意的是，将在该项目中探索的系统并非旨在替代医生。其目的是更好地告知个人，并帮助他们监测其医疗状况表现出的症状。这项工作的更广泛的影响包括一种新型的患者参与诊断过程的方法，以及当早期诊断对于成功治疗至关重要时，个人的预期寿命。该项目的网站（http://rvl1.ecn.purdue.edu/~cbirdev/multimodal.html）将用于传播多媒体信息多模式检索的基本结果，并提供对实验系统的访问。