CISE-MSI: DP: IIS:III: Deep Learning Based Automated Concept and Caption Generation of Medical Images Towards Developing an Effective Decision Support System (DSS)

CISE-MSI：DP：IIS：III：基于深度学习的医学图像自动概念和标题生成，以开发有效的决策支持系统 (DSS)

• 批准号: 2131207
• 负责人: Md Rahman
• 金额: $ 43.98万
• 依托单位: Morgan State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-01-01 至 2024-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2131207&HistoricalAwards=false
• 关键词: CISE; MSI; DP; IIS; III

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Identifying and labeling important features in medical images such as X-rays and ultrasounds is fundamental to both diagnosis itself and to building libraries of images that support education, training, and auditing of medical quality. This work is time-consuming even for trained experts, making it an impactful and important problem domain to study for researchers in computer vision, machine learning (ML), and natural language processing (NLP). These artificial intelligence (AI)-based techniques have made great progress in object recognition and labeling for everyday camera images; however, medical images pose additional challenges because of the need to account for detail and relationships between substructures in the image, the need to generate captions that apply not just to the whole image but to these important substructures, and the need to handle noise and artifacts created in medical image processing. Further, the tolerance for error is low; interpretations need to be coherent, grammatically, and semantically correct in order to be useful. This project focuses on the intersection of biomedical informatics and imaging science, working to develop high quality datasets of human-annotated visual concepts in images that appear in public collections such as open access biomedical journals, then using those datasets to train novel vision, ML, and NLP algorithms. The work will support multi-institutional research and educational collaborations between three minority-serving institutions, providing advanced research and classroom training in AI, ML, and cloud computing to students from groups historically underrepresented in computing. To improve image interpretation and retrieval effectiveness, this project will (1) create a crowdsourcing-based annotation system to clinically annotate important regions of interest (ROIs) of images; (2) advance object detection models to segment images and map medical image ROIs; (3) advance multilabel concept classification techniques by considering correlations between concepts; and (4) apply contextualized embeddings via deep language models to generate the captions. The proposed approaches will be evaluated through comparison with current methods in benchmark datasets, including the ones constructed for this project. The end goal is the development of an AI-based prototype that helps physicians focus on interesting image regions, find relevant comparison images, and describe findings in correct and standard ways, all of which can reduce medical errors and benefit both medical departments and society by reducing the cost per exam. In addition to the research objectives, the project will implement a research-education medical AI training program including cloud-enabled classrooms, cross-institutional mentoring, and partnering with an existing industry internship “pathway to success” initiative to build the science and technology workforce of the future.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该奖项的全部或部分资金来源于《2021 年美国救援计划法案》（公法 117-2）。识别和标记 X 射线和超声波等医学图像中的重要特征对于诊断本身和建立图书馆至关重要即使对于训练有素的专家来说，这项工作也很耗时，这使其成为计算机视觉、机器学习 (ML) 和自然语言研究人员研究的一个有影响力且重要的问题领域。加工（NLP）。这些基于人工智能（AI）的技术在日常相机图像的对象识别和标记方面取得了巨大进步；然而，由于需要考虑图像中子结构之间的细节和关系，医学图像带来了额外的挑战，需要生成不仅适用于整个图像而且适用于这些重要子结构的说明，并且需要处理医学图像处理中产生的噪声和伪影此外，对错误的容忍度需要在语法上保持一致；并按顺序语义正确该项目专注于生物医学信息学和成像科学的交叉点，致力于开发开放获取生物医学期刊等公共收藏中出现的图像中人工注释视觉概念的高质量数据集，然后使用这些数据集来训练新颖的图像。这项工作将支持三个少数族裔服务机构之间的多机构研究和教育合作，为来自计算领域历史上代表性不足的群体的学生提供人工智能、机器学习和云计算方面的高级研究和课堂培训。为了提高图像解释和检索效率，该项目将（1）创建基于众包的注释系统，以临床注释图像的重要感兴趣区域（ROI）；（2）改进对象检测模型以分割图像并映射医学图像 ROI； （3）通过考虑概念之间的相关性来推进多标签概念分类技术；（4）通过深度语言模型应用上下文嵌入来生成标题，将通过与基准数据集中的当前方法进行比较来评估所提出的方法。该项目的最终目标是开发基于人工智能的原型，帮助医生关注有趣的图像区域，找到相关的比较图像，并以正确和标准的方式描述结果，所有这些都可以减少医疗错误并受益。通过降低每次考试的成本，该项目还将实施研究教育医学人工智能培训计划，包括云课堂、跨机构指导以及与现有的行业实习合作。成功之路”倡议建立科学该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Concept Detection and Caption Prediction in ImageCLEFmedical Caption 2023 with Convolutional Neural Networks; Vision and Text-to-Text Transfer Transformers

使用卷积神经网络进行 ImageCLEFmedical Caption 2023 中的概念检测和字幕预测；

• 发表时间: 2023-10
• 期刊: Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023
• 作者: Hasan M;Layode O;Rahman M.
• 通讯作者: Rahman M.

CS_Morgan at ImageCLEFmedical 2022 Caption Task: Deep Learning Based Multi-Label Classification and Transformers for Concept Detection & Caption Prediction

CS_Morgan 在 ImageCLEFmedical 2022 标题任务：基于深度学习的多标签分类和用于概念检测的 Transformers

• 发表时间: 2022-08
• 期刊: CEUR Workshop Proceedings (CEUR-WS.org
• 作者: Rahman, Md M.;Layode, O.
• 通讯作者: Layode, O.

Statistical Analysis of Imbalanced Classification with Training Size Variation and Subsampling on Datasets of Research Papers in Biomedical Literature

生物医学文献研究论文数据集训练规模变化和子采样的不平衡分类统计分析

• 发表时间: 2023-12
• 期刊: Machine Learning and Knowledge Extraction
• 影响因子: 3.9
• 作者: Dixon, J;Rahman, M.
• 通讯作者: Rahman, M.

Media Interestingness Prediction in ImageCLEFfusion 2023 with Dense Architecture-based Ensemble & Scaled; Gradient Boosting Regressor Model

使用基于密集架构的集成在 ImageCLEFfusion 2023 中进行媒体兴趣度预测

• 发表时间: 2023-10
• 期刊: CEUR workshop proceedings
• 作者: Emon, M;Rahman, M.
• 通讯作者: Rahman, M.