CHS:Small:Utilizing synergy between human and computer information processing for complex visual information organization and use

CHS:Small：利用人与计算机信息处理之间的协同作用来组织和使用复杂的视觉信息

基本信息

批准号：
1814450
负责人：
Qi Yu
金额：
$ 49.74万
依托单位：
Rochester Institute of Tech
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2018
资助国家：
美国
起止时间：
2018-07-15 至 2024-06-30
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1814450&HistoricalAwards=false
关键词：
CHS Small Utilizing synergy between

项目摘要

CHS: Small: Utilizing synergy between human and computer information processing for complex visual information organization and useRecent years have brought important advances in the use of computational approaches to automatically extract the meaning of an image (aka, image semantics). But, understanding these images when they relate to specialized areas such as medicine is significantly more challenging because it depends on human expertise. This project brings together human and computer capabilities to discover image semantics by (1) encoding human image inspection and analysis behaviors that represent domain expertise, and (2) algorithmically fusing human expertise with image data. The outcomes will help to provide truly meaningful interpretations of complex images in areas such as medicine, science, and security intelligence. This interdisciplinary project will provide extensive research opportunities for undergraduate and graduate students and for broadening participation in computing. The team will leverage the college?s successful program for Women in Computing and PhD Program that has a strong track record in recruiting students from underrepresented and culturally-diverse groups. The research will contribute novel computational models to capture the complex and unique features of human language and vision related to performing image understanding tasks, and an innovative probabilistic framework to fuse human knowledge data with image features. Interpretable knowledge patterns will be extracted to inform high-level abstractions of human expertise and establish cross-modality relationships. The hierarchical probabilistic framework will promote a systematic fusion of multimodal knowledge data with image content. By fusing data from multiple, complimentary modalities, the framework is robust to sparseness, noise, and ambiguity in human knowledge data while being flexible when one or more data modalities become unavailable. Through nonparametric modeling, the framework can account for the novel semantics resulting from human expertise, hence closely represent the knowledge-based processing in human image understanding.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

CHS：小型：利用人类和计算机信息处理之间的协同作用来组织和使用复杂的视觉信息近年来，在使用计算方法自动提取图像含义（又称图像语义）方面取得了重要进展。但是，当这些图像与医学等专业领域相关时，理解这些图像显然更具挑战性，因为它取决于人类的专业知识。该项目汇集了人类和计算机的能力，通过以下方式发现图像语义：(1) 对代表领域专业知识的人类图像检查和分析行为进行编码，以及 (2) 通过算法将人类专业知识与图像数据融合。研究结果将有助于对医学、科学和安全情报等领域的复杂图像提供真正有意义的解释。这个跨学科项目将为本科生和研究生提供广泛的研究机会，并扩大对计算的参与。该团队将利用学院成功的计算机女性项目和博士项目，该项目在招收代表性不足和文化多元化群体的学生方面拥有良好的记录。该研究将贡献新颖的计算模型来捕获与执行图像理解任务相关的人类语言和视觉的复杂而独特的特征，以及将人类知识数据与图像特征融合的创新概率框架。将提取可解释的知识模式，以告知人类专业知识的高级抽象并建立跨模态关系。分层概率框架将促进多模态知识数据与图像内容的系统融合。通过融合来自多种互补模态的数据，该框架对人类知识数据的稀疏性、噪声和模糊性具有鲁棒性，同时在一种或多种数据模态不可用时保持灵活性。通过非参数建模，该框架可以解释人类专业知识产生的新颖语义，从而密切代表人类图像理解中基于知识的处理。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值进行评估，认为值得支持以及更广泛的影响审查标准。

项目成果

期刊论文数量（26）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

Integrating Bayesian and Discriminative Sparse Kernel Machines for Multi-class Active Learning

DOI：
发表时间：
2019
期刊：
影响因子：
0
作者：
Weishi Shi;Qi Yu
通讯作者：
Weishi Shi;Qi Yu

Knowledge Acquisition for Human-In-The-Loop Image Captioning

DOI：
发表时间：
2023
期刊：
影响因子：
0
作者：
Ervine Zheng;Qi Yu;Rui Li;Pengcheng Shi;Anne R. Haake
通讯作者：
Ervine Zheng;Qi Yu;Rui Li;Pengcheng Shi;Anne R. Haake

An Efficient Many-Class Active Learning Framework for Knowledge-Rich Domains

知识丰富领域的高效多类主动学习框架

DOI：
10.1109/icdm.2018.00164
发表时间：
2018
期刊：
2018 IEEE International Conference on Data Mining (ICDM
影响因子：
0
作者：
Shi, Weishi;Yu, Qi
通讯作者：
Yu, Qi

Integrating Multi-level Tag Recommendation with External Knowledge Bases for Automatic Question Answering

多级标签推荐与外部知识库融合实现自动问答

DOI：
10.1145/3319528
发表时间：
2019
期刊：
ACM Transactions on Internet Technology
影响因子：
5.3
作者：
Lima, Eduardo;Shi, Weishi;Liu, Xumin;Yu, Qi
通讯作者：
Yu, Qi

Balancing Bias and Variance for Active Weakly Supervised Learning

DOI：
10.1145/3534678.3539264
发表时间：
2022-06
期刊：
Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
影响因子：
0
作者：
Hitesh Sapkota;Qi Yu
通讯作者：
Hitesh Sapkota;Qi Yu