RI: Medium: Collaborative Research: Write A Classifier: Learning Fine-Grained Visual Classifiers from Text and Images

RI：媒介：协作研究：编写分类器：从文本和图像中学习细粒度视觉分类器

• 批准号: 1409683
• 负责人: Ahmed Elgammal
• 金额: $ 50万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-06-15 至 2021-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1409683&HistoricalAwards=false
• 关键词: RI; Medium; Collaborative; Research; Write; RI; Medium; Collaborative; Research; Write

This project develops the learning strategy using textual narrative and images makes the learning effective without a huge number of images that a typical visual learning algorithm would need to learn the class boundaries. The research team investigates computational models for joint learning of visual concepts from images and textual descriptions of fine-grained categories, for example, discriminating between bird species. The research activities have broader impact in three fields: computer vision, natural language processing, and machine learning. There is a huge need to develop algorithms to automatically understand the content of images and videos, with numerous potential applications in web searches, image and video archival and retrieval, surveillance applications, robot navigation and others. There are various applications for developing an intelligent system that can use narrative to define and recognize categories.This project addresses two research questions: First, given a visual corpus and a textual corpus about a specific domain, how to jointly and effectively learn visual concepts? Second, given these two modalities how to facilitate learning novel visual concepts using only pure textual descriptions of novel categories in the domain? The research team approaches the problem on three integrated fronts: Learning, Natural Language Processing (NLP), and Computer Vision. On the learning front, the project investigates and develops algorithms suitable for learning and predicting visual classifiers with side textual information. On the NLP front, the project aims to develop novel methods for learning global and local discriminative category-level attributes and their values from text, with feedback from human computation and visual signal. The project investigates supervised and unsupervised methods for detecting visual text, and learning methods for deep language understanding to build such rich domain models from the noisy visual text. On the Vision front, the project addresses the tasks of detection and classification with side textual information. The project investigates models for the shape and appearance of a general category that can specialize to different subordinates, in a way that allows interpreting information from text within a proper geometric context, and handle variability in viewpoints and articulation.

该项目使用文本叙述制定学习策略，图像使学习有效，而无需大量的图像，而典型的视觉学习算法将需要学习类界限。 研究小组研究了从图像和细颗粒类别的文本描述中联合学习视觉概念的计算模型，例如，区分鸟类。 研究活动在三个领域具有更大的影响：计算机视觉，自然语言处理和机器学习。需要开发算法来自动理解图像和视频的内容，并且在Web搜索，图像和视频档案和检索，监视应用程序，机器人导航等中具有许多潜在的应用程序。开发一个可以使用叙述来定义和识别类别的智能系统有各种应用程序。该项目解决了两个研究问题：首先，给定视觉语料库和有关特定领域的文本语料库，如何共同有效地学习视觉概念？其次，鉴于这两种方式如何仅使用域中新颖类别的纯文本描述来促进学习新颖的视觉概念？研究团队在三个综合方面解决了问题：学习，自然语言处理（NLP）和计算机视觉。在学习方面，该项目研究和开发了适合通过侧面文本信息学习和预测视觉分类器的算法。在NLP方面，该项目旨在开发新的方法，以通过人类计算和视觉信号的反馈来学习全球和局部歧视类别级别的属性及其值。该项目调查了被监督和无监督的方法，用于检测视觉文本，以及学习深入的语言理解的学习方法，以从嘈杂的视觉文本中构建这种丰富的领域模型。在视觉方面，该项目通过侧面文本信息解决了检测和分类的任务。该项目研究了可以专门针对不同下属的一般类别的形状和外观的模型，该模型允许在适当的几何环境中解释文本的信息，并处理观点和表达中的可变性。