RI: Medium: Collaborative Research: Text-to-Image Reference Resolution for Image Understanding and Manipulation

RI：媒介：协作研究：用于图像理解和操作的文本到图像参考分辨率

• 批准号: 1563727
• 负责人: Svetlana Lazebnik
• 金额: $ 55万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-06-01 至 2021-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1563727&HistoricalAwards=false
• 关键词: RI; Medium; Collaborative; Research; Text

This project develops new technologies at the interface of computer vision and natural language processing to understand text-to-image relationships. For example, given a captioned image, the project develops techniques which determine which words (e.g. "woman talking on phone", "The farther vehicle") correspond to which image parts. From robotics to human-computer interaction, there are numerous real-world tasks that benefit from practical systems to identify objects in scenes based on language and understand language based on visual context. In particular, the project develops the first language-based image authoring tool which allows users to edit or synthesize realistic imagery using only natural language (e.g. "delete the garbage truck from this photo" or "make an image with three boys chasing a shaggy dog"). Beyond the immediate impact of creating new ways for users to access and author digital images, the broader impacts of this work include three focus areas: the development of new benchmarks for the vision and language communities, outreach and undergraduate research, and leadership in promoting diversity. At the core of the project are new techniques for large-scale text-to-image reference resolution (TIRR) that enable systems to automatically identify the image regions that depict entities described in natural language sentences or commands. These techniques advance image interpretation by enabling systems to perform partial matching between images and sentences, referring expression understanding, and image-based question answering. They also advance image manipulation by enabling systems that can synthesize images starting from a textual description, or modify images based on natural language commands. The main technical contributions of the project are: (1) benchmark datasets for TIRR with comprehensive large-scale gold standard annotations that will make TIRR a standard task for recognition; (2) principled new representations for text-to-image annotations that expose the compositional nature of language using the formalism of the denotation graph; (3) new models for TIRR that perform an explicit alignment (grounding) of words and phrases to image regions guided by the structure of the denotation graph; (4) applications of TIRR methods to referring expression understanding and visual question answering; and (5) applications of TIRR to image creation and manipulation based on natural language input.

该项目在计算机视觉和自然语言处理的界面上开发了新技术，以了解文本对图像的关系。例如，在给定图像的情况下，该项目开发了确定哪个单词（例如，“在电话上说话”，“更远的车辆”）的技术对应于哪个图像零件。从机器人技术到人类计算机的互动，都有许多现实世界中的任务受益于实际系统，可以根据语言识别场景中的对象，并基于视觉上下文理解语言。特别是，该项目开发了基于第一语言的图像创作工具，该工具允许用户仅使用自然语言编辑或综合现实的图像（例如，“从这张照片中删除垃圾卡车”或“与三个男孩一起追逐毛茸茸的狗的图像”）。除了为用户访问和作者数字图像创造新的方法的直接影响外，这项工作的更广泛影响还包括三个重点领域：为愿景和语言社区的新基准制定，外展和本科研究，以及促进多样性的领导力。该项目的核心是大规模文本到图像参考分辨率（TIRR）的新技术，该技术使系统能够自动识别描述自然语言句子或命令中描述的实体的图像区域。这些技术通过使系统能够在图像和句子之间执行部分匹配，参考表达理解以及基于图像的问题回答来推动图像解释。他们还通过启用可以从文本描述开始合成图像的系统来推动图像操纵，或者根据自然语言命令修改图像。该项目的主要技术贡献是：（1）具有全面的大规模黄金标准注释的TIRR基准数据集，这将使TIRR成为识别的标准任务； （2）使用eNotation图的形式主义来揭示语言的构成性质的文本到图像注释的原则新表示； （3）TIRR的新模型，这些模型对图像区域的图像区域进行了显式比对（接地），以表示图形的结构为指导； （4）TIRR方法的应用来参考表达理解和视觉问题回答； （5）TIRR在基于自然语言输入的图像创建和操作中的应用。