CRII: RI: Automatically Understanding the Messages and Goals of Visual Media

CRII：RI：自动理解视觉媒体的信息和目标

• 批准号: 1566270
• 负责人: Adriana Kovashka
• 金额: $ 17.46万
• 依托单位: University of Pittsburgh
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-06-01 至 2019-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1566270&HistoricalAwards=false
• 关键词: CRII; RI; Automatically; Understanding; Messages

This project develops technologies to interpret the visual rhetoric of images. The project advances computer vision through novel solutions to the novel problem of decoding the visual messages in advertisements and artistic photographs, and thus brings computer vision closer to its goal of being able to automatically understand visual content. From a practical standpoint, understanding visual rhetoric can be used to produce image descriptions for the visually impaired that align with how a human would label these images, and thus give them access to the rich content shown in newspapers or on TV. This project is tightly integrated with education. The work is interdisciplinary and can attract undergraduate students to the research from different fields. This research focuses on three media understanding tasks: (1) understanding the persuasive messages conveyed by artistic images and the strategies that those images use to convey their message; (2) exposing a photographer's bias towards their subject, e.g., determining whether a photograph portrays its subject in a positive or negative light; and (3) predicting what part of an artistic photograph a viewer might find most captivating or poignant. To enable decoding of artistic images, a large dataset is collected and annotated with a number of artistic properties and persuasion techniques that are intended for human understanding, then methods are developed to model visual symbolism in artistic images, as well as adapt positive/negative effect methods from sentiment analysis. To predict the photographer's bias towards a subject, a dataset of historical and modern portrayals of minorities and foreigners is collected, then an algorithm is created that reasons about body language and 3D layout and composition of the photo. To predict poignance, eyetracking data on a set of artistic images from famous photographers is collected, then semantic and connotation conflicts between the objects in the photographs are analyzed.

该项目开发解释图像视觉修辞的技术。该项目通过解决广告和艺术照片中视觉信息解码的新问题的新颖解决方案来推进计算机视觉的发展，从而使计算机视觉更接近能够自动理解视觉内容的目标。从实践的角度来看，理解视觉修辞可以用来为视障人士生成与人类标记这些图像的方式一致的图像描述，从而使他们能够访问报纸或电视上显示的丰富内容。该项目与教育紧密结合。这项工作是跨学科的，可以吸引不同领域的本科生进行研究。本研究重点关注三个媒体理解任务：（1）理解艺术图像传达的说服性信息以及这些图像用来传达信息的策略； (2) 暴露摄影师对其拍摄对象的偏见，例如确定照片是否以正面或负面的角度描绘其拍摄对象； (3) 预测艺术照片中观众可能觉得最迷人或最深刻的部分。为了能够解码艺术图像，收集了一个大型数据集，并用许多旨在供人类理解的艺术属性和说服技术进行注释，然后开发了方法来对艺术图像中的视觉象征意义进行建模，并适应积极/消极效果情感分析的方法。为了预测摄影师对某个主题的偏见，我们收集了少数民族和外国人的历史和现代肖像数据集，然后创建了一种算法，可以对照片的肢体语言、3D 布局和构图进行推理。为了预测痛苦，收集了著名摄影师的一组艺术图像的眼动数据，然后分析照片中对象之间的语义和内涵冲突。

项目成果

Predicting Visual Political Bias Using Webly Supervised Data and an Auxiliary Task

• DOI: 10.1007/s11263-021-01506-3
• 发表时间: 2021-08
• 期刊: International Journal of Computer Vision
• 影响因子: 19.5
• 作者: Christopher Thomas;Adriana Kovashka

Learning to Overcome Noise in Weak Caption Supervision for Object Detection

• DOI: 10.1109/tpami.2022.3187350
• 发表时间: 2022-06
• 期刊: IEEE Transactions on Pattern Analysis and Machine Intelligence
• 影响因子: 23.6
• 作者: Mesut Erhan Unal;Keren Ye;Mingda Zhang;Christopher Thomas;Adriana Kovashka;Wei Li;Danfeng Qin