CRII: RI: Automatically Understanding the Messages and Goals of Visual Media

CRII：RI：自动理解视觉媒体的信息和目标

• 批准号: 1566270
• 负责人: Adriana Kovashka
• 金额: $ 17.46万
• 依托单位: University of Pittsburgh
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-06-01 至 2019-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1566270&HistoricalAwards=false
• 关键词: CRII; RI; Automatically; Understanding; Messages

This project develops technologies to interpret the visual rhetoric of images. The project advances computer vision through novel solutions to the novel problem of decoding the visual messages in advertisements and artistic photographs, and thus brings computer vision closer to its goal of being able to automatically understand visual content. From a practical standpoint, understanding visual rhetoric can be used to produce image descriptions for the visually impaired that align with how a human would label these images, and thus give them access to the rich content shown in newspapers or on TV. This project is tightly integrated with education. The work is interdisciplinary and can attract undergraduate students to the research from different fields. This research focuses on three media understanding tasks: (1) understanding the persuasive messages conveyed by artistic images and the strategies that those images use to convey their message; (2) exposing a photographer's bias towards their subject, e.g., determining whether a photograph portrays its subject in a positive or negative light; and (3) predicting what part of an artistic photograph a viewer might find most captivating or poignant. To enable decoding of artistic images, a large dataset is collected and annotated with a number of artistic properties and persuasion techniques that are intended for human understanding, then methods are developed to model visual symbolism in artistic images, as well as adapt positive/negative effect methods from sentiment analysis. To predict the photographer's bias towards a subject, a dataset of historical and modern portrayals of minorities and foreigners is collected, then an algorithm is created that reasons about body language and 3D layout and composition of the photo. To predict poignance, eyetracking data on a set of artistic images from famous photographers is collected, then semantic and connotation conflicts between the objects in the photographs are analyzed.

该项目开发了解释图像的视觉言论的技术。该项目通过新颖的解决方案来推动计算机视觉，以在广告和艺术照片中解码视觉信息的新颖问题，从而使计算机视觉更接近其能够自动理解视觉内容的目标。从实际的角度来看，理解视觉言论可用于制作与人类如何标记这些图像相符的视力障碍的图像描述，从而使他们可以访问报纸或电视上显示的丰富内容。该项目与教育紧密融合。这项工作是跨学科的，可以吸引本科生参加不同领域的研究。这项研究的重点是三个媒体理解任务：（1）了解艺术图像传达的有说服力的信息以及这些图像传达其信息的策略； （2）揭露摄影师对自己的主题的偏见，例如，确定照片是在正面还是负面的光线下描绘其主题； （3）预测观众可能会发现最迷人或凄美的艺术照片的哪一部分。为了启用艺术图像的解码，收集了大型数据集并用许多旨在用于人类理解的艺术特性和说服力的技术进行注释，然后开发出方法来模拟艺术图像中的视觉象征，并适应情感分析中的正面/负面效应方法。为了预测摄影师对一个主题的偏见，收集了少数群体和外国人的历史和现代刻画数据集，然后创建了一种算法，该算法是关于肢体语言以及照片的3D布局和组成的原因。为了预测凄美，收集了来自著名摄影师的一组艺术图像的眼影数据，然后分析了照片中对象之间的语义和内涵冲突。

项目成果

Predicting Visual Political Bias Using Webly Supervised Data and an Auxiliary Task

• DOI: 10.1007/s11263-021-01506-3
• 发表时间: 2021-08
• 期刊: International Journal of Computer Vision
• 影响因子: 19.5
• 作者: Christopher Thomas;Adriana Kovashka

Learning to Overcome Noise in Weak Caption Supervision for Object Detection

• DOI: 10.1109/tpami.2022.3187350
• 发表时间: 2022-06
• 期刊: IEEE Transactions on Pattern Analysis and Machine Intelligence
• 影响因子: 23.6
• 作者: Mesut Erhan Unal;Keren Ye;Mingda Zhang;Christopher Thomas;Adriana Kovashka;Wei Li;Danfeng Qin