CAREER: Weakly-Supervised Visual Scene Understanding: Combining Images and Videos, and Going Beyond Semantic Tags

职业：弱监督视觉场景理解：结合图像和视频，超越语义标签

• 批准号: 1751206
• 负责人: Yong Jae Lee
• 金额: $ 50.05万
• 依托单位: University of California-Davis
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-04-01 至 2021-10-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1751206&HistoricalAwards=false
• 关键词: CAREER; Weakly; Supervised; Visual; Scene

The internet provides an endless supply of images and videos, replete with weakly-annotated meta-data such as text tags, GPS coordinates, timestamps, or social media sentiments. This huge resource of visual data provides an opportunity to create scalable and powerful recognition algorithms that do not depend on expensive human annotations. The research component of this project develops novel visual scene understanding algorithms that can effectively learn from such weakly-annotated visual data. The main novelty is to combine both images and videos together. The developed algorithms could have broad impact in numerous fields including AI, security, and agricultural sciences. In addition to scientific impact, the project performs complementary educational and outreach activities. Specifically, it provides mentorship to high school, undergraduate, and graduate students, teaches new undergraduate and graduate computer vision courses that have been lacking at UC Davis, and organizes an international workshop on weakly-supervised visual scene understanding.This project develops novel algorithms to advance weakly-supervised visual scene understanding in two complementary ways: (1) learning jointly with both images and videos to take advantage of their complementarity, and (2) learning from weak supervisory signals that go beyond standard semantic tags such as timestamps, captions, and relative comparisons. Specifically, it investigates novel approaches to advance tasks like fully-automatic video object segmentation, weakly-supervised object detection, unsupervised learning of object categories, and mining of localized patterns in the image/video data that are correlated with the weak supervisory signal. Throughout, the project explores ways to understand and mitigate noise in the weak labels and to overcome the domain differences between images and videos.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Internet提供了无休止的图像和视频供应，充满了弱宣布的元数据，例如文本标签，GPS坐标，时间戳或社交媒体情感。这种庞大的视觉数据资源为创建不依赖昂贵人类注释的可扩展和强大识别算法提供了机会。该项目的研究组成部分开发了新颖的视觉场景理解算法，这些算法可以有效地从这种弱宣布的视觉数据中学习。主要的新颖性是将图像和视频结合在一起。开发的算法可能在包括人工智能，安全和农业科学在内的许多领域都产生广泛的影响。除了科学影响外，该项目还进行了互补的教育和外展活动。 Specifically, it provides mentorship to high school, undergraduate, and graduate students, teaches new undergraduate and graduate computer vision courses that have been lacking at UC Davis, and organizes an international workshop on weakly-supervised visual scene understanding.This project develops novel algorithms to advance weakly-supervised visual scene understanding in two complementary ways: (1) learning jointly with both images and videos to take advantage of their互补性，以及（2）从弱监督信号中学习，这些信号超出了标准语义标签，例如时间戳，字幕和相对比较。具体而言，它研究了新的方法来推进任务，例如完全自动的视频对象分割，弱监督对象检测，无监督的对象类别的学习以及与弱监督性信号相关的图像/视频数据中局部模式的挖掘。在整个过程中，该项目探索了理解和减轻弱标签中噪音并克服图像和视频之间的领域差异的方法。该奖项反映了NSF的法定任务，并被认为是通过基金会的知识分子和更广泛的影响评估标准的评估来通过评估来支持的。

项目成果

A Visual Attention Grounding Neural Model for Multimodal Machine Translation

• DOI: 10.18653/v1/d18-1400
• 发表时间: 2018-08
• 期刊: ArXiv
• 作者: Mingyang Zhou;Runxiang Cheng;Yong Jae Lee;Zhou Yu

Password-conditioned Anonymization and Deanonymization with Face Identity Transformers

• DOI: 10.1007/978-3-030-58592-1_43
• 发表时间: 2019-11
• 期刊: ArXiv
• 作者: Xiuye Gu;Weixin Luo;M. Ryoo;Yong Jae Lee

You Reap What You Sow: Using Videos to Generate High Precision Object Proposals for Weakly-Supervised Object Detection

• DOI: 10.1109/cvpr.2019.00964
• 发表时间: 2019-06
• 期刊: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Krishna Kumar Singh;Yong Jae Lee

Delving Deeper into Anti-Aliasing in ConvNets

• DOI: 10.1007/s11263-022-01672-y
• 发表时间: 2020-08
• 期刊: International Journal of Computer Vision
• 影响因子: 19.5
• 作者: Xueyan Zou;Fanyi Xiao;Zhiding Yu;Yong Jae Lee

MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generation

• DOI: 10.1109/cvpr42600.2020.00806
• 发表时间: 2019-11
• 期刊: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Yuheng Li;Krishna Kumar Singh;Utkarsh Ojha;Yong Jae Lee