CAREER: Natural Narratives and Multimodal Context as Weak Supervision for Learning Object Categories

职业：自然叙事和多模态上下文作为学习对象类别的弱监督

• 批准号: 2046853
• 负责人: Adriana Kovashka
• 金额: $ 54.71万
• 依托单位: University of Pittsburgh
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-05-01 至 2026-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2046853&HistoricalAwards=false
• 关键词: CAREER; Natural; Narratives; Multimodal; Context

This project develops a framework to train computer vision models for detection of objects from weak, naturally-occurring supervision of language (text or speech) and additional multimodal signals. It considers dynamic settings, where humans interact with their visual environment and refer to the encountered objects, e.g., “Carefully put the tomato plants in the ground” and “Please put the phone down and come set the table,” and captions written for a human audience to complement an image, e.g., news article captions. The challenge of using such language-based supervision for training detection systems is that along with useful signal, the speech contains many irrelevant tokens. The project will benefit society by exploring novel avenues for overcoming this challenge and reducing the need for expensive and potentially unnatural crowdsourced labels for training. It has the potential to make object detection systems more scalable and thus more usable by a broad user base in a variety of settings. The resources and tools developed would allow natural, lightweight learning in different environments, e.g., different languages or types of imagery where the well-known object categories are not useful or where there is a shift in both the pixels as well as the way in which humans refer to objects (different cultures, medicine, art). This project opens possibilities for learning in vivo rather than in vitro; while the focus here is on object categories, multimodal weak supervision is useful for a larger variety of tasks. Research and education are integrated through local community outreach and research mentoring for students from lesser-known universities, new programs for student training including honing graduate students' writing skills, and development of interactive educational modules and demos based on research findings. This project creatively connects two domains, vision-and-language, and object detection, and pioneers training of object detection models with weak language supervision and a large vocabulary of potential classes. The impact of noise in the language channel will be mitigated through three complementary techniques that model visual concreteness of words, to what extent the text refers to the visual environment it appears with, and whether the weakly-supervised models that are learned are logically consistent. Two complementary word-region association mechanisms will be used (metric learning and cross-modal transformers), whose application is novel for weakly-supervised detection. Importantly, to make detection feasible, not only the semantics of image-text pairs, but their discourse relationship, will be captured. To facilitate and disambiguate the association of words to a physical environment, the latter will be represented through additional modalities, namely sound, motion, depth and touch, which are either present in the data or estimated. This project advances knowledge of how multimodal cues contextualize the relation between image and text; no prior work has modeled image-text relationships along multiple channels (sound, depth, touch, motion). Finally, to connect the appearance of objects to the purpose and use of these objects, relationships between objects, properties and actions will be semantically organized in a graph, and grammars to represent activities involving objects will be extracted, still maintaining the weakly-supervised setting.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目开发了一个框架，用于检测来自弱的，自然的监督或语音的对象）和其他多模式信号。 ，“并写有书面范围来补充图像，例如，新闻文章捕获。使用此类基于语言的监督进行培训检测系统的挑战是，与有用的信号一起，演讲包含许多IRELELELELELELELELVANT令牌。LL通过探索探索社会来利益。克服和减少对训练的昂贵和不自然的需求。最终的医学，艺术）。较鲜为人知的大学基于研究发现，磨练了研究生的写作技巧，并开发互动模块和演示。监督和大量的潜在类别的噪声。弱监督的检测是可行的，不仅是图像 - textt对的语义，而且是通过估计的ADITITS来重复的关于沿多个通道的多模式将图像和文本之间的关系（声音，深度，触摸，运动）将在图形上进行语义组织以提取不满意的活动，弱监督的环境。该奖项反映了NSF'Sf'sf'Sf'Story Mission D被认为是值得通过Toundation的智力优点和更广泛的影响审查标准通过评估来获得支持的。

项目成果

Weakly-Supervised Action Detection Guided by Audio Narration

• DOI: 10.1109/cvprw56347.2022.00159
• 发表时间: 2022-05
• 期刊: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
• 作者: Keren Ye;Adriana Kovashka

Improving language-supervised object detection with linguistic structure analysis

• DOI: 10.1109/cvprw59228.2023.00588
• 发表时间: 2023-06
• 期刊: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
• 作者: Arushi Rai;Adriana Kovashka

Boosting Weakly Supervised Object Detection using Fusion and Priors from Hallucinated Depth

• DOI: 10.1109/wacv57701.2024.00079
• 发表时间: 2023-03
• 期刊: 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
• 作者: Cagri Gungor;Adriana Kovashka

Complementary Cues from Audio Help Combat Noise in Weakly-Supervised Object Detection

• DOI: 10.1109/wacv56688.2023.00222
• 发表时间: 2023-01
• 期刊: 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
• 作者: Cagri Gungor;Adriana Kovashka

Learning to Overcome Noise in Weak Caption Supervision for Object Detection

• DOI: 10.1109/tpami.2022.3187350
• 发表时间: 2022-06
• 期刊: IEEE Transactions on Pattern Analysis and Machine Intelligence
• 影响因子: 23.6
• 作者: Mesut Erhan Unal;Keren Ye;Mingda Zhang;Christopher Thomas;Adriana Kovashka;Wei Li;Danfeng Qin