RI: Small: Domain-robust object detection through shape and context

RI：小：通过形状和上下文进行领域稳健的对象检测

• 批准号: 2006885
• 负责人: Adriana Kovashka
• 金额: $ 46.18万
• 依托单位: University of Pittsburgh
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2006885&HistoricalAwards=false
• 关键词: RI; Small; Domain; robust; object

Computer vision has made great advancements in object recognition and detection, but performance drops significantly when the data used at training and deployment time are very different. This is problematic because in many situations, it may be infeasible to retrain the models on a large example set in the domain of interest. For example, artificial intelligence (AI) tools may be developed in one country or region, using that region’s training data, and exported to regions with limited resources to collect new data and retrain models. Unfortunately, the visual environment in the user region may be different from the developer region: some vehicles in India look different from common vehicles in the US; houses often feature bricks on the US East Coast but less frequently on the West Coast; environmental factors (e.g., foliage and smog) may cause models to behave differently. Being robust to domain shifts is important for interpretability and trust when computer vision systems are employed in practice. This project leverages the observation that while the pixels of captured objects change when these objects are shown in different domains (e.g., photographs vs paintings), the overall shape of the objects remains the same. Further, the set of objects that co-occur with the object of interest is also relatively consistent across domains. This project develops new visual representations that capture two global cues: shape and context. While numerous domain adaptations and generalization techniques exist, they have overlooked global cues that can potentially be more robust to domain shifts, based on preliminary experiments. The first proposed representation adapts the medial axis transform (MAT) into a hierarchical, learnable, convolutional representation. MAT computes the "skeleton" of an object, and a representation is developed using a dense feature map to ensure there is enough information for the convolutional network to capture, as well as to build robustness to small shifts. Second, context is represented through graphs containing functionally or semantically related objects, and ambient cues (such as co-occurring text or speech) to improve the model's ability to recognize objects in novel modalities. Techniques for making weakly-supervised techniques more robust to domain shifts are explored, as a way of capturing non-semantic context. Next, these global representations are combined with standard appearance-based ones and are adapted to novel domains or made domain-invariant through domain generalization techniques. The domain robustness of the resulting representations is tested in a variety of domain shift scenarios, including photorealistic and artistic datasets, different capture conditions, and controllable shift scenarios (e.g., blurring and masking), for both object recognition and detection. Code, any artificially created situations (data), clear protocols for how to train models for existing techniques, and detailed benchmarking results (quantitative and qualitative) will be released to ensure reproducibility.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

计算机视觉在对象识别和检测方面取得了巨大的进步，但是当培训和部署时间中使用的数据大不相同时，性能会大大下降。这是有问题的，因为在许多情况下，在感兴趣域中设置的大型示例上重新训练模型可能是不可行的。例如，使用该地区的培训数据，可以在一个国家或地区开发人工智能（AI）工具，并将其导出到具有有限资源的地区以收集新数据和重新训练模型。不幸的是，用户区域的视觉环境可能与开发人员区域不同：印度的某些车辆看起来与美国的普通车辆不同。房屋经常在美国东海岸设有砖，但在西海岸的频率较低。环境因素（例如，叶子和烟雾）可能会导致模型的行为不同。在实践中采用计算机视觉系统时，对域转移的强大转移对于可解释性和信任很重要。该项目利用了这样的观察结果，即捕获的对象的像素在不同域中显示（例如，照片与绘画）时会发生变化，但对象的整体形状保持不变。此外，与感兴趣的对象共同发生的对象集也相对一致。该项目开发了新的视觉表示，可捕获两个全局线索：形状和上下文。尽管存在许多领域的适应和概括技术，但它们忽略了基于初步实验的全球线索，这些线索可能对域的转移有可能更强大。第一个提出的表示将内侧轴变换（MAT）调整为层次，可学习的卷积表示。 MAT计算对象的“骨架”，并使用密集的特征映射开发表示形式，以确保有足够的信息供卷积网络捕获，并为小型移动增强了稳健性。其次，通过包含功能或语义相关对象的图表表示上下文，以及环境提示（例如共发生的文本或语音），以提高模型在新型方式中识别对象的能力。探索了使弱监督技术对领域变化更强大的技术，作为捕获非语义上下文的一种方式。接下来，这些全局表示与标准的外观相结合，并通过域泛化技术适应了新的域或使域不变。在各种域移动方案中测试了所得表示的域鲁棒性，包括光真逼真和艺术数据集，不同的捕获条件以及受控的移位方案（例如，模糊和掩盖），以供对象识别和检测。代码，任何人为创建的情况（数据），如何培训现有技术的模型的清晰协议以及详细的基准测试结果（定量和定性），以确保可重复性。该奖项反映了NSF的法定任务，并通过评估智力和广泛的影响来评估NSF的法定任务，并被视为珍贵的支持。

项目成果

How to Practice VQA on a Resource-limited Target Domain

• DOI: 10.1109/wacv56688.2023.00443
• 发表时间: 2023-01-01
• 期刊: 2023 IEEE/CVF WINTER CONFERENCE ON APPLICATIONS OF COMPUTER VISION (WACV)
• 作者: Zhang，Mingda;Hwa，Rebecca;Kovashka，Adriana
• 通讯作者: Kovashka，Adriana

Contrastive View Design Strategies to Enhance Robustness to Domain Shifts in Downstream Object Detection

• DOI: 10.48550/arxiv.2212.04613
• 发表时间: 2022-12
• 期刊: ArXiv
• 作者: Kyle Buettner;Adriana Kovashka

Towards Shape-regularized Learning for Mitigating Texture Bias in CNNs

• DOI: 10.1145/3591106.3592231
• 发表时间: 2023-06
• 期刊: Proceedings of the 2023 ACM International Conference on Multimedia Retrieval
• 作者: Harsh Sinha;Adriana Kovashka

The Role of Shape for Domain Generalization on Sparsely-Textured Images

• DOI: 10.1109/cvprw56347.2022.00560
• 发表时间: 2022-06
• 期刊: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
• 作者: N. Nazari;Adriana Kovashka