III: Small: Query-By-Sketch: Simplifying Video Clip Retrieval Through A Visual Query Paradigm

III：小：按草图查询：通过可视化查询范式简化视频剪辑检索

• 批准号: 2335881
• 负责人: Kexin Rong
• 金额: $ 60万
• 依托单位: Georgia Tech Research Corporation
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-04-15 至 2027-03-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2335881&HistoricalAwards=false
• 关键词: III; Small; Query; Sketch; Simplifying

This project addresses the growing demand for analyzing movement patterns in videos across diverse applications such as sports analytics, wildlife tracking, urban planning, and autonomous vehicle development. For example, analyzing vehicle trajectories from surveillance videos is essential for improving traffic safety. This project introduces a novel method for querying movement patterns in videos, enabling users to sketch events of interest on a canvas. The main innovation lies in accurately and efficiently matching free-form sketches to real-world trajectories, overcoming challenges posed by ambiguous user intent and variations in perspective, orientation, and camera movements. Consider a user describing a left-turning vehicle event as a 90-degree angle from a top-down perspective; in practice, the turning angles may appear different on video due to varying camera positions relative to vehicles. The project will lead to an open-source video database featuring a sketch-based query interface, making the analysis of movement patterns in videos more accessible and accurate. Research findings will be disseminated through publications at top conferences and incorporated into new database courses at Georgia Tech, as well as research classes for Atlanta-area high school girls interested in pursuing computing careers. Video retrieval from trajectory queries has been explored by the database and machine learning communities using SQL-like and natural language interfaces, but they face limitations due to high query specification time or poor generalizability to unseen videos. This project seeks to address these challenges by introducing a novel visual query paradigm that enables users to sketch exploratory trajectory queries in video analytics through drag-and-drop actions. The project is structured around two research thrusts. The first focuses on developing a human-in-loop similarity search framework that leverages active-learning techniques to solicit user feedback. This process aims to clarify user intent in query specifications and address inaccuracies inherent in human sketching. Domain-specific knowledge will be incorporated as additional predicates in the pre-processing and post-processing stages of similarity search to further enhance retrieval efficiency and quality. The second thrust develops an end-to-end machine learning model that learns a robust similarity measure between user-drawn sketches and trajectories in real-world videos, accounting for variations in camera angles and movements. It will address the lack of diverse and labeled datasets for video retrieval from trajectory queries by developing a self-supervised learning framework based on trajectory simulation. Overall, this project will leverage database-style optimization to reduce both user effort and computational resources required for utilizing vision models in exploratory video analytics, which will help expand the adoption of video analytics.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目满足了在体育分析、野生动物跟踪、城市规划和自动驾驶汽车开发等不同应用中分析视频运动模式日益增长的需求。例如，从监控视频中分析车辆轨迹对于提高交通安全至关重要。该项目引入了一种查询视频中运动模式的新颖方法，使用户能够在画布上勾勒出感兴趣的事件。主要创新在于准确有效地将自由形式的草图与现实世界的轨迹相匹配，克服了用户意图模糊以及视角、方向和相机运动变化带来的挑战。考虑用户将左转车辆事件描述为从自上而下的角度看成 90 度角；实际上，由于摄像机相对于车辆的位置不同，转向角度在视频中可能会出现不同。该项目将推出一个开源视频数据库，该数据库具有基于草图的查询界面，使视频中的运动模式分析更加容易和准确。研究结果将通过顶级会议上的出版物进行传播，并纳入佐治亚理工学院的新数据库课程以及为亚特兰大地区有志于从事计算机职业的高中女生开设的研究课程。数据库和机器学习社区已经使用类似 SQL 和自然语言接口探索了轨迹查询的视频检索，但由于查询规范时间长或对未见过的视频的通用性较差，它们面临着局限性。该项目旨在通过引入一种新颖的视觉查询范例来解决这些挑战，该范例使用户能够通过拖放操作在视频分析中绘制探索性轨迹查询。该项目围绕两个研究重点构建。第一个重点是开发一个人机循环相似性搜索框架，该框架利用主动学习技术来征求用户反馈。此过程旨在澄清查询规范中的用户意图并解决人类草图固有的不准确性。特定领域的知识将作为附加谓词纳入相似性搜索的预处理和后处理阶段，以进一步提高检索效率和质量。第二个重点开发了一种端到端的机器学习模型，该模型可以学习用户绘制的草图和现实视频中的轨迹之间强大的相似性度量，并考虑摄像机角度和运动的变化。它将通过开发基于轨迹模拟的自监督学习框架来解决轨迹查询视频检索中缺乏多样化和标记数据集的问题。总体而言，该项目将利用数据库式优化来减少在探索性视频分析中利用视觉模型所需的用户工作量和计算资源，这将有助于扩大视频分析的采用。该奖项反映了 NSF 的法定使命，并被认为是值得的通过使用基金会的智力优势和更广泛的影响审查标准进行评估来提供支持。