CRII: RI: RUI: Representations for multi-timescale scene dynamics in webcam video streams

CRII：RI：RUI：网络摄像头视频流中多时间尺度场景动态的表示

• 批准号: 2105372
• 负责人: Scott Wehrwein
• 金额: $ 17.47万
• 依托单位: Western Washington University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-06-01 至 2025-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2105372&HistoricalAwards=false
• 关键词: CRII; RI; RUI; Representations; multi

Thousands of webcams positioned all over the world are constantly streaming live HD views of a wide variety of scenes. Thanks to their longevity, these streams capture many visually interesting phenomena that occur not only in real time but over longer scales of time. Although computer vision algorithms have been developed for video and scene understanding, the scale - both in data volume and in time - of these webcam streams require new approaches. This project will develop representations that facilitate techniques to automatically extract insights about short and long-term changes in fixed-view webcam scenes. These techniques can be applied to a variety of areas including security, surveillance, and a diverse array of monitoring use cases including climate science, ecology, and development. Undergraduate students at a teaching-oriented public Primarily Undergraduate Institution will gain valuable skills and experience by being directly involved in all aspects of the research.Existing video understanding and analysis techniques focus almost exclusively on real-time dynamics, such as human actions, while little attention has been paid to longer-term phenomena. The objective of this research is to develop representations for years-long video streams that facilitate analysis and understanding of phenomena at a wide range of time scales. To serve this purpose, the representations need to be compact, allow for full reconstruction of the input, and organize scene content by timescale. To achieve this, a dataset of video streams will be collected and scene-specific models will be trained to encode video frames into compact latent vectors. A novel regularization scheme will impose order in the latent space, arranging the representation of the scene's content by how quickly it varies. The research will enable new ways to detect, monitor, and understand phenomena that occur over weeks, months, and years. Existing video understanding applications such as anomaly detection, action recognition, and video prediction will also benefit by operating in a compact and organized self-supervised latent space. Finally, the research will further progress towards understanding how to encode temporal redundancy in videos and yield insights about general video understanding by helping to determine what is possible with the absence of camera motion and an abundance of observations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

遍布世界各地的数千个网络摄像头不断传输各种场景的实时高清视图。由于它们的寿命长，这些流捕获了许多视觉上有趣的现象，这些现象不仅是实时发生的，而且是在更长的时间尺度上发生的。尽管计算机视觉算法已经开发用于视频和场景理解，但这些网络摄像头流的数据量和时间规模需要新的方法。该项目将开发表示技术，以促进自动提取有关固定视图网络摄像头场景中短期和长期变化的见解的技术。这些技术可应用于多种领域，包括安全、监视和各种监测用例，包括气候科学、生态和发展。以教学为导向的公立本科院校的本科生将通过直接参与研究的各个方面来获得宝贵的技能和经验。现有的视频理解和分析技术几乎只关注实时动态，例如人类行为，而很少有人们对长期现象给予了关注。这项研究的目的是开发长达数年的视频流的表示，以促进在广泛的时间尺度上分析和理解现象。为了达到这个目的，表示需要紧凑，允许完全重建输入，并按时间尺度组织场景内容。为了实现这一目标，将收集视频流数据集，并训练特定于场景的模型，将视频帧编码为紧凑的潜在向量。一种新颖的正则化方案将在潜在空间中强加秩序，根据场景内容变化的速度来安排场景内容的表示。这项研究将提供新的方法来检测、监测和理解几周、几个月和几年内发生的现象。现有的视频理解应用程序（例如异常检测、动作识别和视频预测）也将受益于在紧凑且有组织的自监督潜在空间中运行。最后，该研究将进一步了解如何对视频中的时间冗余进行编码，并通过帮助确定在缺乏摄像机运动和大量观察的情况下可能发生的情况，产生对一般视频理解的见解。该奖项反映了 NSF 的法定使命，并具有通过使用基金会的智力优点和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Visualizing the Passage of Time with Video Temporal Pyramids

使用视频时间金字塔可视化时间的流逝

• DOI: 10.1109/tvcg.2022.3209454
• 发表时间: 2022-08-25
• 期刊: IEEE Transactions on Visualization and Computer Graphics
• 影响因子: 5.2
• 作者: Melissa E. Swift;Wyatt Ayers;Sophie Pallanck;Scott Wehrwein
• 通讯作者: Scott Wehrwein