RI: Small: Collaborative Research: Ontology based Perceptual Organization of Audio-Video Events using Pattern Theory

RI：小型：协作研究：使用模式理论对音频-视频事件进行基于本体的感知组织

• 批准号: 1217676
• 负责人: Sudeep Sarkar
• 金额: $ 24.98万
• 依托单位: University of South Florida
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2012
• 资助国家: 美国
• 起止时间: 2012-09-01 至 2016-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1217676&HistoricalAwards=false
• 关键词: RI; Small; Collaborative; Research; Ontology

It is natural that events of interest in observed scenes manifest themselves across multiple sensing modalities - vision, hearing, smell, etc. The remarkable perceptions of audio-video signals by natural systems, such as humans, also points to superiority of inferences drawn across modalities. It, therefore, seems natural to enhance performance of automated systems by using joint, cross-modal statistical inferences. However, the detection, organization, and understanding of cues and events in real-world scenarios are difficult tasks. This project seeks to develop a pattern-theoretic framework for achieving these goals. The main research items are: (1) development of mathematical quantities to represent audio and visual events and their spatiotemporal relations, (2) use domain-specific ontologies to impose semantic structure and to incorporate prior knowledge, and (3) derive algorithms for Bayesian inferences using efficient adaptations of Markov Chain Monte Carlo sampling. The use of pattern theory allows bridging of gaps between raw signals and high-level, domain-dependent semantics, and helps discovers large groups of audio-visual events likely to represent the same underlying event. This effort combines ideas from perceptual organization in computer vision, computational analysis of auditory signals, pattern theory, and prior developments in ontological structures. The methods developed here are applicable to many scenarios that deploy audio and video sensors, including problems of audio annotations of videos, speaker tracking in teleconferencing, and separation of multiple objects in remote surveillance. Broader impact activities involve the development of teaching modules, innovation and entrepreneurial training of the students, and communication of the findings to the community.

在观察到的场景中感兴趣的事件自然而然地表现出了多种感应方式 - 视觉，听力，气味等。天然系统（例如人类）对音频视频信号的显着感知也表明了跨模态的优越性。因此，通过使用关节，跨模式统计推论来增强自动化系统的性能似乎很自然。但是，在现实世界中，对线索和事件的检测，组织以及理解是艰巨的任务。该项目旨在开发一个实现这些目标的模式理论框架。主要研究项目是：（1）开发数学数量以表示音频和视觉事件及其时空关系，（2）使用特定领域的本体论来强加语义结构并纳入先验知识，并且（3）为贝叶斯的算法提供算法使用马尔可夫链蒙特卡洛采样的有效改编的推论。模式理论的使用允许在原始信号和高级，域依赖性语义之间桥接差距，并帮助发现可能代表同一基础事件的大型音频事件。这项工作结合了感知组织在计算机视觉中的思想，听觉信号的计算分析，模式理论以及本体论结构中的先前发展。此处开发的方法适用于部署音频和视频传感器的许多方案，包括视频的音频注释问题，远程导触点中的扬声器跟踪以及在远程监视中分离多个对象的问题。更广泛的影响活动涉及教学模块，创新和企业家培训的发展以及对社区的调查结果的沟通。