CHS: Small: Audio-Visual Reconstruction for Immersive Virtualized Reality

CHS：小型：沉浸式虚拟现实的视听重建

• 批准号: 1910940
• 负责人: Ming Lin
• 金额: $ 50万
• 依托单位: University of Maryland, College Park
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1910940&HistoricalAwards=false
• 关键词: CHS; Small; Audio; Visual; Reconstruction; CHS; Small; Audio; Visual; Reconstruction

Maintaining the sense of presence is a major challenge in immersive virtual environments. An important aspect of immersion is the feeling of cohesiveness between different senses, including the visual and auditory; for example, an object that looks like wood should also sound like wood. Sound synthesis can improve a user's sensory cohesion when interacting with objects, but it requires accurate real-world material parameters. While much prior work in computer vision has focused on acquiring the geometric shape and visual characteristic of objects, the resulting point-clouds and images can assist in more accurate recovery of audio parameters for sound synthesis, along with acoustic scattering and absorption properties for sound rendering and propagation. A hypothesis of this research is that, conversely, auditory metrics can also assist in determining an object's geometry, including holes and occlusions, in a manner analogous to sonar detection (but of course 3D geometry reconstruction is far more challenging than object detection). The audio-visual reconstruction enabled by this project will have broad impact across many domains, including assistive technology for persons who are visually impaired, multimodal human-centric interfaces, immersive teleconferencing, rapid prototyping of acoustic spaces for urban planning, structural design, and noise control, to name just a few. Project outcomes including scientific advances and software systems will be disseminated through websites, publications, workshops, community outreach, and other professional events.This project explores a novel paradigm of audio-visual reconstruction of real-world scenes, where audio cues are used to guide the classification of objects and materials for 3D geometry reconstruction. At the same time, the visual information can be used to initialize and accelerate the identification of acoustic material parameters. Some of the major research challenges that will be addressed include audio-guided 3D model reconstruction, design of audio-visual neural networks for material and object identification, learning-based acoustic material classification of a large physical or virtual space, and optimization-based acoustic material refinement using geometric and wave-based methods. Perceptually-grounded evaluation and validation of the new methods and applications will be performed.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在沉浸式虚拟环境中，保持存在感是一个主要的挑战。沉浸式的一个重要方面是不同感官（包括视觉和听觉）之间具有凝聚力的感觉。例如，看起来像木头的物体也应该听起来像木头。声音综合可以在与对象交互时可以改善用户的感觉凝聚力，但是它需要准确的现实世界材料参数。尽管计算机视觉中的许多先前工作都集中在获取物体的几何形状和视觉特征上，但所产生的点云和图像可以帮助更准确地恢复音频参数以进行声音合成，以及声音散射和吸收属性，以进行声音渲染和传播。 这项研究的一个假设是，相反，听觉指标还可以以类似于声纳检测的方式来确定对象的几何形状，包括孔和遮挡（但是3D几何重建当然比对象检测更具挑战性）。 该项目实现的视听重建将对许多领域产生广泛的影响，包括针对视力障碍的人的辅助技术，多模式以人为中心的接口，沉浸式的电信，迅速的原始原型，用于城市规划，结构设计和噪声控制的声学空间的快速原型，仅举几例。 包括科学进步和软件系统在内的项目成果将通过网站，出版物，研讨会，社区外展和其他专业活动来分散。本项目探讨了对现实世界情景的视听范围重建的新型范式，其中音频提示用于指导3D egemotiation的3D egemotiaction和材料进行3D equemotiaction的分类。同时，可以使用视觉信息来初始化和加速声学材料参数的识别。将要解决的一些主要研究挑战包括音频指导的3D模型重建，用于材料和对象识别的视听神经网络的设计，大型物理或虚拟空间的基于学习的声学材料分类以及使用基于几何学和波浪方法的基于优化的声学材料改进。 将对新方法和应用程序进行感知地面的评估和验证。该奖项反映了NSF的法定任务，并且使用基金会的知识分子优点和更广泛的影响审查标准，被认为值得通过评估来获得支持。