CISE-ANR: HCC: Small: Omnidirectional BatVision: Learning How to Navigate from Cell Phone Audios

CISE-ANR：HCC：小型：全向 BatVision：学习如何通过手机音频进行导航

• 批准号: 2215542
• 负责人: Stella Yu
• 金额: $ 30万
• 依托单位: Regents of the University of Michigan - Ann Arbor
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-04-01 至 2026-03-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2215542&HistoricalAwards=false
• 关键词: CISE; ANR; HCC; Small; Omnidirectional

This project aims to develop real-time 3D space reconstruction from sound captured not by expensive specialized equipment, but by common-place consumer-grade mobile phones. The approach, inspired by echolocation used by bats, is to develop from sound alone 3D spatial maps that are sufficient for navigation, such as close obstacle avoidance and finding distant exits in a crowded train station. The research will enable 3D vision beyond the line of sight and in low or no light conditions with applications ranging from listening cars that can hear pedestrians around the corner to collective 3D map reconstruction from crowds. Project outcomes will contribute to better computational modeling of sound perception and effective sound-vision integration in robotics, as well as to impactful applications such as navigational aids for visually impaired persons and for fire-fighters in low visibility conditions caused by smoke or darkness. The work will provide a complementary cost-effective alternative to visual 3D mapping that allows everybody to become a 3D content creator.The task of 3D perception from sound is challenging. While stereo audio provides direct cues for horizontal direction of arrival estimation, it only works in well controlled environments. There are no simple mathematical models to map sound to 3D space in real-word settings, as many factors such as device orientations, room layouts, materials, background noises shape sound propagation. This project takes a machine learning approach to infer 3D space from cell phone audios. A large-scale audio-visual dataset will be collected in different environments using a sensor-rig with a binaural microphone, a speaker and an RGB-D stereo. An attached smartphone will record time-synchronized data with its own stereo microphone and cameras. The speaker will emit signals to enable echolocation, but a part of the data will contain only naturally occurring sounds. Several indoor and outdoor environments with LiDAR scanned 3D models will serve as ground-truth. Data will also be collected in public streets to test robustness in realistic situations where LiDAR scans are not possible. Given the dataset, several 3D scene reconstruction tasks will be formulated for both the field of view and full 360° view, first with privileged sensor data and finally from cellphone sensors alone. After collecting large-scale audio-visual data in a variety of environments with binaural microphones and stereo cameras, a model will be trained to map sound data to depth maps extracted from visual data. Once the model is trained, it will be able to “see” the 3D space based on sound inputs alone. The model will then be adapted to achieve the same high quality 3D perception with stereo-microphones and sensors available on a mobile phone.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目旨在开发实时 3D 空间重建，而不是通过昂贵的专用设备，而是通过普通的消费级手机捕获的声音。该方法受到蝙蝠使用的回声定位的启发，是仅从声音开发 3D 空间图。该研究将在低光或无光条件下实现视线之外的 3D 视觉，其应用范围包括能够听到周围行人声音的监听汽车。集体的角落人群的 3D 地图重建将有助于更好地计算机器人中的声音感知和有效的声音视觉集成，以及为视障人士和低能见度条件下的消防员提供导航辅助等有影响力的应用。这项工作将为视觉 3D 映射提供一种具有成本效益的补充方案，让每个人都能成为 3D 内容创作者。从声音中感知 3D 的任务具有挑战性，而立体声音频则为水平方向的到达提供了直接提示。估计，它仅适用于受控良好的环境。在真实环境中，没有简单的数学模型可以将声音映射到 3D 空间，因为设备方向、房间布局、材料、背景噪音等许多因素都会影响声音传播。一种从手机音频推断 3D 空间的方法，将使用带有双耳麦克风、扬声器和 RGB-D 立体声的传感器装置在不同的环境中收集大规模视听数据集。扬声器将使用其自己的立体声麦克风和摄像头发出信号以实现回声定位，但部分数据将仅包含自然发生的声音，并使用激光雷达扫描的 3D 模型作为地面实况。还将在公共街道上收集数据，以测试在无法进行 LiDAR 扫描的实际情况下的稳健性。根据数据集，将针对视野和全 360° 视图制定多个 3D 场景重建任务。使用双耳麦克风和立体声摄像头在各种环境中收集大规模视听数据后，将训练一个模型，将声音数据映射到从视觉数据中提取的深度图。模型经过训练后，将能够仅根据声音输入“看到”3D 空间，然后对该模型进行调整，以通过手机上的立体声麦克风和传感器实现相同的高质量 3D 感知。该奖项由 NSF 授予。法定使命通过使用基金会的智力价值和更广泛的影响审查标准进行评估，并被认为值得支持。