PFI-TT: Cooperative Listening with Networked Audio Devices

PFI-TT：与网络音频设备协作收听

• 批准号: 1919257
• 负责人: Andrew Singer
• 金额: $ 25万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-08-01 至 2023-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1919257&HistoricalAwards=false
• 关键词: PFI; TT; Cooperative; Listening; Networked

The broader impact/commercial potential of this Partnerships for Innovation - Technology Translation (PFI-TT) project is to enable multiple listening devices to work together and improve their individual performance through collaborative signal processing. For both humans and machines, speech comprehension can be difficult in noisy, complex environments with several competing sound sources. Many of these environments may have multiple devices that in turn each contain multiple microphones, such as smart speakers, smart-home appliances, mobile devices, and wearables. Unfortunately, these devices do not currently cooperate to perform spatial sound capture. If they could be connected together to perform large-scale spatial signal processing, such a distributed array could dramatically improve the performance of machine listening, human-computer interaction, and human sensory augmentation tasks. The ability to precisely localize, separate, and enhance sound sources in complex environments would enable new applications that are impossible with current technology. These technologies could be applied to many already-deployed acoustic systems with minimal additional bandwidth and computation requirements. This research has the potential to dramatically impact these and other application areas while training graduate and undergraduate students and post-doctoral researchers from a range of underrepresented groups in lean-startup approaches to technology commercialization. The proposed project will develop technologies to aggregate data from multiple devices containing acoustic arrays, such as smart-home devices and wearables, to improve the spatial sound processing performance of each individual device. The proposed technology uses a hierarchical, distributed processing approach to efficiently aggregate information across networked devices without transmitting full synchronous audio data. The resulting system can leverage the spatial diversity of the distributed array to achieve better performance than a single device in listening tasks, especially in adverse environments with strong noise and interference where current technology often fails. The research objectives of this project are to explore the design trade-offs and scaling behavior of systems at large-scale; determine how best to aggregate distributed array data in the presence of non-idealities such as sample clock mismatch, network latency, and bandwidth constraints; to characterize the performance scaling and design tradeoffs of the system in real-world environments and under various acoustic and network conditions; and to implement and demonstrate the new algorithms on embedded hardware. Source separation and speech recognition experiments will be conducted in both controlled laboratory conditions and real-world environments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

这种伙伴关系对创新 - 技术翻译（PFI -TT）项目的更广泛的影响/商业潜力是使多个聆听设备能够通过协作信号处理来共同努力并提高其个人绩效。对于人类和机器，在具有多种竞争声音来源的嘈杂，复杂的环境中，语音理解都可能很困难。这些环境中的许多可能具有多种设备，而每个设备又包含多个麦克风，例如智能扬声器，智能房屋，移动设备和可穿戴设备。不幸的是，这些设备目前不合作执行空间声音捕获。如果可以将它们连接在一起进行大规模的空间信号处理，那么这种分布式阵列可以大大改善机器聆听，人类计算机相互作用和人类感觉增强任务的性能。精确本地化，分离和增强复杂环境中的声源的能力将使当前技术不可能实现新的应用程序。这些技术可以应用于许多已经部署的声学系统，具有最小的额外带宽和计算要求。这项研究有可能在培训研究生和本科生以及研究生和博士后研究人员中，从一系列人数不足的群体中培训研究生和学科研究人员，以实现技术商业化的方法。拟议的项目将开发技术，以汇总包含声学阵列（例如智能房屋设备和可穿戴设备）的多个设备的数据，以提高每个设备的空间声音处理性能。所提出的技术使用层次结构的分布式处理方法，在不传输完整同步音频数据的情况下有效地汇总了跨网络设备的信息。最终的系统可以利用分布式阵列的空间多样性比在聆听任务中的单个设备获得更好的性能，尤其是在当前技术经常失败的情况下具有强烈噪音和干扰的不利环境中。该项目的研究目标旨在探索大规模系统的设计权衡和扩展行为；确定如何最好地在存在非理想的情况下（例如样本时钟不匹配，网络延迟和带宽约束）汇总分布式数组数据；表征在现实环境以及各种声学和网络条件下系统的性能缩放和设计折衷方案；并在嵌入式硬件上实现和演示新算法。来源分离和语音识别实验将在受控的实验室条件和现实环境中进行。该奖项反映了NSF的法定任务，并且使用基金会的智力优点和更广泛的影响审查标准，被认为值得通过评估来提供支持。

项目成果

Development of an Underground Through-Soil Wireless Acoustic Communication System

• DOI: 10.1109/mwc.2019.1900030
• 发表时间: 2020-02
• 期刊: IEEE Wireless Communications
• 影响因子: 12.9
• 作者: Sijung Yang;Omar Baltaji;A. Singer;Y. Hashash

Immersive Enhancement and Removal of Loudspeaker Sound Using Wireless Assistive Listening Systems and Binaural Hearing Devices

使用无线助听系统和双耳听力设备对扬声器声音进行沉浸式增强和消除

• DOI: 10.1109/icassp49357.2023.10096123
• 发表时间: 2023
• 期刊: ICASSP 2023
• 作者: Corey, Ryan M.;Singer, Andrew C.
• 通讯作者: Singer, Andrew C.

High/Low Model for Scalable Multimicrophone Enhancement of Speech Mixtures

用于可扩展多麦克风增强语音混合物的高/低模型

• DOI: 10.23919/eusipco54536.2021.9616105
• 发表时间: 2021
• 期刊: 2021 29th European Signal Processing Conference (EUSIPCO
• 作者: Corey, Ryan M.;Singer, Andrew C.
• 通讯作者: Singer, Andrew C.

Cooperative Audio Source Separation and Enhancement Using Distributed Microphone Arrays and Wearable Devices

使用分布式麦克风阵列和可穿戴设备进行协作音频源分离和增强

• DOI: 10.1109/camsap45676.2019.9022475
• 发表时间: 2019
• 期刊: IEEE International Workshop on Computational Advances in Multisensor Adaptive Processing (CAMSAP
• 作者: Corey, Ryan M.;Skarha, Matthew D.;Singer, Andrew C.
• 通讯作者: Singer, Andrew C.

Modeling the effects of dynamic range compression on signals in noise

• DOI: 10.1121/10.0005314
• 发表时间: 2021-07-01
• 期刊: JOURNAL OF THE ACOUSTICAL SOCIETY OF AMERICA
• 影响因子: 2.4
• 作者: Corey, Ryan M.;Singer, Andrew C.
• 通讯作者: Singer, Andrew C.