Towards visually-driven speech enhancement for cognitively-inspired multi-modal hearing-aid devices (AV-COGHEAR)

面向认知启发的多模式助听设备的视觉驱动语音增强 (AV-COGHEAR)

• 批准号: EP/M026981/1
• 负责人: Amir Hussain
• 金额: $ 53.29万
• 依托单位: University of Stirling
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2015
• 资助国家: 英国
• 起止时间: 2015 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FM026981%2F1
• 关键词: Towards; visually; driven; speech; enhancement

Current commercial hearing aids use a number of sophisticated enhancement techniques to try and improve the quality of speech signals. However, today's best aids fail to work well in many everyday situations. In particular, they fail in busy social situations where there are many competing speech sources; they fail if the speaker is too far from the listener and swamped by noise. We have identified an opportunity to solve this problem by building hearing aids that can 'see'. This ambitious project aims to develop a new generation of hearing aid technology that extracts speech from noise by using a camera to see what the talker is saying. The wearer of the device will be able to focus their hearing on a target talker and the device will filter out competing sound. This ability, which is beyond that of current technology, has the potential to improve the quality of life of the millions suffering from hearing loss (over 10m in the UK alone).Our approach is consistent with normal hearing. Listeners naturally combine information from both their ears and eyes: we use our eyes to help us hear. When listening to speech, eyes follow the movements of the face and mouth and a sophisticated, multi-stage process uses this information to separate speech from the noise and fill in any gaps. Our hearing aid will act in much the same way. It will exploit visual information from a camera (e.g.using a Google Glass like system), and novel algorithms for intelligently combining audio and visual information, in order to improve speech quality and intelligibility in real-world noisy environments. The project is bringing together a critical mass of researchers with the complementary expertise necessary to make the audio-visual hearing-aid possible. The project will combine new contrasting approaches to audio-visual speech enhancement that have been developed by the Cognitive Computing group at Stirling and the Speech and Hearing Group at Sheffield. The Stirling approach uses the visual signal to filter out noise; whereas the Sheffield approach uses the visual signal to fill in 'gaps' in the speech. The vision processing needed to track a speaker's lip and face movement will use a revolutionary 'bar code' representation developed by the Psychology Division at Stirling. The MRC Institute of Hearing Research (IHR) will provide the expertise needed to evaluate the approach on real hearing loss sufferers. Phonak AG, a leading international hearing aid manufacturer, will provide the advice and guidance necessary to maximise potential for industrial impact.The project has been designed as a series of four workpackages that consider the key research challenges related to each component of the device's design. These questions have been identified by preliminary work at Sheffield and Stirling. Among the challenges are developing improved techniques for visually-driven audio-analysis; designing better metrics for weighting audio and visual evidence; developing techniques for optimally combining the noise-filtering and gap-filling approaches. A further key challenge is that, for a hearing aid to be effective, the processing cannot delay the signal by more than 10ms. In the final year of the project a full integrated, software prototype will be clinically evaluated using listening tests with hearing-impaired volunteers in a range of modern noisy reverberant environments. Evaluation will use a new purpose-built speech corpus that will be designed specifically for testing this new class of multimodal device. The project's clinical research partner, the Scottish Section of MRC IHR, will provide advice on the experimental design and analysis aspects throughout the trials. Industry leader Phonak AG will provide advice and technical support for benchmarking real-time hearing devices. The final clinically-tested prototype will be made available to the whole hearing community as a testbed for further research, development, evaluation and benchmarking.

当前的商业助听器使用多种复杂的增强技术来尝试提高语音信号的质量。但是，今天的最佳援助在许多日常情况下都无法正常工作。特别是，他们在繁忙的社交场合中失败了，那里有许多相互竞争的演讲来源。如果说话者离听众太远并且被噪音淹没了，他们会失败。我们已经通过建立可以“看到”的助听器来确定了解决此问题的机会。这个雄心勃勃的项目旨在开发新一代的助听器技术，该技术通过使用摄像头来看看说话者在说什么，从而从噪音中提取语音。该设备的佩戴者将能够将听力集中在目标谈话者上，该设备将过滤效果。这种能力超出了当前技术的能力，有可能改善遭受听力损失的数百万人的生活质量（仅在英国就超过1000万）。我们的方法与正常的听力是一致的。听众自然地结合了耳朵和眼睛的信息：我们用眼睛帮助我们听到。当听语音时，眼睛遵循面部和嘴巴的运动，以及一个复杂的多阶段过程，使用此信息将语音与噪音分开并填补任何空白。我们的助听器将以相同的方式采取行动。它将利用相机（例如，使用Google Glass（例如系统）和新颖的算法来利用视觉信息，以智能地组合音频和视觉信息，以提高现实世界噪声环境中的语音质量和清晰度。该项目正在汇集大量的研究人员，并拥有使视听助听器成为可能的必要互补专业知识。该项目将结合由斯特林认知计算小组和谢菲尔德的语音和听力组开发的新的对比方法来增强音频语音的增强方法。 Stirling方法使用视觉信号来过滤噪声。而谢菲尔德的方法则使用视觉信号来填补演讲中的“空白”。跟踪说话者的嘴唇和面部运动所需的视觉处理将使用斯特林心理学部门开发的革命性的“条形码”代表。 MRC听力研究所（IHR）将提供评估实际听力损失患者的方法所需的专业知识。国际领先的助听器制造商Phonak AG将提供最大程度地发挥工业影响潜力的建议和指导。该项目被设计为四个工作包，这些工作包考虑与设备设计的每个组件相关的关键研究挑战。这些问题已通过谢菲尔德和斯特林的初步工作来确定。挑战之一是为视觉驱动的音频分析开发改进的技术；设计更好的指标来加权音频和视觉证据；开发技术，可最佳地结合噪声过滤和缝隙填充方法。另一个关键的挑战是，为了使助听器有效，该处理不能将信号延迟超过10ms。在该项目的最后一年中，完整集成的软件原型将使用在一系列现代嘈杂的混响环境中的听力受损志愿者进行临床评估。评估将使用新的专门构建的语音语料库，该语料库专门用于测试这一新类别的多模式设备。该项目的临床研究合作伙伴MRC IHR的苏格兰部分将提供有关整个试验中实验设计和分析方面的建议。行业负责人Phonak AG将为实时听力设备进行基准测试建议和技术支持。最终临床测试的原型将用于整个听证社区，作为进一步研究，开发，评估和基准测试的测试床。

项目成果

Fast Lip Feature Extraction Using Psychologically Motivated Gabor Features

• DOI: 10.1109/ssci.2018.8628931
• 发表时间: 2018-01-01
• 期刊: 2018 IEEE SYMPOSIUM SERIES ON COMPUTATIONAL INTELLIGENCE (IEEE SSCI)
• 作者: Abel, Andrew;Gao, Chengxiang;Hussain, Amir
• 通讯作者: Hussain, Amir

A Novel Spatiotemporal Longitudinal Methodology for Predicting Obesity Using Near Infrared Spectroscopy (NIRS) Cerebral Functional Activity Data

• DOI: 10.1007/s12559-017-9541-x
• 发表时间: 2018-01
• 期刊: Cognitive Computation
• 影响因子: 5.4
• 作者: A. Abdullah;A. Hussain;Imtiaz Hussain Khan

Lip-Reading Driven Deep Learning Approach for Speech Enhancement

• DOI: 10.1109/tetci.2019.2917039
• 发表时间: 2021-06-01
• 期刊: IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE
• 影响因子: 5.3
• 作者: Adeel, Ahsan;Gogate, Mandar;Whitmer, William M.
• 通讯作者: Whitmer, William M.

An Enhanced Binary Particle Swarm Optimization (E-BPSO) algorithm for service placement in hybrid cloud platforms

• DOI: 10.1007/s00521-022-07839-5
• 发表时间: 2018-06
• 期刊: Neural Computing and Applications
• 影响因子: 6
• 作者: Wissem Abbes;Zied Kechaou;Amir Hussain;A. Qahtani;Omar Almutiry;Habib Dhahri;A. Alimi

Cognitively Inspired Audiovisual Speech Filtering: Towards an Intelligent, Fuzzy Based, Multimodal, Two-Stage Speech Enhancement System

认知启发的视听语音过滤：走向智能、模糊、多模态、两阶段语音增强系统

• 发表时间: 2015
• 作者: Abel Andrew