Video-based Speech Enhancement for Persons with Vision and Hearing Loss

针对视力和听力损失人士的基于视频的语音增强

• 批准号: 8443624
• 负责人: Ender Tekin
• 金额: $ 19.88万
• 依托单位: SMITH-KETTLEWELL EYE RESEARCH INSTITUTE
• 依托单位国家: 美国
• 财政年份: 2013
• 资助国家: 美国
• 起止时间: 2013-06-01 至 2015-05-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/8443624
• 关键词: Accounting; Acoustics; Activities of Daily Living; Address; Adult; Age; Age-Years; Aging-Related Process; Algorithms; Amplifiers; Area; Auditory; Blindness; Communication; Comprehension; Computer Vision Systems; Cues; Dependence; Detection; Development; Devices; Digital Signal Processing; Effectiveness; Elderly; Environment; Facial Expression; Feedback; Grant; Hearing Aids; Human; Laboratories; Lead; Learning; Life; Lip structure; Literature; Location; Machine Learning; Measures; Methods; Modality; Modeling; Noise; Output; Performance; Persons; Play; Population; Presbycusis; Process; Quality of life; Research; Role; Self-Help Devices; Sensory; Sensory Aids; Shapes; Signal Transduction; Societies; Source; Speech; Speech Intelligibility; Speech Perception; Staging; System; Techniques; Testing; Time; United States; Vision; Visual; Visual impairment; Voice; base; computerized data processing; design; hearing impairment; improved; interest; novel strategies; performance tests; prototype; public health relevance; research study; social; sound; speech recognition; tool development; visual information

DESCRIPTION (provided by applicant): Video-based Speech Enhancement for Persons with Hearing and Vision Loss Project Summary It is estimated that by 2030, the number of people in the United States over the age of 65 will account for over 20% of the total population. Hearing and vision loss naturally accompanies the aging process. Persons with hearing loss can benefit from observing the visual cues from a speaker such as the shape of the lips and facial expression to greatly improve their ability to comprehend speech. However, persons with vision loss cannot make use of these visual cues, and have a harder time understanding speech, especially in noisy environments. Furthermore, people with normal vision can use visual information to identify a speaker in a group, which allows them to focus on this person. This can greatly benefit a person with hearing loss who may be using a device such as a sound amplifier or a hearing aid. A user with vision loss, however, needs to be provided with this speaker information to make optimal use of such devices. We propose developing a prototype device that will clean the speech signal from a target speaker and improve speech comprehension for persons with hearing and vision loss in everyday situations. In order to accomplish this task, we need to harness the visual cues that have so far largely been ignored in the design of assistive technolo- gies for persons with hearing loss. Our first aim is to learn speaker-independent visual cues that are associated with the target speech signal, and use these audio-visual cues to design speech enhancement algorithms that perform much better in noisy everyday environment than current methods which only utilize the audio signal. We will utilize a video camera and computer vision methods to design advanced digital signal processing techniques to enhance the target speech signals recorded through a microphone. Our second aim is to use the video and audio signals to detect and efficiently localize the visible speaker. The information regarding the location of the speaker of interest can then be used to efficiently perform speaker separation, as well as be provided to the user. Finally, we aim to implement these developed algorithms on a portable prototype system. We will test the performance of this system and improve the user-interface through user experiments in real-world situations as well as laboratory conditions. The end product will show the feasibility and importance of incorporating multiple modalities into sensory assistive devices, and set the stage for future research and development efforts.

描述（由申请人提供）：基于视频的语音增强听力和视力损失项目摘要据估计，到2030年，美国65岁以上的美国人数将占总数的20％以上人口。 听力和视力丧失自然伴随着老化过程。 听力损失的人可以从观察说话者的视觉提示中受益，例如嘴唇的形状和面部表情，从而大大提高他们理解语音的能力。 但是，视力丧失的人无法利用这些视觉提示，并且很难理解语音，尤其是在嘈杂的环境中。 此外，具有正常视觉的人可以使用视觉信息来识别小组中的说话者，这使他们可以专注于这个人。 这可以极大地使听力损失的人受益，他们可能正在使用声音放大器或助听器等设备。 但是，有视力丧失的用户需要提供此扬声器信息，以最佳使用此类设备。 我们建议开发一种原型设备，该设备将清除目标扬声器的语音信号，并改善日常情况下听力和视力丧失的人的语音理解。 为了完成这项任务，我们需要利用迄今为止在辅助技术的设计中为有听力损失的人设计的视觉提示。 我们的第一个目的是学习与目标语音信号相关的扬声器独立的视觉提示，并使用这些视听提示来设计语音增强算法，这些语音增强算法在嘈杂的日常环境中的性能要比仅利用音频信号的当前方法更好。 我们将利用摄像机和计算机视觉方法设计高级数字信号处理技术，以增强通过麦克风记录的目标语音信号。 我们的第二个目的是使用视频和音频信号来检测并有效地定位可见扬声器。 然后可以使用有关感兴趣的位置的信息有效地执行扬声器分离，并提供给用户。 最后，我们旨在在便携式原型系统上实现这些开发的算法。 我们将通过在现实世界中和实验室条件下的用户实验来测试该系统的性能，并通过用户实验来改善用户界面。 最终产品将显示将多种方式纳入感官辅助设备的可行性和重要性，并为未来的研发工作奠定了基础。