Decoding inner speech: An AI approach to transcribing thoughts via EEG & EMG

解码内心言语：一种通过脑电图转录思想的人工智能方法

• 批准号: 10058047
• 负责人: Jose A CORTES-BRIONES
• 金额: $ 52.36万
• 依托单位: YALE UNIVERSITY
• 依托单位国家: 美国
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-15 至 2024-09-14
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/10058047
• 关键词: ALS2 gene; Algorithms; American; Architecture; Area; Articulation; Artificial Intelligence; Attention; Brain; Clinical; Communication; Complex; Data; Data Collection; Data Set; Development; Electroencephalography; Electromyography; Electrophysiology (science); Expert Systems; Fostering; Gender; Genetic Transcription; Hand; Impairment; Individual; Language; Lead; Learning; Letters; Linguistics; Location; Machine Learning; Maps; Measures; Mental Health; Mental disorders; Methods; Modeling; Muscle; Occupations; Output; Patients; Pattern; Performance; Persons; Psyche structure; Psychiatry; Quality of life; Reading; Signal Transduction; Social Interaction; Speech; Stimulus; Sum; System; Technology; Text; Thinking; Time; Training; Translating; Vocabulary; Voice; Writing; algorithm training; blind; computerized data processing; deep learning; deep neural network; design; digital; healthy volunteer; improved; innovation; large datasets; machine learning algorithm; multidisciplinary; performance tests; sound; speech accuracy

ABSTRACT Losing the capacity to communicate through language has a significant negative impact on a person’s autonomy, social interactions, occupation, mental health, and overall quality of life. Many people lose the capacity to speak and write but keep their thinking intact. Inner speech is internally and willfully generated, non-articulated verbal thoughts (e.g., reading in silence). Changes in the activation patterns of the brain’s language-related areas co-occur with inner speech and can be detected with electroencephalography (EEG). Furthermore, while inner speech doesn’t lead to any discernible voice sound or articulation, co-occurring low amplitude electrical discharges in the articulatory muscles can be detected with electromyography (EMG). The information about ongoing inner speech reflected in electrophysiological signals (EEG and EMG) can be used to transcribe inner speech into text or voice. Machine learning algorithms have been used for this purpose, however, the resulting systems have low accuracy and/or are constrained by very small vocabularies (~10 words). Furthermore, these systems need to be trained anew for each user, which significantly increases individual data-collection time. The development of ready-to-use/minimal-training (fine tuning) systems requires large training datasets that algorithms can use to learn high-level features capable of being transferred between individuals. Unfortunately, to date there are no available datasets that are large enough to train these systems. To tackle these issues, I have assembled a multidisciplinary team of collaborators from Google AI, Yale linguistics, and Yale Psychiatry to develop a state-of-the-art deep neural network to transcribe inner speech to text using EEG and EMG signals. This system will incorporate some of the latest advances in artificial intelligence and data processing developed by Google AI. It will be designed to transcribe phonemes, thus, in principle, will be able to transcribe any word. Furthermore, we will collect the largest (x120 times) multi-subject (n=150) electrophysiological (EEG+EMG) inner speech dataset to date (300 hrs. in total) to train the first ready- to-use/minimal-training inner speech transcriber system. The technology resulting from this study has the potential to radically improve the quality of life of thousands of patients by providing them with a fast method of communicating their verbal thoughts. Furthermore, by combining this system with one of the many text-to-speech AIs that are currently available, our system could potentially restore the patients’ capacity to produce conversational speech.

抽象的 失去通过语言进行交流的能力会对一个人的 许多人失去了自主权、社交互动、职业、心理健康和整体生活质量。 能够说和写，但保持完整的思维。 内部言语是内部有意产生的、非明确表达的言语思想（例如，阅读 大脑语言相关区域的激活模式的变化与内部言语同时发生。 而且可以通过脑电图（EEG）检测到，而内部言语不会导致任何结果。 可辨别的声音或发音，发音中同时发生低幅度放电 肌肉可以通过肌电图（EMG）来检测有关正在进行的内部反射言语的信息。 电生理信号（脑电图和肌电图）中的信息可用于将内心言语转录为文本或语音。 机器学习算法已用于此目的，但是所得系统的性能较低 准确性和/或受到非常小的词汇表（~10 个单词）的限制。 对每个用户进行重新培训，这显着增加了个人数据收集时间。 即用型/最小训练（微调）系统需要算法可以使用的大型训练数据集 学习能够在个体之间转移的高级特征不幸的是，迄今为止还没有。 足够大的可用数据集来训练这些系统。 为了解决这些问题，我组建了一个由来自 Google AI、耶鲁大学的多学科合作者组成的团队 语言学和耶鲁大学精神病学共同开发最先进的深度神经网络，将内心言语转录为 使用脑电图和肌电图信号的文本该系统将结合人工技术的一些最新进展。 谷歌人工智能开发的智能和数据处理功能将被设计用于转录电话。 原则上，将能够转录任何单词此外，我们将收集最大（x120倍）的多主题。 (n=150) 迄今为止的电生理 (EEG+EMG) 内部语音数据集（总共 300 小时），用于训练第一个准备好的 使用/最少训练内部语音转录系统。 这项研究产生的技术有可能从根本上改善人们的生活质量 为成千上万的患者提供了一种快速表达口头想法的方法。 此外，通过将该系统与当前可用的众多文本转语音人工智能之一相结合，我们的 系统有可能恢复患者进行对话的能力。