Studies on Multimodal Communication by Integrating Speech and Diagram

语音与图解结合的多模态通信研究

• 批准号: 08458078
• 负责人: DOSHITA Shuji
• 金额: $ 4.67万
• 依托单位: KYOTO UNIVERSITY
• 依托单位国家: 日本
• 项目类别: Grant-in-Aid for Scientific Research (B)
• 财政年份: 1996
• 资助国家: 日本
• 起止时间: 1996 至 1997
• 项目状态: 已结题
• 来源: https://kaken.nii.ac.jp/grant/KAKENHI-PROJECT-08458078/
• 关键词: Multimodal Communication; Dialogue; Speech Recognition; Gesture; Drawing System; Information Integration

The aim of this research is to realize multimodal communication between human and machines through implementing multimodal drawing system with speech input. The results of this research are below :1. Integration of speech, gesture and diagramWe developed multi-mode integration method without assuming the synchronization of each input elements. The validity of this integration method was examined by evaluating the improvement of accuracy of information input in multimodal dictation system.2. Implementation of multimodal interface using speech and pointing gestureWe developed the cooperation mechanism which uses the result of pointing gesture recognition, which is reliable channel, for the recognition of speech, which is relatively unreliable channel. By using the result of pointing gesture recognition, the accuracy of speech recognition raised about 6%. Using this integration method, we implemented a multimodal drawing system with speech input.3. Interpretation of utterance using diagrammatic and contextual information in multimodal drawing system We realized utterance understanding mechanism which uses context management mechanism by stack in order to deal with spontaneous speech (e. g. ellipsis, reference expression, etc.). We integrated this utterance understanding mechanism to multimodal drawing system.

这项研究的目的是通过用语音输入实施多模式绘图系统来实现人与机器之间的多模式通信。这项研究的结果如下：1。语音，手势和图表的集成开发了多模式积分方法，而无需假设每个输入元素的同步。通过评估多模式听写系统中信息输入的准确性的提高来检查这种整合方法的有效性。2。使用语音和指向手势的多模式界面实现了合作机制，该机制使用了指向手势识别的结果，这是可靠的通道，用于识别语音，这是相对不可靠的通道。通过使用指向手势识别的结果，语音识别的准确性提高了约6％。使用这种集成方法，我们实现了一个带有语音输入的多模式绘图系统3。使用示意图和上下文信息在多模式绘图系统中对话语进行解释，我们意识到说话理解机制，该机制使用堆栈使用上下文管理机制来处理自发的语音（例如，省略号，参考表达式等）。我们将这种话语理解机制纳入了多模式绘图系统。

项目成果

Doshita,S.: "UNDERSTANDING AND GENERATING DIALOGUE BY INTEGRATING PRO-CESSING OF SPEECH,LANGUAE AND CONCEPT" Proc.of International Symposium on Spoken Dialogue '96. 1-8 (1996)

Doshita,S.：“通过整合语音、语言和概念处理来理解和生成对话”Proc.of 国际口语研讨会 96。

河原達也: "音声言語を用いた仮想空間との対話による試着システム" 情報処理学会論文誌. (採録決定). (1998)

Tatsuya Kawahara：“使用口语与虚拟空间交互的试穿系统”日本信息处理学会杂志（已接受）（1998 年）。

荒木 雅弘: "マルチモーダル作図システムにおける文脈知識を利用した会話理解" 人工知能学会研究会資料SIG-SLUD-9703-1. (1998)

Masahiro Araki：“在多模态图表系统中使用上下文知识进行对话理解”人工智能研究小组材料 SIG-SLUD-9703-1 (1998)。

Araki, M.: "Automatic Evaluation Environment for Spoken Dialogue Systems" in Mayr, E.et al.ed. "Dialogue Processing in Spoken Language Systems". (1997)

Araki, M.：“口语对话系统的自动评估环境”，Mayr, E.et al.ed。

Doshita, S.: "Human-machine communication by speech" Ohmsha inc.(1998)

Doshita, S.：“通过语音进行人机通信”Ohmsha inc.(1998)