Studies on Multimodal Communication by Integrating Speech and Diagram

语音与图解结合的多模态通信研究

• 批准号: 08458078
• 负责人: DOSHITA Shuji
• 金额: $ 4.67万
• 依托单位: KYOTO UNIVERSITY
• 依托单位国家: 日本
• 项目类别: Grant-in-Aid for Scientific Research (B)
• 财政年份: 1996
• 资助国家: 日本
• 起止时间: 1996 至 1997
• 项目状态: 已结题
• 来源: https://kaken.nii.ac.jp/grant/KAKENHI-PROJECT-08458078/
• 关键词: Multimodal Communication; Dialogue; Speech Recognition; Gesture; Drawing System; Information Integration

The aim of this research is to realize multimodal communication between human and machines through implementing multimodal drawing system with speech input. The results of this research are below :1. Integration of speech, gesture and diagramWe developed multi-mode integration method without assuming the synchronization of each input elements. The validity of this integration method was examined by evaluating the improvement of accuracy of information input in multimodal dictation system.2. Implementation of multimodal interface using speech and pointing gestureWe developed the cooperation mechanism which uses the result of pointing gesture recognition, which is reliable channel, for the recognition of speech, which is relatively unreliable channel. By using the result of pointing gesture recognition, the accuracy of speech recognition raised about 6%. Using this integration method, we implemented a multimodal drawing system with speech input.3. Interpretation of utterance using diagrammatic and contextual information in multimodal drawing system We realized utterance understanding mechanism which uses context management mechanism by stack in order to deal with spontaneous speech (e. g. ellipsis, reference expression, etc.). We integrated this utterance understanding mechanism to multimodal drawing system.

本研究的目的是通过实现具有语音输入的多模态绘图系统来实现人与机器之间的多模态通信。本研究的结果如下： 1．语音、手势和图表的集成我们开发了多模式集成方法，无需假设每个输入元素的同步。通过评价多模态听写系统信息输入准确率的提高情况，检验了该集成方法的有效性。 2．使用语音和指向手势实现多模态界面我们开发了一种协作机制，该机制使用指向手势识别的结果（这是可靠的通道）来识别语音（这是相对不可靠的通道）。利用指向手势识别的结果，语音识别的准确率提高了约6%。利用这种集成方法，我们实现了一个带有语音输入的多模态绘图系统。3．在多模态绘图系统中使用图解和上下文信息解释话语我们实现了话语理解机制，该机制使用堆栈的上下文管理机制来处理自发语音（例如省略号、参考表达等）。我们将这种话语理解机制集成到多模态绘图系统中。

项目成果

Doshita,S.: "UNDERSTANDING AND GENERATING DIALOGUE BY INTEGRATING PRO-CESSING OF SPEECH,LANGUAE AND CONCEPT" Proc.of International Symposium on Spoken Dialogue '96. 1-8 (1996)

Doshita,S.：“通过整合语音、语言和概念处理来理解和生成对话”Proc.of 国际口语研讨会 96。

河原達也: "音声言語を用いた仮想空間との対話による試着システム" 情報処理学会論文誌. (採録決定). (1998)

Tatsuya Kawahara：“使用口语与虚拟空间交互的试穿系统”日本信息处理学会杂志（已接受）（1998 年）。

荒木 雅弘: "マルチモーダル作図システムにおける文脈知識を利用した会話理解" 人工知能学会研究会資料SIG-SLUD-9703-1. (1998)

Masahiro Araki：“在多模态图表系统中使用上下文知识进行对话理解”人工智能研究小组材料 SIG-SLUD-9703-1 (1998)。

Araki, M.: "Automatic Evaluation Environment for Spoken Dialogue Systems" in Mayr, E.et al.ed. "Dialogue Processing in Spoken Language Systems". (1997)

Araki, M.：“口语对话系统的自动评估环境”，Mayr, E.et al.ed。

Doshita, S.: "Human-machine communication by speech" Ohmsha inc.(1998)

Doshita, S.：“通过语音进行人机通信”Ohmsha inc.(1998)