Bayesian-centric Multimodal Hands-free Computer Interaction Technologies for People with Quadriplegia

针对四肢瘫痪患者的以贝叶斯为中心的多模式免提计算机交互技术

• 批准号: 2113485
• 负责人: Xiaojun Bi
• 金额: $ 39.99万
• 依托单位: SUNY at Stony Brook
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-08-01 至 2024-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2113485&HistoricalAwards=false
• 关键词: Bayesian; centric; Multimodal; Hands; free

Interacting with computers remains a challenge for people with quadriplegia. Assistive technologies that enable hands-free interaction with computers are primarily based on eye-gaze, voice, and orally-controlled input modalities, each with its own strengths and weaknesses. However, these assistive technologies do not support collaborative use of multiple input modalities, such as using eye gaze to quickly narrow down the region containing the intended target for executing a spoken command. The overarching goal of the proposed project is to research, design and engineer intelligent and collaborative multimodal hands-free interaction techniques that synergistically combine inputs from different input modalities to accurately predict and act on the user's interaction intent. Synergistic integration of the input modalities and intelligent inferring of the user’s interaction intent amplify the collective strengths of the individual modalities while mitigating their weaknesses. More importantly, these techniques will also learn user-specific interaction patterns from the user’s interaction history for personalizing the prediction of each individual user’s intended action. Overall, the transformative assistive multimodal interaction system, SeeSayClick, that will emerge from this project, will make it far easier for people with quadriplegia to create and consume digital information and thereby fully participate in this digitized economy. The resulting higher productivity of such users will lead to improved access to education and employment opportunities. Lastly, this project will serve as a platform for training students and exposing them to careers in assistive technology development and rehabilitation engineering.The novelty of the envisioned SeeSayClick assistive technology will be the tight integration of multiple interaction modalities that will work together synergistically and resolve ambiguities in interaction, and as a consequence, reduce the interaction burden substantially. The basis for the integration will be rooted in Bayesian inference methods for human computer interaction. These methods provide a principled approach for combining multiple sources of information, possibly noisy, to predict the user's intended interaction action, such as combining: (1) the locational information from gaze with (2) the spoken commands, and (3) prior knowledge from the interaction context to infer the intended target's precise location for selection and execution. By incorporating interaction history as prior into the Bayesian methods, the proposed approach for integrating multiple input modalities will also learn user-specific interaction patterns to personalize the prediction and enhance the prediction accuracy even further for each individual user. Besides cursor operations and command execution, the Bayesian methods will be coupled to a language model for text entry and editing operations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

对于四肢瘫痪的人来说，与计算机的互动仍然是一个挑战。能够与计算机进行免提交互的辅助技术主要基于眼睛注视，声音和口服控制的输入方式，每种方式都具有自己的优点和劣势。但是，这些辅助技术不支持多种输入模式的协作使用，例如使用Eye Chaze快速缩小执行口语命令的预期目标的区域。拟议项目的总体目标是研究，设计和工程师智能和协作多模式的免提交互技术，该技术协同结合了来自不同输入方式的输入，以准确预测和采取用户的交互意图。输入方式的协同整合以及用户互动意图的智能推断会放大单个方式的集体优势，同时减轻其弱点。更重要的是，这些技术还将从用户的交互历史记录中学习特定于用户的交互模式，以个性化每个用户的预期操作的预测。总体而言，将从该项目中出现的变革性辅助多模式交互系统SeesayClick将使四肢瘫痪的人更容易创建和消费数字信息，从而充分参与这种数字化的经济。此类用户的生产率更高将导致改善获得教育和就业机会的机会。最后，该项目将作为培训学生并将他们暴露于辅助技术开发和康复工程领域的职业的平台。所设想的“ SeesayClick seeSayClick辅助技术”的新颖性将是多种互动模式的紧密整合，这些互动模式将协同工作并解决互动中的歧义性，并因此降低了相互作用的效果。集成的基础将植根于人类计算机相互作用的贝叶斯推理方法。这些方法提供了一种组合多个信息源的主要方法，可以预测用户预期的交互作用，例如组合：（1）从目光到（2）口语命令的位置信息以及（3）相互作用上下文中的先验知识以推断预定的目标位置的精确位置，以进行选择和执行。通过将互动历史记录作为先验记录到贝叶斯方法中，提出的集成多个输入模式的方法还将学习特定于用户的交互模式，以个性化预测并进一步提高每个单个用户的预测准确性。除了光标操作和命令执行外，贝叶斯方法还将与文本输入和编辑操作的语言模型结合起来。该奖项反映了NSF的法定任务，并被认为是通过基金会的知识分子的智力优点和更广泛影响的评估标准来通过评估来获得支持的。

项目成果

GlanceWriter: Writing Text by Glancing Over Letters with Gaze

• DOI: 10.1145/3544548.3581269
• 发表时间: 2023-04
• 期刊: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
• 作者: Wenzhe Cui;R. Liu;Zhi Li;Yifan Wang;Andrew Wang;Xia Zhao;S. Rashidian;Furqan Baig;I. Ramakrishnan;Fusheng Wang;Xiaojun Bi

EyeSayCorrect: Eye Gaze and Voice Based Hands-free Text Correction for Mobile Devices

EyeSayCorrect：适用于移动设备的基于眼睛注视和语音的免提文本校正

• DOI: 10.1145/3490099.3511103
• 发表时间: 2022
• 期刊: IUI '22: 27th International Conference on Intelligent User Interfaces
• 作者: Zhao, Maozheng;Huang, Henry;Li, Zhi;Liu, Rui;Cui, Wenzhe;Toshniwal, Kajal;Goel, Ananya;Wang, Andrew;Zhao, Xia;Rashidian, Sina
• 通讯作者: Rashidian, Sina