CHS: Small: Collaborative Research: APERTURE: Augmented Reality based Perception-Sensitive Robotic Gesture

CHS：小型：协作研究：APERTURE：基于增强现实的感知敏感机器人手势

• 批准号: 1909864
• 负责人: Thomas Williams
• 金额: $ 25.73万
• 依托单位: Colorado School of Mines
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-09-01 至 2023-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1909864&HistoricalAwards=false
• 关键词: CHS; Small; Collaborative; Research; APERTURE

How can a robot choose the "best" modality for drawing the attention of a human and communicating the needed information? This is the central theme of the project with the assumption that the human team mates of the robots use augmented reality (AR) based visualization. Specifically, this project plans the following three major research activities for enabling robots to communicate with human teammates in a way that is tailored to their teammates' current mental states: exploring how augmented reality technologies (such as the Microsoft Hololens) can be used to provide robots with new ways to communicate about objects, locations, and people in their environments, especially when used together with communication through spoken language; examining how technologies can non-invasively measure different aspects of teammates' mental states, including how much mental workload, perceptual workload, stress, and frustration they are experiencing; and determining how robots can choose the best way to communicate with their human teammates when the two technologies are combined, (for example, through language alone, AR visualizations alone, or both used together), based on those teammates' individual mental states. This system will then be used to test how it might improve the safety and productivity of underground workers, by allowing robots to communicate in a way that is more effective and less cognitively demanding. While the researchers will be investigating the effectiveness of these integrated technologies specifically within underground work environments, the research will also be applicable to a wide variety of areas, including eldercare, urban search-and-rescue, and space robotics, and will have broad scientific impact across both computer science and cognitive science.The above goals will be achieved through APERTURE, a novel framework integrating head-mounted augmented reality displays, a multimodal suite of noninvasive, lightweight, and field-ready physiological sensors (such as functional near-infrared spectroscopy (fNIRS), Electroencaphalography (EEG), Electrodermal Activity, Electrocardiogram (ECG), and Respiration sensors), and unmanned ground robots, within a cognitive robotic architecture. APERTURE will be built by integrating the Distributed Integrated Affect Reflection Cognition (DIARC) architecture with these robotic, augmented reality, and physiological hardware elements. The project will design and evaluate physiological sensing models, augmented reality gestural cues, and machine learning models for selecting between AR gestural cues based on neurophysiological data. The designed machine learning models will classify users' cognitive and affective states from this sensor data, and help the robots understand when and how to communicate based on users' cognitive and affective states. The novel AR approach to deictic gesture will help robots pick out the objects they are referring to through the use of visualizations displayed in their teammates' augmented reality headsets.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

机器人如何选择“最佳”方式来吸引人类的注意力并传达所需的信息？这是该项目的中心主题，假设机器人的人类队友使用基于增强现实 (AR) 的可视化。具体来说，该项目计划进行以下三项主要研究活动，以使机器人能够以适合其队友当前心理状态的方式与人类队友进行交流：探索如何使用增强现实技术（例如微软Hololens）提供机器人具有新的方式来交流环境中的物体、位置和人，特别是与口语交流一起使用时；研究技术如何能够非侵入性地测量队友心理状态的不同方面，包括他们正在经历多少心理工作量、感知工作量、压力和挫败感；并根据队友的个人心理状态，确定当两种技术结合时，机器人如何选择与人类队友沟通的最佳方式（例如，仅通过语言、单独通过 AR 可视化或两者一起使用）。然后，该系统将用于测试如何通过允许机器人以更有效且认知要求较低的方式进行通信来提高地下工人的安全性和生产力。虽然研究人员将特别研究这些集成技术在地下工作环境中的有效性，但该研究也将适用于多种领域，包括老年人护理、城市搜索和救援以及空间机器人，并将具有广泛的科学依据。上述目标将通过 APERTURE 来实现，APERTURE 是一种新颖的框架，集成了头戴式增强现实显示器、一套非侵入性、轻型和现场就绪的生理传感器（例如功能性近红外传感器）的多模式套件光谱学（fNIRS）、脑电图（EEG）、皮电活动、心电图（ECG）和呼吸传感器）以及认知机器人架构内的无人地面机器人。 APERTURE 将通过将分布式集成情感反射认知 (DIARC) 架构与这些机器人、增强现实和生理硬件元素集成来构建。 该项目将设计和评估生理传感模型、增强现实手势提示以及机器学习模型，用于根据神经生理学数据在 AR 手势提示之间进行选择。设计的机器学习模型将从传感器数据中对用户的认知和情感状态进行分类，并帮助机器人根据用户的认知和情感状态了解何时以及如何进行交流。新颖的 AR 指示手势方法将帮助机器人通过使用其队友的增强现实耳机中显示的可视化来挑选出它们所指的对象。该奖项反映了 NSF 的法定使命，并通过使用基金会的评估进行评估，被认为值得支持。智力价值和更广泛的影响审查标准。

项目成果

Using Augmented Reality to Better Study Human-Robot Interaction

使用增强现实更好地研究人机交互

• 发表时间: 2020-01
• 期刊: and Mixed Reality
• 作者: Williams, Tom;Hirshfield, Leanne;Tran, Nhan;Grant, Trevor;Woodward, Nicholas
• 通讯作者: Woodward, Nicholas

Toward Workload-Based Adaptive Automation: The Utility of fNIRS for Measuring Load in Multiple Resources in the Brain

迈向基于工作负载的自适应自动化：fNIRS 用于测量大脑多种资源负载的实用程序

• DOI: 10.1080/10447318.2023.2266242
• 发表时间: 2023-10-23
• 期刊: International Journal of Human–Computer Interaction
• 作者: Leanne Hirshfield;Christopher Wickens;Emily Doherty;Cara A. Spencer;Tom Williams;Lucas Hayne
• 通讯作者: Lucas Hayne

Towards an Understanding of Physical vs Virtual Robot Appendage Design

了解物理与虚拟机器人附件设计

• 发表时间: 2022-01
• 期刊: and Mixed Reality for Human-Robot Interaction
• 作者: Han, Zhao;Phan, Albert;Castro, Amia;Sandoval Garza, Fernando;Williams, Tom
• 通讯作者: Williams, Tom

Mixed-Reality Robot Behavior Replay: A System Implementation

混合现实机器人行为回放：系统实现

• 发表时间: 2022-01
• 期刊: AAAI Fall Symposium on Artificial Intelligence for Human-Robot Interaction (AI-HRI
• 作者: Han, Zhao;Williams, Tom;Yanco, Holly
• 通讯作者: Yanco, Holly

Crossing Reality: Comparing Physical and Virtual Robot Deixis

穿越现实：比较物理和虚拟机器人指示语

• DOI: 10.1145/3568162.3576972
• 发表时间: 2023-03
• 期刊: ACM/IEEE International Conference on Human-Robot Interaction
• 作者: Han, Zhao;Zhu, Yifei;Phan, Albert;Garza, Fernando Sandoval;Castro, Amia;Williams, Tom
• 通讯作者: Williams, Tom