Integrating Human and Machine Learning for Enabling Co-Adaptive Body-Machine Interfaces

集成人类和机器学习以实现自适应体机接口

• 批准号: 2054406
• 负责人: Ferdinando Mussa-Ivaldi
• 金额: $ 71.42万
• 依托单位: Rehabilitation Institute of Chicago
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2054406&HistoricalAwards=false
• 关键词: Integrating; Human; Machine; Learning; Enabling

This project for the Mind, Machine, and Motor Nexus (M3X) program will advance understanding of how people learn new neuromotor skills, and subsequently apply that understanding to the creation of innovative wearable device controllers called body-machine interfaces (BoMIs). Individuals with neuromuscular impairment -- perhaps due to stroke or spinal cord injury -- may have difficulty carrying out the activities of everyday life. This project explores novel interfaces through which an individual can use their body's residual mobility to issue commands to assistive devices such as computer cursors, wheelchairs, or robotic arms. The project has three main research goals. The first is to improve existing methods for translating small body movements into controller commands for assistive devices. The second is to model the process by which the human user learns over time to use the body-machine interface. The third is to apply the obtained model of the learning process to enable the body-machine interface to adjust to the evolving characteristics of the human user. An interface that does not adapt to changes in its user may significantly degrade in performance over time. On the other hand, an interface whose properties instantly change with every small shift in user behavior will be difficult to control. The ultimate outcome of this project will be human-machine interfaces based on body movement that consider the user and the interface as two components of an integrated system in which each component continually learns from and adapts to the other. The results of the project will lead to assistive devices that more affordable, and provide more versatile control and ease of use. The underlying principles of co-adaptation to be identified through this work are also relevant to rehabilitation from disease or injury, as well as to increasing the capabilities of human-operated robotic systems.Recent work has shown that linear methods such as principal component analysis (PCA) may be effectively used in a body-machine interface (BoMI) to map elements from a higher dimensional feature space of body movements onto a lower dimensional space of device commands. In this project, the features that provide input to the BoMI are generated by multiple inertial measurement units (IMUs) worn by the user; the IMUs report their current orientation in an inertial reference space. The output from the BoMI are commands used to control a sequence of representative devices, specifically a computer cursor, a simulated wheelchair, an actual wheelchair, and a simulated manipulator arm. The three technical goals of the project are as follows: 1) Compare the performance of a linear map based on PCA to a nonlinear map based on an autoencoder network (AEN) for providing input features to the BoMI that translates residual mobility space features into device commands. The AEN is capable of representing a richer variety of features than PCA, but it remains to be shown, for example, whether human users can make effective use of that variety. 2) Obtain a computable representation of the process by which humans learn neuromotor skills. This representation will be based on the premise that humans simultaneously learn both a forward and inverse map of the relationship between neuromotor signals and the resulting physical outcomes. Once learned, the forward map predicts the outcomes that will result from a certain set of signals, while the inverse map is used to generate the signals that correspond to a given desired physical outcome. As a person learns mastery of a neuromotor skill, the forward and inverse maps become more accurate predictors of actual behavior, and the degree of learning can be monitored through estimates of these maps. 3) Incorporate a co-adaptation algorithm into the BoMI for maintaining performance as the user's mastery of the BoMI changes. In most current approaches to human-machine interfaces, the interface is fixed following an initial calibration stage, and the user must learn to control that interface configuration. In this project, the learning representation of objective (2) will be used to monitor and periodically update the BoMI map parameters. The implementation of this objective is aided by parallels between the human learning model and the AEN training method, which automatically generates a decoder network that captures the inverse map between desired device commands and the corresponding residual mobility features needed to produce them.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

这个 Mind、Machine 和 Motor Nexus (M3X) 计划将加深人们对如何学习新的神经运动技能的理解，并随后将这种理解应用于创建称为身体机器接口 (BoMI) 的创新可穿戴设备控制器。患有神经肌肉损伤的人（可能是由于中风或脊髓损伤）可能难以进行日常生活活动。该项目探索了新颖的界面，通过这些界面，个人可以利用身体的剩余活动能力向计算机光标、轮椅或机械臂等辅助设备发出命令。该项目有三个主要研究目标。首先是改进现有方法，将微小的身体动作转化为辅助设备的控制器命令。第二个是对人类用户随着时间的推移学习使用身体-机器界面的过程进行建模。第三是应用所获得的学习过程模型，使人机界面能够适应人类用户不断变化的特征。不适应用户变化的界面可能会随着时间的推移而显着降低性能。另一方面，如果界面的属性随着用户行为的每一个微小变化而立即改变，则将难以控制。该项目的最终成果将是基于身体运动的人机界面，将用户和界面视为集成系统的两个组件，其中每个组件不断学习并适应另一个组件。该项目的成果将导致辅助设备变得更加经济实惠，并提供更通用的控制和易用性。通过这项工作确定的共同适应的基本原则也与疾病或受伤的康复以及提高人类操作的机器人系统的能力相关。最近的工作表明，诸如主成分分析之类的线性方法（ PCA）可以有效地用在身体机器接口（BoMI）中，以将来自身体运动的较高维度特征空间的元素映射到设备命令的较低维度空间。在该项目中，向 BoMI 提供输入的特征是由用户佩戴的多个惯性测量单元 (IMU) 生成的； IMU 报告它们在惯性参考空间中的当前方向。 BoMI 的输出是用于控制一系列代表性设备的命令，特别是计算机光标、模拟轮椅、实际轮椅和模拟机械臂。该项目的三个技术目标如下： 1) 比较基于 PCA 的线性映射与基于自动编码器网络 (AEN) 的非线性映射的性能，以向 BoMI 提供输入特征，BoMI 将剩余移动空间特征转换为设备命令。 AEN 能够表示比 PCA 更丰富的特征，但仍有待证明，例如，人类用户是否可以有效地利用这些特征。 2）获得人类学习神经运动技能过程的可计算表示。这种表示将基于人类同时学习神经运动信号与由此产生的物理结果之间关系的正向和反向映射的前提。一旦学习，正向图就可以预测由一组特定信号产生的结果，而反向图则用于生成与给定的期望物理结果相对应的信号。当一个人学习掌握神经运动技能时，正向和反向图成为实际行为的更准确的预测器，并且可以通过对这些图的估计来监控学习程度。 3）将协同适应算法纳入BoMI，以便在用户对BoMI的掌握程度发生变化时保持性能。在大多数当前的人机界面方法中，界面在初始校准阶段之后是固定的，并且用户必须学会控制该界面配置。在该项目中，目标（2）的学习表示将用于监控并定期更新BoMI地图参数。这一目标的实现得益于人类学习模型和 AEN 训练方法之间的相似之处，该方法自动生成一个解码器网络，捕获所需设备命令与生成这些命令所需的相应剩余移动性特征之间的逆映射。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。