Crowdsourcing Labels and Explanations to Build More Robust, Explainable AI/ML Activity Models

众包标签和解释以构建更强大、可解释的 AI/ML 活动模型

• 批准号: 10833847
• 负责人: Diane Joyce Cook
• 金额: $ 30.56万
• 依托单位: WASHINGTON STATE UNIVERSITY
• 依托单位国家: 美国
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-30 至 2025-05-31
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/10833847
• 关键词: Active Learning; Aging; Agreement; Alzheimer' s disease related dementia; Area; Behavior; Categories; Clinical; Clinical Research; Collection; Communities; Data; Data Collection; Data Set; Elderly; Foundations; Goals; Health; Individual; Intervention; Label; Learning; Machine Learning; Mechanics; Methods; Modeling; Neuropsychology; Outcome; Performance; Population; Process; Research; Sampling; Series; Software Engineering; Technology; Text; Time; Training; Trust; Uncertainty; Visual; Visualization; citizen science; crowdsourcing; data tools; data visualization; diverse data; experience; health application; health assessment; learning strategy; mHealth; machine learning model; multimodality; parent project; recruit; sensor; smart watch; tool; wearable sensor technology

PROJECT SUMMARY / ABSTRACT As the population of individuals 65+ grows from 58 million to 88 million by 2050, so too will the number of individuals who are aging with Alzheimer's disease and related dementias (ADRDs)1. The parent project introduces clinically-driven technological methods to automate assessment of an older adult's functional health from multimodal sensor data. What is lacking in the community, and in our parent project, is the availability of ground-truth smartwatch activity labels. Without a sufficient amount of labeled data, machine learning models cannot learn robust behavior models and use these models for functional health prediction. Additionally, the categories of activities that have corresponding labels are very skewed, further limiting machine learning performance because of the classical imbalanced class distribution problem. In this supplement request, we propose to dramatically increase the availability of labeled smartwatch data for our parent project and for the field. To do this, we will create a mechanism to crowdsource activity labels through Amazon Mechanical Turk. Additionally, we will capitalize on the crowdsourcing opportunity to push the parent project to the next step by laying a foundation for explainable machine learning models. Once our target number of activity labels is reached, we will initiate a second round of crowdsourcing by asking citizen scientists to create one-sentence explanations of the visualized data corresponding to an activity instance. The supplement project will contain four tasks. First, we will create a visualization and data point-selection tool for use in the Amazon Mechanical Turk (AMT) forum for data collection. A baseline active learning strategy will be used to collect an initial set of labels and create a baseline model, after which, the active learning and annotator selection strategies will be refined to collect the remainder of the activity labels. Finally, diverse data points from each modeled category will be displayed to collect a set of text captions for training explanation models. The outcome of this supplement project will be one of the largest sets of activity labeled smartwatch data collected “in the wild.” The labeled datasets created by this supplement will offer a foundation for a multitude of health studies that can utilize activity information observed by continuous wearable sensor readings collected in real-world studies. For the parent project, the amount of labeled data will increase by over 10,000%. The supplement will also offer a starting point for creating explainable mobile health AI/ML tools.

项目概要/摘要 到 2050 年，随着 65 岁以上人口数量从 5800 万增加到 8800 万， 患有阿尔茨海默病和相关痴呆症 (ADRD) 的个体1 父项目。 引入临床驱动的技术方法来自动评估老年人的功能健康 社区和我们的父项目中缺乏的是多模式传感器数据的可用性。 地面实况智能手表活动标签。如果没有足够数量的标记数据，机器学习模型。 无法学习稳健的行为模型并使用这些模型进行功能健康预测。 具有相应标签的活动类别非常倾斜，进一步限制了机器学习 由于经典的不平衡类别分配问题而导致的性能在这个补充请求中，我们。 显着提高我们父项目和 为此，我们将创建一种通过 Amazon Mechanical Turk 众包活动标签的机制。 此外，我们将利用众包机会，通过以下方式将父项目推向下一步： 一旦我们的活动标签数量达到目标，就为可解释的机器学习模型奠定基础。 达成后，我们将启动第二轮众包，要求公民科学家创造一句话 补充项目将包含与活动实例相对应的可视化数据的解释。 首先，我们将创建一个在 Amazon Mechanical 中使用的可视化和数据点选择工具。 Turk (AMT) 数据收集论坛将使用基线主动学习策略来收集初始数据集。 标签并创建基线模型，之后，主动学习和注释器选择策略将是 最后，收集每个建模类别的不同数据点。 将显示收集一组用于训练解释模型的文本标题本补充的结果。 该项目将是“野外”收集的最大的标记智能手表数据的活动之一。 该补充文件创建的数据集将为大量健康研究提供基础，这些研究可以利用 通过在现实世界研究中收集的连续可穿戴传感器读数观察到的活动信息。 父项目中，标记数据量将增加 10,000% 以上。 创建可解释的移动健康 AI/ML 工具的起点。

项目成果

Partnering a Compensatory Application with Activity-Aware Prompting to Improve Use in Individuals with Amnestic Mild Cognitive Impairment: A Randomized Controlled Pilot Clinical Trial.

将补偿性应用程序与活动感知提示相结合，以改善患有遗忘性轻度认知障碍的个体的使用：一项随机对照试点临床试验。

• 发表时间: 2022
• 作者: Schmitter;Brown, Katelyn;Luna, Catherine;Chilton, Reanne;Sumida, Catherine A;Holder, Lawrence;Cook, Diane
• 通讯作者: Cook, Diane