RAPID/Collaborative Research: Human-AI Teaming for Big Data Analytics to Enhance Response to the COVID-19 Pandemic

快速/协作研究：人类与人工智能合作进行大数据分析以增强对 COVID-19 大流行的响应

• 批准号: 2029692
• 负责人: Keri Stephens
• 金额: $ 2.03万
• 依托单位: University of Texas at Austin
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-05-15 至 2021-04-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2029692&HistoricalAwards=false
• 关键词: RAPID; Collaborative; Research; Human; AI

Social media data can provide important clues and local knowledge that can help emergency managers and responders better comprehend and capture the evolving nature of many disasters. Yet humans alone cannot grasp the vast data generated by social media, so computers are used to assist. Very little is currently known about how to leverage the skills of humans and machines when they work together (human-machine teaming) to identify meaningful patterns in social media data. Therefore, the fundamental issues this Rapid Response Research (RAPID) project seeks to address are 1) understanding the process of real-time decisions that human digital volunteers make when they rapidly convert social media data into structured codes the machine (Artificial Intelligence algorithms) can understand, and 2) using this knowledge to improve human-machine teaming. This project advances the field by revealing the unique abilities that both humans and machines bring when working together to comprehend social media patterns during an evolving disaster. It supports education and diversity by providing research experiences to diverse students, as well as generating data useful for interdisciplinary courses teaching teamwork, social media analysis, and human-machine teaming. Finally, the findings can help emergency managers better train their volunteers who comb through social media using their understanding of the local knowledge and built environment to help machines see new patterns in data. Hence, this project supports NSF's mission to promote the progress of science and to advance the nation's health, prosperity, and welfare by articulating the unique value that both humans and computers bring that can lead to better decisions during disasters. The goal of this research is to better understand the real-time decisions that human annotators make under different environmental constraints, and how those contribute to the learning of Artificial Intelligence (AI) models. Under time constraints and information overload, human decision-making capabilities are limited; yet, humans still have a unique ability to understand the contextual references to the structures in the built environment that machines cannot recognize. For example, the meaning of the tweet, “Memorial is overloaded,” -- which means the hospital, called Memorial, is out of beds for patients —- can be lost on AI systems that lack the knowledge of the built environment. This example demonstrates the value that humans in the loop offer in a human-AI teaming context. This research focuses on capturing the ephemeral data from a variety of social media sources and our two research thrusts include: 1) online observations of Community Emergency Response Team (CERT) volunteers and a manager (a collaborator on this project) using think-aloud and cognitive interviewing strategies to reveal the real-time mental models used to make coding decisions for annotation tasks; and 2) an empirical analysis of different sampling algorithms for active (machine) learning paradigms to develop a typology of machine errors under diverse contexts that affect the quality of human decision making for annotation. This research will generate design guidelines that bridge the gap between the mechanisms used for real-time data processing with AI models and the understanding of context contributed by a human user teaming with the AI models. Using theories of human decision-making combined with knowledge of how AI functions, this project provides a real-time, mid-disaster examination of 1) how humans understand, process, and interpret social media messages, and 2) how to refine AI algorithms to optimize active learning paradigm. This understanding will provide a theoretical framework enabling future research to develop protocols to optimize human-AI teaming by using concepts such as motivation and information theory. This work can help emergency managers conduct better training of their CERT volunteers and other annotators and provide clearer guidelines for how to communicate the unique value that humans bring to the annotation process for AI systems. Both our protocols and developed understanding of how humans interact with AI systems will be helpful for global health organizations, local and state-level disaster decision-makers, as well as provide direction for the vast CERT network in the United States.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

社交媒体数据可以提供重要的线索和本地知识，可以帮助应急管理者和响应者更好地理解和捕捉许多灾难的演变性质。然而，仅一个人就无法掌握社交媒体产生的大量数据，因此计算机被用于协助。目前，关于如何利用人类和机器的技能（人机组合）来识别社交媒体数据中有意义的模式时，目前知之甚少。因此，这项快速响应研究（快速）项目试图解决的基本问题是1）了解人类数字志愿者在快速将社交媒体数据转换为机器结构化代码时所做的实时决策的过程（人工智能算法）可以理解，而2）使用此知识来改善人机队伍。该项目通过揭示人类和机器共同努力在进化灾难中理解社交媒体模式时带来的独特能力来推动这一领域。它通过为潜水员的学生提供研究经验，并生成对跨学科课程有用的数据来支持教育和多样性，教授团队合作，社交媒体分析和人机团队。最后，这些发现可以帮助应急管理人员更好地培训他们的志愿者，他们使用对当地知识和建筑环境的理解来通过社交媒体梳理，以帮助机器看到数据中的新模式。因此，该项目支持NSF的使命，以促进科学进步，并通过阐明人类和计算机带来的独特价值来促进国家的健康，繁荣和福利，从而在灾难中带来更好的决策。这项研究的目的是更好地了解人类注释者在不同的环境限制下做出的实时决策，以及那些如何促进人工智能（AI）模型的贡献。在时间限制和信息超载下，人类决策能力有限；然而，人类仍然具有理解机器无法识别的建筑环境中结构的上下文引用的独特能力。例如，推文“纪念馆超载”的含义（这意味着医院称为纪念馆，都在患者的床上 - 可能会在缺乏建筑环境知识的AI系统上丢失。这个示例证明了人类在人类组合背景下提供的价值。这项研究的重点是从各种社交媒体来源捕获短暂的数据，我们的两个研究作用包括：1）在线观察社区应急小组（CERT）志愿者和一名经理（该项目的合作者），使用思想和认知访谈策略，以揭示用于揭示注释任务的实时精神模型； 2）对主动（机器）学习范式的不同采样算法的经验分析，以在潜水员环境下开发机器错误的类型，从而影响注释人类决策的质量。这项研究将生成设计指南，以弥合用AI模型实时数据处理的机制之间的差距，以及对人类用户与AI模型构成的上下文的理解。使用人类决策的理论以及对AI的功能的了解，该项目对1）对人类的理解，处理和解释社交媒体信息以及2）如何完善AI算法来优化主动学习范式进行实时的中心检查。这种理解将提供一个理论框架，使未来的研究能够通过使用诸如动机和信息理论之类的概念来开发协议，以优化人类AI团队。这项工作可以帮助应急管理人员对其证书志愿者和其他注释者进行更好的培训，并为如何传达人类带来的AI系统注释过程的独特价值提供更明确的指南。我们的协议和对人类如何与AI系统互动的理解都将有助于全球卫生组织，地方和州级灾难决策者，并为美国广泛的证书网络提供了指导。该奖项反映了NSF的法定任务，并通过使用该基金会的知识分子功能和广泛影响来评估CRITERIA，并通过评估来诚实地支持NSF的法定任务。