Automatic Multimodal Affect Detection for Research and Clinical Use

用于研究和临床应用的自动多模式情感检测

• 批准号: 9912818
• 负责人: JEFFREY F COHN
• 金额: $ 58.17万
• 依托单位: UNIVERSITY OF PITTSBURGH AT PITTSBURGH
• 依托单位国家: 美国
• 财政年份: 2012
• 资助国家: 美国
• 起止时间: 2012-05-01 至 2022-04-30
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/9912818
• 关键词: Acoustics; Adolescent; Adult; Affect; Affective; Anxiety; Australia; Behavior; Behavioral; Behavioral Sciences; Clinical; Code; Communities; Computer software; Computers; Counseling; Data; Databases; Detection; Development; Disease; Emotional; Emotions; Ensure; Environment; Etiology; Face; Facial Expression; Family; Goals; Human; Intervention; Laboratories; Learning; Linear Models; Linguistics; Machine Learning; Manuals; Measurement; Measures; Mental Depression; Mental Health; Methods; Modality; Modeling; Motion; National Institute of Child Health and Human Development; National Institute of Mental Health; Pain; Parents; Participant; Pattern; Personal Computers; Persuasive Communication; Procedures; Process; Psychopathology; Research; Research Personnel; Risk; Running; Scientist; Software Tools; Speech; Standardization; Subgroup; System; Testing; Time; Training; Training and Education; Triad Acrylic Resin; United States National Institutes of Health; Validation; Verbal Behavior; Visual; Voice; Voice Quality; Work; affective computing; base; clinical practice; conduct problem; deep learning; design; emotional functioning; follow-up; gaze; improved; insight; intelligent tutoring system; interpersonal conflict; multimodality; protective factors; psychological distress; shared database; social skills; tool; treatment response

Project Summary A reliable and valid automated system for quantifying human affective behavior in ecologically important naturalistic environments would be a transformational tool for research and clinical practice. With NIMH support (MH R01-096951), we have made fundamental progress toward this goal. In the proposed project, we extend current capabilities in automated multimodal measurement of affective behavior (visual, acoustic, and verbal) to develop and validate an automated system for detecting the constructs of Positive, Aggressive, and Dysphoric behavior and component lower-level affective behaviors and verbal content. The system is based on the manual Living in Family Environments Coding System that has yielded critical findings related to developmental psychopathology and interpersonal processes in depression and other disorders. Two models will be developed. One will use theoretically-derived features informed by previous research in behavioral science and affective computing; the other empirically derived features informed by Deep Learning. The models will be trained in three separate databases of dyadic and triadic interaction tasks from over 1300 adolescent and adult participants from the US and Australia. Intersystem reliability with manual coding will be evaluated using k-fold cross-validation for both momentary and session level summary scores. Differences between models and in relation to participant factors will be tested using the general linear model. To ensure generalizability, we further will train and test between independent databases as well. To evaluate construct validity of automated coding, we will use the ample validity data available in the three databases to determine whether automated coding achieves the same or better pattern of findings with respect to depression risk and development. Following procedures already in place for sharing databases and software tools, we will design the automated systems for use by non-specialists and make them available for research and clinical use. Achieving these goals will provide behavioral science with powerful tools to examine basic questions in emotion, psychopathology, and interpersonal processes; and clinicians to improve assessment and ability to track change in clinical and interpersonal functioning over time. Relevance For behavioral science, automated coding of affective behavior from multimodal (visual, acoustic, and verbal) input will provide researchers with powerful tools to examine basic questions in emotion, psychopathology, and interpersonal processes. For clinical use, automated measurement will help clinicians to assess vulnerability and protective factors and response to treatment for a wide range of disorders. More generally, automated measurement would contribute to advances in intelligent tutors in education, training in social skills and persuasion in counseling, and affective computing more broadly.

项目概要 一种可靠且有效的自动化系统，用于量化具有重要生态意义的人类情感行为 自然环境将成为研究和临床实践的变革工具。与NIMH 支持（MH R01-096951），我们已经朝着这一目标取得了根本性进展。在拟议的项目中，我们 扩展当前情感行为（视觉、听觉和情感行为）自动多模式测量的能力 口头）开发和验证一个自动化系统，用于检测积极、攻击性和攻击性的结构 烦躁行为和低级情感行为和言语内容的组成部分。该系统基于 《生活在家庭环境编码系统》手册已得出与以下方面相关的重要发现： 抑郁症和其他疾病的发展精神病理学和人际关系过程。两种型号 将被开发。人们将使用从先前的行为研究中得出的理论推导的特征 科学和情感计算；由深度学习提供的其他经验得出的特征。这 模型将在来自 1300 多个二元和三元交互任务的三个独立数据库中进行训练 来自美国和澳大利亚的青少年和成人参与者。 手动编码的系统间可靠性将使用 k 倍交叉验证来评估 和会话级别总结分数。模型之间以及与参与者因素相关的差异将是 使用一般线性模型进行测试。为了确保普遍性，我们将进一步在之间进行训练和测试 还有独立的数据库。为了评估自动编码的构造有效性，我们将使用充足的 三个数据库中可用的有效性数据，以确定自动编码是否达到相同或 关于抑郁症风险和发展的更好的发现模式。以下程序已在 共享数据库和软件工具的地方，我们将设计供非专业人员使用的自动化系统 并使它们可供研究和临床使用。实现这些目标将提供行为科学 拥有强大的工具来检查情感、精神病理学和人际过程中的基本问题；和 临床医生提高评估和跟踪临床和人际功能随时间变化的能力。 关联 对于行为科学，多模态（视觉、听觉和语言）情感行为的自动编码 输入将为研究人员提供强大的工具来检查情感、精神病理学、 和人际交往过程。对于临床使用，自动测量将帮助临床医生评估 脆弱性和保护性因素以及对多种疾病治疗的反应。更一般地说， 自动化测量将有助于教育、社会培训领域智能导师的进步 咨询方面的技能和说服力，以及更广泛的情感计算。