Building normative models of Reinforcement Learning Decision Making Behavior

建立强化学习决策行为的规范模型

• 批准号: 10572615
• 负责人: POORNIMA KUMAR
• 金额: $ 24.6万
• 依托单位: MCLEAN HOSPITAL
• 依托单位国家: 美国
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-12-01 至 2024-11-30
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/10572615
• 关键词: Adopted; Age; Bayesian Method; Behavior; Behavior Disorders; Behavioral; Case/Control Studies; Clinical; Communities; Complex; Computer Models; Conflict (Psychology); Data; Data Reporting; Decision Making; Diagnosis; Diagnostic; Disease; Environment; Ethnic Origin; Failure; Family; Gender; Goals; Grain; Growth; Heterogeneity; Impairment; Individual; Learning; Link; Location; Longevity; Maps; Mathematics; Measures; Mental disorders; Modeling; Normal Range; Outcome; Participant; Patient Self-Report; Patients; Pediatrics; Population; Process; Psychiatry; Psychological reinforcement; Psychopathology; Research; Research Domain Criteria; Risk; Sample Size; Sampling; Shapes; Stimulus; Stratification; Structure; Symptoms; Testing; Time; Uncertainty; Update; aged; clinical phenotype; clinical practice; cohort; computer framework; cost; discounting; flexibility; improved; mathematical model; novel; precision medicine; psychiatric symptom; success; virtual

The main objective in the field of psychiatry is to be able to treat patients at an individual level. To reach this goal of precision medicine, large scale initiatives such as the RDoC have been developed to find new ways to parse heterogeneity in psychiatric disorders. Their success has been slow partly due to a slow transition away from case-control studies based on diagnoses and limitations due to small sample size. Therefore, there is a critical need to find alternative solutions at an affordable cost. One strategy is to identify a complex behavior such as Reinforcement Learning based decision making (RLDM) that is impaired across various psychiatric disorders and adopt a computational framework to explain heterogeneity at an individual level. RLDM is a multifaceted construct involving several sub-processes ranging from estimating values of different options in the environment (valuation), accumulating evidence for these options (sequential sampling), choosing the best option (explore- exploit behavior), estimating the outcome value (salience attribution) and lastly integrating relevant information about outcomes and updating the value of stimuli (learning rate). These sub-processes can be quantified by utilizing computational models. However, prior to building normative models of these RLDM constructs that can be potentially utilized in clinical practice, it is critical to assess the reliability of these RLDM model-derived parameters to avoid translational failures. Therefore, our main goals of this proposal are to (1) parse RLDM sub- processes into mathematically-defined parameters in a large sample using a diverse set of tasks; (2) assess test-retest reliability of these parameters; and finally (3) build normative models of the parameters and chart the heterogeneity at the level of the individual. To achieve this goal, we will acquire behavioral data from six RL tasks, including delay and effort discounting, probabilistic learning, virtual patch foraging, Pavlovian instrumental transfer and approach-avoid conflict tasks. These tasks are each optimized to measure at least two of the RL sub-processes separately. We will collect behavioral data and self-report assessments from a community sample from 1000 participants (aged 18-85). To estimate test-retest reliability of RLDM parameters, we will invite 500 healthy participants from this sample to complete the session again in one week’s time. First, we will apply start- of-art computational models to quantify RLDM behavior in each subject. Second, we will calculate the test-retest reliability of these parameters. Third, we will build normative models to link each of the RLDM construct with age and calculate each subject’s deviation from the norm. Lastly, we will conduct soft clustering on these deviations to identify clusters and investigate their differences in psychopathology and general functioning.

精神病学领域的主要目标是能够在个体层面上治疗患者以实现这一目标。 在精准医学领域，诸如 RDoC 之类的大规模举措已经被开发出来，以寻找新的解析方法 他们在精神疾病方面的异质性进展缓慢，部分原因是从精神疾病的转变缓慢。 基于诊断的病例对照研究由于样本量较小而存在局限性。 需要以可承受的成本找到替代解决方案，一种策略是识别复杂的行为，例如 基于强化学习的决策（RLDM）在各种精神疾病中受到损害 并采用计算框架来解释个体层面的异质性是一个多方面的问题。 涉及多个子过程的构造，范围从估计环境中不同选项的值 （评估），积累这些选项的证据（顺序抽样），选择最佳选项（探索- 利用行为），估计结果价值（显着性归因），最后整合相关信息 关于结果和更新刺激的价值（学习率）这些子过程可以通过以下方式量化。 然而，在建立这些 RLDM 结构的规范模型之前，可以 为了有可能在临床实践中使用，评估这些 RLDM 模型衍生的可靠性至关重要 因此，我们该提案的主要目标是（1）解析 RLDM 子项。 (2) 使用一组不同的任务将过程评估为大样本中数学定义的参数； 重测这些参数的可靠性；最后（3）建立参数的规范模型并绘制图表 为了实现这一目标，我们将从六个强化学习中获取行为数据。 任务，包括延迟和努力折扣、概率学习、虚拟补丁觅食、巴甫洛夫乐器 转移和避免冲突任务这些任务都经过优化以测量至少两个 RL。 我们将从社区样本中收集行为数据和自我报告评估。 1000 名参与者（18-85 岁）为了评估 RLDM 参数的重测可靠性，我们将邀请 500 名参与者。 该样本中的健康参与者在一周内再次完成会话首先，我们将申请开始-。 最先进的计算模型来量化每个受试者的 RLDM 行为 其次，我们将计算重测结果。 第三，我们将规范地构建模型，将每个 RLDM 结构与年龄联系起来。 并计算每个受试者与常态的偏差，最后，我们将对这些偏差进行软聚类。 识别集群并研究它们在精神病理学和一般功能方面的差异。