Brain mechanism of reward prediction under predictable and unpredictable environmental dynamics

In learning goal-directed behaviors, an agent has to consider not only the reward given at each state but also the consequences of dynamic state transitions associated with action selection. To understand brain mechanisms for action learning under predictable and unpredictable environmental dynamics, we measured brain activities by functional magnetic resonance imaging (fMRI) during a Markov decision task with predictable and unpredictable state transitions. Whereas the striatum and orbitofrontal cortex (OFC) were significantly activated both under predictable and unpredictable state transition rules, the dorsolateral prefrontal cortex (DLPFC) was more strongly activated under predictable than under unpredictable state transition rules. We then modelled subjects' choice behaviours using a reinforcement learning model and a Bayesian estimation framework and found that the subjects took larger temporal discount factors under predictable state transition rules. Model-based analysis of fMRI data revealed different engagement of striatum in reward prediction under different state transition dynamics. The ventral striatum was involved in reward prediction under both unpredictable and predictable state transition rules, although the dorsal striatum was dominantly involved in reward prediction under predictable rules. These results suggest different learning systems in the cortico-striatum loops depending on the dynamics of the environment: the OFC-ventral striatum loop is involved in action learning based on the present state, while the DLPFC-dorsal striatum loop is involved in action learning based on predictable future states. (c) 2006 Elsevier Ltd. All rights reserved.

在学习目标导向行为时，个体不仅要考虑每个状态下所给予的奖励，还要考虑与动作选择相关的动态状态转换的后果。为了理解在可预测和不可预测的环境动态下动作学习的大脑机制，我们在具有可预测和不可预测状态转换的马尔可夫决策任务期间，通过功能性磁共振成像（fMRI）测量了大脑活动。尽管纹状体和眶额皮质（OFC）在可预测和不可预测的状态转换规则下均显著激活，但背外侧前额叶皮质（DLPFC）在可预测的状态转换规则下比在不可预测的状态转换规则下激活更强。然后，我们使用强化学习模型和贝叶斯估计框架对受试者的选择行为进行建模，发现受试者在可预测的状态转换规则下采用更大的时间折扣因子。基于模型的fMRI数据分析揭示了在不同的状态转换动态下纹状体在奖励预测中的不同参与情况。腹侧纹状体在不可预测和可预测的状态转换规则下均参与奖励预测，尽管背侧纹状体在可预测规则下主要参与奖励预测。这些结果表明，皮质 - 纹状体环路中的学习系统因环境动态而异：眶额皮质 - 腹侧纹状体环路参与基于当前状态的动作学习，而背外侧前额叶皮质 - 背侧纹状体环路参与基于可预测未来状态的动作学习。（c）2006爱思唯尔有限公司。保留所有权利。