Abstract

Please join us for our Computational Neuroscience journal club on Wednesday 21st February at 2pm UK time in the CBL seminar roo

The title is “Meta Reinforcement Learning”, presented by Luke Johnston and Theoklitos Amvrosiadis.

Summary:

Following on from last week’s journal club on distributional reinforcement learning, this week we examine another widely discussed topic within RL - that of meta reinforcement learning. Meta RL has long been of interest to neuroscientists for the extent to which it better reflects the macroscopic behaviour of animals, where the ability to generalise learning experiences across contexts (ie ‘learning to learn’) forms a vital part of higher-order cognition. However unlike traditional single task learning – where, for example, the mechanisms of classical conditioning are well characterised – the neural basis of meta-learning remains a source of debate.

We begin by taking a look back at the seminal 2018 paper from Botvinick et al. [1], which proposes meta learning to occur within a ‘prefrontal network’ centred largely in the PFC . Specifically, the authors postulate a two-part meta RL system that includes initial dopamine based learning by way of the classical cortico–basal ganglia–thalamo–cortical loop to shape the recurrent connectivity of the PFC network, followed by a second PFC -centred algorithm which is dynamically tailored to the task at hand.

In the second part of the journal club, we will look at a recent paper from the Komiyama lab [2], in which through 2-photon calcium imaging experiments in mice and computational models, the researchers argue that synaptic plasticity within orbitofrontal cortex (OFC) is necessary for learning across sessions, but not for within-session learning in already trained subjects.

[1] Wang, J. X., Kurth-Nelson, Z., Kumaran, D., Tirumala, D., Soyer, H., Leibo, J. Z., ...Botvinick, M. (2018). Prefrontal cortex as a meta-reinforcement learning system. Nat. Neurosci., 21, 860–868. doi: 10.1038/s41593-018-0147-8 [2] Hattori, R., Hedrick, N.G., Jain, A. et al. Meta-reinforcement learning via orbitofrontal cortex. Nat Neurosci 26, 2182–2191 (2023). https://doi.org/10.1038/s41593-023-01485-3 (