Abstract

How do you teach an algorithm to do a backflip or play a game where rewards are sparse? In this seminar we will discuss how algorithms can learn from human preferences as opposed to from pre-specified goal functions.

Removing the need for humans to write goal functions is important because getting them slightly wrong could lead to dangerous behaviour. Here this is only used to learn physical behaviours, but one can imagine that it could apply to learning moral values as well.

We will be looking at the paper ‘Deep Reinforcement Learning from Human Preferences’ (Christiano et. al., 2017). We will discuss the model used and experiments in three domains: simulated robotics, Atari arcade games and novel behaviours.

Link to paper: https://arxiv.org/abs/1706.03741

Slides: https://valuealignment.ml/talks/2017-11-15-deeprl-human-prefs.pdf