Abstract

Reinforcement learning (RL) is increasingly being used to support human decision-making in real-world systems. Before deploying RL-learnt policies, we must verify their safety, particularly in safety-critical domains like healthcare. Counterfactual inference enables offline policy evaluation by predicting how an observed sequence of states and actions (under an existing policy) would have evolved under an alternative policy. However, existing counterfactual inference approaches for MDPs assume a fixed causal model of the underlying system, limiting the validity (and usefulness) of counterfactual inference. We relax these assumptions by computing exact bounds for the counterfactual probabilities across all causal models, leading to more reliable counterfactual analysis. Moreover, we prove closed-form expressions for these bounds, making computation highly efficient and scalable for handling large-scale MDPs. Bio: Milad Kazemi is a postdoctoral researcher at the Department of Informatics, King’s College London. He completed his Ph.D. in Computer Science at Newcastle University in 2023. His research focuses on the intersection of control theory, reinforcement learning, formal methods, and causality, with an emphasis on counterfactual analysis and policy synthesis in sequential decision-making and safety-critical systems