Explainable Reinforcement Learning via Rule Extraction in Complex Visual Environments

Abstract

Deep neural networks have allowed for significant advances within the field of reinforcement learning and autonomous agents. However, in contrast to traditional approaches such as expert systems and hand-crafted control systems, deep neural networks introduce a large amount of ambiguity regarding the decision making of an autonomous agent. Understanding the decision-making process of any autonomous agent is crucial for applications where trusted autonomy is not only paramount, but required before an agent can be deployed. In this thesis, we focus on the following problems of explainability and rule extraction from autonomous agents. 1) How does the neural network architecture impact the performance and the explainability of an agent trained using reinforcement learning. 2) Can rules be defined and extracted from observations of an autonomous agent trained using reinforcement learning. 3) Can complex rules be derived from multiple partial observations of an autonomous agents. For the first problem we investigate the common neural network architectures used in reinforcement learning and how attention mechanisms have been used to improve performance in prior works. We devise a novel spatial temporal attentionbased approach that allows the agent to learn where it should focus its attention in contrast to previous works which favoured constraining networks with guided attention mechanisms. For the second problem we propose a formal definition of a rule for trajectories consisting of state and action pairs. We show that under this definition, rules are extractable using unsupervised learning techniques. Additionally, we investigate the impact of neural network design on an autonomous agent’s ability to learn rules. For the third problem we introduce a novel method for multi-sequence-tosequence based tasks that require visual induction and translation. This method allows us to observe multiple partial visual observations of an agent and extract the over-arching rule set that defines the agent’s behaviour. We also show that this method is robust with respect to noisy signals.