Interpreting the observed behavior of a class of autonomous linear systems using explainable inverse reinforcement learning

Abstract

One of the main challenges faced by society is how to verify the safety of autonomous systems. As the level of autonomy grows, it becomes critical to understand why an autonomous system exhibits a particular behavior and what we need to do to ensure it does not pose risks to people or the surrounding environment. To this end, we need to infer, from observational data, the necessary evidence or decision-making factors to effectively interpret the autonomous system's behavior. Previous approaches in the literature have used explainable models to provide simple input-output mappings to interpret the observer behavior. However, autonomous systems do not pose simple input-output relationships, which compromises the veracity of the interpretations. To alleviate this issue, this article proposes a novel explainable inverse reinforcement learning (EXIRL) that provides the evidence to interpret the behavior of a class of autonomous linear systems. The approach uses a combined model-free and model-based mechanism that improves the learning phase of traditional inverse learning algorithms to accurately infer the evidence from the data. The inferred evidence is used to design counterfactual explanations to interpret the observed behavior and provide the mechanisms to modify it into a desired one. Simulation studies using a DJI Phantom 4 and a quadrotor model are provided to show the benefits and challenges of the proposed work.