Explain the Explainer: Interpreting Model-Agnostic Counterfactual Explanations of a Deep Reinforcement Learning Agent

TY - JOUR

T1 - Explain the Explainer

T2 - Interpreting Model-Agnostic Counterfactual Explanations of a Deep Reinforcement Learning Agent

AU - Chen, Ziheng

AU - Silvestri, Fabrizio

AU - Tolomei, Gabriele

AU - Wang, Jia

AU - Zhu, He

AU - Ahn, Hongshik

N1 - Publisher Copyright: © 2020 IEEE.

PY - 2024/4/1

Y1 - 2024/4/1

N2 - Counterfactual examples (CFs) are one of the most popular methods for attaching post hoc explanations to machine learning models. However, existing CF generation methods either exploit the internals of specific models or depend on each sample's neighborhood; thus, they are hard to generalize for complex models and inefficient for large datasets. This article aims to overcome these limitations and introduces ReLAX, a model-agnostic algorithm to generate optimal counterfactual explanations. Specifically, we formulate the problem of crafting CFs as a sequential decision-making task. We then find the optimal CFs via deep reinforcement learning (DRL) with discrete-continuous hybrid action space. In addition, we develop a distillation algorithm to extract decision rules from the DRL agent's policy in the form of a decision tree to make the process of generating CFs itself interpretable. Extensive experiments conducted on six tabular datasets have shown that ReLAX outperforms existing CF generation baselines, as it produces sparser counterfactuals, is more scalable to complex target models to explain, and generalizes to both the classification and regression tasks. Finally, we show the ability of our method to provide actionable recommendations and distill interpretable policy explanations in two practical real-world use cases.

AB - Counterfactual examples (CFs) are one of the most popular methods for attaching post hoc explanations to machine learning models. However, existing CF generation methods either exploit the internals of specific models or depend on each sample's neighborhood; thus, they are hard to generalize for complex models and inefficient for large datasets. This article aims to overcome these limitations and introduces ReLAX, a model-agnostic algorithm to generate optimal counterfactual explanations. Specifically, we formulate the problem of crafting CFs as a sequential decision-making task. We then find the optimal CFs via deep reinforcement learning (DRL) with discrete-continuous hybrid action space. In addition, we develop a distillation algorithm to extract decision rules from the DRL agent's policy in the form of a decision tree to make the process of generating CFs itself interpretable. Extensive experiments conducted on six tabular datasets have shown that ReLAX outperforms existing CF generation baselines, as it produces sparser counterfactuals, is more scalable to complex target models to explain, and generalizes to both the classification and regression tasks. Finally, we show the ability of our method to provide actionable recommendations and distill interpretable policy explanations in two practical real-world use cases.

KW - Counterfactual explanations

KW - deep reinforcement learning (DRL)

KW - explainable artificial intelligence (XAI)

KW - machine learning (ML) explainability

UR - http://www.scopus.com/inward/record.url?scp=85144015335&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85144015335&partnerID=8YFLogxK

U2 - 10.1109/TAI.2022.3223892

DO - 10.1109/TAI.2022.3223892

M3 - Article

SN - 2691-4581

VL - 5

SP - 1443

EP - 1457

JO - IEEE Transactions on Artificial Intelligence

JF - IEEE Transactions on Artificial Intelligence

IS - 4

ER -