Explainability in Security

So what does explainability mean in cybersecurity?

Description:

In this project, we aim to systematize the state-of-the-art literature that uses explainable artifical intellignece (XAI) for security applications in order to consolidate the notion of explainability within cybersecurity (Nadeem et al., 2023). The systematization is done in terms of key stakeholders (explanation users), application objectives (explanation uses), task domains, and preferred explanation methods.

Code repo: XAI pipeline

References

EuroS&P

SoK: Explainable Machine Learning for Computer Security Applications

Nadeem, Azqa, Vos, Daniel, Cao, Clinton, Pajola, Luca, Dieck, Simon, Baumgartner, Robert, and Verwer, Sicco

In IEEE European Symposium on Security and Privacy (Euro S&P), 2023

Abstract PDF Code

Explainable Artificial Intelligence (XAI) is a promising solution to improve the transparency of machine learning (ML) pipelines. We systematize the increasingly growing (but fragmented) microcosm of studies that develop and utilize XAI methods for defensive and offensive cybersecurity tasks. We identify 3 cybersecurity stakeholders, i.e., model users, designers, and adversaries, that utilize XAI for 5 different objectives within an ML pipeline, namely 1) XAI-enabled decision support, 2) applied XAI for security tasks, 3) model verification via XAI, 4) explanation verification & robustness, and 5) offensive use of explanations. We further classify the literature w.r.t. the targeted security domain. Our analysis of the literature indicates that many of the XAI applications are designed with little understanding of how they might be integrated into analyst workflows – user studies for explanation evaluation are conducted in only 14% of the cases. The literature also rarely disentangles the role of the various stakeholders. Particularly, the role of the model designer is minimized within the security literature. To this end, we present an illustrative use case accentuating the role of model designers. We demonstrate cases where XAI can help in model verification and cases where it may lead to erroneous conclusions instead. The systematization and use case enable us to challenge several assumptions and present open problems that can help shape the future of XAI within cybersecurity.