awesome-xai

Explainability toolkit

A curated collection of papers, methods, and resources for understanding and improving machine learning models' explainability

Awesome Explainable AI (XAI) and Interpretable ML Papers and Resources

GitHub
142 stars
8 watching
13 forks
last commit: over 3 years ago
Linked from 2 awesome lists

awesomeawesome-listexplanationssaliency-mapssaliency-methodsxai

Awesome XAI / Papers / Landmarks
Explanation in Artificial Intelligence: Insights from the Social Sciences This paper provides an introduction to the social science research into explanations. The author provides 4 major findings: (1) explanations are constrastive, (2) explanations are selected, (3) probabilities probably don't matter, (4) explanations are social. These fit into the general theme that explanations are -contextual-
Sanity Checks for Saliency Maps An important read for anyone using saliency maps. This paper proposes two experiments to determine whether saliency maps are useful: (1) model parameter randomization test compares maps from trained and untrained models, (2) data randomization test compares maps from models trained on the original dataset and models trained on the same dataset with randomized labels. They find that "some widely deployed saliency methods are independent of both the data the model was trained on, and the model parameters"

Awesome XAI / Papers / Surveys
Explainable Deep Learning: A Field Guide for the Uninitiated An in-depth description of XAI focused on technqiues for deep learning

Awesome XAI / Papers / Evaluations
Quantifying Explainability of Saliency Methods in Deep Neural Networks An analysis of how different heatmap-based saliency methods perform based on experimentation with a generated dataset

Awesome XAI / Papers / XAI Methods
Ada-SISE Adaptive semantice inpute sampling for explanation
ALE Accumulated local effects plot
ALIME Autoencoder Based Approach for Local Interpretability
Anchors High-Precision Model-Agnostic Explanations
Auditing Auditing black-box models
BayLIME Bayesian local interpretable model-agnostic explanations
Break Down Break down plots for additive attributions
CAM Class activation mapping
CDT Confident interpretation of Bayesian decision tree ensembles
CICE Centered ICE plot
CMM Combined multiple models metalearner
Conj Rules Using sampling and queries to extract rules from trained neural networks
CP Contribution propogation
DecText Extracting decision trees from trained neural networks
DeepLIFT Deep label-specific feature learning for image annotation
DTD Deep Taylor decomposition
ExplainD Explanations of evidence in additive classifiers
FIRM Feature importance ranking measure
Fong, et. al. Meaninful perturbations model
G-REX Rule extraction using genetic algorithms
Gibbons, et. al. Explain random forest using decision tree
GoldenEye Exploring classifiers by randomization
GPD Gaussian process decisions
GPDT Genetic program to evolve decision trees
GradCAM Gradient-weighted Class Activation Mapping
GradCAM++ Generalized gradient-based visual explanations
Hara, et. al. Making tree ensembles interpretable
ICE Individual conditional expectation plots
IG Integrated gradients
inTrees Interpreting tree ensembles with inTrees
IOFP Iterative orthoganol feature projection
IP Information plane visualization
KL-LIME Kullback-Leibler Projections based LIME
Krishnan, et. al. Extracting decision trees from trained neural networks
Lei, et. al. Rationalizing neural predictions with generator and encoder
LIME Local Interpretable Model-Agnostic Explanations
LOCO Leave-one covariate out
LORE Local rule-based explanations
Lou, et. al. Accurate intelligibile models with pairwise interactions
LRP Layer-wise relevance propogation
MCR Model class reliance
MES Model explanation system
MFI Feature importance measure for non-linear algorithms
NID Neural interpretation diagram
OptiLIME Optimized LIME
PALM Partition aware local model
PDA Prediction Difference Analysis: Visualize deep neural network decisions
PDP Partial dependence plots
POIMs Positional oligomer importance matrices for understanding SVM signal detectors
ProfWeight Transfer information from deep network to simpler model
Prospector Interactive partial dependence diagnostics
QII Quantitative input influence
REFNE Extracting symbolic rules from trained neural network ensembles
RETAIN Reverse time attention model
RISE Randomized input sampling for explanation
RxREN Reverse engineering neural networks for rule extraction
SHAP A unified approach to interpretting model predictions
SIDU Similarity, difference, and uniqueness input perturbation
Simonynan, et. al Visualizing CNN classes
Singh, et. al Programs as black-box explanations
STA Interpreting models via Single Tree Approximation
Strumbelj, et. al. Explanation of individual classifications using game theory
SVM+P Rule extraction from support vector machines
TCAV Testing with concept activation vectors
Tolomei, et. al. Interpretable predictions of tree-ensembles via actionable feature tweaking
Tree Metrics Making sense of a forest of trees
TreeSHAP Consistent feature attribute for tree ensembles
TreeView Feature-space partitioning
TREPAN Extracting tree-structured representations of trained networks
TSP Tree space prototypes
VBP Visual back-propagation
VEC Variable effect characteristic curve
VIN Variable interaction network
X-TREPAN Adapted etraction of comprehensible decision tree in ANNs
Xu, et. al. Show, attend, tell attention model

Awesome XAI / Papers / Interpretable Models
Decision List Like a decision tree with no branches
Decision Trees The tree provides an interpretation
Explainable Boosting Machine Method that predicts based on learned vector graphs of features
k-Nearest Neighbors The prototypical clustering method
Linear Regression Easily plottable and understandable regression
Logistic Regression Easily plottable and understandable classification
Naive Bayes Good classification, poor estimation using conditional probabilities
RuleFit Sparse linear model as decision rules including feature interactions

Awesome XAI / Papers / Critiques
Attention is not Explanation Authors perform a series of NLP experiments which argue attention does not provide meaningful explanations. They also demosntrate that different attentions can generate similar model outputs
Attention is not --not-- Explanation This is a rebutal to the above paper. Authors argue that multiple explanations can be valid and that the and that attention can produce valid explanation, if not -the- valid explanation
Do Not Trust Additive Explanations Authors argue that addditive explanations (e.g. LIME, SHAP, Break Down) fail to take feature ineractions into account and are thus unreliable
Please Stop Permuting Features An Explanation and Alternatives Authors demonstrate why permuting features is misleading, especially where there is strong feature dependence. They offer several previously described alternatives
Stop Explaining Black Box Machine Learning Models for High States Decisions and Use Interpretable Models Instead Authors present a number of issues with explainable ML and challenges to interpretable ML: (1) constructing optimal logical models, (2) constructing optimal sparse scoring systems, (3) defining interpretability and creating methods for specific methods. They also offer an argument for why interpretable models might exist in many different domains
The (Un)reliability of Saliency Methods Authors demonstrate how saliency methods vary attribution when adding a constant shift to the input data. They argue that methods should fulfill , that a saliency method mirror the sensistivity of the model with respect to transformations of the input

Awesome XAI / Repositories
EthicalML/xai 1,125 about 3 years ago A toolkit for XAI which is focused exclusively on tabular data. It implements a variety of data and model evaluation techniques
MAIF/shapash 2,739 24 days ago SHAP and LIME-based front-end explainer
PAIR-code/what-if-tool 917 2 months ago A tool for Tensorboard or Notebooks which allows investigating model performance and fairness
slundberg/shap 22,876 12 days ago A Python module for using Shapley Additive Explanations

Awesome XAI / Videos
Debate: Interpretability is necessary for ML A debate on whether interpretability is necessary for ML with Rich Caruana and Patrice Simard for and Kilian Weinberger and Yann LeCun against

Awesome XAI / Follow
The Institute for Ethical AI & Machine Learning A UK-based research center that performs research into ethical AI/ML, which frequently involves XAI
Tim Miller One of the preeminent researchers in XAI
Rich Caruana The man behind Explainable Boosting Machines

Backlinks from these awesome lists:

More related projects: