Explainable AI — Python, SHAP & LIME
Lab ID: LAB-XAI-001 · Version: 0.3 · Status: Proposed reference design
What is Explainable AI?
Explainable Artificial Intelligence (XAI) refers to methods and practices that help people understand how an AI or machine-learning model produces its outputs. Explanations may describe a model’s overall behavior or examine a specific prediction—for example, which features contributed to a credit-risk score and how they influenced that result.
This lab uses SHAP and LIME to explore feature contributions, compare explanations, and evaluate their consistency and limitations. These methods support model review, debugging, and governance; they do not, by themselves, establish causation, fairness, or correctness.
Purpose of the lab
Examine how machine-learning predictions can be explained and how explanation quality should be evaluated. The initial lab uses tabular models and SHAP and LIME to compare explanations of individual predictions and broader model behavior. It supports model review by preserving the conditions, assumptions, and limitations behind each explanation.
Technical architecture
A versioned Python environment loads a public or synthetic tabular dataset. A documented preprocessing pipeline produces training and evaluation inputs. An interpretable baseline and a more complex comparison model generate predictions. SHAP and LIME produce explanations using recorded reference data and settings. Evaluation notebooks compare predictive performance, attribution behavior, and explanation stability. Reports link findings to the dataset, model, code, and environment versions.
Baseline architecture
Components, purpose, and testing opportunities
| Component or feature | Brief description and purpose | Proposed implementation and tests |
|---|---|---|
| Python environment | A reproducible runtime supports repeatable experiments. | Pin compatible package versions, record seeds and hardware assumptions, and rerun a documented experiment. S27 ↗ |
| Dataset and data dictionary | A controlled dataset defines inputs, targets, and feature meanings. | Use synthetic or licensed public data; assess missingness, imbalance, proxy variables, and dataset limitations. S28 ↗ |
| Preprocessing pipeline | Transforms inputs consistently for training, evaluation, and explanation. | Test encoding, scaling, feature-name alignment, leakage prevention, and explanation mapping to original features. S29 ↗ |
| Interpretable baseline | A simple model provides a comparison whose behavior can be inspected more directly. | Train a suitable linear model or shallow tree and evaluate performance and interpretability tradeoffs. S30 ↗ |
| Comparison model | A more complex model provides a subject for post-hoc explanation. | Train a suitable tree ensemble and compare held-out predictive performance with the baseline. S31 ↗ |
| SHAP — SHapley Additive exPlanations | Feature attributions explain predictions relative to a chosen reference under the selected explainer’s assumptions. | Examine individual attributions and aggregate summaries; vary reference data and compare correlated-feature behavior. S32 ↗ |
| LIME — Local Interpretable Model-agnostic Explanations | A local surrogate approximates a model around a selected instance. | Vary seeds, neighborhood sampling, and feature settings; evaluate local fit and explanation variability. S33 ↗ |
| Explanation visualizations | Graphical summaries help reviewers inspect individual and aggregate behavior. | Compare attribution charts with model predictions, units, reference values, and feature definitions. S34 ↗ |
| Evaluation notebook | Structured checks assess explanation usefulness and limits. | Assess repeatability, sensitivity, local surrogate fidelity, and behavior across selected examples and groups. S35 ↗ |
| Governance evidence record | Connects an explanation to its context and review decision. | Preserve model and dataset versions, selected records, explainer settings, findings, limitations, and reviewer notes. S36 ↗ |
Implementation stages, test scenarios & evidence
Implementation stages
1. Define a narrow prediction task and document the intended use and explanation audience.
2. Establish a versioned environment, dataset, and leakage-resistant evaluation split.
3. Train and evaluate a simple baseline and a comparison model.
4. Explain selected predictions using appropriate SHAP and LIME configurations.
5. Compare explanations across repeated runs, reference choices, and representative examples.
6. Produce an evidence report describing what the explanations support and what remains uncertain.
Initial test scenarios and evidence
| Test | Question and expected learning | Evidence to retain |
|---|---|---|
| XAI-01 — Reproducibility | Can the experiment be repeated under recorded conditions? | Dependency versions, seeds, data split, predictions, and numerical tolerances. |
| XAI-02 — Individual prediction | Which features are attributed influence for a selected prediction? | Input record, prediction, reference, attribution chart, and configuration. |
| XAI-03 — Aggregate behavior | What patterns emerge when individual explanations are summarized? | Evaluation population, aggregation method, summaries, and subgroup caveats. |
| XAI-04 — LIME stability | How much does an explanation vary across seeds and neighborhood settings? | Repeated runs, local fit measures, feature rankings, and variability. |
| XAI-05 — SHAP reference sensitivity | How do attributions change when the reference population changes? | Reference-selection rationale, matched predictions, and attribution comparisons. |
| XAI-06 — Correlated inputs | How do explanation methods distribute influence across correlated features? | Controlled feature relationships, model outputs, explanations, and interpretation limits. |
| XAI-07 — Method comparison | Where do SHAP and LIME agree or diverge for the same prediction? | Matched records, compatible output interpretation, settings, and a reasoned comparison. |
| XAI-08 — Governance interpretation | Does the explanation answer the reviewer’s stated question? | Audience, question, evidence, limitations, and review outcome. |
Requirements and limits
Feature attributions describe model behavior under particular assumptions; they do not establish causation, fairness, correctness, or regulatory compliance. Agreement between methods is not proof of truth. Divergence should prompt examination of references, neighborhoods, feature dependence, and output scales. Sensitive training data may also appear in explanation artifacts, so evidence handling requires appropriate access controls.
Future research connection
Proposed paper: Explainable Artificial Intelligence: Methods, Evaluation, and Governance.
The later paper can compare intrinsically interpretable models, permutation importance, partial dependence, accumulated local effects, SHAP, LIME, counterfactual explanations, and gradient-based attribution. It should distinguish explanation scope, intended audience, fidelity, stability, computational cost, and methodological limitations. Generative-AI explanations and retrieval provenance warrant separate treatment; fluent generated rationales should not be assumed to reveal a model’s actual reasoning. The initial lab supplies practical experiments for the paper rather than attempting to implement every method immediately.
Explainable AI sources
Python, scikit-learn, SHAP and LIME documentation and original research support the experimental methods. Governance references inform the evidence record; they do not validate a particular explanation or model.
- S27 · venv — Creation of virtual environments — Python 3.15.2 documentationRetrieved 30 October 2026
- S28 · 9. Dataset loading utilities — scikit-learn 1.9.1 documentationRetrieved 30 October 2026
- S29 · 12. Common pitfalls and recommended practices — scikit-learn 1.9.1 documentationRetrieved 30 October 2026
- S30 · 1.1. Linear Models — scikit-learn 1.9.1 documentationRetrieved 30 October 2026
- S31 · 1.11. Ensembles: Gradient boosting, random forests, bagging, voting, stacking — scikit-learn 1.9.1 documentationRetrieved 30 October 2026
- S32 · Welcome to the SHAP documentation — SHAP latest documentationRetrieved 30 October 2026
- S33 · Local Interpretable Model-Agnostic Explanations (lime) — lime 0.1 documentationRetrieved 30 October 2026
- S34 · API Reference — SHAP latest documentationRetrieved 30 October 2026
- S35 · [1602.04938] "Why Should I Trust You?": Explaining the Predictions of Any ClassifierRetrieved 30 October 2026
- S36 · AI Risk Management Framework | NISTRetrieved 30 October 2026