xai-toolkit
Provides a production-ready path for scalable model explainability and data access through integration with Databricks and Unity Catalog.
Enables users to query machine learning model behavior and individual prediction reasoning using natural language directly within the VS Code Copilot interface.
Facilitates integration with Kedro-based machine learning pipelines to provide narrative explainability for model outputs within existing workflows.
Supports integration with MLflow model registries to track and generate human-readable explanations for registered machine learning models.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@xai-toolkit Explain why customer 8821 was predicted to churn"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
xai-toolkit
ML model explainability as plain-English narratives, exposed via MCP.
Ask your model why in natural language. Get a deterministic, reproducible English answer backed by SHAP analysis — directly inside VS Code Copilot.
User: "Why was sample 42 classified as degraded?"
Copilot: The model classified this sample as degraded (probability: 0.91)
primarily because of 3 factors: total_acid_number = 4.8 (pushing toward
the positive class by +0.28), water_content_ppm = 312.0 (+0.19),
and viscosity_40c = 48.2 (+0.14). The top opposing factor is
flash_point = 215.0 (pushing away from the positive class by -0.06).No SHAP plots to decipher. No code to write. English that a decision-maker can act on.
Quick Start
# 1. Install dependencies
uv sync
# 2. Train the models (run once)
uv run python scripts/train_lubricant_model.py
# 3. Run the test suite (should show 100+ passing)
uv run python -m pytest tests/ -v
# 4. Open VS Code — the MCP server starts automatically via .vscode/mcp.json
# Open Copilot chat in agent mode and ask:
# "What models are available?"Requirements: Python 3.11+, uv, VS Code with GitHub Copilot
What It Does
Seven MCP tools answer the most common explainability questions:
Ask | Tool | Returns |
"What models are available?" |
| Model IDs, types, accuracy |
"Tell me about the data" |
| Sample count, class distribution, stats |
"What does this model do?" |
| Model type, accuracy, top 5 features |
"Which features matter most?" |
| Ranked feature importance with magnitudes |
"Why was sample N classified as X?" |
| Narrative + SHAP bar chart |
"Show me the full SHAP breakdown" |
| Narrative + waterfall plot |
"How does feature F affect predictions?" |
| Narrative + PDP/ICE plot |
Every response includes a complete audit trail: model ID, timestamp, tool version, and a SHA256 hash of the input data. Same question + same data = same answer, every time.
Architecture
VS Code Copilot (Sonnet 4.5)
│ natural language question
▼
MCP Client (built into Copilot)
│ JSON-RPC over stdio
▼
xai-toolkit MCP Server (server.py — thin adapter only)
│
├── explainers.py SHAP values, PDP/ICE, global importance, data hashing
├── narrators.py Structured data → deterministic English paragraphs
├── plots.py matplotlib → base64 PNG (bar, waterfall, PDP+ICE)
├── schemas.py Pydantic contracts — single source of truth
└── registry.py ModelRegistry — load and serve multiple model typesDesign principle: The LLM is the presenter, not the analyst. All computation and narrative generation happens in pure Python. The LLM chooses the right tool and wraps the pre-computed result conversationally. This guarantees reproducibility — the LLM cannot hallucinate SHAP values.
Project Layout
xai-mcp/
├── src/xai_toolkit/ # Source package
├── tests/ # 100+ pytest tests
├── docs/
│ ├── decisions/ # 7 Architecture Decision Records
│ └── scalability-path.md # PoC → Production roadmap
├── scenarios/ # YAML acceptance criteria (day1–day5)
├── scripts/ # Model training scripts
├── models/ # Trained model artifacts
├── data/ # Test datasets
└── .vscode/mcp.json # MCP server configurationRunning Tests
# Full test suite
uv run python -m pytest tests/ -v
# Write snapshot golden files (run once after first install)
uv run python -m pytest tests/test_snapshots.py --snapshot-update -v
# Just the fast unit tests (no model loading)
uv run python -m pytest tests/test_narrators.py tests/test_explainers.py tests/test_reproducibility.py -v
# Second model integration tests (requires trained RF model)
uv run python -m pytest tests/test_second_model.py -vAdding a New Model
Train and save your model following the convention in
scripts/train_lubricant_model.pyAdd one line to the startup block in
server.py:registry.load_from_disk("your_model_id", MODELS_DIR, DATA_DIR)Run
uv run python -m pytest tests/test_second_model.py -v— all tools should work for your new model with zero code changes
See AGENTS.md for full coding standards and architecture guidance.
Architecture Decision Records
Seven decisions documented in docs/decisions/:
ADR | Decision |
001 | Pure functions separated from MCP layer |
002 | Deterministic narratives — no LLM calls |
003 | stdio → Streamable HTTP migration path |
004 | Pydantic schemas as single source of truth |
005 | Consistent tool output structure |
006 | ModelRegistry pattern |
007 | Single-agent architecture |
Production Path
This local PoC becomes a production service by changing infrastructure, not code.
See docs/scalability-path.md for the full roadmap,
including Databricks integration, MLflow model registry, Unity Catalog data access,
and the integration path with the existing Kedro-based XAI pipeline.
Estimated effort to production: 2–4 weeks (platform team, not application code).
Related
FastMCP — MCP server framework used here
SHAP — explainability library
Model Context Protocol — the standard this implements
Available Tools
20 toolscompare_featuresB
Rank features by importance and describe which matter most.
Returns a ranked list of features with their magnitude, direction,
and comparative language — all in plain English.
Args:
model_id: ID of a registered model (e.g., "gbc_lubricant_quality").
top_n: Number of top features to include (default: 10).
Inputs: model_id (required), top_n.
compare_predictionsA
Compare what two models predict for the same sample and explain why.
Returns whether the models agree, their confidence levels, and which
features they share or diverge on — all in plain English. Use this
to build trust in predictions by checking cross-model consistency.
Args:
model_id_1: ID of the first model (e.g., "gbc_lubricant_quality").
model_id_2: ID of the second model (e.g., "rf_lubricant_quality").
sample_index: Row index in the test dataset to compare (0-based).
Inputs: model_id_1 (required), model_id_2 (required), sample_index (required).
describe_datasetA
Describe the dataset associated with a model.
Returns number of samples, features, class distribution, missing values,
and basic statistics — all in plain English.
Args:
model_id: ID of a registered model (e.g., "gbc_lubricant_quality").
Inputs: model_id (required).
detect_driftA
Detect data drift between a model's training data and test data.
Checks the result store first for precomputed drift results.
Falls back to on-the-fly computation if not found.
Numeric features are tested with PSI (primary) and KS (supporting).
Categorical features are tested with chi-squared.
Args:
model_id: ID of a registered model (e.g., "gbc_lubricant_quality").
Inputs: model_id (required).
detect_feature_driftB
Detect drift for a single feature between training and test data.
Returns detailed drift analysis for one feature: statistical test
results, severity, how the distribution shifted (direction, magnitude),
and reference vs. current distribution summaries.
Args:
model_id: ID of a registered model (e.g., "gbc_lubricant_quality").
feature_name: Name of the feature to analyze (e.g., "mean radius").
Inputs: model_id (required), feature_name (required).
explain_predictionA
Explain why a single sample received its classification.
Returns a plain-English narrative explaining which features drove
the model's prediction for the given sample, backed by SHAP values.
Optionally includes a SHAP bar chart (tornado plot) visualization.
Checks the result store first for precomputed explanations.
Falls back to on-the-fly SHAP computation if not found.
Args:
model_id: ID of a registered model (e.g., "gbc_lubricant_quality").
sample_index: Row index in the test dataset to explain (0-based).
include_plot: If True, include a SHAP bar chart as base64 PNG (default: True).
Inputs: model_id (required), sample_index (required), include_plot.
explain_prediction_waterfallA
Show a SHAP waterfall plot for a single prediction.
The waterfall shows how the base prediction builds up to the final
prediction feature by feature. This is the most detailed SHAP visualization.
Args:
model_id: ID of a registered model (e.g., "gbc_lubricant_quality").
sample_index: Row index in the test dataset to explain (0-based).
Inputs: model_id (required), sample_index (required).
get_glass_floorA
Retrieve the Glass Floor separation principles for presenting model explanations alongside business context.
Call this when you need to present both deterministic model outputs and AI-interpreted business guidance. Returns the two-layer separation protocol.
Inputs: no named parameters.
get_partial_dependenceA
Show how a single feature affects predictions across its range.
Returns a narrative describing the relationship between the feature
and the model's predicted probability. Optionally includes a PDP + ICE
plot (model-agnostic visualization, not SHAP-based).
PDP (bold line) shows the average effect. ICE (gray lines) show individual
sample effects, revealing heterogeneity the average hides.
Args:
model_id: ID of a registered model (e.g., "gbc_lubricant_quality").
feature_name: Name of the feature to analyze (e.g., "mean radius").
include_plot: If True, include a PDP+ICE plot as base64 PNG (default: True).
Inputs: model_id (required), feature_name (required), include_plot.
get_skillC
Retrieve one skill by id/version with guardrails and checksum.
Inputs: skill_id (required), version.
get_taste_contextA
Retrieve organizational taste — what experts think good explanations look like.
Returns aggregated feedback from experts across business lines.
Use this to understand audience preferences before presenting results.
For example, if reliability engineers consistently rate explanations as
"too_technical", the LLM can adjust its presentation framing.
All filters are optional. Omit all for a full summary.
Args:
model_id: Filter to a specific model.
audience_role: Filter to a specific role (e.g., "reliability_engineer").
business_line: Filter to a specific business line (e.g., "lubricants").
tool_name: Filter to a specific tool (e.g., "explain_prediction").
Inputs: model_id, audience_role, business_line, tool_name.
get_xai_methodologyA
Retrieve the XAI analysis methodology guide.
Call this BEFORE starting any model explanation to understand the correct tool sequence (explanation funnel), Glass Floor protocol, and anti-patterns to avoid. Returns the full workflow guide.
Inputs: no named parameters.
list_drift_alertsA
Browse batch drift findings across features and models.
Returns precomputed drift results from the result store. This is a
discovery tool — it does NOT fall back to on-the-fly computation.
Args:
model_id: Filter to a specific model. If None, returns results
for all models with stored drift data.
severity: Filter by severity ("none", "moderate", "severe").
run_id: Filter to a specific batch run.
Inputs: model_id, severity, run_id.
list_explained_samplesA
Browse which samples have precomputed explanations.
Returns a summary of available precomputed explanations from the
result store. This is a discovery tool — it does NOT compute
explanations on the fly.
Args:
model_id: Model identifier.
run_id: Filter to a specific batch run.
Inputs: model_id (required), run_id.
list_modelsA
List all registered models with their metadata.
Returns model IDs, types, dataset names, feature counts, and accuracy. Use this to discover what models are available before asking questions.
Inputs: no named parameters.
list_skillsB
List available versioned context skills and guardrail metadata.
Inputs: no named parameters.
record_feedbackA
Record expert feedback on a toolkit narrative.
Call this when an expert evaluates the quality of an explanation.
The narrative is hashed to create an auditable link between the
feedback and the exact output that was evaluated.
Valid ratings: excellent, useful, too_technical, too_vague, missing_context
Args:
model_id: Which model produced the output being evaluated.
tool_name: Which tool produced it (e.g., "explain_prediction").
narrative: The exact narrative text being evaluated (will be hashed).
rating: Expert's assessment (excellent/useful/too_technical/too_vague/missing_context).
audience_role: Evaluator's role (e.g., "reliability_engineer").
business_line: Business context (e.g., "lubricants").
expert_comment: Optional free-text elaboration.
Inputs: model_id (required), tool_name (required), narrative (required), rating (required), +3 more.
retrieve_business_contextA
Retrieve relevant business context from the knowledge base.
Searches loaded business documents (e.g., clinical protocols, operational
rules) for sections relevant to the query. Returns ranked chunks with
source provenance for the Glass Floor presentation pattern.
Use this AFTER an explainability tool to find actionable business guidance.
For example, after explain_prediction returns a high-probability malignant
classification, call this with a query like 'high risk malignant biopsy'
to retrieve the relevant clinical protocol sections.
The provenance_label is always 'ai-interpreted' — any synthesis from
these chunks by the LLM is NOT deterministic and must be clearly
distinguished from grounded tool outputs.
Args:
query: Natural language search query (e.g., 'high risk biopsy referral').
top_k: Maximum number of chunks to return (default: 5).
Inputs: query (required), top_k.
standard_briefingA
Generate a concise, predefined briefing from persisted batch results.
This tool is retrieval-first and does not run SHAP/drift computations.
It is designed as a reusable daily/weekly briefing entry point for
stakeholders who ask a similar set of baseline questions each run.
Args:
model_id: Optional model filter. If None, includes all models with
persisted result-store artifacts.
run_id: Optional run filter.
top_cases: Maximum number of highlighted explained samples per model.
Inputs: model_id, run_id, top_cases.
summarize_modelA
Summarize what a model does and what drives its decisions.
Returns model type, accuracy, number of features, and the top features
ranked by importance — all in plain English.
Args:
model_id: ID of a registered model (e.g., "gbc_lubricant_quality").
Inputs: model_id (required).
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 20 tools
Most tools have distinct purposes, but some overlap exists between explain_prediction and explain_prediction_waterfall (both explain individual predictions using SHAP), and between detect_drift and detect_feature_drift (both detect drift, with one being a subset of the other). The descriptions clarify the differences, but an agent might occasionally misselect between these pairs.
Tool names follow a consistent snake_case pattern throughout, with clear verb_noun structures (e.g., compare_features, describe_dataset, detect_drift). There are no deviations in naming conventions, making the set predictable and easy to parse.
With 20 tools, the count is slightly high but reasonable for a comprehensive XAI toolkit covering explanation, drift detection, feedback, and context retrieval. It feels slightly heavy but not overwhelming, as each tool appears to serve a specific function in the domain.
The toolset provides complete coverage for the XAI domain, including model listing, prediction explanation, drift detection, feedback recording, and business context integration. There are no obvious gaps; it supports a full workflow from model discovery to explanation and evaluation.
Related MCP Connectors
Market forecasts with their own track record, inside Claude, ChatGPT and other MCP apps.
Deterministic context layer for your codebase: change impact, blast radius, answers with receipts.
Multi-expert decision intelligence with transparent synthesis and auditable workflows.
Train, explain, optimise and deploy transparent glass-box ML models via workflow tools.