interp-playground
One-line summary: Interp-playground's MCP server exposes a language model's internals (Gemma 2 2B + Gemma Scope SAEs) as tools, so an agent or human can run prompts and inspect which interpretable features fire — though the supplied schema lists only run_prompt.
Run any prompt through Gemma 2 2B locally and get back the SAE features active in the text.
See top features (10 per layer, at their peak token) with Neuronpedia labels, density, and the tokens each feature promotes.
Get per-token feature lists (
top_k, default 5) for fine-grained, token-by-token views of activation.Read exact activation values — raw
actplusrel(activation relative to the feature's typical max) — with rankings byrel.Choose layers to read from (residual stream, 0–25; default layer 12).
Optionally generate first (
max_new_tokens) and analyze prompt plus completion together.Filter noise: dense features (>10% of tokens) hidden unless
include_dense,<bos>omitted, position artifacts flagged.Per the README, additional tools also exist beyond this schema:
feature_info,steer,ablate,logit_lens,eval_sentences, andgrade_explanation(steering with random controls, ablation, layer-wise prediction, and explanation scoring).
Provides tools for inspecting and steering Google's Gemma 2 2B model using Gemma Scope residual-stream SAEs, including activation lookup, feature steering, and ablation.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@interp-playground Show top SAE features for 'The bridge crossed the river'."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
interp-playground
Tools for looking inside a language model: activations, sparse-autoencoder (SAE) features, steering, and ablation. The goal is an MCP server that lets an AI agent and a human investigate a model's internals together.
The motivation: humans and AIs should be able to understand how a model reasons, especially if chain-of-thought stops being legible English. This makes that concrete: an AI using tools to understand another model.
It's a learning project. Current target: Gemma 2 2B with Google DeepMind's Gemma Scope residual-stream SAEs, running locally on Apple Silicon (MPS).
Status
Nix + uv environment; MPS verified against CPU
Reproduce a known SAE feature by hand (activations, steering, random control)
MCP server:
run_prompt,feature_info,steer,ablate,logit_lensFirst agent-driven investigations (see FINDINGS.md)
Related MCP server: chuk-mcp-lazarus
Setup
Requires Nix with flakes. The flake provides Python 3.12 and uv; uv manages Python packages.
nix develop # or: direnv allow
uv syncGemma 2 is gated on Hugging Face. Accept the license at google/gemma-2-2b, then log in once:
uv run hf auth loginThe model (~10 GB in fp32) and SAEs download on first use.
MCP server
claude mcp add interp-playground -- uv run --directory /path/to/interp-playground interp-playground-mcp
uv run python scripts/mcp_smoke.py # end-to-end check over stdiorun_prompt returns the top SAE features across the text and per token, with
Neuronpedia labels, plus exact per-token activations for any features you ask
about (features=[...]). Activations are also given relative to each feature's
typical max (rel). Two kinds of features are hidden from rankings by default:
dense ones (active on >10% of tokens) and position artifacts (detected at load
time with probe prompts). The model stays loaded between calls. Feature metadata
comes from Neuronpedia's bulk exports and is cached in ~/.cache/interp-playground/.
steer adds a feature's direction during generation at several scales (default
1x/2x/4x the feature's max activation), each next to a random-direction control
of the same size. It reports the completions plus kl at the prompt's end (how
far the distribution moved) and the mean next-token mass on the feature's
promoted tokens, or on caller-chosen target_tokens, across the whole
completion, so feature-specific effects can be separated from generic disruption
and thematic steering is measured where it shows.
ablate removes a feature's own contribution (or clamps it) at the positions
where it fires and reports the change in next-token predictions, per-position
log-prob deltas, and optionally completions. feature_info gathers a feature's
label, density, promoted/suppressed tokens, nearest features by decoder
direction, and Neuronpedia's top examples. logit_lens shows what each layer
would predict at a position, and when the final answer forms.
eval_sentences and grade_explanation run an explanation eval: the agent
predicts a feature's activations on a mixed pool (held-out Neuronpedia
examples, random text from other features, its own near-misses), first from
the label alone and again after investigating, and the server scores both
sets with Spearman correlation, firing F1 by source, and peak-token hits.
Results append to results/explanation_evals.jsonl; scripts/eval_report.py
prints the label-vs-investigated table (--run LABEL for one batch). Design:
docs/superpowers/specs/2026-10-18-explanation-eval-design.md.
A blind batch is three steps: scripts/select_features.py picks features by
seeded shuffle, scripts/check_pools.py builds each pool and lists the
features that cannot be evaluated, and scripts/blind_eval_workflow.js drives
four agents per feature through the MCP server. Every choice that shapes the
numbers, with predictions written before each run, is in
results/DECISIONS.md.
uv run pytest # unit tests (no model needed)Scripts
# Does MPS give the same internals as CPU? (fp32: yes; bf16: 1-4% drift)
uv run python scripts/check_mps.py
# Where does a feature fire, token by token?
uv run python scripts/feature_acts.py --feature 7272 "We drove across the old bridge."
# Steer generation along a feature direction, or a random one as a control
uv run python scripts/steer.py --feature 7272 --scales 0 50 100 200
uv run python scripts/steer.py --random 0 --scales 0 50 100 200
uv run python scripts/tool.py logit_lens prompt="Romeo and Juliet was written by" top_k=3 # any MCP tool, no server
uv run python scripts/eval_report.py # label vs investigated scores per featureFindings so far
The full log is in FINDINGS.md. Highlights:
Layer-12 feature 7272, labelled "references to various types of bridges" on Neuronpedia:
Activations: fires at 35-50 on the word "bridge" in every sense (physical, dental, card game, metaphorical) and at 4-22 on bridge-like concepts without the word (overpass, ferry, river). The label describes the top of the activation range; the low range is broader.
Steering: at scale 70-90 the output drifts toward rivers and "connecting the community"; at 100+ it produces "bridge" and degrades. Random directions of the same norm never produce "bridge" and stay fluent up to ~200, so the effect is specific to the feature. (One prompt, two random seeds: suggestive, not conclusive.)
Position artifacts. Features 1178, 4667, 6889 and 11087 (layer 12) fire on
whatever token comes first, whether "Bananas", "def" or "1", and nowhere else.
Their density is ~1/127, i.e. once per Neuronpedia text window. Their
auto-generated labels ("words that indicate agreement, concession...") are
confabulated meanings for a purely positional signal. Found by Claude using
run_prompt; this is why the server now detects and hides them.
Explanation eval. Does investigating a feature produce a better explanation than its Neuronpedia label? Two blind batches, 48 layer-12 features, each agent predicting a feature's activations on a 16-sentence pool:
From the label alone, mean firing F1 was 0.66 and 0.81 in the two batches. With
feature_info, fourrun_prompts and one causal probe it was 0.88 and 0.90. Paired, on the pre-registered primary metric: +0.22 (p = 0.007) in the first batch, +0.08 (p = 0.077, not significant) in the second, +0.15 (25 wins, 17 ties, 6 losses, p = 0.0009) pooled.The gain comes from a minority of bad labels. Where the label's own F1 was below 0.6 (10 features) investigating added +0.53; where it was 0.9 or more (20 features) it added nothing. "References to a crisis" is a Spanish-language detector; "scientific measurements" is a syntactic slot; three of 50 sampled features are mid-text
<bos>detectors under topical labels.A third arm in the second batch, given the label and the top examples but no experiments, got F1 0.89 against the investigator's 0.90. On this pool, reading eight examples accounts for nearly all of the gain.
38 of 48 causal probes barely moved the next token.
A follow-up on 24 features tested sentences the examples do not resemble: ordinary text where the feature fires, and text where its nearest features fire. Fresh agents predicted from the stored explanations. Experiments against examples: +0.05 F1 (p = 0.18, not significant, and -0.01 under a second scoring threshold). Against the bare label, an explanation cut false alarms on neighboring features' text from 0.33 to 0.10 and missed more of the weak firings, with no significant change in F1. Most firings in ordinary text are weak, and one feature's top examples hid a second trigger.
Known limits: one SAE and layer, one model family writing and reading the
explanations, 24 to 25 features per batch, and F1 figures that depend on
where "predicted to fire" is drawn when firings are weak. Records in
results/, write-up in FINDINGS sections 12-16, decisions and predictions
written before each run in results/DECISIONS.md.
Roadmap
The initial tool set is complete. Design principles:
Compact, structured JSON. Agents do badly with large activation dumps; return top-k lists, deltas and exact values for features asked about.
Swappable backend. Keep model access behind an interface so larger models, or remote execution via NDIF, can be added later.
Long-lived process. Load the model once, not per call.
Next steps suggested by the findings:
A judge for thematic steering. Completion-averaged token mass (section 10) is graded but still lexical and misses "loved ones"; score completions against a theme with a small model or embedding similarity.
Agent explanation evals are built (
eval_sentences,grade_explanation); next is running them over more features and adding a judge for the explanations themselves.Multi-feature ablation and attention-head attribution, to find what does carry the "bridge" prediction at layers 19-23.
Future directions:
Larger models: Gemma 2 9B locally; much larger via NDIF.
Probes on a reasoning model's chain of thought: does the model "know" its answer before its written reasoning says so? A CoT-faithfulness question.
Train our own SAEs where pretrained ones don't cover a model or layer (activation caching, storage, streaming, shuffling).
Evaluate the agent's explanations: do its claims about features hold up under intervention? An open question in automated interpretability.
References: SAELens, nnsight, TransformerLens, Neuronpedia, Gemma Scope, and ARENA's mechanistic interpretability chapter.
Ablation vs. steering disagree, and both are right. Zero-ablating 7272 on "We drove across the old" changes nothing (the feature is inactive there), but subtracting its direction at the context positions removes "bridge" from the prediction. The logit lens shows the prediction forming at layers 19-23 from the earlier tokens. "Feature inactive" and "direction unused" are different claims.
Concept features steer; token features cliff. A "grief and loss" feature at 4x its max activation yields fluent text about honouring loved ones; "bridges" and "dogs" at the same relative scale produce the word, then garbage.
The logit lens shows retrieval before commitment. On "Romeo and Juliet was
written by", " Shakespeare" leads at layer 20 but the answer is " William" at
layer 25; p_final stays near 0 until layer 24. Read top tokens from layer 15
on, not just p_final.
Labels are search keys, not ground truth. 9435 "San Francisco" is a "San" string feature (fires on "Sanitize" and "Sanofi", not on "Bay Area").
Notes
fp32, not bf16. bf16 on MPS drifts 1-4% in the residual stream, enough to flip JumpReLU features near their thresholds.
nnsight 0.7: only objects that are themselves
.save()d leave a trace. Wrap lists of saved tensors innnsight.save([...]).transformer-lensis pinned below 4.0 until sae-lens supports it.
License
MIT
Available Tools
1 toolrun_promptA
Run a prompt through the model and report which SAE features are active.
Returns, per layer:
- top_features: the 10 features most strongly active anywhere in the text,
each at its peak token, with its Neuronpedia label, density (fraction of
all tokens it fires on) and the output tokens it promotes.
- tokens: for each token, its top_k features.
`act` is the raw activation. `rel` is act / the feature's typical max
activation, so rel near 1 means firing about as hard as it ever does.
Rankings use rel. Features active on >10% of all tokens carry little
meaning and are hidden unless include_dense is true. The <bos> token is
omitted. The first real token often shows position artifacts: features
with very large activations unrelated to its meaning.
Args:
prompt: Text to run. Gemma 2 2B is a base model, not a chat model.
layers: Residual-stream layers to read (0-25). Default [12].
top_k: Features to list per token.
max_new_tokens: If > 0, greedily generate this many tokens first and
analyze prompt + completion.
include_dense: Include features active on >10% of tokens.
Inputs: prompt (required), top_k, layers, include_dense, +1 more. Returns: object
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 1 tool
With only one tool, there is no risk of selecting the wrong tool; its purpose is unambiguous. The internally complex output does not create tool-selection overlap.
The sole tool name run_prompt follows a clear verb_noun snake_case convention. There are no other names to contradict the pattern, so consistency is trivially maintained.
One tool is very thin for a server branded as a playground. While the operation is rich, the absence of companion tools for discovery, comparison, or feature inspection makes the count feel too low for the apparent scope.
The surface covers only running a prompt and viewing active SAE features. It lacks obvious operations such as listing models/layers, searching or inspecting individual features, or comparing prompts, leaving significant gaps for an interpretability playground.
Maintenance
Related MCP Connectors
Memory, reflection and safety for AI agents. Hosted or local, no vector database.
Self-hosted AI prompt library: prompts, collections, tags, teams, chains. 29 MCP tools for agents.
AI Agent Utility Layer for agent safety, web monitoring, crypto research and tool discovery.
AI Reasoning Cache & Consensus Layer with 11 MCP tools via Streamable HTTP.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceMCP server for mechanistic interpretability research, enabling agents to drive probe-causality and SAE-feature experiments via 8 typed tools on user's own compute (Colab).Apache 2.0
- AlicenseNot gradedqualityDmaintenanceMechanistic interpretability MCP server wrapping chuk-lazarus, enabling model loading, activation extraction, probe training, steering, and ablation via MCP tools.21Apache 2.0
- AlicenseAqualityCmaintenanceMCP server for analyzing transformer language model internals, exposing tools to trace attention heads and neurons responsible for predictions, get activation statistics, ablate components, and sketch circuits.443 PyPI1MIT
- FlicenseNot gradedqualityAmaintenanceEnables AI agents and developers to query traces, sessions, exceptions, prompts, datasets, annotation queues, scores, and metrics for local debugging and observability. Runs locally over stdio or HTTP with selective tool-group loading, read-only mode, and compact or file-dump output to control token usage.-