Skip to main content
Glama
interp-playground

interp-playground

by agaskell

interp-playground

Tools for looking inside a language model: activations, sparse-autoencoder (SAE) features, steering, and ablation. The goal is an MCP server that lets an AI agent and a human investigate a model's internals together.

The motivation: humans and AIs should be able to understand how a model reasons, especially if chain-of-thought stops being legible English. This makes that concrete: an AI using tools to understand another model.

It's a learning project. Current target: Gemma 2 2B with Google DeepMind's Gemma Scope residual-stream SAEs, running locally on Apple Silicon (MPS).

Status

  • Nix + uv environment; MPS verified against CPU

  • Reproduce a known SAE feature by hand (activations, steering, random control)

  • MCP server: run_prompt, feature_info, steer, ablate, logit_lens

  • First agent-driven investigations (see FINDINGS.md)

Related MCP server: chuk-mcp-lazarus

Setup

Requires Nix with flakes. The flake provides Python 3.12 and uv; uv manages Python packages.

nix develop            # or: direnv allow
uv sync

Gemma 2 is gated on Hugging Face. Accept the license at google/gemma-2-2b, then log in once:

uv run hf auth login

The model (~10 GB in fp32) and SAEs download on first use.

MCP server

claude mcp add interp-playground -- uv run --directory /path/to/interp-playground interp-playground-mcp
uv run python scripts/mcp_smoke.py   # end-to-end check over stdio

run_prompt returns the top SAE features across the text and per token, with Neuronpedia labels, plus exact per-token activations for any features you ask about (features=[...]). Activations are also given relative to each feature's typical max (rel). Two kinds of features are hidden from rankings by default: dense ones (active on >10% of tokens) and position artifacts (detected at load time with probe prompts). The model stays loaded between calls. Feature metadata comes from Neuronpedia's bulk exports and is cached in ~/.cache/interp-playground/.

steer adds a feature's direction during generation at several scales (default 1x/2x/4x the feature's max activation), each next to a random-direction control of the same size. It reports the completions plus kl at the prompt's end (how far the distribution moved) and the mean next-token mass on the feature's promoted tokens, or on caller-chosen target_tokens, across the whole completion, so feature-specific effects can be separated from generic disruption and thematic steering is measured where it shows.

ablate removes a feature's own contribution (or clamps it) at the positions where it fires and reports the change in next-token predictions, per-position log-prob deltas, and optionally completions. feature_info gathers a feature's label, density, promoted/suppressed tokens, nearest features by decoder direction, and Neuronpedia's top examples. logit_lens shows what each layer would predict at a position, and when the final answer forms.

eval_sentences and grade_explanation run an explanation eval: the agent predicts a feature's activations on a mixed pool (held-out Neuronpedia examples, random text from other features, its own near-misses), first from the label alone and again after investigating, and the server scores both sets with Spearman correlation, firing F1 by source, and peak-token hits. Results append to results/explanation_evals.jsonl; scripts/eval_report.py prints the label-vs-investigated table (--run LABEL for one batch). Design: docs/superpowers/specs/2026-10-18-explanation-eval-design.md.

A blind batch is three steps: scripts/select_features.py picks features by seeded shuffle, scripts/check_pools.py builds each pool and lists the features that cannot be evaluated, and scripts/blind_eval_workflow.js drives four agents per feature through the MCP server. Every choice that shapes the numbers, with predictions written before each run, is in results/DECISIONS.md.

uv run pytest            # unit tests (no model needed)

Scripts

# Does MPS give the same internals as CPU? (fp32: yes; bf16: 1-4% drift)
uv run python scripts/check_mps.py

# Where does a feature fire, token by token?
uv run python scripts/feature_acts.py --feature 7272 "We drove across the old bridge."

# Steer generation along a feature direction, or a random one as a control
uv run python scripts/steer.py --feature 7272 --scales 0 50 100 200
uv run python scripts/steer.py --random 0 --scales 0 50 100 200
uv run python scripts/tool.py logit_lens prompt="Romeo and Juliet was written by" top_k=3   # any MCP tool, no server
uv run python scripts/eval_report.py   # label vs investigated scores per feature

Findings so far

The full log is in FINDINGS.md. Highlights:

Layer-12 feature 7272, labelled "references to various types of bridges" on Neuronpedia:

  • Activations: fires at 35-50 on the word "bridge" in every sense (physical, dental, card game, metaphorical) and at 4-22 on bridge-like concepts without the word (overpass, ferry, river). The label describes the top of the activation range; the low range is broader.

  • Steering: at scale 70-90 the output drifts toward rivers and "connecting the community"; at 100+ it produces "bridge" and degrades. Random directions of the same norm never produce "bridge" and stay fluent up to ~200, so the effect is specific to the feature. (One prompt, two random seeds: suggestive, not conclusive.)

Position artifacts. Features 1178, 4667, 6889 and 11087 (layer 12) fire on whatever token comes first, whether "Bananas", "def" or "1", and nowhere else. Their density is ~1/127, i.e. once per Neuronpedia text window. Their auto-generated labels ("words that indicate agreement, concession...") are confabulated meanings for a purely positional signal. Found by Claude using run_prompt; this is why the server now detects and hides them.

Explanation eval. Does investigating a feature produce a better explanation than its Neuronpedia label? Two blind batches, 48 layer-12 features, each agent predicting a feature's activations on a 16-sentence pool:

  • From the label alone, mean firing F1 was 0.66 and 0.81 in the two batches. With feature_info, four run_prompts and one causal probe it was 0.88 and 0.90. Paired, on the pre-registered primary metric: +0.22 (p = 0.007) in the first batch, +0.08 (p = 0.077, not significant) in the second, +0.15 (25 wins, 17 ties, 6 losses, p = 0.0009) pooled.

  • The gain comes from a minority of bad labels. Where the label's own F1 was below 0.6 (10 features) investigating added +0.53; where it was 0.9 or more (20 features) it added nothing. "References to a crisis" is a Spanish-language detector; "scientific measurements" is a syntactic slot; three of 50 sampled features are mid-text <bos> detectors under topical labels.

  • A third arm in the second batch, given the label and the top examples but no experiments, got F1 0.89 against the investigator's 0.90. On this pool, reading eight examples accounts for nearly all of the gain.

  • 38 of 48 causal probes barely moved the next token.

  • A follow-up on 24 features tested sentences the examples do not resemble: ordinary text where the feature fires, and text where its nearest features fire. Fresh agents predicted from the stored explanations. Experiments against examples: +0.05 F1 (p = 0.18, not significant, and -0.01 under a second scoring threshold). Against the bare label, an explanation cut false alarms on neighboring features' text from 0.33 to 0.10 and missed more of the weak firings, with no significant change in F1. Most firings in ordinary text are weak, and one feature's top examples hid a second trigger.

Known limits: one SAE and layer, one model family writing and reading the explanations, 24 to 25 features per batch, and F1 figures that depend on where "predicted to fire" is drawn when firings are weak. Records in results/, write-up in FINDINGS sections 12-16, decisions and predictions written before each run in results/DECISIONS.md.

Roadmap

The initial tool set is complete. Design principles:

  • Compact, structured JSON. Agents do badly with large activation dumps; return top-k lists, deltas and exact values for features asked about.

  • Swappable backend. Keep model access behind an interface so larger models, or remote execution via NDIF, can be added later.

  • Long-lived process. Load the model once, not per call.

Next steps suggested by the findings:

  • A judge for thematic steering. Completion-averaged token mass (section 10) is graded but still lexical and misses "loved ones"; score completions against a theme with a small model or embedding similarity.

  • Agent explanation evals are built (eval_sentences, grade_explanation); next is running them over more features and adding a judge for the explanations themselves.

  • Multi-feature ablation and attention-head attribution, to find what does carry the "bridge" prediction at layers 19-23.

Future directions:

  • Larger models: Gemma 2 9B locally; much larger via NDIF.

  • Probes on a reasoning model's chain of thought: does the model "know" its answer before its written reasoning says so? A CoT-faithfulness question.

  • Train our own SAEs where pretrained ones don't cover a model or layer (activation caching, storage, streaming, shuffling).

  • Evaluate the agent's explanations: do its claims about features hold up under intervention? An open question in automated interpretability.

References: SAELens, nnsight, TransformerLens, Neuronpedia, Gemma Scope, and ARENA's mechanistic interpretability chapter.

Ablation vs. steering disagree, and both are right. Zero-ablating 7272 on "We drove across the old" changes nothing (the feature is inactive there), but subtracting its direction at the context positions removes "bridge" from the prediction. The logit lens shows the prediction forming at layers 19-23 from the earlier tokens. "Feature inactive" and "direction unused" are different claims.

Concept features steer; token features cliff. A "grief and loss" feature at 4x its max activation yields fluent text about honouring loved ones; "bridges" and "dogs" at the same relative scale produce the word, then garbage.

The logit lens shows retrieval before commitment. On "Romeo and Juliet was written by", " Shakespeare" leads at layer 20 but the answer is " William" at layer 25; p_final stays near 0 until layer 24. Read top tokens from layer 15 on, not just p_final.

Labels are search keys, not ground truth. 9435 "San Francisco" is a "San" string feature (fires on "Sanitize" and "Sanofi", not on "Bay Area").

Notes

  • fp32, not bf16. bf16 on MPS drifts 1-4% in the residual stream, enough to flip JumpReLU features near their thresholds.

  • nnsight 0.7: only objects that are themselves .save()d leave a trace. Wrap lists of saved tensors in nnsight.save([...]).

  • transformer-lens is pinned below 4.0 until sae-lens supports it.

License

MIT

Available Tools

1 tool
run_promptA

Run a prompt through the model and report which SAE features are active.

Returns, per layer:
- top_features: the 10 features most strongly active anywhere in the text,
  each at its peak token, with its Neuronpedia label, density (fraction of
  all tokens it fires on) and the output tokens it promotes.
- tokens: for each token, its top_k features.

`act` is the raw activation. `rel` is act / the feature's typical max
activation, so rel near 1 means firing about as hard as it ever does.
Rankings use rel. Features active on >10% of all tokens carry little
meaning and are hidden unless include_dense is true. The <bos> token is
omitted. The first real token often shows position artifacts: features
with very large activations unrelated to its meaning.

Args:
    prompt: Text to run. Gemma 2 2B is a base model, not a chat model.
    layers: Residual-stream layers to read (0-25). Default [12].
    top_k: Features to list per token.
    max_new_tokens: If > 0, greedily generate this many tokens first and
        analyze prompt + completion.
    include_dense: Include features active on >10% of tokens.

Inputs: prompt (required), top_k, layers, include_dense, +1 more. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.3/5.0

    Scored across 1 tool

    Disambiguation5/5

    With only one tool, there is no risk of selecting the wrong tool; its purpose is unambiguous. The internally complex output does not create tool-selection overlap.

    Naming Consistency5/5

    The sole tool name run_prompt follows a clear verb_noun snake_case convention. There are no other names to contradict the pattern, so consistency is trivially maintained.

    Tool Count2/5

    One tool is very thin for a server branded as a playground. While the operation is rich, the absence of companion tools for discovery, comparison, or feature inspection makes the count feel too low for the apparent scope.

    Completeness2/5

    The surface covers only running a prompt and viewing active SAE features. It lacks obvious operations such as listing models/layers, searching or inspecting individual features, or comparing prompts, leaving significant gaps for an interpretability playground.

    Maintenance

    ActivityActive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      C
      maintenance
      MCP server for mechanistic interpretability research, enabling agents to drive probe-causality and SAE-feature experiments via 8 typed tools on user's own compute (Colab).
      Apache 2.0
    • A
      license
      Not graded
      quality
      D
      maintenance
      Mechanistic interpretability MCP server wrapping chuk-lazarus, enabling model loading, activation extraction, probe training, steering, and ablation via MCP tools.
      21
      Apache 2.0
    • A
      license
      A
      quality
      C
      maintenance
      MCP server for analyzing transformer language model internals, exposing tools to trace attention heads and neurons responsible for predictions, get activation statistics, ablate components, and sketch circuits.
      4
      43 PyPI
      1
      MIT
    • F
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents and developers to query traces, sessions, exceptions, prompts, datasets, annotation queues, scores, and metrics for local debugging and observability. Runs locally over stdio or HTTP with selective tool-group loading, read-only mode, and compact or file-dump output to control token usage.
      -