ui-diff-mcp

MCP server for comparing mobile app screenshots against mockup designs using free-first visual model diffing. Calorix is the primary integration target.

Design spec: docs/superpowers/specs/2026-06-12-ui-diff-mcp-research-design.md

Free-First Default

The default mode (free) never calls paid OpenRouter routes. It probes direct Gemini routes first, then direct Mistral routes, OpenCode Zen's image-capable mimo-v2.5-free, native NVIDIA free VLM endpoints, and finally OpenRouter :free routes. Every route must pass the role's real image-count and JSON probe before selection. Paid OpenRouter models are disabled unless mode: "paid" is passed and UI_DIFF_ENABLE_PAID_MODE=1 is set.

OpenCode currently also lists deepseek-v4-flash-free, but its model metadata is text-only. It is intentionally excluded from auditor, reviewer, and target-recovery roles because those roles require four or five crop images. OpenCode documents its free models as limited-time routes, so catalog presence never replaces runtime probes.

Gemini model-list probing on this machine showed gemini-3.1-pro-preview, gemini-3.5-flash, and gemini-3.1-flash-lite are visible. A direct live call to gemini-3.1-pro-preview returned free-tier quota limit 0, so it remains first in the quality ranking but is expected to fail closed unless quota is available. gemini-3.5-flash is the current direct Gemini live-gate model.

Mistral model-list probing showed vision-capable routes including mistral-large-2512, mistral-medium-2604, mistral-small-2603, ministral-14b-2512, and ministral-8b-2512. Live role probing on this machine showed ministral-14b-2512 and ministral-8b-2512 correctly handle the five-image payload required by auditor/reviewer/recovery. mistral-large-2512, mistral-medium-2604, and mistral-small-2603 miscounted or misclassified simple probe images, so they are not current pipeline routes.

Before starting a free-model run, the pipeline estimates the required request count and checks available quota against the OpenRouter key info endpoint. If estimated calls exceed available free quota, the run exits immediately with status: "insufficient_free_quota" rather than consuming quota silently.

Related MCP server: MCP Component Review

Modes

mode selects the provider/model route policy. It does not select how much of the UI to inspect.

Diff Scopes

Use diffScope to select the visual scope independently from provider mode.

{ "kind": "screen" }

Audits the whole screen first: global placement, major color/appearance changes, broad shape/border/layer differences, and whole-screen diff masks. Target-level recovery is bypassed.

{ "kind": "regions", "regions": ["top", "nav"] }

Audits only selected deterministic regions. Current region names are top, middle, bottom, header, content, and nav. Target recovery is restricted to uncovered components inside the selected regions.

{ "kind": "target", "query": "scan button" }

Resolves a target by locator label, visible text, and element type, then audits only the best matching pair. If the target cannot be resolved, the report includes a warning and does not pretend the target was checked.

{ "kind": "full" }

Default. Runs screen/region summaries, scope-level VLM audit where deterministic triggers fire, and the existing target-level audit/recovery path.

Example MCP payload:

{
  "expectedImagePath": "C:/mockups/Today.png",
  "actualImagePath": "C:/screenshots/today.png",
  "mode": "free",
  "diffScope": { "kind": "regions", "regions": ["nav"] }
}

Artifacts As Machine Evidence

All generated images (pixel diff, directional overlay, crop pairs, recovery crops) are machine evidence consumed by audit and recovery models. They are not a manual inspection workflow. Do not rely on visual artifact review as a substitute for structured report.json output.

final-diff-groups-overlay.png shows visual clusters of final diffs. It is not a semantic parent/child hierarchy: a whole-screen cluster and localized clusters can be siblings when they explain different criteria. Use semantic-hierarchy-overlay.png and semantic-hierarchy-legend.json to inspect the detected UI structure such as screen, nav, macro card, macro circle, and repeated cards.

Report Parts And Usage Accounting

report.json is the slim manifest. Large report sections are written as referenced JSON parts under artifacts/parts/ instead of being duplicated inline:

• elements.json

• pairs.json

• diffs.json

• unresolved-regions.json

• debug-summary.json

• usage-summary.json

• scope-summary.json

reportParts[].path is relative to the report.json directory. read_ui_diff_report hydrates these parts before returning the report, so existing MCP consumers still receive a full schema-valid report even though the on-disk manifest keeps elements, pairs, diffs, and unresolved regions compact.

usageSummary is first-class run-level accounting. It records input tokens, output tokens, total tokens, reasoning tokens, successful calls, failed calls, fallbacks, route exhaustion, and duration totals by phase, role, and provider/model route. If a provider reports only total tokens, input/output are left as zero and totalOnlyUsageCalls increments; the MCP does not invent a fake split.

LocateAnything Live Gate Sizing

LOCATEANYTHING_MAX_DIMENSION controls the largest image dimension sent to the LocateAnything sidecar. The default remains 1200 for detail.

600 is a local timeout workaround, not a quality default. It shrinks a 1206x2622 Calorix mockup to roughly 276x600 for the locator, which can hide small icons, thin borders, and text. Prefer the highest dimension that fits the sidecar budget, and run the sequential locator benchmark before production sign-off:

$env:UI_DIFF_LIVE_EXPECTED_IMAGE = "C:\Users\xursc\projects\calorix\docs\design-handoff\placeholder-app\reference-images\today--dark.png"
$env:LOCATEANYTHING_SIDECAR_URL = "http://127.0.0.1:39731"
$env:UI_DIFF_LOCATOR_BENCHMARK_DIMENSIONS = "600,900,1200"
npm run benchmark:locator

For Calorix release evidence, do not set UI_DIFF_LIVE_ACTUAL_IMAGE by default. The live gates build the debug APK only when stale, install it when needed, open calorix://debug/reseed, and capture a fresh ADB screenshot into C:\Users\xursc\projects\calorix\.ui-diff\captures. Set UI_DIFF_LIVE_ACTUAL_IMAGE only when you intentionally want to compare against a historical file; that path is logged as an explicit override and is not fresh release evidence.

Calorix live gates validate the canonical default expected reference before any device or pipeline work: C:\Users\xursc\projects\calorix\docs\design-handoff\placeholder-app\reference-images\today--dark.png, its adjacent reference-images-manifest.json, and SHA-256 73BA85F25489C8D45BEAB57DD1B317138870CE8360FE0F4399AB0737A5E505F1. An explicit UI_DIFF_LIVE_EXPECTED_IMAGE remains supported, but must be readable; reference-images-buggy and good-screenshots are never fallback sources.

Calorix reports persist a report-safe inputProvenance record whose expected and actual SHA-256 identities are computed from the exact image bytes by the pipeline. A manifest entry is included only after the pipeline verifies that its recorded hash matches the expected image bytes. Acquisition sources such as auto_capture and env_override are stored separately with verification:"caller_attested"; they are not presented as independently verified facts. Public MCP requests cannot supply computed hashes. On resume, omitted provenance inherits the stored effective record; an explicit replacement is accepted only when the recomputed expected and actual image identities match the resumed report.

Every report records locatorInputSizing, including original image size, sent image size, scale, maxDimension, and whether actual elements were independently located or projected from expected elements. Runs also save the exact image payloads sent to the sidecar as locator-input-expected.png and, in dual-locator mode, locator-input-actual.png; these appear in runArtifacts as locator_input_expected and locator_input_actual.

Locator debugging uses a three-step artifact chain:

• locator-input-expected.png: the exact expected-image bytes sent to LocateAnything.

• locator-expected-overlay.png: the located target boxes drawn on the normalized expected image.

• locator-actual-projected-overlay.png: in default single-pass projection mode, the expected target boxes projected onto the normalized actual screenshot. This does not mean the actual screenshot was sent to LocateAnything; it shows where the expected targets land on the actual image before the diff/audit stages inspect those projected regions.

The overlay files are indexed in runArtifacts as locator_expected_overlay, locator_actual_overlay, and locator_overlay_legend. The legend maps compact overlay labels such as E001 and P001 back to exact element IDs, types, query IDs, source, and boxes.

Installation

npm install

Verification

npm run verify

Build

npm run build

Running the server

node dist/src/index.js

Environment Variables

Copy .env.example and fill in the relevant keys.

This implementation requires no user-authored target map, ROI map, ignore mask, or anchor dump.

Configuring OpenCode Zen Free Models

No local OpenCode daemon or opencode run process is required. The MCP calls the OpenAI-compatible Zen API directly. The current free route works with the default public credential; set OPENCODE_API_KEY only when OpenCode provides a dedicated key.

$env:RUN_OPENCODE_LIVE="1"
npm run verify:opencode-live

The gate verifies the live catalog, a one-image structured response, and one deduplicated five-image probe shared across auditor, reviewer, and target recovery.

Configuring Direct Gemini Models

Set GEMINI_API_KEY to an AI Studio key. The pipeline probes direct Gemini routes before using them; gemini-3.1-pro-preview may be visible but quota-blocked on the free tier, so gemini-3.5-flash is the current live-gate route.

$env:GEMINI_API_KEY="..."
$env:RUN_GEMINI_LIVE="1"
npm run verify:gemini-live

Configuring Direct Mistral Models

Set MISTRAL_API_KEY to a Mistral API key. The pipeline starts with ministral-14b-2512 because it passed the same five-image role probe used by the auditor/reviewer/recovery gates; ministral-8b-2512 is the next Mistral fallback.

$env:MISTRAL_API_KEY="..."
$env:RUN_MISTRAL_LIVE="1"
npm run verify:mistral-live

Configuring Native NVIDIA Free Models

Set NVIDIA_API_KEY to a NVIDIA Build API key. The pipeline probes native NVIDIA candidates from CANONICAL_MODEL_RANKING and selects the highest-quality passing model. Use NVIDIA_VLM_BASE_URL to point at a self-hosted NIM instead.

$env:NVIDIA_API_KEY="nvapi-..."
# $env:NVIDIA_VLM_BASE_URL="http://localhost:8000/v1"  # for self-hosted NIM

Run the NVIDIA live gate to verify:

$env:RUN_NVIDIA_LIVE="1"
npm run verify:nvidia-live

Configuring OpenRouter Free Models

Set OPENROUTER_API_KEY to an OpenRouter key (free-tier account works). The pipeline:

1. Estimates required calls (probes + audit + recovery + review).

2. Queries GET https://openrouter.ai/api/v1/key for limit_remaining.

3. Exits with insufficient_free_quota if estimated calls exceed available quota.

4. Throttles OpenRouter free calls to ≤ 18 requests/minute.

$env:OPENROUTER_API_KEY="sk-or-..."

Run the free live gate to verify:

$env:RUN_FREE_LIVE="1"
npm run verify:free-live

LocateAnything Sidecar

The MCP calls a sidecar endpoint at POST /v1/locate-ui-elements. Start the local wrapper after installing NVIDIA's Eagle Embodied package:

git clone https://github.com/NVlabs/Eagle.git C:\Users\xursc\projects\Eagle
python -m venv C:\Users\xursc\projects\.venvs\ui-diff-mcp-locateanything
C:\Users\xursc\projects\.venvs\ui-diff-mcp-locateanything\Scripts\python.exe -m pip install --upgrade pip setuptools wheel
C:\Users\xursc\projects\.venvs\ui-diff-mcp-locateanything\Scripts\python.exe -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
cd C:\Users\xursc\projects\Eagle\Embodied
C:\Users\xursc\projects\.venvs\ui-diff-mcp-locateanything\Scripts\python.exe -m pip install --no-deps -e .
cd C:\Users\xursc\projects\ui-diff-mcp
C:\Users\xursc\projects\.venvs\ui-diff-mcp-locateanything\Scripts\python.exe -m pip install -r sidecars\locateanything\requirements.txt
$env:LOCATEANYTHING_EAGLE_EMBODIED_DIR="C:\Users\xursc\projects\Eagle\Embodied"
$env:LOCATEANYTHING_PYTHON="C:\Users\xursc\projects\.venvs\ui-diff-mcp-locateanything\Scripts\python.exe"
.\scripts\start-locateanything-sidecar.ps1

The startup script and Calorix live-test helper prefer LOCATEANYTHING_PYTHON, then the known local venv at C:\Users\xursc\projects\.venvs\ui-diff-mcp-locateanything\Scripts\python.exe, then plain python. The script prints the interpreter it selected; if /health returns an error, live tests fail fast with that load error instead of waiting for the full readiness timeout.

LOCATEANYTHING_SKIP_MODEL=1 is only a diagnostic shortcut. It makes /health.ready true without loading the 3B model and still runs CV/OCR/optional lanes, but a run made with that flag is not full LocateAnything-model release evidence.

The TypeScript client sends image bytes with each locator request, so LOCATEANYTHING_SIDECAR_URL can point to a remote GPU service that exposes the same contract.

Parser-only sidecar tests:

C:\Users\xursc\projects\.venvs\ui-diff-mcp-locateanything\Scripts\python.exe -m unittest sidecars.locateanything.test_parser

MCP Integration

Claude Code (project-scoped .mcp.json)

{
  "mcpServers": {
    "ui-diff": {
      "command": "node",
      "args": ["C:\\Users\\xursc\\projects\\ui-diff-mcp\\dist\\src\\index.js"],
      "env": {
        "OPENROUTER_API_KEY": "<your-key>"
      }
    }
  }
}

Codex (project-scoped .codex/config.toml)

[mcp_servers.ui-diff]
command = "node"
args = ['C:\Users\xursc\projects\ui-diff-mcp\dist\src\index.js']
enabled = true

Live Release Gates

npm run verify is deterministic and does not call external APIs. Before production use, run the live gates:

Bounded Smoke vs Full Classification

A bounded smoke run (UI_DIFF_MAX_AUDIT_PAIRS set) is explicitly not full visual classification:

• auditLimited: true in compact output and report.json.

• visualClassificationStatus: "incomplete" unless all pairs happened to be within the limit.

• auditScope.auditedPairs / auditScope.totalPairs records the actual vs total pair count.

A full all-target run must show auditLimited: false. Use verify:calorix-full-live to confirm.

See docs/release/production-readiness-checklist.md for the complete sign-off sequence.