Skip to main content
Glama

lewm-mcp

MCP server for LeWorldModel — visual anomaly detection for Claude Code agents and other MCP clients.

Uses a JEPA-style ViT encoder to compute surprise scores between frames, enabling agents to detect unexpected UI changes, video anomalies, and state mismatches.

Quick start

npx lewm-mcp

Or install globally:

npm install -g lewm-mcp
lewm-mcp

Related MCP server: Vision-Graft MCP

Python requirements

The model runs in a Python subprocess. Install dependencies:

pip install torch transformers Pillow numpy
# For video analysis:
pip install opencv-python

Remote inference: To run on a GPU/MPS server (e.g. 100.105.97.18), start lewm-mcp there and connect via MCP over SSH or Tailscale.

Tools

load_model

Load the ViT encoder into memory. Call once before other tools.

{ "checkpoint": "/path/to/checkpoint" }

Returns model info (param count, device, embed_dim, status).

get_model_status

Check if the model is loaded, which checkpoint, param count, device (mps/cuda/cpu).

analyze_screenshot

Encode a screenshot and compute surprise vs a previous frame.

{
  "source": "/path/to/screenshot.png",
  "previous_source": "/path/to/previous.png",
  "anomaly_threshold": 2.0
}

source accepts a file path or base64-encoded image data.

Returns: embedding, surprise_score, normalized_surprise, cosine_similarity, mse, anomaly.

compare_states

Compare expected vs actual screen states in embedding space.

{
  "expected": "/path/to/expected.png",
  "actual": "/path/to/actual.png",
  "anomaly_threshold": 0.1
}

Returns: cosine_similarity, mse, surprise_score, match, anomaly.

analyze_video

Extract frames from a video, run through ViT, return surprise timeline.

{
  "video_path": "/path/to/recording.mp4",
  "frame_sample_rate": 1,
  "sigma_threshold": 2.0,
  "top_n": 5
}

Returns: timestamps, surprise_scores, normalized_scores, anomaly_windows, top_anomalies.

run_surprise_detection

Full pipeline on a directory of screenshots or a video file.

{
  "directory": "/path/to/screenshots/",
  "threshold_multiplier": 2.0
}

Returns: timeline, exceeded_threshold, stats.

Architecture

Claude Code agent
       │
       │ MCP (stdio)
       ▼
  lewm-mcp (TypeScript)
       │
       │ stdin/stdout JSON protocol
       ▼
  model.py (Python subprocess)
       │
       ▼
  transformers ViTModel (tiny: hidden=192, layers=3, patch=16)
  runs on: mps → cuda → cpu

The Python process stays alive between tool calls — the model loads once and stays warm.

Configure in Claude Code

Add to ~/.claude/claude_desktop_config.json (or equivalent MCP config):

{
  "mcpServers": {
    "lewm-mcp": {
      "command": "npx",
      "args": ["lewm-mcp"]
    }
  }
}

Model details

Default model: tiny ViT initialized with random weights.

  • hidden_size: 192

  • num_hidden_layers: 3

  • num_attention_heads: 3

  • patch_size: 16

  • image_size: 224

Pass a checkpoint path to load_model to use a fine-tuned or pretrained checkpoint (must be a transformers ViTModel checkpoint).

Environment variables

Variable

Default

Description

LEWM_PYTHON

python3

Python executable to use for model subprocess

License

MIT

Available Tools

6 tools
analyze_screenshotB

Encode a screenshot through the ViT encoder and optionally compute a surprise score vs a previous frame. Returns embedding vector, cosine similarity, MSE, and anomaly flag.

Inputs: source (required), previous_source, anomaly_threshold.

analyze_videoA

Extract frames from a video file, run them through the ViT encoder, and compute frame-to-frame surprise scores. Returns timestamp array, surprise scores, z-score normalized scores, anomaly windows (>2σ spikes), and top N anomaly timestamps.

Inputs: video_path (required), top_n, sigma_threshold, frame_sample_rate.

compare_statesA

Compare two screenshots in embedding space. Useful for 'does this screen match what I expected?' Returns cosine similarity, MSE, surprise score, and match/anomaly flags.

Inputs: actual (required), expected (required), anomaly_threshold.

get_model_statusA

Check whether the model is loaded, which checkpoint is active, parameter count, and which device (mps/cuda/cpu) is in use.

Inputs: no named parameters.

load_modelA

Load the ViT world-model encoder into memory. Call this before other tools for faster inference. Defaults to a tiny pretrained ViT (hidden_size=192, 3 layers, patch_size=16).

Inputs: checkpoint.

run_surprise_detectionA

Run full surprise detection pipeline on a directory of screenshots or a video file. Returns annotated timeline, list of frames exceeding threshold, and summary stats.

Inputs: directory, video_path, threshold_multiplier.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.8/5.0

    Scored across 6 tools

    Disambiguation3/5

    Some tools have overlapping purposes: analyze_screenshot with a previous frame and compare_states both compare two screenshots and return similar metrics. Similarly, analyze_video and run_surprise_detection both process videos and compute surprise scores, though run_surprise_detection is a higher-level pipeline.

    Naming Consistency5/5

    All tool names follow a consistent verb_noun pattern (e.g., load_model, analyze_screenshot, get_model_status). The naming is predictable and easy to infer.

    Tool Count5/5

    Six tools is a well-scoped count for a surprise-detection server. Each tool addresses a distinct part of the workflow (model management, single-image analysis, pairwise comparison, video analysis, full pipeline) without redundancy.

    Completeness4/5

    The server covers the core workflow: load model, check status, analyze images/videos, compare states, and run a full detection pipeline. Minor gaps exist, such as no explicit way to unload the model or customize the loaded checkpoint beyond defaults, but these are not critical.

    Related MCP Connectors

    Related MCP Servers