lewm-mcp
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@lewm-mcp analyze this screenshot for anomalies"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
lewm-mcp
MCP server for LeWorldModel — visual anomaly detection for Claude Code agents and other MCP clients.
Uses a JEPA-style ViT encoder to compute surprise scores between frames, enabling agents to detect unexpected UI changes, video anomalies, and state mismatches.
Quick start
npx lewm-mcpOr install globally:
npm install -g lewm-mcp
lewm-mcpRelated MCP server: Vision-Graft MCP
Python requirements
The model runs in a Python subprocess. Install dependencies:
pip install torch transformers Pillow numpy
# For video analysis:
pip install opencv-pythonRemote inference: To run on a GPU/MPS server (e.g.
100.105.97.18), startlewm-mcpthere and connect via MCP over SSH or Tailscale.
Tools
load_model
Load the ViT encoder into memory. Call once before other tools.
{ "checkpoint": "/path/to/checkpoint" }Returns model info (param count, device, embed_dim, status).
get_model_status
Check if the model is loaded, which checkpoint, param count, device (mps/cuda/cpu).
analyze_screenshot
Encode a screenshot and compute surprise vs a previous frame.
{
"source": "/path/to/screenshot.png",
"previous_source": "/path/to/previous.png",
"anomaly_threshold": 2.0
}source accepts a file path or base64-encoded image data.
Returns: embedding, surprise_score, normalized_surprise, cosine_similarity, mse, anomaly.
compare_states
Compare expected vs actual screen states in embedding space.
{
"expected": "/path/to/expected.png",
"actual": "/path/to/actual.png",
"anomaly_threshold": 0.1
}Returns: cosine_similarity, mse, surprise_score, match, anomaly.
analyze_video
Extract frames from a video, run through ViT, return surprise timeline.
{
"video_path": "/path/to/recording.mp4",
"frame_sample_rate": 1,
"sigma_threshold": 2.0,
"top_n": 5
}Returns: timestamps, surprise_scores, normalized_scores, anomaly_windows, top_anomalies.
run_surprise_detection
Full pipeline on a directory of screenshots or a video file.
{
"directory": "/path/to/screenshots/",
"threshold_multiplier": 2.0
}Returns: timeline, exceeded_threshold, stats.
Architecture
Claude Code agent
│
│ MCP (stdio)
▼
lewm-mcp (TypeScript)
│
│ stdin/stdout JSON protocol
▼
model.py (Python subprocess)
│
▼
transformers ViTModel (tiny: hidden=192, layers=3, patch=16)
runs on: mps → cuda → cpuThe Python process stays alive between tool calls — the model loads once and stays warm.
Configure in Claude Code
Add to ~/.claude/claude_desktop_config.json (or equivalent MCP config):
{
"mcpServers": {
"lewm-mcp": {
"command": "npx",
"args": ["lewm-mcp"]
}
}
}Model details
Default model: tiny ViT initialized with random weights.
hidden_size: 192num_hidden_layers: 3num_attention_heads: 3patch_size: 16image_size: 224
Pass a checkpoint path to load_model to use a fine-tuned or pretrained checkpoint (must be a transformers ViTModel checkpoint).
Environment variables
Variable | Default | Description |
|
| Python executable to use for model subprocess |
License
MIT
Available Tools
6 toolsanalyze_screenshotB
Encode a screenshot through the ViT encoder and optionally compute a surprise score vs a previous frame. Returns embedding vector, cosine similarity, MSE, and anomaly flag.
Inputs: source (required), previous_source, anomaly_threshold.
analyze_videoA
Extract frames from a video file, run them through the ViT encoder, and compute frame-to-frame surprise scores. Returns timestamp array, surprise scores, z-score normalized scores, anomaly windows (>2σ spikes), and top N anomaly timestamps.
Inputs: video_path (required), top_n, sigma_threshold, frame_sample_rate.
compare_statesA
Compare two screenshots in embedding space. Useful for 'does this screen match what I expected?' Returns cosine similarity, MSE, surprise score, and match/anomaly flags.
Inputs: actual (required), expected (required), anomaly_threshold.
get_model_statusA
Check whether the model is loaded, which checkpoint is active, parameter count, and which device (mps/cuda/cpu) is in use.
Inputs: no named parameters.
load_modelA
Load the ViT world-model encoder into memory. Call this before other tools for faster inference. Defaults to a tiny pretrained ViT (hidden_size=192, 3 layers, patch_size=16).
Inputs: checkpoint.
run_surprise_detectionA
Run full surprise detection pipeline on a directory of screenshots or a video file. Returns annotated timeline, list of frames exceeding threshold, and summary stats.
Inputs: directory, video_path, threshold_multiplier.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 6 tools
Some tools have overlapping purposes: analyze_screenshot with a previous frame and compare_states both compare two screenshots and return similar metrics. Similarly, analyze_video and run_surprise_detection both process videos and compute surprise scores, though run_surprise_detection is a higher-level pipeline.
All tool names follow a consistent verb_noun pattern (e.g., load_model, analyze_screenshot, get_model_status). The naming is predictable and easy to infer.
Six tools is a well-scoped count for a surprise-detection server. Each tool addresses a distinct part of the workflow (model management, single-image analysis, pairwise comparison, video analysis, full pipeline) without redundancy.
The server covers the core workflow: load model, check status, analyze images/videos, compare states, and run a full detection pipeline. Minor gaps exist, such as no explicit way to unload the model or customize the loaded checkpoint beyond defaults, but these are not critical.
Related MCP Connectors
Visual testing for coding agents: your agent sees every UI change it makes and triages each diff.
Video scene understanding for AI agents via the Primate Vision API.
Give agents eyes on any web page: structured context, and changes explained in plain language.
Browser-backed QA with evidence and fix-ready reports for coding agents.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceEnables agentic coding workflows in Claude Code through a multi-candidate patch evaluation loop that generates code variants, validates builds, scores results with mandatory vision testing, and automatically selects the best implementation.MIT
- AlicenseAqualityDmaintenanceEnables blind AI coding models to debug visual bugs by capturing screenshots, analyzing them with a vision-capable model, and returning a written report.15MIT
- AlicenseNot gradedqualityDmaintenanceProvides structured visual analysis tools for coding agents, enabling UI analysis, screenshot comparison, OCR, and crop-based refinement through VLM.MIT
- AlicenseNot gradedqualityAmaintenanceEnables coding agents to watch and analyze videos by extracting scene-aware frames, transcribing speech, and generating shot timelines, all constrained by a token budget to fit LLM context limits.1MIT