Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@lewm-mcp analyze this screenshot for anomalies"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
MCP server for LeWorldModel — visual anomaly detection for Claude Code agents and other MCP clients.
Uses a JEPA-style ViT encoder to compute surprise scores between frames, enabling agents to detect unexpected UI changes, video anomalies, and state mismatches.
npx lewm-mcpOr install globally:
npm install -g lewm-mcp
lewm-mcpRelated MCP server: websight
The model runs in a Python subprocess. Install dependencies:
pip install torch transformers Pillow numpy
# For video analysis:
pip install opencv-pythonRemote inference: To run on a GPU/MPS server (e.g.
100.105.97.18), startlewm-mcpthere and connect via MCP over SSH or Tailscale.
load_modelLoad the ViT encoder into memory. Call once before other tools.
{ "checkpoint": "/path/to/checkpoint" }Returns model info (param count, device, embed_dim, status).
get_model_statusCheck if the model is loaded, which checkpoint, param count, device (mps/cuda/cpu).
analyze_screenshotEncode a screenshot and compute surprise vs a previous frame.
{
"source": "/path/to/screenshot.png",
"previous_source": "/path/to/previous.png",
"anomaly_threshold": 2.0
}source accepts a file path or base64-encoded image data.
Returns: embedding, surprise_score, normalized_surprise, cosine_similarity, mse, anomaly.
compare_statesCompare expected vs actual screen states in embedding space.
{
"expected": "/path/to/expected.png",
"actual": "/path/to/actual.png",
"anomaly_threshold": 0.1
}Returns: cosine_similarity, mse, surprise_score, match, anomaly.
analyze_videoExtract frames from a video, run through ViT, return surprise timeline.
{
"video_path": "/path/to/recording.mp4",
"frame_sample_rate": 1,
"sigma_threshold": 2.0,
"top_n": 5
}Returns: timestamps, surprise_scores, normalized_scores, anomaly_windows, top_anomalies.
run_surprise_detectionFull pipeline on a directory of screenshots or a video file.
{
"directory": "/path/to/screenshots/",
"threshold_multiplier": 2.0
}Returns: timeline, exceeded_threshold, stats.
Claude Code agent
│
│ MCP (stdio)
▼
lewm-mcp (TypeScript)
│
│ stdin/stdout JSON protocol
▼
model.py (Python subprocess)
│
▼
transformers ViTModel (tiny: hidden=192, layers=3, patch=16)
runs on: mps → cuda → cpuThe Python process stays alive between tool calls — the model loads once and stays warm.
Add to ~/.claude/claude_desktop_config.json (or equivalent MCP config):
{
"mcpServers": {
"lewm-mcp": {
"command": "npx",
"args": ["lewm-mcp"]
}
}
}Default model: tiny ViT initialized with random weights.
hidden_size: 192
num_hidden_layers: 3
num_attention_heads: 3
patch_size: 16
image_size: 224
Pass a checkpoint path to load_model to use a fine-tuned or pretrained checkpoint (must be a transformers ViTModel checkpoint).
Variable | Default | Description |
|
| Python executable to use for model subprocess |
MIT
Encode a screenshot through the ViT encoder and optionally compute a surprise score vs a previous frame. Returns embedding vector, cosine similarity, MSE, and anomaly flag.
| Name | Required | Description | Default |
|---|---|---|---|
| source | Yes | File path to an image (png/jpg/webp) OR base64-encoded image data (with or without data URL prefix). | |
| previous_source | No | Optional: file path or base64 of a previous screenshot to compare against. If omitted, uses the last screenshot passed to this tool. | |
| anomaly_threshold | No | Normalized surprise multiplier above which to flag as anomaly (default 2.0 = 2× baseline). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description bears the full burden of disclosing behavioral traits. It states the main operation and outputs but fails to mention the stateful behavior that if previous_source is omitted, the tool uses the last screenshot passed to it. This is a significant gap that could lead to incorrect usage. It also does not disclose any side effects or requirements (e.g., model loaded).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that is front-loaded with the main action and resource, and it efficiently lists the return values without any filler or unnecessary detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description gives a concise overview of the tool's function and outputs, which is minimally adequate. However, it lacks important context such as the meaning of 'surprise score', output format details (no output schema exists), and the stateful default for previous_source. Given the tool's moderate complexity and absence of annotations, more explanation would be beneficial.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all parameters are already documented with useful semantics. The tool description does not add additional parameter-level meaning beyond the schema, but it does mention general outputs that relate to parameters. Baseline 3 is appropriate when the schema handles the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function with a specific verb ('Encode'), a resource ('screenshot through the ViT encoder'), and optional behavior ('compute a surprise score'). It also lists the return values. However, it does not differentiate itself from sibling tools like compare_states or run_surprise_detection, which may overlap in functionality.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool is used for encoding screenshots and optionally comparing them with a previous frame, but it does not provide explicit guidance on when to choose this tool over alternatives like compare_states or run_surprise_detection. No exclusions or prerequisites are mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract frames from a video file, run them through the ViT encoder, and compute frame-to-frame surprise scores. Returns timestamp array, surprise scores, z-score normalized scores, anomaly windows (>2σ spikes), and top N anomaly timestamps.
| Name | Required | Description | Default |
|---|---|---|---|
| top_n | No | Number of top anomaly timestamps to return (default 5). | |
| video_path | Yes | Path to an mp4 or webm video file. | |
| sigma_threshold | No | Z-score threshold to flag as anomaly window (default 2.0). | |
| frame_sample_rate | No | Sample one frame every N seconds (default 1). Increase for long videos. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It transparently describes the processing pipeline (frame extraction, ViT encoding, surprise computation) and explicitly lists returned outputs (timestamps, surprise scores, z-scores, anomaly windows, top N anomalies). This gives agents a clear picture of behavior beyond basic input/output.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the primary action, and a compact list of return values. Every sentence contributes meaningful information without redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description explains return values, which is important given no output schema. However, it omits prerequisites or dependencies, such as whether a model must be loaded first (sibling load_model), and does not mention potential failure modes or performance caveats. This is a notable gap for a complex pipeline.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds value by linking 'top N anomaly timestamps' to top_n and '>2σ spikes' to sigma_threshold, reinforcing the schema's parameter descriptions. For instance, the anomaly window threshold is directly tied to sigma_threshold.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action sequence: 'Extract frames from a video file, run them through the ViT encoder, and compute frame-to-frame surprise scores.' This specific verb+resource distinguishes it from siblings like analyze_screenshot (screenshots vs. video) and run_surprise_detection (generic detection vs. video-specific).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied but not explicitly stated. The phrase 'from a video file' suggests it is for video anomaly detection, but there are no explicit when-to-use vs. alternatives or exclusions. Sibling tools like load_model and run_surprise_detection are not referenced.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Compare two screenshots in embedding space. Useful for 'does this screen match what I expected?' Returns cosine similarity, MSE, surprise score, and match/anomaly flags.
| Name | Required | Description | Default |
|---|---|---|---|
| actual | Yes | File path or base64 of the actual/current state. | |
| expected | Yes | File path or base64 of the expected/reference state. | |
| anomaly_threshold | No | Surprise score above which to flag as anomaly (default 0.1). Lower = stricter. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It lists the return values (cosine similarity, MSE, surprise score, match/anomaly flags), but does not mention prerequisites like model loading, potential side effects, or error behavior. This is a moderate gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is only two sentences, front-loaded with the purpose and outputs. It is concise and every sentence earns its place without repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a relatively simple comparison tool with no output schema, the description adequately covers purpose, use case, and return values. However, it omits prerequisites (e.g., need to load a model) and edge cases, so it is not fully complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema description coverage is 100%, so each parameter is already explained. The tool description adds no extra parameter semantics (e.g., format or constraints) beyond what the schema provides, so the baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it compares two screenshots in embedding space, with a specific verb (compare) and resource (two screenshots). It also provides a use case ('does this screen match what I expected?'), which distinguishes it from sibling tools that analyze single images or videos.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit context for when to use it: 'Useful for does this screen match what I expected?' It implies this tool is for comparing expected vs actual states, but it does not explicitly name alternatives or state when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Check whether the model is loaded, which checkpoint is active, parameter count, and which device (mps/cuda/cpu) is in use.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It accurately describes the read-only nature ('check') and the information returned, but does not mention potential edge cases (e.g., behavior when model is not loaded) or confirm non-destructiveness beyond the verb choice.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, well-structured sentence that front-loads the verb and lists all key query aspects without unnecessary detail. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple status-check tool with no parameters and no output schema, the description sufficiently covers what the tool does and what information it provides. It lacks explicit mention of return format, but the listed items imply the response content, making it complete for this use case.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has zero parameters, so there are no parameter semantics to describe. The baseline for 0 params is 4, and the description appropriately focuses on output rather than inputs.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses the specific verb 'check' and clearly identifies the resource: model status. It enumerates the exact aspects it reports on (loaded state, active checkpoint, parameter count, device), making it distinct from sibling tools like load_model or analyze_screenshot.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. There are no explicit or implicit usage instructions beyond the action itself. The context suggests it could be used before load_model, but this is not stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Load the ViT world-model encoder into memory. Call this before other tools for faster inference. Defaults to a tiny pretrained ViT (hidden_size=192, 3 layers, patch_size=16).
| Name | Required | Description | Default |
|---|---|---|---|
| checkpoint | No | Optional path to a local model checkpoint directory. Omit to use the default tiny ViT. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the full burden. It does not disclose important behavioral traits such as memory footprint, idempotency, or behavior on repeated calls. The default model details are informative but not about behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences, front-loaded with purpose and usage. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple with one optional parameter, and the description is adequate for a basic load operation. However, it lacks details on side effects, error conditions, or how it interacts with sibling tools like get_model_status, making it not fully complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already fully describes the 'checkpoint' parameter (optional path, omit for default). The description adds specifics about the default ViT architecture, but this does not change parameter usage. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Load') and resource ('ViT world-model encoder'), clearly distinguishing this setup tool from the analysis-oriented sibling tools. It is immediately obvious what the tool does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly instructs 'Call this before other tools for faster inference,' providing a clear when-to-use context. It doesn't mention exclusions or alternatives, but for a load/setup tool this is sufficient.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run full surprise detection pipeline on a directory of screenshots or a video file. Returns annotated timeline, list of frames exceeding threshold, and summary stats.
| Name | Required | Description | Default |
|---|---|---|---|
| directory | No | Path to a directory of image files (sorted alphabetically). Provide this or video_path. | |
| video_path | No | Path to a video file. Provide this or directory. | |
| threshold_multiplier | No | Z-score multiplier for anomaly detection (default 2.0 = 2σ above mean). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
There are no annotations to signal safety or side effects. The description only mentions the return values, not whether the pipeline modifies files, requires a pre-loaded model (hinted by siblings load_model and get_model_status), or has performance implications. For a tool likely to depend on model state, this is a significant transparency gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that leads with the action, then states inputs and outputs. Every word is useful; there is no redundancy or extraneous information. It fits the ideal model of front-loaded, efficient prose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a pipeline tool with three parameters and no output schema, the description gives a good roll-up of what it does and returns. However, it omits the relationship with model loading (sibling load_model) and does not explain the processing steps or how threshold_multiplier affects results, which might be relevant for an agent deciding to use it. Overall, it is nearly complete but could add a prerequisite note.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema provides descriptions for all three parameters, matching the 100% schema coverage baseline. The description adds no parameter-specific detail beyond mentioning 'directory' and 'video file' as input types, which is already in the schema. It earns the baseline score of 3.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific action ('Run full surprise detection pipeline') with explicit input types (directory or video file) and output components (annotated timeline, frames exceeding threshold, summary stats). It distinguishes itself from siblings like analyze_screenshot and analyze_video by emphasizing the full pipeline on a batch of inputs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for processing a directory of screenshots or a video file, which is clear context. However, it does not explicitly mention when to use this tool versus alternatives such as analyze_screenshot for single images or analyze_video for individual videos, nor does it state exclusions or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0analyze_screenshotanalyze_videocompare_statesget_model_statusload_modelrun_surprise_detectionScored across 6 tools
Some tools have overlapping purposes: analyze_screenshot with a previous frame and compare_states both compare two screenshots and return similar metrics. Similarly, analyze_video and run_surprise_detection both process videos and compute surprise scores, though run_surprise_detection is a higher-level pipeline.
All tool names follow a consistent verb_noun pattern (e.g., load_model, analyze_screenshot, get_model_status). The naming is predictable and easy to infer.
Six tools is a well-scoped count for a surprise-detection server. Each tool addresses a distinct part of the workflow (model management, single-image analysis, pairwise comparison, video analysis, full pipeline) without redundancy.
The server covers the core workflow: load model, check status, analyze images/videos, compare states, and run a full detection pipeline. Minor gaps exist, such as no explicit way to unload the model or customize the loaded checkpoint beyond defaults, but these are not critical.
Video scene understanding for AI agents via the Primate Vision API.
Browser-backed QA with evidence and fix-ready reports for coding agents.
Codebase intelligence for agents: 152 structured artifacts across 21 programs, one call.
UI design from prompts, screenshots, and URLs for AI coding agents and theme tokens.
