Screen Observer MCP

Give your AI agent eyes on the Windows desktop — without giving up privacy.

A local, read-only MCP server that exposes your screen as a bounded, privacy-filtered JSON state, plus optional on-demand screenshots. Built for Claude Code and any other MCP-compatible client.

✨ Why Screen Observer MCP?

Most AI agents are blind. They can't tell whether your build just finished, whether a dialog is blocking your script, or what window is currently focused. Screen Observer MCP fixes that by giving the agent a structured, queryable, bounded view of your screen:

• JSON-first. The agent sees a semantic screen state — active window, focused element, UI tree, change summary — not an opaque pixel stream.

• Privacy by default. Password fields, sensitive titles, and configurable regions are redacted in source coordinates before any image is encoded. Nothing is written to disk.

• Agent-explicit. The agent decides when to start and stop observation. No background daemons, no cross-process IPC, no surprises.

• Event-driven waits. Block on wait_for_title("Build successful") or wait_for_idle(3000) instead of polling the model.

• Bounded memory. Frames live in a ring buffer (RING_DEFAULT_FRAMES, MAX_RING_BYTES). Stop wipes everything in RAM.

Related MCP server: blade-computer-use

🚀 Quickstart

1. Install (editable, dev)

py -3.12 -m venv .venv
.venv\Scripts\python -m pip install -e ".[dev]"

2. Run the MCP server

.venv\Scripts\screen-observer mcp

3. Wire it into Claude Code

Add to your Claude Code MCP config (%APPDATA%\Claude\claude_desktop_config.json or .mcp.json):

{
  "mcpServers": {
    "screen-observer": {
      "command": ".venv\\Scripts\\screen-observer.exe",
      "args": ["mcp"]
    }
  }
}

Or use the prebuilt onedir artifact:

{
  "mcpServers": {
    "screen-observer": {
      "command": "C:\\path\\to\\dist\\screen-observer\\screen-observer.exe",
      "args": ["mcp"]
    }
  }
}

The packaged executable depends on its _internal\ directory — copy the whole dist\screen-observer\ folder, not just the .exe.

🧰 The 10 MCP Tools

The lifecycle contract

start ──► read (loop, with wait_for_change/title/idle) ──► stop

• Before start and after stop, every read tool returns the structured observer_not_started error.

• stop is the data boundary — it clears all published frame/state artifacts immediately. Nothing is persisted to disk.

• JSON paths carry everything a text-only client needs. PNG paths require a multimodal/vision-capable client to interpret the rendered pixels.

🎬 Three agent patterns

1. Explicit polling — the agent drives every step

screen_observe_start                  # ready=true, firstRevision, capabilities, firstFrame
loop:
  screen_wait_for_change(since_revision, timeout_ms = 5000)
  inspect state → decide whether the task is done
screen_observe_stop                   # summary carries session counters

Use this when the agent knows exactly which UI element to inspect.

2. Wait for a title substring — let the server block

screen_observe_start
screen_wait_for_title(
    title_contains = "Build successful",
    since_revision = <firstRevision>,
    timeout_ms    = 120000)
# matched=true → matchedAtRevision, observedTitle
screen_observe_stop

Perfect for npm run build / pytest / cargo test terminals.

3. Wait for screen idle — completion by silence

screen_observe_start
screen_wait_for_idle(idle_ms = 3000, since_revision = <firstRevision>, timeout_ms = 120000)
# idleReached=true → idleMs, lastObservedRevision
screen_observe_stop

For tasks that finish without a recognizable title.

🐚 PowerShell wrapper

For humans and one-shot scripts:

# Block until the terminal shows "Build successful":
scripts\observe-until.ps1 -WaitForTitle 'Build successful' -TimeoutSec 180

# Block until the screen stops changing for 3 s:
scripts\observe-until.ps1 -WaitForIdleMs 3000 -TimeoutSec 60

The script writes the start/stop summary to the pipeline and exits non-zero on timeout.

🔒 Privacy & data lifecycle

We take this seriously, because screens leak.

• In-memory only. Screenshots, videos, and history are not written to disk by the application. Frames live in a bounded ring buffer (MAX_RING_FRAMES, MAX_RING_BYTES in src/screen_observer/domain/limits.py).

• Opt-in images. include_image=true is the only path that ever encodes PNG. JSON-only clients never see pixels.

• Source-coordinate redaction. Password element names and values are scrubbed. Configured process, title, and physical-pixel regions are redacted before resize and PNG encoding.

• No leaked telemetry. Base64 image data and full UI text dumps are never written to diagnostic logs.

• Stop is the data boundary. screen_observe_stop clears every published artifact immediately.

• Honest disclaimer. The application cannot guarantee that Windows will never page process memory to disk.

🏗️ Architecture

DXGI Desktop Duplication (dxcam)
        ↓
ChangeDetector + RingBuffer (in-memory, bounded)
        ↓
StateService (single source of truth)
        ↓
  ┌──────────────┐    ┌────────────────┐
  │  CLI adapter │    │  MCP stdio     │  ← FastMCP, 10 tools
  │  (humans)    │    │  (agents)      │
  └──────────────┘    └────────────────┘

• Capture backend: DXGI Desktop Duplication (dxcam) on Windows 11; mss injectable for synthetic tests. The PyInstaller onedir build must collect_all("dxcam") to bundle the native DXGI/D3D11 binaries.

• Privacy filter: runs before resize/encode.

• One process for now. Capture + state + MCP server live in the same process to keep deployment simple. The MCP layer is the only public interface — agents never touch internal services directly.

• JSON-only by default. The MCP protocol reserves stdout; diagnostics go to stderr; tool handlers return structured safe errors instead of Python tracebacks.

📦 Build a portable Windows package

# Build the onedir artifact:
powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\build_windows.ps1

# Smoke test the packaged CLI/MCP:
powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\smoke_packaged.ps1

Artifact lands at dist\screen-observer\screen-observer.exe plus its _internal\ folder.

A onefile package has not been built or validated — stick with onedir for now.

🛠️ Development

# Run the full test suite:
.venv\Scripts\python -m pytest -q

# Lint:
.venv\Scripts\python -m ruff check src tests

# Strict type check:
.venv\Scripts\python -m mypy

# Dependency sanity:
.venv\Scripts\python -m pip check

Test entry points:

• tests/adapters/test_windows_integration.py — Windows capture / UIA / window adapter integration.

• tests/interfaces/test_mcp_server.py — stdio protocol and tool contracts.

• tests/integration/test_packaged_smoke.py — packaged onedir artifact smoke (gated by SCREEN_OBSERVER_PACKAGED_TEST=1).

Project layout

src/screen_observer/
├── adapters/        # DXGI / Windows / UIA / fake capture backends
├── domain/          # Pure models, limits, errors, privacy rules
├── services/        # StateService, ChangeDetector, RingBuffer, ImageEncoder
├── interfaces/      # CLI adapter, MCP stdio server
└── main.py          # Entry point

🗺️ When to use Screen Observer MCP

✅ Great fit

• Driving CI / build / test runs from an agent and waiting for completion.

• Verifying desktop app behavior after a script change (does the dialog appear? did the window move?).

• Capturing screenshots of the current UI for a vision-capable agent to interpret.

• Building automation that needs to know the active window before clicking.

⚠️ Not a fit

• 30/60 FPS video analysis — this is stateful, not streaming.

• Remote screen sharing — strictly local, no network surface.

• Mouse/keyboard automation in v1 — coming later; see open issues.

• Cross-platform — Windows 11 only for real capture (the interfaces are platform-neutral for testing).

🤝 Contributing

Issues and PRs welcome. Before opening a PR:

1. Run pytest, ruff check, and mypy — all must pass.

2. Add or update tests for any behavioral change.

3. Keep MCP tool contracts backward-compatible (additive only).

📄 License

Apache License 2.0 — Copyright © 2026 Aura.

You may use, modify, and distribute this project (including for commercial purposes) under the terms of the Apache License, Version 2.0. A copy of the license is included in this repository at LICENSE.

🙏 Acknowledgments

• Model Context Protocol — the transport that makes this possible.

• dxcam — clean DXGI Desktop Duplication bindings.

• pywinauto — UI Automation access.

• FastMCP — the MCP Python SDK.