# Screen Observer MCP

> **Give your AI agent eyes on the Windows desktop — without giving up privacy.**
>
> A local, read-only MCP server that exposes your screen as a bounded, privacy-filtered JSON state, plus optional on-demand screenshots. Built for [Claude Code](https://code.claude.com) and any other MCP-compatible client.

[![Platform: Windows 11](https://img.shields.io/badge/platform-Windows%2011-0078d4?logo=windows11&logoColor=white)](https://www.microsoft.com/windows)
[![Python 3.12](https://img.shields.io/badge/python-3.12-3776ab?logo=python&logoColor=white)](https://www.python.org/downloads/)
[![MCP](https://img.shields.io/badge/MCP-stdio-blueviolet)](https://modelcontextprotocol.io)
[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-261230?logo=ruff&logoColor=white)](https://docs.astral.sh/ruff/)
[![Type check: mypy --strict](https://img.shields.io/badge/type%20check-mypy%20--strict-blue)](https://mypy.readthedocs.io/)

---

## ✨ Why Screen Observer MCP?

Most AI agents are blind. They can't tell whether your build just finished, whether a dialog is blocking your script, or what window is currently focused. **Screen Observer MCP** fixes that by giving the agent a **structured, queryable, bounded view of your screen**:

- **JSON-first.** The agent sees a semantic screen state — active window, focused element, UI tree, change summary — not an opaque pixel stream.
- **Privacy by default.** Password fields, sensitive titles, and configurable regions are redacted *in source coordinates* before any image is encoded. Nothing is written to disk.
- **Agent-explicit.** The agent decides *when* to start and stop observation. No background daemons, no cross-process IPC, no surprises.
- **Event-driven waits.** Block on `wait_for_title("Build successful")` or `wait_for_idle(3000)` instead of polling the model.
- **Bounded memory.** Frames live in a ring buffer (`RING_DEFAULT_FRAMES`, `MAX_RING_BYTES`). Stop wipes everything in RAM.

---

## 🚀 Quickstart

### 1. Install (editable, dev)

```powershell
py -3.12 -m venv .venv
.venv\Scripts\python -m pip install -e ".[dev]"
```

### 2. Run the MCP server

```powershell
.venv\Scripts\screen-observer mcp
```

### 3. Wire it into Claude Code

Add to your Claude Code MCP config (`%APPDATA%\Claude\claude_desktop_config.json` or `.mcp.json`):

```json
{
  "mcpServers": {
    "screen-observer": {
      "command": ".venv\\Scripts\\screen-observer.exe",
      "args": ["mcp"]
    }
  }
}
```

Or use the prebuilt `onedir` artifact:

```json
{
  "mcpServers": {
    "screen-observer": {
      "command": "C:\\path\\to\\dist\\screen-observer\\screen-observer.exe",
      "args": ["mcp"]
    }
  }
}
```

> The packaged executable depends on its `_internal\` directory — copy the whole `dist\screen-observer\` folder, not just the `.exe`.

---

## 🧰 The 10 MCP Tools

| Tool | What it does |
|---|---|
| `screen_observe_start` | **Begin** an observation session. Synchronously publishes the first redacted frame, returns `ready: true`, `firstRevision`, `capabilities`, and a compact `firstFrame` summary. |
| `screen_observe_stop` | **End** the session. Joins the collector, clears every in-memory frame and current state. Returns a `summary` block (elapsed time, frames, active-window changes) so you can audit the window. |
| `screen_get_state` | Current screen state — geometry, active window, focused element, UI tree, change summary. JSON by default; pass `include_image=true` to attach a Base64 PNG. |
| `screen_wait_for_change` | Block until the published revision advances, or the timeout fires. |
| `screen_wait_for_title` | Block until the active window title contains a substring. Example: `wait_for_title("Build successful")`. Returns `observedRedacted: true` instead of erroring if the title was filtered. |
| `screen_wait_for_idle` | Block until the screen stops changing for *N* ms — perfect for "is the task done yet?" without knowing the marker text. |
| `screen_get_ui_tree` | Bounded UI Automation snapshot for a target element and depth. |
| `screen_get_region` | One physical-pixel region as an in-memory Base64 PNG. No file is written. |
| `screen_get_frame_history` | Pull up to *N* recent redacted frames from the in-memory ring. |
| `screen_get_frame` | Pull one specific redacted frame by `revision`. |

### The lifecycle contract

```text
start ──► read (loop, with wait_for_change/title/idle) ──► stop
```

- **Before `start` and after `stop`**, every read tool returns the structured `observer_not_started` error.
- **`stop` is the data boundary** — it clears all published frame/state artifacts immediately. Nothing is persisted to disk.
- **JSON paths** carry everything a text-only client needs. **PNG paths** require a multimodal/vision-capable client to interpret the rendered pixels.

---

## 🎬 Three agent patterns

### 1. Explicit polling — the agent drives every step

```text
screen_observe_start                  # ready=true, firstRevision, capabilities, firstFrame
loop:
  screen_wait_for_change(since_revision, timeout_ms = 5000)
  inspect state → decide whether the task is done
screen_observe_stop                   # summary carries session counters
```

Use this when the agent knows exactly which UI element to inspect.

### 2. Wait for a title substring — let the server block

```text
screen_observe_start
screen_wait_for_title(
    title_contains = "Build successful",
    since_revision = <firstRevision>,
    timeout_ms    = 120000)
# matched=true → matchedAtRevision, observedTitle
screen_observe_stop
```

Perfect for `npm run build` / `pytest` / `cargo test` terminals.

### 3. Wait for screen idle — completion by silence

```text
screen_observe_start
screen_wait_for_idle(idle_ms = 3000, since_revision = <firstRevision>, timeout_ms = 120000)
# idleReached=true → idleMs, lastObservedRevision
screen_observe_stop
```

For tasks that finish without a recognizable title.

### 🐚 PowerShell wrapper

For humans and one-shot scripts:

```powershell
# Block until the terminal shows "Build successful":
scripts\observe-until.ps1 -WaitForTitle 'Build successful' -TimeoutSec 180

# Block until the screen stops changing for 3 s:
scripts\observe-until.ps1 -WaitForIdleMs 3000 -TimeoutSec 60
```

The script writes the start/stop summary to the pipeline and exits non-zero on timeout.

---

## 🔒 Privacy & data lifecycle

We take this seriously, because screens leak.

- **In-memory only.** Screenshots, videos, and history are *not* written to disk by the application. Frames live in a bounded ring buffer (`MAX_RING_FRAMES`, `MAX_RING_BYTES` in `src/screen_observer/domain/limits.py`).
- **Opt-in images.** `include_image=true` is the only path that ever encodes PNG. JSON-only clients never see pixels.
- **Source-coordinate redaction.** Password element names and values are scrubbed. Configured process, title, and physical-pixel regions are redacted *before* resize and PNG encoding.
- **No leaked telemetry.** Base64 image data and full UI text dumps are never written to diagnostic logs.
- **Stop is the data boundary.** `screen_observe_stop` clears every published artifact immediately.
- **Honest disclaimer.** The application cannot guarantee that Windows will never page process memory to disk.

---

## 🏗️ Architecture

```text
DXGI Desktop Duplication (dxcam)
        ↓
ChangeDetector + RingBuffer (in-memory, bounded)
        ↓
StateService (single source of truth)
        ↓
  ┌──────────────┐    ┌────────────────┐
  │  CLI adapter │    │  MCP stdio     │  ← FastMCP, 10 tools
  │  (humans)    │    │  (agents)      │
  └──────────────┘    └────────────────┘
```

- **Capture backend:** DXGI Desktop Duplication (`dxcam`) on Windows 11; `mss` injectable for synthetic tests. The PyInstaller `onedir` build must `collect_all("dxcam")` to bundle the native DXGI/D3D11 binaries.
- **Privacy filter:** runs before resize/encode.
- **One process for now.** Capture + state + MCP server live in the same process to keep deployment simple. The MCP layer is the only public interface — agents never touch internal services directly.
- **JSON-only by default.** The MCP protocol reserves `stdout`; diagnostics go to `stderr`; tool handlers return structured safe errors instead of Python tracebacks.

---

## 📦 Build a portable Windows package

```powershell
# Build the onedir artifact:
powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\build_windows.ps1

# Smoke test the packaged CLI/MCP:
powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\smoke_packaged.ps1
```

Artifact lands at `dist\screen-observer\screen-observer.exe` plus its `_internal\` folder.

A `onefile` package has **not** been built or validated — stick with `onedir` for now.

---

## 🛠️ Development

```powershell
# Run the full test suite:
.venv\Scripts\python -m pytest -q

# Lint:
.venv\Scripts\python -m ruff check src tests

# Strict type check:
.venv\Scripts\python -m mypy

# Dependency sanity:
.venv\Scripts\python -m pip check
```

Test entry points:

- `tests/adapters/test_windows_integration.py` — Windows capture / UIA / window adapter integration.
- `tests/interfaces/test_mcp_server.py` — stdio protocol and tool contracts.
- `tests/integration/test_packaged_smoke.py` — packaged `onedir` artifact smoke (gated by `SCREEN_OBSERVER_PACKAGED_TEST=1`).

### Project layout

```
src/screen_observer/
├── adapters/        # DXGI / Windows / UIA / fake capture backends
├── domain/          # Pure models, limits, errors, privacy rules
├── services/        # StateService, ChangeDetector, RingBuffer, ImageEncoder
├── interfaces/      # CLI adapter, MCP stdio server
└── main.py          # Entry point
```

---

## 🗺️ When to use Screen Observer MCP

✅ **Great fit**

- Driving CI / build / test runs from an agent and waiting for completion.
- Verifying desktop app behavior after a script change (does the dialog appear? did the window move?).
- Capturing screenshots of the *current* UI for a vision-capable agent to interpret.
- Building automation that needs to know the active window before clicking.

⚠️ **Not a fit**

- 30/60 FPS video analysis — this is stateful, not streaming.
- Remote screen sharing — strictly local, no network surface.
- Mouse/keyboard automation in v1 — coming later; see [open issues](https://github.com/wuhaostudio/screen-observer-mcp/issues).
- Cross-platform — Windows 11 only for real capture (the interfaces are platform-neutral for testing).

---

## 🤝 Contributing

Issues and PRs welcome. Before opening a PR:

1. Run `pytest`, `ruff check`, and `mypy` — all must pass.
2. Add or update tests for any behavioral change.
3. Keep MCP tool contracts backward-compatible (additive only).

---

## 📄 License

[Apache License 2.0](LICENSE) — Copyright © 2026 Aura.

You may use, modify, and distribute this project (including for commercial purposes) under the terms of the Apache License, Version 2.0. A copy of the license is included in this repository at [`LICENSE`](LICENSE).

---

## 🙏 Acknowledgments

- [Model Context Protocol](https://modelcontextprotocol.io) — the transport that makes this possible.
- [dxcam](https://github.com/ra1nty/DXcam) — clean DXGI Desktop Duplication bindings.
- [pywinauto](https://pywinauto.readthedocs.io) — UI Automation access.
- [FastMCP](https://github.com/modelcontextprotocol/python-sdk) — the MCP Python SDK.