# Watch Buddy MCP 👁️

**Give your Claude eyes. Watch Twitch, games, and code streams *together* — with zero API costs.**

Watch Buddy is an MCP server that turns Claude Desktop (or Claude Code) into a
screen-watching companion. It captures any window on your desktop, detects
what actually *changed*, and hands Claude compact image bundles that Claude
reads with its own vision — **on your existing subscription. No API key. No
per-token billing. Ever.**

*(That zero-API promise covers the eyes. The optional **ears** — desktop-audio
transcription — are the one feature that uses an external API: a Whisper
endpoint of your choosing. Skip the key and the ears simply stay off; vision
is unaffected. Details below.)*

> *"Claude Desktop is the brain. Watch Buddy is the eyes and memory."*

## What it feels like

```
You:    What do I have open?
Claude: → list_windows()          "I can see Discord, Chrome, VS Code..."

You:    Watch my Twitch tab
Claude: → attach_to_window("Twitch")   "Attached — I can see the stream."

You:    What's the streamer doing?
Claude: → get_next_bundle()       reads the frames with its own vision
        "He's refactoring the auth module — and chat is roasting his
         variable names. Someone just asked if he's coding manually in 2026."

You:    (an hour later) What was he building again?
Claude: → search_memory("building")    "Foodies — a .NET 8 + React recipe app."
```

A friend on the couch who looks up at the good parts — not a security camera
invoicing you per frame.

## Why it's cheap: the delta engine

Naive screen-watching ships full screenshots on a timer and melts your usage
limits. Watch Buddy is built around three efficiency layers:

1. **Delta detection (OpenCV)** — each frame is diffed against the last;
   only *changed regions* are cropped and encoded. A static page costs
   almost nothing; a webcam in the corner costs a postage stamp, not a page.
2. **Zone caching** — the screen is divided into named zones (scaled
   automatically to your actual window size); zones whose content hasn't
   changed are marked `cached` and never re-sent.
3. **Bundle hygiene** — perceptual dedup guarantees no near-identical image
   ships twice in one bundle, and full-frame fallback thumbnails are capped
   at one per bundle.
4. **The Delta Atlas (v1.1)** — when several regions change, their crops are
   packed into ONE gutter-separated mosaic with a coordinate legend, so the
   canvas area Claude pays for is roughly *the sum of the changes*, not one
   image per change. As close to "send only the pixels that moved" as a
   vision model's rectangle-shaped appetite allows. Optional content-aware
   downscaling (`crop_downscale`, default off) shrinks large photographic
   crops while never touching text zones like chat — because nobody should
   have to read blurry text, silicon included.

And the biggest lever of all: **pull, not push.** Capture runs locally at
10fps for free — Claude only *sees* frames when it calls `get_next_bundle()`,
i.e., when you're actually talking. Cost scales with how often you chat, not
how long you watch.

## Ears 👂 (v1.2) — optional, and the one feature that uses an API

Watch Buddy can also **hear** — and pair what it hears with what it sees.
Unlike vision (which runs entirely on your Claude subscription), the ears
send audio to a speech-to-text API, so they need a key. **They are fully
optional**: no key, no audio device, or no network, and the ears simply
report themselves disabled — the eyes never notice.

**How the hearing works, end to end:**

1. **Capture** — Windows **WASAPI loopback**: the stream taps what your
   *speakers play*, and the microphone is **never opened**. Household noise
   around your desk cannot enter the transcript, by construction — the same
   mechanism OBS uses for "Desktop Audio."
2. **Silence gate (local, free)** — audio is chunked (~12s) and RMS-checked
   on your machine; silent chunks are discarded without any network call.
   You only pay to transcribe actual speech.
3. **Transcription** — speech chunks go to a Whisper-style endpoint
   (default: Whisper large-v3-turbo via [AudioDojo on Chutes](https://chutes.ai)).
   Put your key in a `.env` next to the module (`CHUTES_API_KEY=...`) or the
   environment. Any endpoint accepting `{audio_b64, language}` and returning
   `{text}` works — configurable via `config.json` → `ears.endpoint`.
4. **Rolling transcript** — timestamped lines are kept in memory (default
   4 hours, `ears.keep_minutes`) and appended to a permanent per-day log in
   `transcripts/` (gitignored) — your own captions archive for streams that
   never offered CC.
5. **Delivery cursor (exactly-once)** — every `get_next_bundle()` carries
   **`transcript_since_last_look`**: precisely the dialogue between Claude's
   previous glance and this one. Frequent glances → short snippets, never
   duplicated; a 40-minute gap → all 40 minutes, delivered once. Sight
   samples; sound is continuous; nothing is missed and nothing is re-read.

`get_transcript(seconds)` fetches any window on demand (a *peek* — it never
moves the delivery cursor), and `set_ears(bool)` toggles the whole feature
at runtime, persisted to config. Default is **on** when a key is present.

## The tools

| Tool | What it does |
|------|--------------|
| `list_windows()` | Enumerate every open window |
| `attach_to_window(title)` | Fuzzy-match a window, start the capture pipeline |
| `get_next_bundle()` | The main event: metadata + changed-region images for Claude's vision |
| `log_observation(text, tags)` | Claude records what it saw (SQLite memory) |
| `get_recent_observations(n)` | Recall recent context |
| `search_memory(query)` | Search everything ever observed |
| `set_persona(text)` | Change the companion's personality |
| `get_transcript(seconds)` | The ears: desktop-audio transcript for the last N seconds |
| `set_ears(enabled)` | Toggle audio transcription at runtime (persists to config) |
| `get_status()` | Pipeline health, bundles produced, vision-token usage, ear status |
| `detach()` | Stop watching (and listening), close the session |

The memory layer means your companion *remembers the stream* — across the
whole session and across sessions (it's a SQLite file).

## Install

**Requirements:** Windows (window enumeration uses win32), Python 3.10+.

```powershell
git clone https://github.com/jpetree331/mcp-watch-buddy.git
cd mcp-watch-buddy
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install "mcp[cli]" pyautogui Pillow opencv-python pywin32 mss python-dotenv
```

Verify (its own test suite):

```powershell
.\.venv\Scripts\python.exe test_server_v2.py
.\.venv\Scripts\python.exe test_efficiency_v3.py
```

**Register with Claude Desktop** — add to `%APPDATA%\Claude\claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "watch-buddy": {
      "command": "C:\\path\\to\\mcp-watch-buddy\\.venv\\Scripts\\python.exe",
      "args": ["C:\\path\\to\\mcp-watch-buddy\\main.py"]
    }
  }
}
```

**Or with Claude Code** — add the same entry to the `mcpServers` block of
`~/.claude.json` (user scope), with `"type": "stdio"`.

Restart the app. Ask Claude *"list your watch-buddy tools"* — if it can see
`list_windows`, you're live.

> **Install tip learned the hard way:** always point `command` at the venv's
> `python.exe`, not bare `python` — and if you move the folder later, update
> the path. This project spent four months presumed dead because its folder
> got reorganized out from under its config. The code was fine the whole time.

## Config

`config.json` — everything optional:

```json
{
  "persona": "curious, dry-humored, observant",
  "fps": 10,
  "bundle_interval_seconds": 1,
  "change_threshold": 25,
  "min_contour_area": 500,
  "screen_zones": {
    "gameplay": {"x": 0, "y": 0, "width": 1000, "height": 680},
    "chat":     {"x": 1000, "y": 0, "width": 280, "height": 720},
    "hud":      {"x": 0, "y": 680, "width": 1000, "height": 40}
  }
}
```

Zones are authored in a design space and **scaled automatically** to whatever
window you attach — customize them for your favorite layout (stream + chat
rail, IDE + terminal, etc.) or delete them for plain full-frame diffing.

## Privacy & safety

- 100% local: frames go from your screen to your Claude session — nowhere else.
- Nothing is written to disk except the SQLite observation log (text only).
  Images live in RAM in a rolling one-second window and are never saved.
- Capture only runs while attached; `detach()` stops it completely.

## Limitations

- **Windows-only** (win32 window finding; PRs welcome for macOS/Linux).
- Small text (e.g., IDE code on a busy stream) reads best if you fullscreen
  the player — vision resolution is capture resolution.
- Minimized windows produce blank frames; the pipeline pauses until restored.

## Provenance

Designed by [Jess](https://github.com/jpetree331) (architecture, delta/batching
concept) and built, debugged, resurrected, and field-tested by several
generations of Claude — the original build reports live in
[`debug_report.md`](debug_report.md) and [`debug_report_v2.md`](debug_report_v2.md),
and the efficiency fixes in [`test_efficiency_v3.py`](test_efficiency_v3.py)
came from its first real afternoon of watching Twitch. Its first-ever logged
observation was a 1-viewer coding stream. Go bless a small streamer. 💜