screencast-desktop

A Claude Code plugin that lets an agent operate a Windows application and turn that session into a finished demo video — where the camera starts zooming toward each click half a second before the click happens.

• Drive the desktop with vision in the loop. Every action returns a fresh screenshot in the same reply, so the agent works look → act → look instead of writing a script blind. Controls are found by name through Windows UI Automation; Electron apps that expose no control tree fall back to screenshots and coordinates (or to their DOM over CDP, when a debugging port is open).

• Record a window, not a screen. Capture goes through Windows Graphics Capture, bound to a window handle, so the desktop behind the app never enters the frame and it does not matter which monitor or GPU the window lives on.

• The video is built from an event log, not from the footage. Every click, every typed string and the full cursor path are timestamped as they happen. Camera moves are computed from that log.

• Screen Studio-grade finish, generated: eased camera pushes and pans, a drawn cursor with a soft shadow, click ripples, motion blur, rounded corners on a gradient backdrop, vignette and grain — plus dead-air removal that collapses the agent's own thinking time.

• Narration that lands on the beat. Optional ElevenLabs voice-over, generated before the run so the click falls inside the sentence describing it.

• A fuller montage layer, one script away. Title and outro cards, captions, and a complete sound design pass — click track, camera whooshes, landing impacts, closing riser, ducked music bed, loudness normalisation — live in cinematic.py, sfx.py and sfx_bank.py. They are driven by server/make_showcase.py rather than by the desktop_render tool; see Known limitations.

• Re-render for free. Rendering never touches the application: change the zoom cap, the narration or the effects and rebuild the same take as often as you like.

Why it's different

Every screen recorder that does "auto zoom" — Screen Studio and its Windows imitators — works the same way: it records first, then goes back through mouse hooks or the footage itself to guess where the interesting moments were. That ordering has a hard consequence nobody can engineer around: the zoom cannot begin before the click, because at the moment of the click the recorder has only just learned that a click is coming. The best it can do is start moving on the click and arrive shortly after. Human editors do the opposite — they lead the viewer in, so the eye is already at the button when it is pressed.

Here the agent generates the actions, so the coordinates and the timings are known before a single frame is composited. The camera can be given a lead-in (LEAD_IN = 0.55 s in server/camera.py), so by the time the button is pressed the shot has already arrived and settled. The same foreknowledge buys three more things a post-hoc tool cannot have:

• No pumping. Consecutive clicks in the same region are merged into one steady shot instead of the camera zooming in and out on every list item.

• Narration that fits. The speech is generated before the run (voice.plan()), its real duration is measured with ffprobe, and the agent's pauses are set from those numbers — so the click lands inside the sentence that describes it, with no manual nudging. Recording first and narrating afterwards always ends with the voice saying "I click Save" a second after Save was already clicked.

• A machine-readable record of what happened. desktop_describe_take reads a take back as a numbered procedure ("3. [12.4s] click Multiply by") — a better artifact than a pile of screenshots, and enough to write a reusable skill from.

Where this sits among the neighbours

The idea is not obscure — it is just hard to reach on Windows. Four browser-side projects implementing "edit from the action log" appeared within two weeks of March 2026 (argo, testreel, pagecast, playwright-recast), because Playwright hands you the log for free. On Windows the log has to be built alongside the input driver, and in the same five months nothing appeared: the recorders have no agent, and the agents produce no frames.

Related MCP server: windows-gui-mcp

Requirements

Python packages

Pulled from the imports of every module in server/:

pip install mcp pillow opencv-python numpy windows-capture uiautomation

No package is needed for the privacy scan: it reads the screen with the OCR that ships with Windows, driven through PowerShell.

Installation

1. Get the plugin

git clone https://github.com/JHamidun/screencast-desktop.git

2. Register it in Claude Code

The repository is its own marketplace (.claude-plugin/marketplace.json), so point Claude Code at the clone and install from there:

/plugin marketplace add <path-to-clone>
/plugin install screencast-desktop

.mcp.json registers both servers with ${CLAUDE_PLUGIN_ROOT}-relative paths, so nothing needs a global install and the clone can live anywhere.

3. Run setup

/screencast-desktop:setup

That runs server/doctor.py, which checks the things that break silently:

• DPI awareness — a process that has not declared itself DPI-aware is told the screen is 2560×1440 when it is really 3840×2160, and every click misses by the scaling factor.

• Monitor layout — coordinates are shared across all screens and go negative on secondary monitors.

• ffmpeg and available encoders.

• Window capture, for real — it captures ~25 frames and checks they are not all identical.

• Narration — whether a key is present and whether the configured voice id still exists on the account (a deleted voice otherwise fails with a bare 404).

It writes machine.json with what it found.

4. Fetch the UI Automation binary

The windows-ui server is an external binary — sbroenne/mcp-windows (MIT). It is deliberately not vendored into this repository: it is ~60 MB, it is someone else's project, and pinning a copy here would only ship a stale one. Download it on demand:

python server/fetch_ui_binary.py          # --force to re-download

The script resolves the latest GitHub release, verifies the archive against the SHA256SUMS.txt published with it, and refuses to install anything on a mismatch. It lands in bin/, which is where .mcp.json expects it.

5. Confirm both servers are up

claude mcp list      # expect: screencast, windows-ui

Quick start

Record a demo of the Windows Calculator. The /screencast-desktop:record command walks the agent through this, but here is what it actually does, with the real tool names.

1 — Stage the window. Put it on a secondary monitor if there is one, so it does not sit on top of your work:

desktop_monitors()
desktop_place_window(window="Calculator", monitor=1, fit=0.7)

Read the reply. Applications are not obliged to become the size they are told — a UWP window asked for 2380×1490 here came back 3967×2426 and hung off the screen. desktop_place_window measures the result, corrects it, and says outright whether the window fits.

2 — Check the frame for anything private.

desktop_screenshot(window="Calculator")
desktop_privacy_check(window="Calculator")

The check reports, it does not block: it OCRs the frame and flags card numbers, API keys, email addresses, phone numbers and personal names. Turn on Do Not Disturb before you record.

3 — Rehearse. Walk the route with the real tools and confirm from the returned screenshots that you are hitting what you think you are hitting. Nothing is being recorded yet:

ui_snapshot(windowHandle=…)          # windows-ui: controls by name — try this first
desktop_snapshot(window="Calculator")# or the built-in UIA walk, which returns e1, e2, … refs
desktop_click(ref="e7")

4 — Reset the app. A search box left open from the rehearsal ends up in the take.

5 — Record the real one.

desktop_record_start(window="Calculator")
desktop_click(ref="e12")                 # every click from here is logged for the camera
desktop_type("128")
desktop_click(ref="e19")
desktop_record_stop()

desktop_record_stop reports the duration, the frame count and how many clicks made it into the log, then tells you the out_dir to render.

6 — Render, then look at it.

desktop_render(out_dir="%USERPROFILE%/screencasts/take-143502", max_zoom=2.0)
desktop_render_status(out_dir="…")       # rendering runs in a child process

Open the file and check with your eyes: did the camera arrive where it should, are there black bars at any edge, is the cursor visible. A wrong coordinate produces a technically valid file in which the camera looks at nothing. If it is off, re-run desktop_render with different settings — the application is not launched again and the screen is not re-recorded.

Tool reference

screencast server — desktop_monitors, desktop_windows, desktop_screenshot, desktop_snapshot, desktop_dom, desktop_dom_launch, desktop_click, desktop_type, desktop_key, desktop_move_mouse, desktop_scroll, desktop_focus, desktop_launch, desktop_place_window, desktop_privacy_check, desktop_record_start, desktop_record_stop, desktop_render, desktop_render_status, desktop_describe_take.

desktop_click, desktop_type, desktop_key, desktop_scroll, desktop_focus and desktop_launch all take see="shot" (default) or see="none" — the second saves context when you already know what the screen looks like.

windows-ui server (exposed subset) — ui_snapshot, ui_find, ui_click, ui_type, ui_select, ui_read, ui_wait, window_management, app.

How it works

The same thing in text, for anyone reading this in a terminal:

  AGENT                                                   server/
  ─────                                                   ───────
  desktop_click / desktop_type / …                        desktop_server.py
        │                                                 (MCP, FastMCP)
        ├──► real SendInput: cursor eased to the target,  driver.py
        │    clicked, keys sent                           ── moves + clicks
        │                                                    the real desktop
        │
        ├──► EVENT LOG  t, kind, x, y, label, dur         driver.py → events.json
        │    + the sampled cursor path (track)               ◄── the ground truth
        │
        └──► screenshot back to the agent in the same reply

  desktop_record_start                                    recorder_proc.py (child process)
        └──► Windows Graphics Capture, bound to the HWND  wgc.py
             ├─ frames arrive only when the picture       ── writer thread re-sends
             │  changes …                                    the last frame on a
             └─ … so a writer thread feeds ffmpeg at a       fixed clock
                constant rate, logging the true
                wall-clock time of every frame
                                                          → raw.mp4 + frame_times.json

  desktop_render                                          render_proc.py (child process)
        │
        ├─ 1. TIMELINE   collapse the dead air            timeline.py
        │      keep 1.1 s before and 1.5 s after every
        │      action, squeeze the gaps to 0.55 s
        │
        ├─ 2. CAMERA     event log → keyframes            camera.py
        │      lead-in 0.55 s BEFORE each click,
        │      nearby clicks merged into one shot,
        │      pan instead of pumping in and out
        │
        ├─ 3. COMPOSITOR one affine matrix per frame      composer.py
        │      recording on a gradient backdrop, rounded     + cinematic.py
        │      corners, drop shadow, drawn cursor, click     (vignette, grain;
        │      ripples, motion blur, breathing idle,          title cards and
        │      vignette, grain                                captions available)
        │                                                 → silent.mp4
        │
        └─ 4. SOUND      optional narration               voice.py
               ElevenLabs TTS mixed onto the cut          → demo.mp4

  make_showcase.py — the fuller montage, run as a script rather than a tool:
        the same four stages plus title/outro cards, captions, and the whole
        sound design pass (clicks, whooshes, impacts, riser, ducked music bed,
        loudness normalisation)                           sfx.py + sfx_bank.py

Two design decisions explain most of the file layout:

Capture and render run in child processes. Importing the capture library or OpenCV inside the MCP server process wedges it, and a render takes minutes, which no tool call should hold open. recorder_proc.py and render_proc.py exist for that reason alone. They communicate through files (started.json, stop, render.log), and both are started with stdin=DEVNULL — a child that inherits the server's stdin starts eating the JSON-RPC requests meant for the server.

Frames are matched by timestamp, not by index. If the machine falls behind, source frame N is not at N/fps. frame_times.json carries the real capture time of every frame, and the compositor looks frames up through it — which is what keeps the camera on the clicks when the machine stutters.

Configuration

Render

desktop_render(out_dir, name="demo.mp4", max_zoom=2.0, narration="") — narration takes a JSON list of {"text": …, "at": seconds}.

Everything else is a module constant, edited in place:

desktop_record_start(window, out_dir="", fps=30) defaults to ~/screencasts/take-HHMMSS.

If max_zoom is not given, camera.build() picks a cap itself so at least 70 % of the window's height stays in frame — a tall narrow window fitted into a 16:9 frame is already small, and forcing 2× there crops away the part that gives the action its meaning. (On the Calculator it cut off the display showing the result.)

Narration

Set ELEVENLABS_API_KEY in the environment or in a .env file in the plugin root (KEY=value, one per line — the file is git-ignored). Optionally pin a voice with ELEVENLABS_VOICE_ID; with none set, the first voice on the account is used. Model: eleven_multilingual_v2. Point SCREENCAST_ENV_FILE elsewhere if you keep your keys somewhere else.

voice.resolve_voice() checks configured ids against the account's actual voice roster before using one, because a deleted voice otherwise fails with an unexplained 404.

Without a key nothing breaks. doctor.py reports it as a warning, not a failure, and desktop_render produces a silent video — which is also what it produces with a key when no narration argument is passed.

The sound design layer degrades rather than dies without a key: sfx_bank.build() needs ElevenLabs to generate the palette, but sfx_bank.build_synthetic() synthesises the same families offline with numpy (the *_syn1.wav / *_syn2.wav files), and sfx.click_samples() falls back to synth_click() when no sample assets are found. So an offline machine still gets clicks, whooshes and impacts — it only loses the voice.

Known limitations

Honest list. These are real, currently true, and mostly things that were hit during development. What is planned about them, in priority order with the measurements behind each item, is in ROADMAP.md.

• The desktop_render tool renders less than the codebase can. It calls composer.compose() without intro, outro or captions, and it mixes narration only — no click track, no whooshes, no music bed. Everything else is implemented and working, but currently reachable only through server/make_showcase.py, which is a script with its own hard-coded take path and beat list. Wiring those parameters through the tool is the most obvious open task in this repository.

• No drag. There is no drag or drag-and-drop tool. Mouse-down and mouse-up are always emitted at the same position, so anything requiring a press-move-release gesture — sliders, reordering, canvas drawing, resizing by grip — is out of reach.

• Key presses are not written to the event log. desktop_key sends the keystroke but does not log it, so the camera never reacts to keyboard-only steps and they do not appear in desktop_describe_take. Clicks and typed text (desktop_type) are logged; individual key presses are not.

• UWP windows must be addressed by their frame window. Windows Graphics Capture takes a top-level window handle. For a UWP/Store app that is the visible ApplicationFrameWindow — the inner CoreWindow is not a usable capture target. In practice: resolve the app by its visible window title (which is what the tools do) and do not try to reach past it.

• desktop_screenshot is a crop of the screen, not a window capture. It grabs the screen region the window occupies. Anything overlapping the window — another window, a notification toast, a tooltip — appears in the screenshot. (The recording does not have this problem: WGC captures the window itself.) Make sure the target window is on top before you trust a screenshot.

• Electron apps expose almost nothing to UI Automation. Measured on Windows 11: a typical Electron app returns 2 to 6 named elements — window wrappers, not interfaces. Fall back to screenshots and coordinates, or use desktop_dom when a debugging port is available. desktop_dom_launch opens a second copy of the app with a separate profile, which is not signed in.

• NVENC is effectively required. Both wgc.WindowRecorder and composer.compose() default to h264_nvenc. libx264 paths are implemented and doctor.py detects the right encoder into machine.json, but nothing wires that choice through automatically yet. On a machine without NVENC you have to pass the encoder yourself.

• Large files. Film grain is applied per frame, which defeats inter-frame compression — a short demo is heavier than the same footage without grain. Set grain_amount=0 if size matters more than the look.

• A take is capped at 15 minutes. recorder_proc.py stops itself, so a forgotten recording cannot run forever.

• Windows only, and only the interactive desktop. SendInput, UI Automation, WGC and the WinRT OCR are all Windows APIs; none of this works over an unattended session, in a service, or on a locked screen.

• The window is a live application. State survives between takes — a search box left open from a rehearsal turned a typed word into "githubgithub". Reset the app before the clean take.

• The privacy check reports, it does not block. Matching is literal; a name in a menu is not a leak, and OCR misses things. Look at the frame yourself before publishing.

Credits and licenses

• sbroenne/mcp-windows (MIT) — the windows-ui MCP server that does UI Automation. Not vendored here; downloaded on demand by server/fetch_ui_binary.py, checksum-verified against the release's own SHA256SUMS.txt.

• ffmpeg — capture encoding, the audio mix and ffprobe duration measurement. Called as an external binary; not bundled. Licensing depends on the build you install (LGPL or GPL).

• Sound bank. The .wav files under server/sfx_bank/ are generated, not sampled: server/sfx_bank.py builds them once from prompt recipes through the ElevenLabs sound-generation API and caches them, and every candidate is screened by measurement (a "click" whose peak is 200 ms in is rejected however good it sounds). A second family of files — *_syn1.wav, *_syn2.wav — is synthesised entirely offline in sfx_bank.build_synthetic() with numpy, so a machine with no API key still gets sound design. No third-party sample library is redistributed here. Point SCREENCAST_SFX_DIR at your own sample folder and sfx.py will prefer those clicks and music beds over the shipped bank; with nothing set it uses server/sfx_bank/, and with nothing found it synthesises. No hard dependency either way.

• ElevenLabs — optional, for narration and for building the sound bank. Bring your own key; the plugin ships no audio generated from anyone's voice.

• Fonts — titles and captions use Segoe UI, which ships with Windows, with Arial as a fallback. No font files are redistributed.

• Model Context Protocol Python SDK (MIT) — the server framework.

The plugin's own code is released under the MIT License. See LICENSE.

Contributing

Issues and pull requests are welcome. A few things that make review quick:

• Windows only. Test on a real desktop; there is no CI that can click buttons for you.

• Run the doctor first (python server/doctor.py) and paste its output into a bug report — most problems here are environmental (DPI scaling, monitor layout, missing encoder) and the doctor names them directly.

• If it broke silently, say so in a comment. This codebase is full of notes explaining why a line is the way it is, because nearly every one of them is a failure that looked like success: identical frames that pass for a recording, a camera drifting off the edge, an empty event log producing a video with no zoom at all. Keeping those notes is deliberate.

• Changes to camera or timeline constants need a before/after clip. They are judgement calls about how the result looks, and no test can settle them.