VoltageInputMcp

Ein MCP-Server, der es einem Frontier-Modell erlaubt, einen Computer mit Eingabegeschwindigkeit statt Werkzeug-Aufrufgeschwindigkeit zu steuern.

Das Problem

Computer-Use-Werkzeuge machen für jede Aktion einen Roundtrip zu einem Remote-Modell: Screenshot hoch, Entscheidung runter, ein Klick spinunkt: for filling in: This is fine for filling out a form and useless for anything that requires a sequence of inputs delivered quickly — playing a game, operating a modal dialog, driving a timeline, any UI where the third input waits for the first two to already have landed. The bottleneck is not the model's intelligence. It's that this intelligence is 800 ms away while inputs need to be 8 ms apart.

The shape of the answer

Separate deciding from doing — do the deciding in the orchestrator, do the doing on the same machine where the keyboard is.

  ┌─────────────────────────────────────────────────────────────────┐
  │  Layer 1  —  the orchestrator (Claude, or any MCP client)       │
  │  Writes a Playbook: states, what to look for, what is allowed,  │
  │  when to move on. Thinks once, up front. Watches and corrects.  │
  └───────────────────────────┬─────────────────────────────────────┘
                              │  MCP
  ┌───────────────────────────▼─────────────────────────────────────┐
  │  Layer 2  —  two small local models, on your GPU                │
  │                                                                 │
  │   vision (Qwen2.5-VL-3B)     "of these specific things,         │
  │                               which are on screen, and where?"  │
  │   actuator (Qwen3-1.7B)      "given that, which inputs?"        │
  │                                                                 │
  │  Neither plans. Both answer one closed question per cycle.      │
  └───────────────────────────┬─────────────────────────────────────┘
                              │
  ┌───────────────────────────▼─────────────────────────────────────┐
  │  safety governor  →  /dev/uinput  →  the actual desktop         │
  └─────────────────────────────────────────────────────────────────┘

The orchestrator is the brain. The small models are the arms. The arms exist solely to help the orchestrator gather information; they are classic "limited models," and that is a feature.

Where the speed actually comes from

Not from the small models being fast — a 3B VLM still costs ~0.3 seconds. The speed comes from four things, ordered by impact:

Bursts — The actuator does not send one input. It sends a burst: a timed sequence of inputs executed by a dedicated executor, with no model involved.

g:0;c:l;w:150;t:"README.md";k:enter;w:80;k:ctrl+s

That burst is one decision and 7 inputs spread over enf: it's milliseconds, scheduled to the millisecond. A 40-action burst still costs one decision. Input rate is set by the burst, not the model.

Reflexes — Rules that trigger in microseconds, fired by small screen probes (a single pixel, a region average), between decisions; no model at all.

{"id": "heal", "when": "probe('health') < 0.25", "do": "k:q;w:60", "cooldown_ms": 800}

Skipping perception — Most cycles end up looking at an unchanged screen. A 40 µs frame-diff decides whether to spend 300 ms on the vision model or keep using the last observation. On typical desktop loads it skips the VLM most cycles.

Prompt-cache locality — Prompts are arranged static-first, so llama.cpp reuses the KV cache, re-prefill only the changed tail.

Why the small models are reliable despite being small

Because we don't ask them to be reliable — we constrain them.

Under llama.cpp, both models have a GBNF grammar that is regenerated each cycle, based on current context. The grammar is not advisory. It masks the logits so only tokens that continue a valid parse are available. Concretely, the actuator cannot:

• emit a malformed burst

• name a key that the policy forbids — such key isn't present in the grammar

• reference an element that wasn't observed — the index space is the element list of this cycle

• propose a state transition that isn't in the App's Playbook

The vision model likewise cannot invent an UI-element name: its label vocabulary is the watch list you supply, plus a small general-purpose set. So a sees("address bar") guard talks about a closed set, not arbitrary words a 3B model might produce.

No retry loop, no defensive JSON parsing, because malformed output is not improbable — it is unrepresentable.

The Playbook

You don't give the small models a goal. You give them a state machine. Transitions are guard expressions evaluated by the choreography, not by the model.

{
  "name": "open_downloads",
  "goal": "Open the file manager at ~/Downloads. Delete nothing, confirm nothing.",
  "initial": "launch",
  "policy": {
    "dry_run": true,
    "allow_verbs": ["g", "c", "k", "t", "w"],
    "deny_labels": ["delete", "trash", "confirm", "empty trash"]
  },
  "budget": { "max_cycles": 60, "max_seconds": 90 },
  "states": {
    "launch": {
      "brief": "Open the application launcher and start the file manager.",
      "watch": ["application launcher", "search field", "file manager icon"],
      "on_enter": "k:meta;w:400",
      "transitions": [
        { "when": "sees('search field')", "to": "type_name" },
        { "when": "cycles() > 6", "to": "@failure", "note": "launcher never opened" }
      ]
    },
    "navigate": {
      "brief": "Focus the location bar with ctrl+l, type the path, press Enter.",
      "watch": ["location bar", "file list", "error message"],
      "on_enter": "k:ctrl+l;w:200",
      "transitions": [
        { "when": "text('Downloads')", "to": "@success" },
        { "when": "sees('error message')", "to": "@failure" }
      ]
    }
  },
  "success_when": "text('Downloads') and not flag('loading')"
}

voltage_reference returns the full DSL, JSON Schema, and guard function table so an orchestrator can write a Playbook without reading the repository.

Performance tuning

All numbers below are measured on the reference machine (RTX 3050 6 GB laptop, Qwen2.5-VL-3B + Qwen3-1.7B under llama.cpp), not derived.

Both models are decode-bound. Output-tokens are the only lever that matters.

That was surprising — the original design assumed vision would be prefill-bound, but it's instead with decode. Decode runs at ~22 ms/token. Prefill is cheap.

Three consequences, each changed a default:

• max_elements is the dominant vision cost. Default is 3. Raising to 6 adds ~2.0 s per perception cycle. Use the number of elements your guards actually need.

• Shrinking downscale_to doesn’t help, it usually hurts. 448×252 was measured 2–3 slower than 448×504 — a blurrier image makes the model less sure, so it yields more tokens. Keep the largest size that fits.

• The actuator’s note field cost 55% of its latency. It is purely diagnostic: at 48 characters it measured 412 ms/cycle, at 0 → 140 ms. Default is now 12.

Elements are encoded as [label_index, x1, y1, x2, y2] rather than {"l":"address bar","b":[...],"c":0.9} — measured therefore 27–29% fewer tokens and 32–41% lower latency. Indexing into the closed watch vocabulary is also safer: the model can’t write a label at all.

GBNF evaluation runs once on the CPU per sampled token, so actuator gets more CPU scan than the vision model (see above). **Giving allow_keys also turns a safety feature into a latency improvement.

Two settings that fail silently if wrong:

• GGML_CUDA_FA_ALL_QUANTS=ON at build time. We serve q8_0 KV-cache and flash attention. Without that llama.cpp doesn’t compile fast kernels for this KV-cache, and falls back to slow path — no error, just Y. scripts/build-llama.sh sets it.

• GGML_CUDA_ENABLE_UNIFIED_MEMORY=0 at runtime. If it’s 1, VRAM overflow silently spills over PCIe instead of failing. Everything still works but is ~10× slower. serve.sh pins it off.

Measure rather than guess:

.venv/bin/voltage bench

It drives both backends with the exact prompt shapes the loop uses and reports cold vs. prompt-cached latency, ms-per-visual-token at three input sizes, and the resulting cycle time. A prompt-cache speedup below ~1.5× can mean something dynamic leaked into the prompt prefix.

Comparing models

The obvious experiment — "which model writes better bursts" — measures the wrong thing. The grammar already guarantees every burst is valid: so a bigger model can’t win on syntax. What actually decides whether a configuration is usable:

1. Grounding accuracy. A model that’s 200 ms faster and 40 px off is useless — the mouse sound misses. The measured metric is center distance in screen pixels, not IoU, because clicks land on the center.

2. Decision quality under constraint. Given the same scene, does it choose the correct legal action, and does it chain a whole sequence into a single burst rather than emitting timid actions one at a time?

3. Latency, which only matters once 1 and 2 are good.

.venv/bin/voltage fixture desktop      # capture a real screen
.venv/bin/voltage compare              # score whatever is running now

Ground truth comes from real screenshots labeled by (the orchestrating) model — same reference the system uses at runtime. Synthesic UI is a trap: a drawn rectangle doesn’t read as a button to a real‑model trained on real UIs, so scoring for it tests the wrong skill.

Results accumulate across runs: serve A → compare → serve B → compare → read the table. voltage compare --list prints it without re-running.

Fixtures are yours and not committed. Add fixtures/ to .gitignore if screenshots are private.

Safety

The component generating inputs is a 1.7B model. The governor is a layer that isn’t advisory: every action goes through it, including reflex actions and your own code.

• dry_run is the default. A new Playbook parses, checks, and journals every burst while touching nothing.

• Whole-batch refusal. Half-executing a polemic move is worse than not doing it.

• deny_labels blocks clicks on anything named Delete / Confirm / Purchase / Allow, wherever it appears — that catches the dialog that pops up in an unexpected place.

• Region fences, key allowlists, forbidden chords (ctrl+alt+delete, alt f4), forbidden text patterns (rm -rf, sudo), burst-size and inputs-per-second caps.

• Four independent stops: voltage stop (writes a file — works over SSH), a deadman timer that fires in the background if the loop wedges, physical input contention — touch the real mouse and it halts, and Playbook budgets.

• Held keys are always released — aborting, crashing, timing out. When a run is interrupted between d:shift and u:shift, Shift must not stay down.

Install

cd voltage-input-mcp && ./scripts/setup.sh

That checks /dev/uinput access, installs system dependencies, creates the venv, and prints what’s missing. Then:

./scripts/fetch-models.sh lean && ./scripts/serve.sh lean

.venv/bin/voltage doctor

Launching from an MCP client

MCP clients start servers with orders a locked environment — PATH, HOME, take little else. That’s a sensible default but it breaks screen capture: reaching the compositor requires DBUS_SESSION_BUS_ADDRESS and WAYLAND_DISPLAY. Input injection still works without them (uinput is a device file, not a session service), so the failure looks partial: data is handled, screenshots are not.

Thus pass them explicitly:

claude mcp add voltage-input \
  -e WAYLAND_DISPLAY="$WAYLAND_DISPLAY" \
  -e DISPLAY="$DISPLAY" \
  -e DBUS_SESSION_BUS_ADDRESS="$DBUS_SESSION_BUS_ADDRESS" \
  -e XDG_RUNTIME_DIR="$XDG_RUNTIME_DIR" \
  -- /absolute/path/to/voltage-input-mcp/.venv/bin/voltage-input-mcp

voltage_doctor reports exactly which ones are missing, so that’s the first place to look if capture is broken.

Requirements

• Linux with /dev/uinput on X11, Wayland, or console — it injects below the display server

• Python 3.11+

• A GPU with ~5 GB free for the lean profile; voltage profiles shows what fits yours

• llama.cpp for the fast path, or Ollama for a slower zero-build path

Verified against KDE Plasma 6 mo on Wayland (KWin), CUDA, Python 3.14.

MCP tools

Support

• ARCHITECTURE.md — how the loop works, why each choice was made, where 2ns last

• PLAYBOOK.md — the guide to writing playbooks

Status

Built and tested as far as possible without model binaries on disk. 149 tests cover the burst DSL, the guard sandbox, the safety governor, playbook compilation, GBNF generation, the uinput wire encoding, and the run loop itself (run with stub models, including a check that on_change perception indeed skips the vision model on a static screen).

The MCP server was driven end‑to‑end over stdio by a real client: 13 tools, correct schemas, execute_burst accepted a valid burst and denied sudo rm -rf / (matches both policy rules).

Was nicht ausgeführt wurde, ist ein Live‑Modell: Dafür müssen Gewichte heruntergeladen und llama.cpp gebaut werden – was scripts/ einrichtet. Zwei Dinge wurden beim Build außerdem bewusst nicht ausgelöst – der Portal-Berechtigungsdialog und jegliche echte Eingabe‑Injektion – da beide auf deinen Desktop einwirken.

Reihenfolge der Schritte von hier aus:

./scripts/setup.sh          # reports what needs sudo, doesn't run it
./scripts/build-llama.sh    # ~15 min with CUDA
./scripts/fetch-models.sh lean
./scripts/serve.sh lean
.venv/bin/voltage doctor    # should now say READY

Dann in einem MCP‑Client: voltage_calibrate (beobachte, wie sich der Cursor tatsächlich bewegt), voltage_observe (prüfe, ob das Vision‑Modell deine Labels findet), danach ein dry_run‑Playbook und lies voltage_journal, bevor du jemals dry_run=false setzt.

Autorenschaft

Durchgängig geschrieben von Claude Opus 5 (Anthropic) in einer einzigen Sitzung – Architektur, Implementierung, Tests und Dokumentation. Ein Mensch hat die Idee vorgegeben, die Rahmenbedingungen festgelegt (KDE Wayland, 6 GB VRAM, „schneller als Computer‑Use“) und das Ergebnis überprüft, aber den Code nicht geschrieben.

Die in diesem Repo enthaltene Erfahrungen mit der Plattform stammen aus dem Erkunden der Maschine während des Builds und nicht aus Annahmen: dass KWin ScreenShot2 ausführbaren Programmen verweigert, die nicht auf der Allowlist stehen; dass grim unter KWin nicht funktioniert; dass MCP‑Clients den Session‑Bus entfernen. Jede Erkenntnis ist an der Stelle im Code dokumentiert, an der sie eine Entscheidung erzwungen hat.

LICENSE nennt keine Einzelperson als Urheberrechtsinhaber, und die Begründung ist dort ausformuliert.

Lizenz

MIT. Siehe LICENSE.