dibs

A local hub for AI Agent Computer Use. Let your AI agents call dibs on using your desktop.

It lets multiple agents share one Windows desktop by exposing screenshot, mouse, keyboard, and window control over HTTP and MCP, so a Claude Code session, a scheduled automation task, and a browsing agent can all drive the same machine without fighting each other — or without shoving the user out of their own chair.

The mental model: agents register, one has dibs on "the desk" at a time (an exclusive lease on the mouse/keyboard), the human always wins, every action is logged, there's one big pause button, and there's an on-screen overlay so anyone glancing at the monitor can see which agent is doing what.

Install

uv venv --python 3.12
uv sync

That creates .venv and installs everything (FastAPI, uvicorn, mss, pyautogui, pywin32, pynput, the mcp SDK, httpx, uiautomation). No extra dependencies to add yourself.

Related MCP server: windows-computer-use-mcp

Run

uv run dibs serve

Prints the dashboard URL and the admin token, then serves REST (/v1/*), MCP (/mcp), and the dashboard (/) all on one port (7474 by default).

For it to start automatically at Windows logon:

.\scripts\install-task.ps1

To have it restarted automatically if it ever stops answering (the task only restarts on a crash it can see):

powershell -ExecutionPolicy Bypass -File scripts\install-watchdog.ps1

That registers dibs-watchdog, which hits the unauthenticated GET /healthz every 5 minutes and starts the dibs task when it fails. scripts\uninstall-task.ps1 removes both tasks.

On this machine Register-ScheduledTask needs an elevated shell (UAC-filtered admin token): open Windows Terminal as administrator and run the installer from there. The task itself runs unelevated as your interactive user. The installer stops any hand-started dibs serve first so the task can take port 7474.

This registers a Scheduled Task named dibs, not a Windows service — a service runs in session 0, which can't see the desktop or send input at all. The task runs in your interactive logon session with a hidden window instead. .\scripts\uninstall-task.ps1 removes it.

First run

uv run dibs serve prints an admin token the first time it runs (stored once in data/secrets.json). You'll need it for anything that isn't loopback-open. Grab it again anytime: uv run dibs token.

Dashboard

http://127.0.0.1:7474 — a single-page, no-build-step dashboard:

• Live screenshot (polls every second) with a screen picker for multi-monitor setups.

• Mode selector (Ask me / Hands-off / Locked) and a human-presence chip.

• A consent card that appears whenever an agent is waiting on you, with a countdown and Allow / Deny buttons.

• The desk card: who holds it, the wait queue, recent consent decisions, force-release and take-the-desk-back buttons.

• Agents table (revoke a token), stats, and an audit tail with screenshot thumbnails.

It asks for the admin token the first time it hits a 401 and remembers it in the browser. Looks fine on a phone.

Register an agent

uv run dibs register --name my-agent --purpose "testing things"

Prints the new agent's token once — store it. Equivalent over curl (works without a token from loopback if allow_local_open_registration is true, the default):

curl -s -X POST http://127.0.0.1:7474/v1/agents \
  -H "Content-Type: application/json" \
  -d '{"name":"my-agent","purpose":"testing things"}'

Use it — REST

Every action except wait needs dibs first, screenshots included: nobody looks at your screen without you saying yes.

TOKEN=<agent token>

# acquire the desk (long-polls up to wait_s if someone else holds it)
curl -s -X POST http://127.0.0.1:7474/v1/lease \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"ttl_s":60,"wait_s":10}'

# screenshot (needs dibs like everything else)
curl -s -o shot.png "http://127.0.0.1:7474/v1/screenshot.png?screen=0" \
  -H "Authorization: Bearer $TOKEN"

# click and type
curl -s -X POST http://127.0.0.1:7474/v1/actions \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"action":"left_click","coordinate":[400,300]}'
curl -s -X POST http://127.0.0.1:7474/v1/actions \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"action":"type","text":"hello"}'

# scroll_pages -- more reliable than `scroll` against pages/apps that smooth-scroll (Chrome
# can collapse a burst of wheel events into a much smaller move than requested); clicks
# coordinate first if given, then sends Page_Up/Page_Down `scroll_amount` times
curl -s -X POST http://127.0.0.1:7474/v1/actions \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"action":"scroll_pages","scroll_direction":"down","scroll_amount":2,"coordinate":[900,700]}'

# batch (stops at first failure)
curl -s -X POST http://127.0.0.1:7474/v1/actions/batch \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"actions":[{"action":"left_click","coordinate":[400,300]},{"action":"type","text":"hi"}],"auto_lease":true}'

# find + click a labeled element instead of guessing pixel coordinates
curl -s -X POST http://127.0.0.1:7474/v1/actions \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"action":"find","text":"Add to cart","near":"Blue Widget"}'
curl -s -X POST http://127.0.0.1:7474/v1/actions \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"action":"click_element","text":"Add to cart","near":"Blue Widget"}'

# screenshot_after: opt in on any single action or batch to get a screenshot back with the
# result, taken right after the action ran (or after the last successful action of a batch,
# even one that stopped early on error) -- same shape as the `screenshot` action's own image
curl -s -X POST http://127.0.0.1:7474/v1/actions \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"action":"key","text":"Return","screenshot_after":true}'
# -> {"ok":true,"result":"OK","screenshot":{"png_base64":"...","width":...,"height":...,"scale":...,"screen":0}}
curl -s -X POST http://127.0.0.1:7474/v1/actions/batch \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"actions":[{"action":"left_click","coordinate":[400,300]}],"auto_lease":true,"screenshot_after":true}'
# -> {"results":[...],"screenshot":{...}}

POST /v1/lease responds one of four ways depending on mode and who's around:

• granted (200) — you hold it.

• queued (202) — someone else holds it; wait_s ran out, keep polling.

• awaiting_consent (202) — mode is ask and a human is active; a consent prompt is up (overlay + dashboard + hotkeys), keep polling with the same request.

• denied (403) — locked mode, human said no, or the request timed out; check reason and retry_after_s.

Use it — MCP from Claude Code

claude mcp add --transport http dibs http://127.0.0.1:7474/mcp \
  --header "Authorization: Bearer <token>"

Exposes nine tools: computer (one tool, action param selects the behavior — mirrors Anthropic's computer_toolset_20260801, plus dibs extras), desk_status, acquire_desk, release_desk, list_windows, focus_window, ui_tree, find, click_element. computer always auto-leases, and a denial comes back as a tool error naming the holder, the consent countdown, or when it'll auto-resume — so the model can decide whether to wait or back off.

UI Automation: ui_tree / find / click_element

Reading a full-screen screenshot and guessing pixel coordinates is expensive and fragile — a scrolled page can move the target out from under a hard-coded click. These three actions read the real Windows UI Automation tree instead, so an agent can say "click the element named 'Add to cart' nearest to 'Blue Widget'" and get it right even after a scroll:

• ui_tree(hwnd?, title?, max_depth?=6, max_nodes?=400, roles?) — a compact indented text tree plus structured nodes ({id, role, name, value?, rect, rect_shot, depth, enabled, offscreen}). rect is absolute screen pixels; rect_shot is screenshot-space pixels (the same coordinate system screenshot/zoom/clicks use), so a returned rect can be clicked directly.

• find(text, role?, near?, hwnd?, title?, exact?) — case-insensitive substring match on name/value, optional role filter, near picks the match closest to a named landmark when several elements share a name. Returns the best match plus up to 5 alternates (center/center_shot included), or a not_found error listing nearby element names.

• click_element(text, role?, near?, hwnd?, title?, exact?, button?, double?) — find, then click the centre. Goes through the same click path as left_click, so motion, the overlay flash, and the audit log all work the same way.

Window resolution for all three: hwnd if given, else a case-insensitive substring of title, else the current foreground window.

Chromium/Electron apps (Chrome, VS Code, Slack, the Edge WebView...) only publish their accessibility tree once an assistive-tech client connects. uiautomation usually triggers that on the first query, but sometimes only a beat later — dibs retries a couple of times with a short pause before giving up. If a Chromium-based app still comes back nearly empty, launch it with --force-renderer-accessibility to force the tree on from startup.

Use it — Python client

clients/python/dibs_client.py is a small sync httpx wrapper, no dependency on the dibs package itself:

from dibs_client import DibsClient

client = DibsClient("http://127.0.0.1:7474", token="<agent token>")
client.acquire(ttl_s=60)
client.click(400, 300)
png, w, h, scale = client.screenshot()
client.release()

• examples/claude_agent.py — a real Claude computer-use loop (computer_toolset_20260801) that forwards tool calls straight to dibs. Only makes a live API call if you have Anthropic credentials; otherwise it just proves the wiring.

• examples/two_agents_demo.py — two agents contend for the desk against a running server, printing the FIFO queueing timeline (acquire, queued, long-poll, granted).

Living with a human

Three modes (dibs mode ask|hands_off|locked, or the dashboard selector):

In ask mode, if you're active at the keyboard an agent has to ask first: a prompt shows up on screen (bottom-right, doesn't steal focus), on the dashboard as a consent card, with a countdown. Answer with the buttons, or the hotkeys Ctrl+Alt+Shift+Y (allow) / Ctrl+Alt+Shift+N (deny). Saying yes grants a 5-minute consent window so the same agent doesn't have to ask again right away.

Takeover always wins: touch the mouse or keyboard while an agent holds the desk, and everything pauses immediately, the agent loses its dibs, and its next action gets a clear "a human took the desk" error. It resumes on its own after you've been idle for 20 seconds. Ctrl+Alt+Shift+R does the same thing explicitly (take the desk back right now, no agent input required). Ctrl+Alt+Shift+P is the separate manual pause/resume toggle — manual pauses never auto-resume, you have to un-pause them yourself.

What you see on screen

An always-on-top, click-through overlay makes agent activity visible to anyone looking at the monitor:

• Cyan halo (color configurable) around the cursor with the current agent's name, whenever an agent holds the desk. Hidden otherwise.

• Top banner: which agent, its purpose, countdown to when its dibs run out. Green the whole time you have the desk (after a takeover or Ctrl+Alt+Shift+R); red only for a manual pause.

• Quick flash ring on every click, small "typing…" tag while text goes in.

• The consent prompt described above, bottom-right.

Turn it off with overlay.enabled: false — the hub never depends on it (no display, or Tk failing to start, just logs a warning and carries on).

Safety

• Pause: dashboard button, Ctrl+Alt+Shift+P, or dibs pause — stops all input actions immediately (423 to callers). Only wait is exempt: nothing looks at or touches the screen while paused.) still work.

• Failsafe: fling the physical mouse to the top-left screen corner during an action (the pyautogui convention) and it aborts and pauses.

• allow_launch is off by default — agents can't start arbitrary processes unless you turn it on.

• Audit log: every action, successful or not, read-only or not, is written to data/dibs.db (SQLite). Screenshots and zooms are saved to data/shots/ (rolling, keep_screenshots deep, default 200) and linked from each audit row.

Exposing to other machines

By default dibs only listens on 127.0.0.1 — nothing outside this machine can reach it. To let other boxes on your Tailscale (e.g. a remote automation runner) drive it, set host: 0.0.0.0 in config.yaml (copy config.example.yaml to start). The moment you do that, loopback's free pass goes away: every route needs a bearer token, including registration and the dashboard.

Config reference

config.yaml in the working directory, or DIBS_CONFIG=<path>. Every key is also settable via env var DIBS_<KEY> (__ for nesting, e.g. DIBS_PRESENCE__IDLE_AFTER_S=10). See config.example.yaml for a starting copy.

Layout

Dev

uv run pytest                 # everything that doesn't need a real display
uv run pytest -m display      # exercises the real screen/mouse/keyboard/overlay on this box
uv run ruff check .           # lint python code
uv run ruff format .          # format python code
uv run mypy .                 # typecheck python code

npm install                   # install JS dependencies
npm run lint                  # lint dashboard JS
npm run format                # format dashboard files

E2E tests

tests/test_browser_e2e.py (marked display) drives a real, Playwright-launched Chromium window through the dibs hub in-process, against the real desktop -- proving that dibs coordinate math (screenshot-space -> absolute pixels, accounting for OS DPI scaling and dibs' own screenshot downscale) actually lands clicks/scrolls where a real screenshot-driven agent would compute them to go, not just that the internal call chain runs. It never touches the live :7474 server or the user's own windows -- only its own throwaway Chromium instance.

Setup (once): uv run playwright install chromium

Run: uv run pytest -q -m display tests/test_browser_e2e.py

Auto-skips if playwright or its Chromium browser isn't installed, or if not on Windows.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Ensure you run the test suite (uv run pytest) before submitting.