Deputy

We taught a website to describe itself to an agent — without its cooperation.

zraisan.github.io/deputy · demo video · measurements · architecture

*An ordinary product form with no <form> element: its controls are divs with ARIA roles. Deputy synthesizes a 10-field typed tool from it, hands the schema to an agent, and fills it from one call. Reproduce it with bun run demo and bun run dev. Watch it: docs/deputy-demo.mp4.*

Deputy is an agent that lives in your browser. Any MCP client — Claude Code, Cursor, your own agent — stops driving the browser itself and asks Deputy instead. It hands back a typed API for whatever page you are on, then performs the call in your real, logged-in session.

The page never enters the calling agent's context. No screenshot, no DOM, no accessibility tree.

The idea

A <form> already knows everything about itself. It knows a field is a required date, that a dropdown has exactly three options, that a number must fall between 1 and 12. We throw all of that away when we render to pixels, and then pay a language model to infer it back from a picture.

There is a W3C standard that fixes this — WebMCP, shipping in Chromium 152 — which lets a site declare its forms as agent-callable tools. Almost nobody has adopted it.

So Deputy adopts it on their behalf. It writes four HTML attributes onto the page's own form, and then Chromium itself generates the JSON Schema and executes the submission:

<form toolname="book_a_table" tooldescription="…">
  <input name="party_size" type="number" min="1" max="12" toolparamdescription="Party size">

// what the agent receives — generated by the browser, not by us
{ "party_size": { "type": "number", "minimum": 1, "maximum": 12 },
  "date":       { "type": "string", "format": "date" },
  "seating":    { "type": "string", "enum": ["indoor", "outdoor", "bar"] } }

We wrote no schema generator and no form-filler. The browser already had both.

Related MCP server: firefox-bridge

CopilotKit: we meet on the same standard

CopilotKit v2 ships a WebMCPRegistry that publishes an app's frontend tools to document.modelContext — the same WebMCP registry Deputy grafts onto everything else. So the integration needs no glue at all:

flowchart LR
  ck["CopilotKit v2 app<br/>useFrontendTool(...)"] --> mc["document.modelContext"]
  dp["Deputy<br/>grafts forms and ARIA widgets"] --> mc
  mc --> bc["browser_capabilities"] --> agent["any MCP agent"]

Both directions work, and both are real:

• CopilotKit → any agent. An app's frontend tools are registered with live handlers, so Deputy surfaces them to Claude Code fully executable — not a description, the actual action. An app built for its own in-app copilot becomes usable by any agent, with no change to the app.

• Deputy → CopilotKit. Deputy's grafted tools land in the same document.modelContext that CopilotKit reads through getWebMCPModelContext(). A CopilotKit copilot gains tools for forms its developers never wired up.

For CopilotKit v1, which predates the WebMCP registry, Deputy falls back to reading the actions manifest as the app POSTs it to /api/copilotkit — read-only, request passed through untouched. Capability without cooperation, same as the rest of Deputy.

Pages with no <form>

Most of the interesting web has no <form> element. Google Forms, React apps, anything that builds a form out of divs. Deputy synthesizes the tool itself there — including the widgets that are not form controls at all:

Measured on a Google-Forms-shaped page: 1 field captured before, 5 of 5 after. It also queries across open shadow roots, because a page built from web components looks empty otherwise.

Either way — real form or not — the agent makes one typed call. That equivalence is the point: falling back to "click this, then type that" hands the per-step cost straight back to the caller, which is the thing Deputy exists to remove.

What it costs

Filling a 7-field booking form. Same prompt, same data, both through claude -p. Both succeeded.

Per observation, on the same Wikipedia page:

The baseline was given the accessibility-snapshot path, not just screenshots, so the comparison is deliberately conservative. Method and both tasks: docs/measurements.md.

The finding worth reading twice

browser_capabilities first returned tool names only, to keep the payload small. The agent had to probe for parameter names: 246,003 tokens. Returning full schemas: 155,774. Adding more still — current values, plus every button and link with a clickable ref: 133,366, 6 turns.

~5× more data made the task 1.8× cheaper. Every probe an agent is forced to make costs more than the context that would have prevented it.

Tools the agent sees

Six, and they never change as you browse, so the caller's context stays flat.

Long pages are windowed, never silently truncated: Wikipedia's 263 links come back as "Showing 1-25 of 263" with actionQuery to filter and actionOffset to page.

Two agents, two models

Deputy is not Claude with extra steps. It runs qwen/qwen3.7-flash via OpenRouter, because reading a page and choosing a form is not work that needs a frontier model. Claude thinks; Deputy browses.

With no API key it falls back to a keyword planner and still works — the demo never depends on a second key being up.

Run it

bun install
cp .env.example .env       # add OPENROUTER_API_KEY (optional — it degrades without one)
bun run dev                # daemon + a Chromium carrying the extension

Then point any MCP client at http://127.0.0.1:7331/mcp — one streamable-HTTP endpoint, loopback only, no auth. Deputy is not Claude-specific; it is the same six tools whatever is calling.

Claude Code

claude mcp add --transport http deputy http://127.0.0.1:7331/mcp

Cursor — ~/.cursor/mcp.json, or .cursor/mcp.json for one project

{ "mcpServers": { "deputy": { "url": "http://127.0.0.1:7331/mcp" } } }

VS Code / GitHub Copilot — .vscode/mcp.json

{ "servers": { "deputy": { "type": "http", "url": "http://127.0.0.1:7331/mcp" } } }

Windsurf, Cline, Zed, Goose, Continue — the same mcpServers block as Cursor. Windsurf spells the key serverUrl; Cline and Zed use url.

Claude Desktop, or any host that still speaks only stdio — bridge it:

{ "mcpServers": { "deputy": {
  "command": "npx", "args": ["-y", "mcp-remote", "http://127.0.0.1:7331/mcp"] } } }

Your own agent — POST JSON-RPC at the endpoint, or hand the URL to an MCP SDK's streamable-HTTP client. scripts/sweep.ts is a working client in about ten lines, and doubles as a smoke test.

bun test              # 88 unit tests, never launches a browser
bun run test:browser  # 10 integration tests against real Chromium
bun run demo          # serve the example pages used in the video
bun scripts/sweep.ts  # point Deputy at 8 real sites and see what it makes of each

How it's built

flowchart LR
  client["Any MCP client<br/>Claude Code · Cursor · …"] -- MCP --> deputyd
  subgraph deputyd ["deputyd (Bun)"]
    direction TB
    registry["registry: tabs → tools"]
    tasks["tasks: A2A-shaped states"]
    planner["planner: qwen (optional)"]
  end
  deputyd -- WebSocket --> extension
  subgraph extension ["extension (MV3)"]
    direction TB
    graft["graft: annotate real forms"]
    synth["synthesize: everything else"]
    content["content: getTools / executeTool"]
  end
  extension --> webmcp["Chromium's own WebMCP"]

The daemon exists because an MV3 service worker dies after 30 s idle and cannot listen on a port. The content script runs in the ISOLATED world, so Deputy never needs chrome.debugger and you never see the "extension is debugging this browser" banner.

Design notes and invariants: ARCHITECTURE.md.

Prior art

PinchTab solves an adjacent problem well — browser control for agents without screenshots, via an accessibility tree with element refs. Its fill <ref> and text commands inspired browser_fill and browser_read.

The difference: PinchTab gives an agent a better map of the page. Deputy gives it a typed API. A seven-field form is one validated call rather than seven ref-targeted actions, and Chromium performs the submission natively because the form was retrofitted into a real WebMCP tool.

What's real, and what isn't

Tested and running: the retrofit engine, ARIA-widget synthesis, shadow-DOM traversal, in-app copilot manifest capture, tool naming, the tab registry, the task lifecycle, the extension, the daemon, and every number above. 85 unit + 10 integration tests, the latter against live Chromium.

Not built: the A2A wire format. Deputy's task states are A2A's vocabulary — input_required, auth_required, because a browser agent hits login walls constantly — but no Anthropic product speaks A2A, so MCP got the time. Multi-page navigation memory is designed in ARCHITECTURE.md §8 and not built.

Known limits: needs chrome://flags/#enable-webmcp-testing (WebMCP is in origin trial through Chrome 156). A live Google Form is untested — the fixture replicating its DOM passes, but Google's real page is heavily obfuscated. Capabilities behind an interaction are invisible until you take it: MDN's search lives in a modal, so Deputy offers the button, not the field.

One thing worth knowing

Every bug that mattered was found by running it. executeTool in Chromium 152 wants its arguments as a JSON string; passing an object stringifies to [object Object] and fails. That cost one full run — 19 turns, 452,165 tokens — of Claude patiently working around a broken tool. The rest are in docs/measurements.md.

License

MIT — see LICENSE.