annotate-mcp

Draw on a live page, and your coding agent reads what you drew.

An MCP server that opens a real browser and injects a comment/pencil overlay into every page it loads — your app, a staging deploy, someone else's site. You mark things up like a PDF annotation layer; the agent gets back your note plus the DOM elements it lands on.

It is project-agnostic by construction. Nothing is installed into the app under review: no component, no route, no dev-server dependency, no framework assumption.

Why it can be agnostic

Two Playwright primitives:

• addInitScript injects src/overlay.js into every page and frame before any page script runs, for the life of the browser context. Navigation, reloads, and cross-origin hops all keep the overlay.

• exposeBinding gives that injected script a real function into Node, so marks round-trip to the server instead of being trapped in page memory.

State lives in Node, not the page. The overlay is a pure view: on load it asks the server for the marks, on every change it pushes them back. That inversion is what makes a mark left on /dashboard still exist after you navigate to /settings, reload, or close the tab — none of which localStorage would survive cleanly across origins.

Related MCP server: ui-annotator-mcp

Quick start

Clone it anywhere — it does not have to live inside the project you are reviewing:

git clone <this-repo> annotate-mcp
cd annotate-mcp
npm install          # builds automatically via `prepare`; pnpm and bun work too

Then point your MCP client at dist/index.js. For Claude Code, add it to the project you want to annotate — .mcp.json in that project's root:

{
  "mcpServers": {
    "annotate": {
      "command": "node",
      "args": ["/absolute/path/to/annotate-mcp/dist/index.js"],
      "timeout": 28800000
    }
  }
}

Restart the session, accept the "new MCP server" prompt, and ask the agent to open the browser. Full per-client instructions, including one-line install and where marks are stored: docs/INSTALL.md.

Uses your installed Google Chrome by default (familiar browser, no 150MB download), falling back to Playwright's bundled Chromium. The profile is persistent, so logins survive between sessions — you need to be signed in to review the flows worth reviewing.

Using it

Ask the agent to open the browser, then mark things up. Toolbar starts bottom-right; drag it anywhere by the grip on its left edge and it stays there, per browser.

Esc backs out one level at a time: open note → unlinking → linking mode → active tool → panel. Without it you can get stuck in comment mode unable to click the page underneath.

Working with existing marks

• Click a pin or a drawing to reopen its note — edit the text, or delete it from there. Clicking never stacks a second comment on top of the first, and it works whichever tool is in hand, including the pencil you just drew with.

• Drag a pin to move it. It re-resolves what it points at on drop, so a moved pin cites the element it landed on and not the one it left.

• Link (chain icon in the note box) makes one comment cover several places: press it, then click another mark to pull it in, click empty page to drop a new linked pin, or draw to add a linked drawing. Comments and drawings link to each other freely — a group can be any mix. Linked marks share one text (editing any of them rewrites all of them) and are joined on screen by a dashed thread. Press Done or Esc to leave linking mode.

• Unlink — the eraser in the link banner. It appears once a group exists, and clicking a member takes it out while leaving the mark, its text, and the rest of the group alone. It erases links and nothing else; the toolbar eraser is the one that deletes marks. Take the second-to-last member out and the group dissolves on its own, since one mark is not a link.

Sent vs pending

The panel has two tabs. Send moves a mark from To send into Sent, where it is dimmed and cannot be sent again — a queue that never shrinks stops telling you what is outstanding. The ink stays on the page as a record; the eye toggle in the Sent tab takes it off the page entirely when the archive starts covering the app you are still reviewing.

Editing a sent mark's text is the one way back: rewriting it makes it new information, so it rejoins the queue.

Reflow

Marks are anchored to the element they land on, as a fraction of that element's box. Resize the window and the ink follows its component — pins are repositioned, drawings are translated without being stretched, and the stored coordinates are rewritten so the agent, the screenshot crop, and the screen never disagree.

If the element is not rendered at the new size (a nav that collapsed, a card dropped from the layout), its mark is deliberately left where it was rather than dragged somewhere arbitrary, and drawn hollow/dashed so it reads as "this is about something not on screen" instead of silently labelling whatever slid underneath.

Tools

The hand-off, and the three states of Send

An MCP server answers calls; it cannot start a turn in an agent's session. So a send only reaches the agent on its own while something is parked in annotate_wait.

annotate_wait parks for an hour by default (max 8h). Parking is free — a blocked call spends no tokens — and on clients that background long calls it doesn't block the agent either: Claude Code moves any MCP call still running after two minutes into a background task, so it keeps working and the batch arrives as a task notification the moment you press Send. Long parks are kept alive with progress pings, because a client will abort an stdio call that has been silent too long — 30 minutes, in Claude Code's case.

Two ceilings are worth knowing:

Nothing sent is ever lost. A send with nobody parked is held, and the next park collects it immediately instead of waiting for a fresh one — so the gap between an agent taking one batch and parking again costs a delay, not your feedback.

That leaves three states, and the panel says which one you are in — but only when it is not the obvious one:

No copy anywhere names a model or vendor — the panel says "the agent", so the tool reads the same whichever assistant is on the other end of the MCP connection. Falling asleep while you are working raises a one-off notice, since that is the moment the button quietly changes meaning. Loading a page with nothing listening does not — that is ordinary, and the line under Send already covers it.

What a mark carries

The note is what gets acted on; the DOM capture is how it gets found in source.

### 1. ✏️ drawing
> currency should come first

- where: stroke over 30,369 → 160×30 (document coords)
- viewport: 782×495 @1x — tablet/laptop
- elements under the drawing (most specific first):
  - testid=`price-badge` · name="€25.00"
    - selector: `[data-testid="price-badge"]`
    - box: 40,379 140×38

A pin records the element clicked plus three ancestors. A stroke hit-tests a grid across its bounding box, so "I circled these three cards" survives as structured data rather than only as pixels — that is what makes a drawing actionable instead of merely suggestive.

Deliberately not captured: a file:line source mapping. React 19 dropped _debugSource from fibers and Next 16 is on SWC, so that needs a build-plugin dependency in every project — which would destroy the agnosticism. testid plus visible text finds the component in one grep.

Output

Written to <cwd>/.annotations/ (override with ANNOTATE_DIR):

• store.json — canonical state, reloaded on restart

• annotations.md — every mark, rewritten on change

• inbox.md — only the batch you last sent. The separation is the point: one file is everything ever marked, the other is what you're asking about right now.

Environment

Tests

pnpm build && pnpm test

test/smoke.mjs drives the real overlay in a real browser with real clicks and drags. It covers what typechecking cannot: injection ordering, the Node↔page binding, stroke hit-testing, mark survival across a full page load, selective send, the eraser, reopening and editing a mark, pin dragging, linking and unlinking across kinds, linked notes staying in step, ink surviving the note box, the sent/pending split, and marks following their element through a reflow.

Known limits

• One browser. You annotate in the window this server opens, not your everyday Chrome.

• No interaction tools. The server navigates (annotate_open) and screenshots, but exposes no click/type/hover — a human drives this window. If you want the agent to drive the app too, that is the Playwright MCP's job, or add the tools here; the Playwright page is already in hand.

• Iframes are skipped. The overlay mounts only in the top frame — otherwise a page with three iframes gets four toolbars.

• Re-anchoring resolves a selector. Short CSS paths can match more than one element, so a mark on a generic div > button may re-anchor to the wrong sibling after a reflow. Marks on anything carrying a testid or id are exact.

• Only pins can be dragged. A drawing's position follows its element on reflow, but there is no handle to move the ink by hand — erase and redraw.

Contributing

Bug reports: yes, please. Especially "the overlay broke on this site" — the injected script has to survive whatever CSS and JS the host page throws at it, and the failures worth knowing about are on pages I will never think to test.

Pull requests: not by default. This is a small tool with a specific shape, and most of it is one injected script that cannot import anything, cannot ship a stylesheet, and has to work inside a stranger's cascade — constraints that are invisible until you have been bitten by them. Fork it and make it yours; that is what the licence is for. If you have something you think genuinely belongs upstream, open an issue first and we can talk.

License

MIT © Nikolaos Psycharis