Annotation Overlay MCP

DOM-aware visual annotation overlay for web feedback. Draw arrows, boxes, text, freehand strokes, click elements, and annotate text selections — all in one toolbar. Submit structured annotations to Claude Code via MCP.

6 interaction modes. Zero JS dependencies. Chrome Extension + MCP Server.

[Browser Page]  ←→  [Chrome Extension]  ←→  [MCP Server]  ←→  [Claude Code]
   overlay.js         bridge.js              HTTP + stdio       read/clear tools

Quick Start

1. Install & Start MCP Server

git clone https://github.com/cc10143/annotation-overlay-mcp.git
cd annotation-overlay-mcp
npm install
npm start
# → HTTP server on port 3847 + MCP stdio transport connected

2. Load Chrome Extension

1. Open chrome://extensions

2. Enable Developer mode

3. Click Load unpacked

4. Select the extension/ directory

The overlay auto-injects on every page, but the toolbar stays hidden during normal browsing (it's a tool for agents, not a resident UI). To annotate, either press Ctrl+Shift+A, or — in the feedback flow — have the agent call set_annotation_mode(true), which shows the toolbar on the active tab.

3. Configure Claude Code MCP

// ~/.claude/settings.json
{
  "mcpServers": {
    "annotation-overlay-mcp": {
      "command": "node",
      "args": ["D:/KaiFa/annotation-overlay/server/index.js"]
    }
  }
}

Or use the CLI:

claude mcp add annotation-overlay-mcp -- node D:/KaiFa/annotation-overlay/server/index.js

Related MCP server: Browser Feedback MCP

Usage

Annotation Workflow

1. Agent launches the annotation browser (primary) — Claude Code runs the annotation-browser-launch skill: Playwright opens chromium with the extension + a persistent profile, navigates to the target page, and shows the toolbar. (Fallback: the agent calls set_annotation_mode(true) on the user's own Chrome.)

2. Annotate — use any of the 6 tools to mark issues

3. Submit — click Submit to send structured JSON to the MCP server. The extension automatically captures a viewport screenshot with annotations overlaid and uploads it; the server saves it to ~/.annotation-overlay/screenshots/. Submit also hides the toolbar and deactivates the overlay.

4. Agent reads — Claude Code calls read_annotations → gets annotations + screenshotPath → feeds the screenshot to a vision-capable tool → processes feedback

5. Fix & verify — after the fix, the agent refreshes the annotation browser (via CDP reconnect), then compares a fresh screenshot against the annotated screenshotPath to confirm the fix actually landed

6. Next round — the agent re-activates the overlay and repeats

Tools

Drawing annotates immediately — no comment prompt interrupts your flow. To add a comment to an existing annotation, switch to the Select tool and double-click the drawing (or the numbered badge). Only the Text tool still prompts, because its text is the annotation itself.

Keyboard Shortcuts

Colors

5 preset colors: red #e94560, blue #4080f0, green #2ecc71, yellow #f1c40f, purple #9b59b6.

Screenshot

On Submit, the extension captures the current viewport with annotations overlaid — the canvas strokes and numbered badges are DOM, so they appear in the shot naturally; only the toolbar is hidden during capture. The screenshot is uploaded to the MCP server and saved to ~/.annotation-overlay/screenshots/anno-<timestamp>.png.

read_annotations returns the absolute screenshotPath in its response. Pass it to a vision-capable tool (e.g. a vision MCP) to see exactly what the user is pointing at — this is how a non-multimodal agent gets visual context for the annotations.

For before/after verification, the agent calls capture_page after the fix reloads the page. The extension captures the current viewport (clean — the overlay is inactive until toggled) and the server exposes it as afterScreenshotPath in read_annotations. Feeding both screenshotPath (annotated "before") and afterScreenshotPath (clean "after") to a vision-capable tool lets the agent confirm a fix landed instead of working blind.

Extension only. Without the extension bridge (standalone script injection), submit proceeds without a screenshot.

MCP Tools

read_annotations

Read all pending annotations. The response also includes screenshotPath — the absolute path to the last annotated viewport screenshot (saved on Submit) — and afterScreenshotPath — the absolute path to the last capture_page screenshot (clean post-fix viewport), both for vision-capable consumption.

Each annotation includes:

{
  "id": "uuid",
  "type": "circle",                       // arrow | circle | text | freehand | select | textsel
  "comment": "user feedback text",        // empty until double-clicked in select mode
  "selector": "div.card:nth-child(1) > button.btn-primary",
  "fallbackSelectors": [
    { "type": "id", "value": "#submit-btn" },
    { "type": "cssPath", "value": "div.card:nth-child(1) > ..." },
    { "type": "contentHash", "value": "Buy Now-a3f8b2c1" }
  ],
  "tagName": "BUTTON",
  "classes": ["btn-primary"],
  "elementText": "Buy Now",
  "contentHash": "Buy Now-a3f8b2c1",
  "region": {                             // box/freehand only: the container element
    "selector": "#card",                  //   covering the drawn area (deepest ancestor
    "fallbackSelectors": [                //   whose rect covers ≥60% of the drawing)
      { "type": "id", "value": "#card" },
      { "type": "cssPath", "value": "div.card" },
      { "type": "contentHash", "value": "Card A-..." }
    ],
    "tagName": "div",
    "classes": ["card"]
  },
  "position": { "x": 100, "y": 200, "w": 200, "h": 100 },
  "color": "#e94560"
}

screenshotPath — added to the response when a screenshot was captured on submit, e.g. "C:\Users\you\.annotation-overlay\screenshots\anno-2026-08-12T06-27-48.png". Pass it to a vision-capable tool to see the page with annotations overlaid.

afterScreenshotPath — added to the response after an agent-requested capture_page, e.g. "C:\Users\you\.annotation-overlay\screenshots\anno-2026-08-12T06-30-12.png". This is the clean post-fix viewport; compare it against screenshotPath to verify a fix landed. afterScreenshotTabUrl is the URL of the tab that was captured — confirm it matches the page you expected (the capture is of the active tab). All three are cleared by clear_annotations.

httpOwnedExternally — true when another annotation-overlay instance holds the HTTP port (e.g. a stale npm start, or an orphaned MCP child from an abrupt Claude Code shutdown), with externalHttpVersion naming that instance. read_annotations/clear_annotations still work against the shared store, but capture_page/set_annotation_mode may fail until the stale process is killed (netstat -ano | grep :3847 → taskkill //PID <pid> //F). The server no longer crashes on a port conflict — it keeps the MCP tools alive and reports this flag instead.

position shape varies by type (viewport coordinates):

region (box/freehand only) is the container element that best represents the drawn area — e.g. the card <div> the user circled — with the same fallback chain as selector. Use it to locate the region when the box center happens to sit on a child element.

capture_page

Capture the current page viewport and save it as the "after" state for before/after verification. The server asks the extension (via its existing ~10s badge poll) to capture the active tab of the focused window; the shot is clean — the extension hides any annotation overlay chrome before capturing. Returns the absolute path to the saved PNG plus the URL of the captured tab, e.g.:

{ "ok": true, "path": "C:\\Users\\you\\.annotation-overlay\\screenshots\\anno-2026-08-12T06-30-12.png", "tabUrl": "https://example.com/" }

Verify tabUrl matches the page you expect — capture_page captures the active tab, so if the user switched tabs it would show the wrong page (the server can't tell). May take up to ~15s (extension poll + capture). Requires the extension to be loaded. Pair the result with screenshotPath from read_annotations to confirm a fix landed.

set_annotation_mode

Show or hide the annotation toolbar on the active tab, so the user can give visual feedback. The overlay is injected on every page but the toolbar is hidden by default — this is how the agent turns it on before asking the user to annotate, and off after processing:

{ "ok": true, "enabled": true, "tabUrl": "https://example.com/" }

The extension relays the change to the page's overlay via its ~10s badge poll. May take up to ~15s. Requires the extension to be loaded and the target page to be the active tab. The user can also toggle manually with Ctrl+Shift+A.

clear_annotations

Clear all stored annotations. Call after processing feedback.

Claude Code Skills

The repo ships two skills (in skills/) that encode the annotation feedback loop. Install them to ~/.claude/skills/ with:

npm run install-skills

Edit the skills in skills/ (the repo is the source of truth), then re-run npm run install-skills to sync.

Selector Fallback Chain

When the agent regenerates the page, CSS selectors may break. Each annotation carries a fallback chain:

1. id — #element-id (most stable)

2. cssPath — div.card:nth-child(1) > button.btn-primary

3. contentHash — Buy Now-a3f8b2c1 (first 40 chars of text + djb2 hash)

The agent should try each fallback in order when resolving elements after page changes.

Standalone Use (Without Extension)

The overlay can be injected into any page via script tag or browser console:

<script src="annotation-overlay.js"></script>

Or via Tandem evaluate / Playwright:

// Tandem
tandem_devtools_evaluate({ function: "..." }) // paste annotation-overlay.js contents

// Playwright
await page.evaluate(fs.readFileSync('annotation-overlay.js', 'utf-8'));

Public API:

__annotationOverlay.activate()    // show toolbar
__annotationOverlay.deactivate()  // hide overlay
__annotationOverlay.serialize()   // → JSON string
__annotationOverlay.clear()       // remove all annotations
__annotationOverlay.submit()      // send to MCP server via direct fetch

Architecture

┌──────────────────────────────────────────────────────┐
│  annotation-overlay.js  (src/, ~1600 lines)          │
│                                                      │
│  ┌──────────┐  ┌──────────┐  ┌──────────────────┐   │
│  │ Toolbar  │  │  Canvas  │  │  DOM Bridge      │   │
│  │  6 tools │  │  2D ctx  │  │  elementFromPoint │   │
│  │  5 colors│  │  DPR     │  │  CSS selector gen │   │
│  │  Submit  │  │  undo    │  │  contentHash      │   │
│  └────┬─────┘  └────┬─────┘  └────────┬─────────┘   │
│       │             │                │               │
│       └─────────────┴───────┬────────┘               │
│                             │                         │
│                    ┌────────▼────────┐                │
│                    │  Annotation Model│               │
│                    │  + Serializer   │               │
│                    │  + Fallback     │               │
│                    └────────┬────────┘               │
└─────────────────────────────┼────────────────────────┘
                              │ postMessage
┌─────────────────────────────┼────────────────────────┐
│  Chrome Extension           ▼                         │
│  ┌──────────────┐  ┌──────────────┐                  │
│  │  bridge.js   │  │  service-    │                  │
│  │  (ISOLATED)  │  │  worker.js   │                  │
│  │  inject +    │  │  badge +     │                  │
│  │  relay       │  │  relay       │                  │
│  └──────┬───────┘  └──────┬───────┘                  │
└─────────┼──────────────────┼──────────────────────────┘
          │                  │ HTTP (port 3847)
┌─────────▼──────────────────▼──────────────────────────┐
│  MCP Server (Node.js, single process)                  │
│  ┌──────────────┐  ┌──────────────┐                   │
│  │  Express API │  │  MCP stdio   │                   │
│  │  POST/GET/   │  │  read_       │                   │
│  │  DELETE ann  │  │  clear tools │                   │
│  └──────┬───────┘  └──────┬───────┘                   │
│         └────────┬─────────┘                           │
│         ┌────────▼────────┐                            │
│         │  File Store      │                            │
│         └─────────────────┘                            │
└────────────────────────────────────────────────────────┘

On Submit the extension captures the annotated viewport; the server saves it to ~/.annotation-overlay/screenshots/ and read_annotations surfaces the path via screenshotPath.

Comparison

Configuration

License

MIT — Copyright (c) 2026 gaogao