ui-annotate

Point at the UI, tell your coding agent what is wrong. A browser extension for Chrome and Firefox: click an element of your running dev app, write a note, and Claude Code, Codex or Cursor receives the selector, DOM context, component name and the exact source file:line. The agent answers and resolves right on the page.

English · Русский

Why

Describing UI problems in a chat is slow and lossy: "the grey button under the price, no, the other one". Screenshots carry no code location, so the agent greps for text and guesses. ui-annotate attaches what the agent needs to every note: which element, which component, which line of which file. Nothing gets installed into your app: the extension reads framework internals of the dev build.

Related MCP server: earmark

Features

• Element, text, multi-select and area annotations with intent (fix, change, question, approve) and severity.

• Source location without app changes for Vue, React 18/19, Svelte 4/5, Angular and Solid; exact template lines with common build plugins (see Source location).

• Two-way thread on the page: the agent acknowledges, asks questions, resolves with a summary; you reply in the overlay.

• Agent integrations: an MCP server (stdio and Streamable HTTP), hooks that put open annotations into the agent's context and keep it from stopping while work is open, a Claude Code plugin with a live channel.

• Automatic mode: every new annotation of a site goes to a background claude, codex or cursor-agent run in the project folder, limited to reading and editing.

• Project root detection: the daemon finds the repository behind localhost:PORT by itself, so paths are relative and annotations reach only the agent working in that repository.

• Local only: one daemon on 127.0.0.1, SQLite storage, token pairing done automatically.

How it works

flowchart LR
  subgraph Browser
    page["dev app page"] -- "framework internals<br/>(MAIN world)" --> overlay["overlay<br/>(content script)"]
    overlay --> bg["extension background"]
  end
  bg -- "HTTP + token" --> daemon["ui-annotate daemon<br/>127.0.0.1:4848, SQLite"]
  daemon -- "MCP tools, hooks, channel" --> agents["Claude Code · Codex · Cursor"]
  daemon -. "automatic mode" .-> runs["background agent run<br/>in the project root"]
  agents -- "acknowledge · reply · resolve" --> daemon
  daemon -- "live updates (SSE)" --> bg

Quick start

Want your agent to set it up? Give it docs/agents/install.md.

Requirements: Node.js ≥ 22.13 (the daemon uses node:sqlite), pnpm 9, Chrome ≥ 116 or Firefox ≥ 128.

1. Build

git clone https://github.com/AnarchyGhost/ui-annotate.git
cd ui-annotate
pnpm install
pnpm build          # extension → apps/extension/.output/{chrome-mv3,firefox-mv3}

2. Load the extension

• Chrome: chrome://extensions → Developer mode → Load unpacked → apps/extension/.output/chrome-mv3.

• Firefox: about:debugging#/runtime/this-firefox → Load Temporary Add-on → apps/extension/.output/firefox-mv3/manifest.json. A temporary add-on is removed when Firefox restarts; Firefox Developer Edition, Nightly and forks such as Zen keep an unsigned build installed with xpinstall.signatures.required = false.

3. Connect your agent (any combination)

# Claude Code: MCP server, skill, hooks and the channel in one plugin
claude plugin marketplace add "$PWD"
claude plugin install ui-annotate@ui-annotate-local

# Codex: MCP server in ~/.codex/config.toml, hooks in ~/.codex/hooks.json
pnpm exec ui-annotate init codex --write     # then approve the hooks once in Codex: /hooks

# Cursor: .cursor/mcp.json and .cursor/hooks.json of the project you run it in
cd /path/to/your/app && /path/to/ui-annotate/node_modules/.bin/ui-annotate init cursor --write

init writes the absolute path of the Node.js that runs it, so run it with Node ≥ 22.13 first in PATH. The daemon starts by itself with the first MCP connection; pnpm exec ui-annotate serve runs it by hand.

4. Annotate. Open your dev app, click the extension icon (or press Ctrl/Cmd+Shift+F), turn on annotation mode (Ctrl/Cmd+Shift+1), click an element and type the note. Enter saves.

5. Let the agent work. Ask it to "handle the UI feedback", or simply keep working: the hooks show open annotations in its context. Replies and resolutions appear on the page live.

What the agent receives

get_pending returns a markdown block per annotation plus the same data as JSON. The first line is the location the agent opens first:

`src/components/PlanCard.vue:17:7` `<PlanCard>` — #1 fix/important [pending] id=a_vOPD0BMk

> Upgrade is the main action here: make it primary, like Invite

- element: `button "Upgrade plan"`
  selector: `[data-testid="upgrade"]`
  path: `main.content > div.grid > section.card > div.actions > button[data-testid=upgrade]`
  text: "Upgrade plan"
  parents: App (src/App.vue)
- page: http://127.0.0.1:5180/ (1280×800)

The agent works through annotations with MCP tools and the thread shows up in the overlay:

MCP tools

The same tools are served over Streamable HTTP at http://127.0.0.1:4848/mcp with the token as Authorization: Bearer (pnpm exec ui-annotate token). watch_annotations keeps a cursor per MCP session, so annotations created while the agent was busy come with the next call.

Hooks and the channel

• Claude Code and Codex: SessionStart and UserPromptSubmit add a digest of open annotations (≤ 10 000 characters) and stay silent when there are none; Stop blocks the end of a turn while annotations of this project are open, once (stop_hook_active).

• Cursor: the digest comes from sessionStart (beforeSubmitPrompt cannot add context); stop sends a follow-up message at most twice.

• Claude Code channel (research preview, interactive sessions): claude --dangerously-load-development-channels plugin:ui-annotate@ui-annotate-local delivers each new annotation as a <channel source="...ui-annotate..."> message without typing anything.

The daemon is shared by all projects. Hooks and the channel show annotations of pages whose project root contains the agent's working directory; pages without a root appear only as a count and never block.

Automatic mode

Popup → This site → Automatic mode hands every new annotation of the site to a background agent run, with no agent session open. Pick the agent and a delay: the run starts after that many quiet seconds, so notes left in a row go into one run.

The daemon starts the CLI in the page's project root and reports in the thread: "working on it", then resolved or dismissed with the agent's summary, a question as a reply, or the failure reason (the annotation stays open). One run per project at a time.

The CLIs are looked up in the daemon's PATH; UI_ANNOTATE_CLAUDE_BIN, UI_ANNOTATE_CODEX_BIN and UI_ANNOTATE_CURSOR_BIN override them. A run stops after 15 minutes. The agent edits your code with no review step, so turn the mode on for projects under version control.

Project root

The project root of a page makes source paths relative and scopes its annotations to agent sessions in that folder. It is found without setup, in this order of precedence:

1. the Project root field of the popup (manual override);

2. link_page(url) called by an agent: the repository it runs in (the Claude Code skill calls it after starting a dev server);

3. the dev server behind a localhost page: the process listening on the port (/proc on Linux, lsof on macOS), its working directory, lifted to the git top level;

4. absolute source paths in the first annotation, lifted to the git top level and accepted only with a project marker (package.json, angular.json, …).

The home directory, the file system root and processes outside home (proxies, containers) never become a root. The popup shows what was detected and how.

Using the overlay

Rebind them in the browser: Change shortcuts in the popup opens the shortcut settings.

Markers follow their element and fall back to the stored position with a dashed border when it is gone. Load on every visit in the popup injects the overlay at page load; origins other than localhost and 127.0.0.1 ask for the host permission first.

Source location

Verified on the dev servers in fixtures/ by apps/extension/e2e/resolve.spec.ts:

Build-plugin attributes take priority: data-insp-path (code-inspector-plugin), vite-plugin-vue-inspector, data-locatorjs, data-source-loc. Without any source the annotation still carries the DOM context and a reason: production-build, no-framework or no-line.

Security model

The daemon listens on 127.0.0.1 only and checks every request:

1. Host is 127.0.0.1:<port> or localhost:<port> (DNS rebinding).

2. Origin, when present, is the extension origin: a web page cannot call the API even with the token. Without Origin, Sec-Fetch-Site must be absent, none or same-origin.

3. Authorization: Bearer <token> (other local processes and extensions).

POST /pair hands the token only to the extension: the fixed Chrome ID, or the first Firefox install that pairs (pnpm exec ui-annotate unpair resets it). Overlay inputs accept only trusted user events, so page scripts cannot write annotations. Page-derived text reaches the agent flattened and marked as data, and source paths outside the project root are dropped.

Demo

examples/demo is the Vue app of the screenshots. pnpm screenshots (after pnpm build) starts it with a test daemon, drives the extension in Chrome for Testing and writes docs/images/*.png.

Development

See AGENTS.md: layout, commands, conventions and how to verify a change. It is written for coding agents and works as a contributor guide too.

Status

Early stage. Not published to the Chrome Web Store, addons.mozilla.org or npm yet: install from source as above. Tested on Linux with Chrome for Testing 151–156, Google Chrome 155, Firefox 158 and Zen 1.23; the macOS lsof path of project root detection is untested.

License

MIT