Semantic Hints MCP

A small local stdio MCP server that exposes compact semantic observations from a UI annotated with data-semtag-* hints.

It is a companion to the official Playwright MCP, not a replacement. It only reads semantics; all browser actions (click, type, navigate) stay in Playwright MCP. The intended split:

1. semantic_snapshot → a compact map of hinted elements (~2–3× smaller than a Playwright ARIA snapshot of the same page).

2. Pick a semantic id, e.g. checkout.submit.

3. Act via Playwright MCP using the selector [data-semtag-id='checkout.submit'].

4. semantic_observe → the current value/state of one hinted element.

Browser / session architecture

The semantic-hints MCP uses the Playwright library directly. It does not call the Playwright MCP internally.

However, the semantic-hints MCP and the official Playwright MCP must observe/control the same browser state during an experiment. Avoid accidentally creating two unrelated browser sessions where:

• Playwright MCP clicks in browser A

• semantic-hints MCP observes browser B

That would invalidate the workflow.

Semantic-hints MCP must not silently launch a separate browser by default.

Preferred design: one shared Chromium over CDP

1. Start Chromium with remote debugging enabled, e.g.:

   chromium --remote-debugging-port=9222

   (or the equivalent using your locally installed Chrome/Chromium binary).

2. Configure the official Playwright MCP to connect to that shared browser, if supported by the installed version/config.

3. The semantic-hints MCP connects to the same browser via Playwright:

   chromium.connectOverCDP("http://127.0.0.1:9222")

4. It reuses the active page where possible. If no suitable page exists and a URL is provided, it may open/navigate a page. If no page exists and no URL is provided, it returns a clear error asking for a URL or active page.

Configuration

Behavior:

• If SEMANTIC_HINTS_CDP_URL is reachable, connect over CDP (the normal mode).

• If the CDP connection fails and SEMANTIC_HINTS_LAUNCH_BROWSER is false, fail clearly with setup instructions — never silently launch.

• If SEMANTIC_HINTS_LAUNCH_BROWSER is true, the server may launch its own browser, but every tool response then carries a warning field making clear this is a standalone session that may not share state with the Playwright MCP.

The normal research workflow should use the shared CDP browser mode.

Related MCP server: BAP MCP

Tools

semantic_snapshot

Returns hinted elements grouped by their data-semtag-role — navigation, action, option, input, select, toggle, slider, observable, region, collection — plus other, collections, url and screen.

Empty groups are omitted, and there is no counts block: the arrays state their own lengths, so repeating them would only spend tokens.

{
  "url": "http://localhost:8082/products", // optional: navigate first; else read current page
  "scope": "products.filters",             // optional: data-semtag-id or CSS selector subtree
  "includeHidden": false                    // optional: include hidden elements (default false)
}

Per element (fields included only when present): id, name, action, state, target, controls, options (selects only), value (roles that hold one), enabled, visible. Never returns HTML, class names, or DOM subtrees.

There is no role field — the group key is the role, stated once for the whole group instead of once per entry. The exception is other, where the key says nothing: entries there keep the raw role string so an off-vocabulary value is visible rather than silently erased. semantic_observe always includes role, having no group around it.

The ten roles and the seven data-semtag-* attributes are defined in semtag-injector-skill/semtag-inject/references/hint-design.md, the source of truth for this server, the hint-injection prompt, and the browser-agent prompt alike. Nothing is inferred from tags or ARIA, and nothing is repaired: an element with a data-semtag-action but no data-semtag-role is other, not an action. other is how a hinting gap surfaces, so it is a thing to go fix in the app.

enabled and visible are omitted unless false — an element carrying neither is enabled and visible. Since snapshots exclude hidden elements by default, a "visible": true on every entry would only restate the filter.

Folded collections

A repeated collection is most of a real snapshot, and nearly all of it is repetition: every card restates its role and target, every price cell restates its role and state. So a collection is folded into one columnar block — shared attributes stated once, each item reduced to a row of only what varies:

"collections": [{
  "id": "products.grid",
  "name": "Products",
  "idPattern": "products.grid.item.{key}[.{control}]",
  "item":         { "role": "navigation" },
  "itemControls": { "price": { "role": "observable", "state": "product.price" },
                    "remove": { "role": "action", "name": "Remove",
                                "action": "remove-product" } },
  "fields": ["key", "name", "target", "price.value", "remove"],
  "items": [
    ["alpha", "Alpha Phone",  "product.detail", "£10.00", true],
    ["beta",  "Beta Laptop",  "product.detail", "£20.00", true],
    ["gamma", "Gamma Bundle", "bundle.detail",  "£30.00", null]
  ]
}]

• item / itemControls[c] hold the fields every item agrees on. (itemControls, not controls, because the container may carry its own data-semtag-controls.)

• fields is the column header: "key" first, then the item's own varying fields by bare name, then "<control>.<field>". items rows align to it, and null marks an absent cell.

• An element only some items have gets a bare "<control>" presence column holding true / null — above, gamma has no Remove button. It is needed because a uniform optional control otherwise hoists away completely, leaving no cell that could be null. The item itself can be optional too, and then takes a column named "item".

• name is dropped from a cell when it equals value: for a text observable the accessible name is the displayed value.

• The bucket an element would have occupied is not carried. On real pages it only ever restated role ("navigation"/"navigation"), and role with action / target / state already says everything an agent acts on.

Folding costs no addressability. idPattern states how to rebuild any member's real data-semtag-id — products.grid.item.beta.price — which semantic_observe and [data-semtag-id="…"] still resolve exactly as before.

Folded members and their container leave the six flat buckets, so nothing is stated twice.

Folding needs two things: at least two items using the <prefix>.item.<key> id convention, and a DOM ancestor declaring data-semtag-role="collection". Ancestry, not id equality — a container is often named a little differently from the prefix its rows use (admin.products.list holding admin.products.item.42), and that is a naming nicety, not a reason to give up the compression. Hence id (the container's own hint) and idPattern (built from the rows' prefix) can differ. A repeated group with no collection ancestor at all is a hinting gap in the app, not something the extractor guesses at: it stays flat, and the hint-review overlay flags it.

semantic_observe

{ "id": "cart.total" }

Resolves [data-semtag-id="<id>"] and returns its compact current state. Returns a clear error if zero or multiple elements match.

Setup

cd semantic-hints-mcp
npm install
npx playwright install chromium   # one-time browser download
npm run build

See Configuration above for the environment variables.

Register in Claude Code

From this folder, after npm run build:

claude mcp add semantic-hints \
  --env SEMANTIC_HINTS_CDP_URL=http://127.0.0.1:9222 \
  --env SEMANTIC_HINTS_TARGET_URL=http://localhost:8080 \
  -- node "$(pwd)/dist/index.js"

Or add it to .mcp.json / your Claude Code config manually:

{
  "mcpServers": {
    "semantic-hints": {
      "command": "node",
      "args": ["/absolute/path/to/semantic-hints-mcp/dist/index.js"],
      "env": {
        "SEMANTIC_HINTS_CDP_URL": "http://127.0.0.1:9222",
        "SEMANTIC_HINTS_TARGET_URL": "http://localhost:8080"
      }
    }
  }
}

(For development without a build step, use "command": "npx", "args": ["tsx", "/abs/path/src/index.ts"].)

Usage example

Expected Claude Code workflow:

1. Start the app under test, e.g.

   npm run dev          # → http://localhost:8080 for the annotated WebTestBench app

2. Start a shared Chromium with CDP enabled, e.g.

   chromium --remote-debugging-port=9222

3. Register/start both MCP servers:

   • the official Playwright MCP, configured for the shared browser if possible

   • the semantic-hints MCP, configured with SEMANTIC_HINTS_CDP_URL=http://127.0.0.1:9222

4. Agent calls:

   semantic_snapshot({ "url": "http://localhost:8080" })

5. Agent interacts using the official Playwright MCP:

   browser_click({ "target": "[data-semtag-id='checkout.submit']" })

6. Agent calls:

   semantic_observe({ "id": "cart.total" })

The click (step 5) and observe (step 6) must operate on the same browser page/session — which is exactly what the shared CDP browser guarantees.

Tests

npm test

Vitest loads test/fixture.html in headless Chromium and checks: grouped compact output, hidden-element handling, scope, observable/input value reads, missing & duplicate-ID errors, and that output contains no raw HTML/DOM.