# Semantic Hints MCP

A small **local stdio MCP server** that exposes compact semantic observations from
a UI annotated with `data-semtag-*` hints.

It is a **companion to the official Playwright MCP, not a replacement.** It only
*reads* semantics; all browser actions (click, type, navigate) stay in Playwright
MCP. The intended split:

1. `semantic_snapshot` → a compact map of hinted elements (~2–3× smaller than a
   Playwright ARIA snapshot of the same page).
2. Pick a semantic `id`, e.g. `checkout.submit`.
3. Act via **Playwright MCP** using the selector `[data-semtag-id='checkout.submit']`.
4. `semantic_observe` → the current value/state of one hinted element.

## Browser / session architecture

The semantic-hints MCP uses the **Playwright library directly**. It does **not**
call the Playwright MCP internally.

However, the semantic-hints MCP and the official Playwright MCP **must observe/control
the same browser state** during an experiment. Avoid accidentally creating two
unrelated browser sessions where:

- Playwright MCP clicks in browser **A**
- semantic-hints MCP observes browser **B**

That would invalidate the workflow.

> **Semantic-hints MCP must not silently launch a separate browser by default.**

### Preferred design: one shared Chromium over CDP

1. Start Chromium with remote debugging enabled, e.g.:

   ```bash
   chromium --remote-debugging-port=9222
   ```

   (or the equivalent using your locally installed Chrome/Chromium binary).

2. Configure the official Playwright MCP to connect to that shared browser, if
   supported by the installed version/config.

3. The semantic-hints MCP connects to the **same** browser via Playwright:

   ```js
   chromium.connectOverCDP("http://127.0.0.1:9222")
   ```

4. It **reuses the active page** where possible. If no suitable page exists and a
   URL is provided, it may open/navigate a page. If no page exists and no URL is
   provided, it returns a clear error asking for a URL or active page.

### Configuration

| Variable                       | Default                 | Purpose                                                                 |
| ------------------------------ | ----------------------- | ----------------------------------------------------------------------- |
| `SEMANTIC_HINTS_CDP_URL`       | `http://127.0.0.1:9222` | CDP endpoint of the shared Chromium to attach to.                       |
| `SEMANTIC_HINTS_TARGET_URL`    | (unset)                 | Optional default app URL, opened when a tool is called with no `url`.   |
| `SEMANTIC_HINTS_LAUNCH_BROWSER`| `false`                 | If `true`, may launch a private browser when the CDP connection fails.  |
| `SEMANTIC_HINTS_HEADLESS`      | `true`                  | Headless mode for the standalone-launch fallback only.                  |

Behavior:

- If `SEMANTIC_HINTS_CDP_URL` is reachable, connect over CDP (the normal mode).
- If the CDP connection fails and `SEMANTIC_HINTS_LAUNCH_BROWSER` is `false`, **fail
  clearly with setup instructions** — never silently launch.
- If `SEMANTIC_HINTS_LAUNCH_BROWSER` is `true`, the server may launch its own browser,
  but every tool response then carries a `warning` field making clear this is a
  **standalone session that may not share state** with the Playwright MCP.

The normal research workflow should use the shared CDP browser mode.

## Tools

### `semantic_snapshot`

Returns hinted elements grouped **by their `data-semtag-role`** — `navigation`,
`action`, `option`, `input`, `select`, `toggle`, `slider`, `observable`,
`region`, `collection` — plus `other`, `collections`, `url` and `screen`.

Empty groups are omitted, and there is no `counts` block: the arrays state
their own lengths, so repeating them would only spend tokens.

```jsonc
{
  "url": "http://localhost:8082/products", // optional: navigate first; else read current page
  "scope": "products.filters",             // optional: data-semtag-id or CSS selector subtree
  "includeHidden": false                    // optional: include hidden elements (default false)
}
```

Per element (fields included only when present): `id`, `name`, `action`,
`state`, `target`, `controls`, `options` (selects only), `value` (roles that
hold one), `enabled`, `visible`. Never returns HTML, class names, or DOM
subtrees.

There is no `role` field — the group key is the role, stated once for the whole
group instead of once per entry. The exception is `other`, where the key says
nothing: entries there keep the raw `role` string so an off-vocabulary value is
visible rather than silently erased. `semantic_observe` always includes `role`,
having no group around it.

The ten roles and the seven `data-semtag-*` attributes are defined in
`semtag-injector-skill/semtag-inject/references/hint-design.md`, the source of
truth for this server, the
hint-injection prompt, and the browser-agent prompt alike. Nothing is inferred
from tags or ARIA, and nothing is repaired: an element with a
`data-semtag-action` but no `data-semtag-role` is `other`, not an action. `other`
is how a hinting gap surfaces, so it is a thing to go fix in the app.

`enabled` and `visible` are omitted unless `false` — an element carrying neither
is enabled and visible. Since snapshots exclude hidden elements by default, a
`"visible": true` on every entry would only restate the filter.

#### Folded collections

A repeated collection is most of a real snapshot, and nearly all of it is
repetition: every card restates its role and target, every price cell restates
its role and state. So a collection is folded into **one columnar block** —
shared attributes stated once, each item reduced to a row of only what varies:

```jsonc
"collections": [{
  "id": "products.grid",
  "name": "Products",
  "idPattern": "products.grid.item.{key}[.{control}]",
  "item":         { "role": "navigation" },
  "itemControls": { "price": { "role": "observable", "state": "product.price" },
                    "remove": { "role": "action", "name": "Remove",
                                "action": "remove-product" } },
  "fields": ["key", "name", "target", "price.value", "remove"],
  "items": [
    ["alpha", "Alpha Phone",  "product.detail", "£10.00", true],
    ["beta",  "Beta Laptop",  "product.detail", "£20.00", true],
    ["gamma", "Gamma Bundle", "bundle.detail",  "£30.00", null]
  ]
}]
```

- `item` / `itemControls[c]` hold the fields every item agrees on.
  (`itemControls`, not `controls`, because the container may carry its own
  `data-semtag-controls`.)
- `fields` is the column header: `"key"` first, then the item's own varying
  fields by bare name, then `"<control>.<field>"`. `items` rows align to it, and
  `null` marks an absent cell.
- An element only *some* items have gets a bare `"<control>"` presence column
  holding `true` / `null` — above, `gamma` has no Remove button. It is needed
  because a uniform optional control otherwise hoists away completely, leaving
  no cell that could be null. The item itself can be optional too, and then
  takes a column named `"item"`.
- `name` is dropped from a cell when it equals `value`: for a text observable
  the accessible name *is* the displayed value.
- The bucket an element would have occupied is **not** carried. On real pages it
  only ever restated `role` (`"navigation"`/`"navigation"`), and `role` with
  `action` / `target` / `state` already says everything an agent acts on.

**Folding costs no addressability.** `idPattern` states how to rebuild any
member's real `data-semtag-id` — `products.grid.item.beta.price` — which
`semantic_observe` and `[data-semtag-id="…"]` still resolve exactly as before.

Folded members and their container leave the six flat buckets, so nothing is
stated twice.

Folding needs two things: at least two items using the `<prefix>.item.<key>` id
convention, and a **DOM ancestor** declaring `data-semtag-role="collection"`.
Ancestry, not id equality — a container is often named a little differently from
the prefix its rows use (`admin.products.list` holding
`admin.products.item.42`), and that is a naming nicety, not a reason to give up
the compression. Hence `id` (the container's own hint) and `idPattern` (built
from the rows' prefix) can differ. A repeated group with no collection ancestor
at all is a hinting gap in the app, not something the extractor guesses at: it
stays flat, and the hint-review overlay flags it.

### `semantic_observe`

```jsonc
{ "id": "cart.total" }
```

Resolves `[data-semtag-id="<id>"]` and returns its compact current state. Returns a
clear error if zero or multiple elements match.

## Setup

```bash
cd semantic-hints-mcp
npm install
npx playwright install chromium   # one-time browser download
npm run build
```

See [Configuration](#configuration) above for the environment variables.

## Register in Claude Code

From this folder, after `npm run build`:

```bash
claude mcp add semantic-hints \
  --env SEMANTIC_HINTS_CDP_URL=http://127.0.0.1:9222 \
  --env SEMANTIC_HINTS_TARGET_URL=http://localhost:8080 \
  -- node "$(pwd)/dist/index.js"
```

Or add it to `.mcp.json` / your Claude Code config manually:

```jsonc
{
  "mcpServers": {
    "semantic-hints": {
      "command": "node",
      "args": ["/absolute/path/to/semantic-hints-mcp/dist/index.js"],
      "env": {
        "SEMANTIC_HINTS_CDP_URL": "http://127.0.0.1:9222",
        "SEMANTIC_HINTS_TARGET_URL": "http://localhost:8080"
      }
    }
  }
}
```

(For development without a build step, use `"command": "npx", "args": ["tsx",
"/abs/path/src/index.ts"]`.)

## Usage example

Expected Claude Code workflow:

1. Start the app under test, e.g.

   ```bash
   npm run dev          # → http://localhost:8080 for the annotated WebTestBench app
   ```

2. Start a shared Chromium with CDP enabled, e.g.

   ```bash
   chromium --remote-debugging-port=9222
   ```

3. Register/start **both** MCP servers:
   - the official Playwright MCP, configured for the shared browser if possible
   - the semantic-hints MCP, configured with `SEMANTIC_HINTS_CDP_URL=http://127.0.0.1:9222`

4. Agent calls:

   ```jsonc
   semantic_snapshot({ "url": "http://localhost:8080" })
   ```

5. Agent interacts using the official Playwright MCP:

   ```jsonc
   browser_click({ "target": "[data-semtag-id='checkout.submit']" })
   ```

6. Agent calls:

   ```jsonc
   semantic_observe({ "id": "cart.total" })
   ```

The click (step 5) and observe (step 6) **must operate on the same browser
page/session** — which is exactly what the shared CDP browser guarantees.

## Tests

```bash
npm test
```

Vitest loads `test/fixture.html` in headless Chromium and checks: grouped compact
output, hidden-element handling, `scope`, observable/input value reads, missing &
duplicate-ID errors, and that output contains no raw HTML/DOM.