Semantic Hints MCP
Semantic Hints MCP is a lightweight server that reads compact UI semantic hints from a browser and exposes them as structured data, acting as a companion to the Playwright MCP for efficient agent workflows.
semantic_snapshot: Returns a compact, categorized map of UI elements annotated withdata-agent-id, grouped by:regions,actions,inputs,observables,navigation, andother. Options include:Navigate to a URL before extracting
Scope extraction to a subtree via a
data-agent-idor CSS selectorInclude hidden elements (excluded by default)
Returns element metadata (id, role, name, state, value, etc.) — never raw HTML, class names, or DOM subtrees
semantic_observe: Reads the live value/state of a single hinted element by itsdata-agent-id. Returns an error if no element or multiple elements match.
Typical workflow: Use semantic_snapshot to identify element IDs → pass those IDs to Playwright MCP (e.g., [data-agent-id='checkout.submit']) to perform actions → use semantic_observe to verify resulting state.
Integration: Shares a Chromium browser session with Playwright MCP via CDP (recommended), so both tools operate on the same live page. Configurable via environment variables (SEMANTIC_HINTS_CDP_URL, etc.), with a fallback to launching a private browser if no shared session is available.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Semantic Hints MCP Take a semantic snapshot of the current page"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Semantic Hints MCP
A small local stdio MCP server that exposes compact semantic observations from
a UI annotated with data-semtag-* hints.
It is a companion to the official Playwright MCP, not a replacement. It only reads semantics; all browser actions (click, type, navigate) stay in Playwright MCP. The intended split:
semantic_snapshot→ a compact map of hinted elements (~2–3× smaller than a Playwright ARIA snapshot of the same page).Pick a semantic
id, e.g.checkout.submit.Act via Playwright MCP using the selector
[data-semtag-id='checkout.submit'].semantic_observe→ the current value/state of one hinted element.
Browser / session architecture
The semantic-hints MCP uses the Playwright library directly. It does not call the Playwright MCP internally.
However, the semantic-hints MCP and the official Playwright MCP must observe/control the same browser state during an experiment. Avoid accidentally creating two unrelated browser sessions where:
Playwright MCP clicks in browser A
semantic-hints MCP observes browser B
That would invalidate the workflow.
Semantic-hints MCP must not silently launch a separate browser by default.
Preferred design: one shared Chromium over CDP
Start Chromium with remote debugging enabled, e.g.:
chromium --remote-debugging-port=9222(or the equivalent using your locally installed Chrome/Chromium binary).
Configure the official Playwright MCP to connect to that shared browser, if supported by the installed version/config.
The semantic-hints MCP connects to the same browser via Playwright:
chromium.connectOverCDP("http://127.0.0.1:9222")It reuses the active page where possible. If no suitable page exists and a URL is provided, it may open/navigate a page. If no page exists and no URL is provided, it returns a clear error asking for a URL or active page.
Configuration
Variable | Default | Purpose |
|
| CDP endpoint of the shared Chromium to attach to. |
| (unset) | Optional default app URL, opened when a tool is called with no |
|
| If |
|
| Headless mode for the standalone-launch fallback only. |
Behavior:
If
SEMANTIC_HINTS_CDP_URLis reachable, connect over CDP (the normal mode).If the CDP connection fails and
SEMANTIC_HINTS_LAUNCH_BROWSERisfalse, fail clearly with setup instructions — never silently launch.If
SEMANTIC_HINTS_LAUNCH_BROWSERistrue, the server may launch its own browser, but every tool response then carries awarningfield making clear this is a standalone session that may not share state with the Playwright MCP.
The normal research workflow should use the shared CDP browser mode.
Related MCP server: brocogni
Tools
semantic_snapshot
Returns hinted elements grouped by their data-semtag-role — navigation,
action, option, input, select, toggle, slider, observable,
region, collection — plus other, collections, url and screen.
Empty groups are omitted, and there is no counts block: the arrays state
their own lengths, so repeating them would only spend tokens.
{
"url": "http://localhost:8082/products", // optional: navigate first; else read current page
"scope": "products.filters", // optional: data-semtag-id or CSS selector subtree
"includeHidden": false // optional: include hidden elements (default false)
}Per element (fields included only when present): id, name, action,
state, target, controls, options (selects only), value (roles that
hold one), enabled, visible. Never returns HTML, class names, or DOM
subtrees.
There is no role field — the group key is the role, stated once for the whole
group instead of once per entry. The exception is other, where the key says
nothing: entries there keep the raw role string so an off-vocabulary value is
visible rather than silently erased. semantic_observe always includes role,
having no group around it.
The ten roles and the seven data-semtag-* attributes are defined in
semtag-injector-skill/semtag-inject/references/hint-design.md, the source of
truth for this server, the
hint-injection prompt, and the browser-agent prompt alike. Nothing is inferred
from tags or ARIA, and nothing is repaired: an element with a
data-semtag-action but no data-semtag-role is other, not an action. other
is how a hinting gap surfaces, so it is a thing to go fix in the app.
enabled and visible are omitted unless false — an element carrying neither
is enabled and visible. Since snapshots exclude hidden elements by default, a
"visible": true on every entry would only restate the filter.
Folded collections
A repeated collection is most of a real snapshot, and nearly all of it is repetition: every card restates its role and target, every price cell restates its role and state. So a collection is folded into one columnar block — shared attributes stated once, each item reduced to a row of only what varies:
"collections": [{
"id": "products.grid",
"name": "Products",
"idPattern": "products.grid.item.{key}[.{control}]",
"item": { "role": "navigation" },
"itemControls": { "price": { "role": "observable", "state": "product.price" },
"remove": { "role": "action", "name": "Remove",
"action": "remove-product" } },
"fields": ["key", "name", "target", "price.value", "remove"],
"items": [
["alpha", "Alpha Phone", "product.detail", "£10.00", true],
["beta", "Beta Laptop", "product.detail", "£20.00", true],
["gamma", "Gamma Bundle", "bundle.detail", "£30.00", null]
]
}]item/itemControls[c]hold the fields every item agrees on. (itemControls, notcontrols, because the container may carry its owndata-semtag-controls.)fieldsis the column header:"key"first, then the item's own varying fields by bare name, then"<control>.<field>".itemsrows align to it, andnullmarks an absent cell.An element only some items have gets a bare
"<control>"presence column holdingtrue/null— above,gammahas no Remove button. It is needed because a uniform optional control otherwise hoists away completely, leaving no cell that could be null. The item itself can be optional too, and then takes a column named"item".nameis dropped from a cell when it equalsvalue: for a text observable the accessible name is the displayed value.The bucket an element would have occupied is not carried. On real pages it only ever restated
role("navigation"/"navigation"), androlewithaction/target/statealready says everything an agent acts on.
Folding costs no addressability. idPattern states how to rebuild any
member's real data-semtag-id — products.grid.item.beta.price — which
semantic_observe and [data-semtag-id="…"] still resolve exactly as before.
Folded members and their container leave the six flat buckets, so nothing is stated twice.
Folding needs two things: at least two items using the <prefix>.item.<key> id
convention, and a DOM ancestor declaring data-semtag-role="collection".
Ancestry, not id equality — a container is often named a little differently from
the prefix its rows use (admin.products.list holding
admin.products.item.42), and that is a naming nicety, not a reason to give up
the compression. Hence id (the container's own hint) and idPattern (built
from the rows' prefix) can differ. A repeated group with no collection ancestor
at all is a hinting gap in the app, not something the extractor guesses at: it
stays flat, and the hint-review overlay flags it.
semantic_observe
{ "id": "cart.total" }Resolves [data-semtag-id="<id>"] and returns its compact current state. Returns a
clear error if zero or multiple elements match.
Setup
cd semantic-hints-mcp
npm install
npx playwright install chromium # one-time browser download
npm run buildSee Configuration above for the environment variables.
Register in Claude Code
From this folder, after npm run build:
claude mcp add semantic-hints \
--env SEMANTIC_HINTS_CDP_URL=http://127.0.0.1:9222 \
--env SEMANTIC_HINTS_TARGET_URL=http://localhost:8080 \
-- node "$(pwd)/dist/index.js"Or add it to .mcp.json / your Claude Code config manually:
{
"mcpServers": {
"semantic-hints": {
"command": "node",
"args": ["/absolute/path/to/semantic-hints-mcp/dist/index.js"],
"env": {
"SEMANTIC_HINTS_CDP_URL": "http://127.0.0.1:9222",
"SEMANTIC_HINTS_TARGET_URL": "http://localhost:8080"
}
}
}
}(For development without a build step, use "command": "npx", "args": ["tsx", "/abs/path/src/index.ts"].)
Usage example
Expected Claude Code workflow:
Start the app under test, e.g.
npm run dev # → http://localhost:8080 for the annotated WebTestBench appStart a shared Chromium with CDP enabled, e.g.
chromium --remote-debugging-port=9222Register/start both MCP servers:
the official Playwright MCP, configured for the shared browser if possible
the semantic-hints MCP, configured with
SEMANTIC_HINTS_CDP_URL=http://127.0.0.1:9222
Agent calls:
semantic_snapshot({ "url": "http://localhost:8080" })Agent interacts using the official Playwright MCP:
browser_click({ "target": "[data-semtag-id='checkout.submit']" })Agent calls:
semantic_observe({ "id": "cart.total" })
The click (step 5) and observe (step 6) must operate on the same browser page/session — which is exactly what the shared CDP browser guarantees.
Tests
npm testVitest loads test/fixture.html in headless Chromium and checks: grouped compact
output, hidden-element handling, scope, observable/input value reads, missing &
duplicate-ID errors, and that output contains no raw HTML/DOM.
Available Tools
2 toolssemantic_observeSemantic observeA
Return the current compact value/state of a single hinted element, resolved via [data-agent-id='']. Returns a clear error if no element or multiple elements match. Use after a snapshot to read the live value/state of one element. Returns compact JSON only.
Inputs: id (required).
semantic_snapshotSemantic snapshotA
Return a compact map of hinted UI elements (only elements carrying data-agent-id), grouped into regions/actions/inputs/observables/navigation/other. Use this as a cheap first observation instead of a full accessibility snapshot. To act on an element, pick its id and use the Playwright MCP with selector [data-agent-id='']. Returns compact JSON only — never HTML, DOM, or class names.
Inputs: url, scope, includeHidden.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 2 tools
The two tools have clearly distinct purposes: semantic_snapshot provides an overview of all hinted elements, while semantic_observe retrieves the state of a single element. No ambiguity.
Both tools follow a consistent verb_noun pattern with the 'semantic_' prefix, using snake_case. The naming is predictable and clear.
With only 2 tools, the server is tightly scoped to its purpose of observing semantic hints. This is appropriate for a focused utility; each tool earns its place.
The tool surface covers the essential operations for the domain: getting a full map of elements and reading a single element's state. No obvious gaps for an observation-only tool.
Maintenance
Related MCP Connectors
Official Browserless MCP server: browser automation, scraping, screenshots and PDFs for AI agents.
Web scraping, browser automation, screenshots and AI data extraction for AI agents via MCP.
Hosted browser for AI agents: screenshots, post-JS DOM, console, WCAG. No install, no API key.
Live browser debugging for AI assistants — DOM, console, network via MCP.
Related MCP Servers
- AlicenseNot gradedqualityNot gradedmaintenanceA Playwright-based MCP server that exposes a live browser as a traceable, inspectable, debuggable and controllable execution environment for AI agents.5,213 npm57-
- AlicenseAqualityCmaintenancesemantic browser observation for AI agents via MCP, 100% local, zero telemetry1215 npm9MIT
- AlicenseAqualityBmaintenanceTerminal-native DOM perception layer for AppSec and AI. Hook into live Chrome sessions, compress HTML tokens by 98.9%, and map attack surfaces deterministically.4094 PyPI3Apache 2.0
- AlicenseNot gradedqualityNot gradedmaintenanceEnables autonomous web agents to prune raw HTML into accessibility-friendly semantic trees with 80%+ token reduction, auto-map form input schemas, convert HTML/Markdown tables into type-inferred JSON records, and score agent execution trajectories. Runs as a zero-dependency, stdlib-only MCP stdio server for Claude Desktop, Cursor, and Windsurf.MIT