# mcp-ui-web-recon

An MCP server (plus a CLI for CI) that scans web UI screens and writes **one `.properties` locator file per screen**. Each element gets an XPath that has been checked against the live DOM to match exactly one node.

```properties
# --- form:loginForm ---
# input[email] | label-for | high
txt_emailAddress=//input[@id=//label[normalize-space()='Email address']/@for]
# button[submit] | test-attribute | high
btn_loginSubmit=//*[@data-testid='login-submit']
#   alt (text): //button[normalize-space()='Sign in']

# --- section:Pro plan ---
# button | anchor>text | medium
btn_addToCart_2=//section[.//h2[normalize-space()='Pro plan']]//button[normalize-space()='Add to cart']

# --- page ---
# button | context-text | medium
btn_buy=//tr[.//td[normalize-space()='Keyboard']]//button[normalize-space()='Buy']
```

Use it from Claude Code, Claude Desktop, VS Code / Copilot, Cursor or any other MCP client. You can also run it straight from a pipeline with `recon-cli`.

It also handles:

- **apps behind a login**: form, SSO, TOTP MFA, manual SSO/MFA handoff, Basic/NTLM, tokens, with session reuse;
- **screens you reach by following test-case steps**: modals, wizards, checkout, iframes;
- **apps on an internal network**: proxy with auth/PAC/env/system settings, internal DNS, internal CA, mTLS, a remote browser on a jump host, or the MCP server itself running inside the network over HTTP.

| Doc | For |
|-----|-----|
| [docs/AUTH_AND_NETWORK.md](docs/AUTH_AND_NETWORK.md) | How to set up login, test-case flows and internal-network access |
| [docs/INTAKE_CHECKLIST.md](docs/INTAKE_CHECKLIST.md) | **What you must provide** (URLs, test accounts, MFA, proxy, certificates …) |
| [docs/REQUIREMENTS.md](docs/REQUIREMENTS.md) | Use cases, requirements, acceptance criteria, traceability to tests |
| [examples/](examples) | One annotated config per scenario |

---

## Install

Requires Node.js 18.18+.

```bash
git clone https://github.com/ginglove/mcp-ui-web-recon.git
```

```bash
cd mcp-ui-web-recon && npm install && npx playwright install chromium && npm run build
```

> If Chromium is already installed somewhere else, set `RECON_CHROMIUM_PATH=/path/to/chrome` and skip `playwright install`.

### Register with your MCP client

**Claude Code** (from the directory where you want the output):

```bash
claude mcp add ui-web-recon -e RECON_OUTPUT_DIR=./recon-output -- node /abs/path/to/mcp-ui-web-recon/dist/index.js
```

**Claude Desktop / Cursor / VS Code** (`claude_desktop_config.json`, `.cursor/mcp.json`, `.vscode/mcp.json` → `"servers"`):

```json
{
  "mcpServers": {
    "ui-web-recon": {
      "command": "node",
      "args": ["/abs/path/to/mcp-ui-web-recon/dist/index.js"],
      "env": { "RECON_OUTPUT_DIR": "/abs/path/to/your-test-project/src/test/resources/locators" }
    }
  }
}
```

A ready-to-use `.mcp.json` is included for working inside this repo.

---

## The prompt

The server ships an MCP prompt called **`scan-web-ui`** (in Claude Code: `/mcp__ui-web-recon__scan-web-ui`). It takes `urls`, `outputDir`, `scope` and `loginSteps`. It tells the agent to:

1. log in if needed, then save the session for reuse,
2. scan a list of URLs with `recon_scan_urls`, or discover screens with `recon_crawl`,
3. reach modals, tabs and wizard steps with `recon_interact`, then scan them as their own screens,
4. check every file with `recon_verify`,
5. report LOW-confidence locators and suggest a `data-testid` for each.

If your client doesn't support MCP prompts, a copy-paste version is in [`prompts/scan-web-ui.md`](prompts/scan-web-ui.md).

Example request to the agent:

> Scan https://staging.myshop.com/login, /products, /cart and /checkout into `src/test/resources/locators`. Log in first as `$QA_USER` / `$QA_PASS`. Also scan the "Add address" modal on checkout as `CheckoutAddAddressModal`.

---

## Tools

| Tool | Purpose |
|------|---------|
| `recon_open` | Launch the browser (headless or headed, viewport, locale, `storageStatePath`) and navigate. |
| `recon_interact` | Run steps on the live page: `goto`, `click`, `dblclick`, `hover`, `fill`, `select`, `check`, `uncheck`, `press`, `waitForXPath`, `wait`. Used for login and for reaching dialogs, tabs and wizard steps. Typed values are never echoed back. |
| `recon_scan_screen` | Scan a URL, **or the current page if `url` is omitted**, and write `<ScreenName>.properties`. |
| `recon_scan_urls` | Batch scan: a list of URLs or `{url, screenName, waitForXPath}` objects, one file each. |
| `recon_crawl` | Breadth-first crawl of same-origin links (`maxPages`, `maxDepth`, `include`/`exclude` regexes). Skips logout/delete links, binary files and HTTP errors. |
| `recon_verify` | Re-check every XPath in a `.properties` file against a page, including iframe sections. Reports unique / ambiguous / missing / invalid. |
| `recon_save_session` | Save cookies and localStorage to a JSON file so later scans start logged in. |
| `recon_close` | Close the browser. |

**Config-driven tools** (login, test-case flows, internal network):

| Tool | Purpose |
|------|---------|
| `recon_init_config` | Write a starter `recon.config.json` for a scenario: `public`, `form-login`, `sso-mfa`, `test-case-flow`, `internal-proxy` or `remote-browser`. Also writes `.env.recon.example`, the JSON schema and `.gitignore` entries. |
| `recon_validate_config` | List exactly what is still missing: placeholders, env values and secrets, files, success marker. With `checkAccess`, it also opens the site through the configured network and explains any failure. |
| `recon_check_access` | Diagnose one URL: DNS/VPN, proxy, 407, certificate, mTLS, timeout, 401/403, login wall. |
| `recon_login` | Log in using the config's auth (form / steps+TOTP / manual / storageState / token) and save or reuse the session. |
| `recon_run_config` | Run everything: public screens logged out, then login, then screens with their pre-steps, then test-case flows with `scan` checkpoints. |

`recon_open` accepts `configFile` (+ `login: true`) and `proxy`. `recon_interact` accepts `frame`, `upload`, `fillTotp`, `waitForUrl` and `scan` steps, so an agent can follow a test case and capture screens as it goes.

Prompts: `scan-web-ui`, **`recon-from-test-case`** (paste a test case and it captures every screen the test visits), **`recon-setup`** (interviews you for login and network details and builds a validated config).

Resources: `recon://xpath-strategy` (the strategy table below) and `recon://intake-checklist` (what you must provide).

### Scan options (all optional)

| Option | Default | Meaning |
|--------|---------|---------|
| `scope` | `interactive+content` | `interactive` = controls only; `interactive+content` adds headings, labels, images, tables and text; `all` = every element that can be identified |
| `testAttributes` | data-testid, data-test, data-qa, data-cy, … | Your project's test attributes, highest priority first |
| `includeHidden` | `false` | Also list hidden elements |
| `includeFrames` | `true` | Scan iframes into separate sections |
| `maxElements` | `500` | Limit per document |
| `waitForXPath`, `settleMs` | – / 500 | Wait for SPA content before scanning |
| `autoScroll` | `true` | Scroll first so lazy-loaded content is in the DOM |
| `keyPrefix` | `false` | Prefix keys with the screen name (`loginPage.btn_signIn`), useful when all files are merged into one |
| `escapeUnicode` | `true` | Write non-ASCII as `\uXXXX`, so `java.util.Properties.load(InputStream)` reads it correctly |
| `includeAlternatives` | `true` | Write fallback XPaths as `# alt` comments, for self-healing or manual review |
| `writeJson` | `false` | Also write `<ScreenName>.json` with full metadata (tag, text, rect, strategy, alternatives) |

---

## XPath strategy

For each element, the engine builds candidates and runs each one with `document.evaluate` in the page. A candidate is kept **only if it matches exactly one node, and that node is the target**. The highest-ranked candidate that passes is used.

| # | Strategy | Example | Confidence |
|---|----------|---------|-----------|
| 1 | Dedicated test attribute | `//*[@data-testid='login-submit']` | high |
| 2 | Stable `id` | `//input[@id='username']` | high |
| 3 | `name` (+`value` for radios) | `//input[@name='gender' and @value='f']` | high |
| 4 | `aria-label` | `//button[@aria-label='Close']` | high |
| 5 | Associated `<label>` (for / wrapping / aria-labelledby) | `//input[@id=//label[normalize-space()='Email']/@for]` | high |
| 6 | `placeholder` / `alt` / `title` | `//input[@placeholder='Search']` | medium |
| 7 | Visible text | `//button[normalize-space()='Sign in']` | medium |
| 8 | role+text, type+value, href | `//*[@role='tab' and normalize-space()='Billing']` | medium |
| 9 | Anchored to a unique ancestor (id / testid / aria-label / landmark / heading) | `//form[@id='login']//button[normalize-space()='Submit']` | medium |
| 10 | Context text (table rows, cards, list items) | `//tr[.//td[normalize-space()='Mouse']]//button[normalize-space()='Buy']` | medium |
| 11 | Stable CSS class | `//div[contains(concat(' ',normalize-space(@class),' '),' card ')]` | low |
| 12 | Index into the best ambiguous match | `(//button[normalize-space()='Add'])[2]` | low |
| 13 | Absolute path (last resort) | `/html/body/div[2]/main/button` | low |

Rules built into the engine:

- **Dynamic values are rejected**, so they never appear in a locator: `ember1234`, React `:r5:`, GUIDs, hashes, `mui-123`, `j_id…`, long numbers, CSS-in-JS classes (`css-1x2y3z`, `sc-…`, `jss12`), CSS-module suffixes, Tailwind utilities, and state classes (`active`, `is-open`).
- `normalize-space()` is used instead of `text()`, so whitespace, line breaks and nested tags don't break matches.
- Quotes are handled correctly in XPath 1.0: `Don't "panic"` becomes `concat('Don',"'",'t "panic"')`.
- SVG and MathML elements are matched with `local-name()`, because a plain `//svg` matches nothing in an HTML document.
- Text containing digits (prices, counters, dates) ranks lower. Long text uses `contains()`.
- Custom-styled checkboxes and radios that are visually hidden are kept if their label is visible.
- **Iframes** get their own section, headed by the iframe XPath you switch into. Nested frames are chained with `>>`.
- Shadow DOM can't be reached with XPath. It is reported as a warning in the file header.

### Key naming

Keys are `<prefix>_<camelCaseName>`. The name comes from, in order: test id, aria-label, label, placeholder, value, text, title, alt, name, then id. Diacritics are folded to ASCII (`Đăng nhập` → `dangNhap`). Duplicate keys get `_2`, `_3`, and so on.

`btn` button · `txt` text input/textarea · `lnk` link · `chk` checkbox/switch · `rdo` radio · `ddl` select/combobox · `opt` option · `tab` · `mnu` menu item · `hdr` heading · `lbl` label/text · `img` · `tbl` table · `cell` · `form` · `frm` iframe · `dlg` dialog · `nav` · `file` · `sld` slider

---

## Using the files in your framework

**Java / Selenium**

```java
Properties loc = new Properties();
try (InputStream in = getClass().getResourceAsStream("/locators/LoginPage.properties")) { loc.load(in); }
driver.findElement(By.xpath(loc.getProperty("txt_emailAddress"))).sendKeys(user);
driver.findElement(By.xpath(loc.getProperty("btn_loginSubmit"))).click();
// iframe section: driver.switchTo().frame(driver.findElement(By.xpath("//iframe[@name='newsletter']")));
```

**Playwright (TS)**: `page.locator(\`xpath=${props["btn_loginSubmit"]}\`)`. **Robot Framework / Katalon / Serenity**: load the file as a variable or resource file and use `xpath=${btn_loginSubmit}`.

---

## CLI (CI / no LLM)

Config-driven (login, flows, proxy). See [docs/AUTH_AND_NETWORK.md](docs/AUTH_AND_NETWORK.md):

```bash
npx recon-cli init form-login
```

```bash
npx recon-cli validate recon.config.json --check-access
```

```bash
npx recon-cli login recon.config.json --manual
```

```bash
npx recon-cli run recon.config.json
```

`validate` lists what you still need to provide. `login --manual` is a one-time human login for SSO/MFA; it saves `.auth/state.json`. `run` exits with code 1 if any login, screen or flow fails.

Single URLs:

```bash
npx recon-cli scan https://app.example.com/login https://app.example.com/home --out locators
```

```bash
npx recon-cli crawl https://app.example.com/ --max-pages 30 --max-depth 2 --exclude "admin" --out locators
```

```bash
npx recon-cli verify locators/LoginPage.properties https://app.example.com/login
```

Flags: `--config recon.config.json` (applies its proxy, certificates, DNS, remote browser and saved session), `--proxy URL|env|system`, `--proxy-bypass`, `--scope`, `--name`, `--state storage.json`, `--headed`, `--json`, `--utf8`, `--key-prefix`, `--no-frames`, `--no-alternatives`, `--wait-xpath`, `--settle`, `--max-elements`. `verify` exits with code 1 if any locator is not unique, so you can use it as a nightly check for locator drift.

---

## Running inside an internal network (HTTP mode)

```bash
RECON_HTTP_TOKEN="$(openssl rand -hex 24)" node dist/index.js --http --host 0.0.0.0 --port 8931
```

```bash
claude mcp add --transport http ui-web-recon http://internal-host:8931/mcp --header "Authorization: Bearer <token>"
```

The token is mandatory. Each MCP session gets its own browser. Put the port behind TLS and a firewall.

## Security notes

- `recon_save_session` and `--state` files contain live session cookies. Keep them out of git (`.auth/` is a good place to put them, and add it to `.gitignore`).
- Credentials, TOTP seeds, proxy passwords and certificate passphrases live in environment variables or `.env.recon`. The config refers to them as `${env:NAME}`. Validation reports missing values by name only, and typed values are never echoed back.
- Session files are written with mode 0600. `recon-cli init` adds `.auth/`, `.env.recon` and `certs/` to `.gitignore`.
- HTTP mode needs a bearer token of at least 16 characters and listens on 127.0.0.1 unless you pass `--host`.
- The crawler stays on the start URL's origin and skips logout/delete/unsubscribe links by default.

## Development

```bash
npm test
```

```bash
npm run inspect
```

`npm test` runs the vitest suite against local fixtures. `npm run inspect` opens the MCP Inspector against the built server.

Layout:

- `src/engine/`: `inPage.ts` (the in-browser XPath engine), `naming.ts`, `properties.ts`, `scanner.ts`, `crawler.ts`, `actions.ts` (steps, TOTP), `auth.ts`, `browser.ts` (proxy, mTLS, DNS, remote browser)
- `src/config/`: `schema.ts`, `load.ts` (placeholders, envFile), `runner.ts`, `diagnose.ts`, `templates.ts`, `init.ts`
- `src/server.ts` (MCP tools, prompts, resources), `src/guides.ts`, `src/index.ts` (stdio / HTTP), `src/cli.ts`

`npm run gen:docs` regenerates `docs/INTAKE_CHECKLIST.md`, `examples/`, `prompts/` and the JSON schema from the source.