# urlscan-mcp

urlscan.io as MCP tools, sized so an investigation can actually run inside a context window.

## The problem

One urlscan result document is 5.2 MB of request timings, response headers and cookie
values. Four MCP servers for urlscan already exist. Each wraps the API endpoint for
endpoint and hands that document to the model whole, which spends most of a context
window to answer "is this phishing".

Measured end to end on a real scan, 525 requests across 102 domains:

| | bytes |
|---|---|
| what urlscan sends | 5,215,387 |
| the same document as compact JSON | 3,426,957 |
| **what one `get_scan_result` call puts in the agent's context** | **13,092** |

**398× smaller.**

13,092 is the honest figure and it is larger than the object itself.
The shaped model is 5,528 bytes.
An MCP result carries structured content *and* a text rendering of the same object.
So the agent receives roughly twice the model size, and quoting 5,528 would be
measuring the wrong thing.

The ceiling is enforced, not hoped for. `shape_result` serialises its output and halves
the widest indicator list until it fits. Before that, a one-million-character page title
produced a 1.5 MB "summary". `test_shaped_result_fits_a_context_window` fails the build
against the same 5.2 MB document, stored in `tests/fixtures/result_live.json.gz`.

## Tools

| Tool | API key | |
|---|---|---|
| `pivot` | no | Scans touching a domain, IP, ASN, hash or URL. Picks the field for you |
| `search_scans` | no | ElasticSearch query with cursor paging |
| `get_screenshot` | no | The screenshot, as an image block a vision model reads |
| `get_scan_result` | yes | Verdict, IP, ASN, TLS issuer, contacted infrastructure |
| `get_dom` | yes | Windows around a search string in the captured DOM |
| `scan_url` | yes | Submit a URL. Returns a UUID, not a result |

## A worked investigation

19 suspected phishing URLs, triaged 2026-08-14. 17 scanned; 2 rejected by urlscan with
`400 DNS Error`, confirmed NXDOMAIN.

Four of the 19 were `*.icefactory.cl` subdomains. Pivoting on the landing IP turned four
URLs into a campaign:

```
pivot("186.64.119.45")        → 2,412 scans, AS52368 ZAM LTDA, CL
pivot("icefactory.cl")        → 2,451 scans, all hex-named subdomains
get_scan_result(uuid)         → "Adobe Acrobat - Secure PDF Document", pt-BR lure
get_dom(uuid, contains="<form")
    <form id="loginForm" method="POST"
          action="https://corpsanmarcello[.]com/report/pdf-reprt.php">
pivot("corpsanmarcello.com")  → 0 scans
```

The collector has **zero** urlscan coverage. It is never browsed, only posted to, so it
appears in no verdict, no screenshot, and no blocklist derived from scan data. Reading
the DOM was the only way to reach it.

**Two pairs show why the verdict field is not the answer.** `a77e56.icefactory.cl` scored
100 and `029d21.icefactory.cl` scored 0, same IP, same kit, same minute.
`finityvoice.weebly.com` ("Sign in to Xfinity.") scored 100 and
`ithelpdeskwebformnotice.weebly.com` ("Outlook") scored 0, both posting to Weebly's own
`formSubmitAjax.php`, so the credentials land in the attacker's Weebly inbox with no
suspicious external POST for a scanner to flag.

One caution, learned the same afternoon: pivoting on `188.114.96.3` and `216.198.79.131`
looked like shared infrastructure and was Cloudflare and Vercel anycast. An IP pivot on a
CDN edge returns 10,000 unrelated results.

## Why it is built this way

**Errors are instructions, not statuses.**
A scan submitted ten seconds ago returns HTTP 404, and so does a UUID that never existed.
The response cannot distinguish them.
So the message says both, and says when to stop polling.
An agent told "not found" abandons a scan twenty seconds from ready.
An agent told "still running" about a typo polls forever.
Each class in `errors.py` exists because the agent should do something different: poll,
stop, wait for a quota window, fix a variable.

**Shaped by default, raw on request.**
`detail="full"` attaches the whole document, and refuses above 1 MB because MCP delivers
it twice.
Hiding the raw document outright would make an agent conclude the field does not exist.

**A submission is never re-sent.**
Reads retry on 429 and 5xx. The one POST does not.
A timeout means the request arrived, so retrying burns a second submission quota and
creates a scan the caller is never told about.

**Indicators are escaped and quoted.**
An indicator lifted from an attacker-authored report cannot break out of its field.
A reviewer brute-forced all 1,114,112 Unicode codepoints through the query builder: none
produce an unescaped quote inside the wrapping quotes.

**Results are labelled untrusted, in the data.**
Titles, brands, server banners and DOM text were authored by the scanned site.
urlscan is a public corpus anyone can submit to, so an attacker can put chosen text in
front of an analyst who never visited their page.
This server also exposes `scan_url`, which fetches an arbitrary URL, so that text has an
outbound channel available to it.
Every result carries a `content_warning` field, because a caution in a tool description
is read once while the data arrives on every call.

**Quota is spent deliberately.**
The three key-gated tools fail before the network when no key is set, rather than
spending a round trip to be told 403.
`scan_url` defaults to `unlisted`: a public scan is visible to whoever operates the URL,
which during a live incident tells an attacker they were caught.

## What measurement changed

Each of these contradicts something documented, and none would have surfaced from
reading.

**Three endpoints need a key, not one.** The published API docs describe `result` and
`dom` as optional-auth. Both return `403 "You're not logged in!"` without one.

**403 is two different failures.** urlscan also returns 403 for a query naming a field
the account cannot search, with a message naming the field. Mapping every 403 to "missing
API key" sends the agent to fix a key that is already correct.

**And that error does not always arrive.** Only unknown *top-level* fields 403. A typo in
a dotted field, `page.asnn`, returns 200 with zero hits, which reads as "no scans match"
rather than "you asked wrongly". Three of the seven fields this server emits are dotted,
so each was verified against the live API.

**The SDK moved.** `mcp` 2.0.0 renamed `FastMCP` to `MCPServer`, moved `Context`, renamed
the tool schema attributes to snake_case, and deprecated the logging capability
`ctx.info()` uses. Code written from v1 memory would be wrong in four places.

## What adversarial review broke

Four independent reviews ran against this: correctness, security, claim verification, and
a second model. They found real defects, which is why they are listed.

| | |
|---|---|
| API key sent to a redirect target | httpx strips `Authorization` across origins and keeps custom headers. Redirects are no longer followed |
| `../../../user/quotas` as a `uuid` | Normalised to a different authenticated endpoint with the key attached. UUIDs are validated before reaching a path |
| `shape_result` raised on 6 input shapes | In a function documented "never raises". Now total across 20 hostile documents |
| `malicious: "false"` read as True | `bool("false")` is True, on the field that decides the verdict |
| `get_dom` returned 6× `max_chars` | The separator between windows was never charged against the budget |
| A counted match absent from the returned text | U+0130, the Turkish dotted capital I, is the only codepoint whose `.lower()` changes length, so offsets from a lowercased copy addressed the wrong place. Turkish phishing is in scope |
| A failed POST was re-sent | Duplicate scan submission |
| A 1M-character title produced a 1.5 MB summary | Per-item caps multiplied above the ceiling |
| Obfuscated loopback accepted | `http://127.0.0.1/` refused while `http://2130706433/` passed |
| No response size limit | 80 MB body accepted at 160 MB peak heap |

**The cause was one habit.** The robustness tests were written from failure shapes that
were imagined, so they agreed with the implementation. Mutation testing showed four
deliberate defects surviving the original suite, including a naive `bool()` replacing the
verdict coercion, because the string `"false"` was never fed to it.

The suite went from 132 tests to 278. Every test added corresponds to a finding someone
reproduced.

## Install

```bash
git clone https://github.com/Moussa93x/urlscan-mcp
cd urlscan-mcp
uv sync
```

Claude Code:

```bash
claude mcp add urlscan --env URLSCAN_API_KEY=your-key -- uv --directory /path/to/urlscan-mcp run urlscan-mcp
```

Claude Desktop, in `claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "urlscan": {
      "command": "uv",
      "args": ["--directory", "/path/to/urlscan-mcp", "run", "urlscan-mcp"],
      "env": { "URLSCAN_API_KEY": "your-key" }
    }
  }
}
```

Without `URLSCAN_API_KEY` the server starts, warns on stderr, and serves the three
keyless tools. A free account issues a key at <https://urlscan.io/user/signup>: 5,000
public scans, 1,000 searches and 10,000 retrievals a day.

| Variable | Default | |
|---|---|---|
| `URLSCAN_API_KEY` | none | Required by `scan_url`, `get_scan_result`, `get_dom` |
| `URLSCAN_BASE_URL` | `https://urlscan.io` | Plain HTTP refused except on loopback |
| `URLSCAN_TIMEOUT` | `30` | Seconds per request |
| `URLSCAN_MAX_SEARCH_SIZE` | `100` | Ceiling on `size`, clamped rather than rejected |
| `URLSCAN_MAX_RETRIES` | `3` | Total attempts, so `1` disables retrying |

## Layout

```
tools/            one module per investigative step
  submit.py       scan_url
  result.py       get_scan_result
  search.py       search_scans
  pivot.py        pivot
  artifacts.py    get_dom, get_screenshot
shaping.py        raw document -> models. Pure, total, byte-bounded
models.py         Pydantic response models; field descriptions are the schema
client.py         HTTP, auth, retries, rate-limit headers, error mapping
indicators.py     indicator detection and query construction
errors.py         one class per action an agent should take
config.py         environment, validated at the boundary
server.py         MCPServer + lifespan; the only entry point
```

The client knows about HTTP, the tools know about MCP, neither knows the other's
concerns. Adding the paid-tier endpoints is a new module in `tools/` plus one line in
`tools/__init__.py`.

## Testing

```bash
uv sync --extra dev
uv run pytest                          # 278 tests, no network
uv run pytest --cov=src/urlscan_mcp    # 97%
uv run python tests/smoke_stdio.py     # real subprocess over stdio
URLSCAN_LIVE=1 uv run pytest -m live   # 4 tests against the real API
```

The offline suite runs the full MCP protocol in-process through `Client(server)` against
`httpx.MockTransport`: no network, no subprocess, no port.

`smoke_stdio.py` spawns the console entry point as a subprocess and lists its tools,
because the in-process tests cannot catch a broken script or a stray write to stdout,
which would corrupt every message on the stdio transport.

The live suite submits a scan, polls it through the 404-while-running window, retrieves
it, greps the DOM and pulls the screenshot.

CI runs 3.10 through 3.13, pins each interpreter explicitly, and asserts the running
version matches the matrix entry. Without that, uv resolves any interpreter satisfying
`>=3.10` and four green jobs can all be the same Python.

Design notes, including what was considered and rejected: [docs/design.md](docs/design.md).

## Licence

MIT.