# local-web-mcp

Version 0.3.0 · AGPL-3.0

*This tool is developed with support from AI, but it has been evaluated by a
human before upload.*

An MCP server that gives Claude a **fallback** web fetcher running on your own
machine. When the built-in fetcher is blocked, this one tries from your IP, your
connection, and optionally your logged-in session.

One tool: `fetch_url_locally`.

## The problem it solves

Hosted fetchers run from datacentre IP ranges, which a lot of sites refuse
outright. A local stdio MCP server does not: the client spawns the process on
your machine, so requests originate from your ordinary connection.

Verified working on sites the hosted fetcher could not read, including CNBC,
Politiken and Bloomberg.

## What it does not change

Worth stating plainly, because the framing invites overclaiming.

- Model inference still runs on Anthropic's servers.
- Fetched page text is still sent to the model as tool output. Anything read
  through an authenticated session goes with it.
- It changes *who talks to the website*, not *where the model runs*.

## The hard boundary

**Can the page be read without executing JavaScript?**

If yes, this handles it. If no, nothing at the header, TLS or cookie layer will
get there, and no amount of User-Agent tuning helps. Sites using DataDome,
Cloudflare Turnstile, PerimeterX and similar serve a challenge that must be
*executed* to pass. For those, use a browser-based tool or open the page
yourself.

This is a boundary to know, not a bug to fix.

## Verdicts

Every response is labelled. The dangerous case is not an obvious 403, it is a
**200 that is not the page**: a bot interstitial, a consent wall, an empty
client-rendered shell. Those get summarised as though they were the article.

| Verdict | Meaning | What to do |
|---|---|---|
| `ok` | Real content | Use it |
| `challenge` | Bot protection detected | Browser required, stop |
| `blocked` | Refused outright (403), usually bot detection | Browser required, stop |
| `js_required` | Client-rendered shell, no content without JS | Browser required, stop |
| `login_required` | 401, login wall, or subscription wall | Cookies may help |
| `http_error` | Any other non-2xx status (404, 429, 5xx) | Retry later for 429/5xx only |
| `empty` | Under 200 characters extracted | Failed read, not an empty page |

Anything other than `ok` attaches an explicit warning telling the model not to
treat the body as content.

Body evidence outranks the status code. A publisher that serves a paywalled
article as HTTP 403 is reported as `login_required`, not `blocked`, because
the useful advice there is to export cookies rather than to reach for a
browser.

Detection matches on **vendor infrastructure** rather than wording: challenge
pages are localised, so `captcha-delivery.com` is a reliable signal where
"verifying your device" is not. Vendor cookies in response headers count as
evidence when paired with a refusal status.

## The tool

`fetch_url_locally(url, max_chars=20000, use_session=true)`

| Parameter | Type | Default | Purpose |
|---|---|---|---|
| `url` | string | required | The page to fetch. `http` and `https` only |
| `max_chars` | integer | `20000` | Cap on returned text, clamped to 200000. Truncation is reported in the header |
| `use_session` | boolean | `true` | Send cookies. Pass `false` to test whether a page is genuinely public |

The reply is a short header followed by the extracted text:

```
URL: https://example.com/article
Status: 200
Verdict: ok
Title: The headline
Session: authenticated (the user's cookies were sent)

The article text...
```

`Session:` appears only when cookies were actually loaded and scoped to that
host. A verdict other than `ok` adds a `WARNING:` line telling the model not to
treat the body as content.

Failures come back as an error naming the cause, for example
`[blocked_host] Host '192.168.1.5' resolves to a private or internal address`.
Codes: `invalid_url`, `blocked_scheme`, `blocked_host`, `dns_failure`, `timeout`,
`too_many_redirects`, `upstream_error`, `invalid_argument`.

## Requirements

- Python 3.11+ (uses `asyncio.timeout`). With uv you do not need to install
  it yourself; uv fetches a suitable interpreter
- Claude Desktop or Claude Code. Stdio servers do not work in the browser or the
  mobile app, which require a publicly reachable HTTPS endpoint.

## Install

With [uv](https://docs.astral.sh/uv/), which fetches a suitable Python and the
dependencies itself:

```bash
uv --directory /absolute/path/to/local-web-mcp run local-web-mcp
```

**Run that once in a terminal before registering the server.** The first run
resolves and builds the environment, which takes long enough that a client
launching it cold may give up and report the server as failed even though the
configuration is correct.

It prints `Starting local-web` and then waits for a client on stdin, which is
what a working server looks like. Once you see that line, the environment is
built: press Ctrl+C and carry on.

Or with a plain virtual environment:

```bash
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
```

## Register with Claude Desktop

Add to `claude_desktop_config.json` (macOS:
`~/Library/Application Support/Claude/`, Linux: `~/.config/Claude/`):

```json
{
  "mcpServers": {
    "local-web": {
      "command": "uv",
      "args": [
        "--directory",
        "/absolute/path/to/local-web-mcp",
        "run",
        "local-web-mcp"
      ]
    }
  }
}
```

If `uv` is not on the PATH that the client sees, use its absolute path as
`command`. The virtual-environment equivalent is:

```json
{
  "mcpServers": {
    "local-web": {
      "command": "/absolute/path/to/.venv/bin/python",
      "args": ["/absolute/path/to/local_web_mcp.py"]
    }
  }
}
```

Claude Code:

```bash
claude mcp add local-web -- uv --directory /absolute/path/to/local-web-mcp run local-web-mcp
```

## Configuration

All optional. Every setting has a working default.

| Variable | Default | Purpose |
|---|---|---|
| `LOCALWEB_COOKIE_FILE` | unset | Path to a cookie jar |
| `LOCALWEB_COOKIE_DOMAINS` | unset | Domains the jar may be used for. Set this whenever you set a cookie file |
| `LOCALWEB_ALLOW_PRIVATE` | `false` | Allow private, loopback and link-local targets |
| `LOCALWEB_ALLOWLIST` | empty | Hostnames exempt from the IP check |
| `LOCALWEB_USER_AGENT` | Chrome UA | Sent on every request |
| `LOCALWEB_ACCEPT_LANGUAGE` | `da,en-GB;q=0.9,en;q=0.8` | Language preference. Sites use it to pick a language and sometimes a regional edition |
| `LOCALWEB_CONNECT_TIMEOUT` | `5` | Seconds |
| `LOCALWEB_READ_TIMEOUT` | `20` | Seconds |
| `LOCALWEB_TOTAL_TIMEOUT` | `40` | Hard ceiling per call |
| `LOCALWEB_MAX_BYTES` | `3000000` | Response body cap |
| `LOCALWEB_MAX_REDIRECTS` | `5` | Redirect hops, shared with meta refreshes |

The default `Accept-Language` asks for Danish first. That is a deliberate
default rather than a neutral one, so set `LOCALWEB_ACCEPT_LANGUAGE` if you
want English or anything else:

```json
"env": { "LOCALWEB_ACCEPT_LANGUAGE": "en-GB,en;q=0.9" }
```

## Session cookies

Point `LOCALWEB_COOKIE_FILE` at an export from a browser extension. The loader
accepts Netscape `cookies.txt` with any header or none, and JSON exports from the
common extensions. See `cookies.txt.example`.

**Always set `LOCALWEB_COOKIE_DOMAINS`.** Without it the whole jar is live and
any fetch can carry an unrelated session. With it, every cookie outside those
domains is dropped at load time, before the jar reaches a client. Matching covers
subdomains (`example.com` covers `www.example.com`) but not lookalikes
(`evil-example.com`).

Three things to keep in mind:

1. Content read through your session is sent to the model as tool output.
2. A cookie file is a credential file. `chmod 600`, and it is gitignored here.
3. Cookies expire. A previously working source returning `login_required` is
   usually a stale export rather than lost access.

Pass `use_session=false` to check whether a page is genuinely public.

## Reaching a local network

Private addresses are blocked by default, and deliberately so. The model chooses
URLs partly from text it has just read, so a fetched page can attempt to steer it
at `192.168.1.1`. The blocklist means a prompt injection in page content cannot
turn this into a network scanner.

Prefer a narrow allowlist over the blanket switch:

```json
"env": { "LOCALWEB_ALLOWLIST": "nas.local,nas" }
```

## Security

- Scheme restricted to http and https; URLs carrying credentials rejected
- Hostname resolved and **every** returned address validated before the request
- IPv6-mapped IPv4 (`::ffff:10.0.0.1`) unwrapped before checking
- Cloud metadata endpoints blocked unconditionally: the hostname is resolved
  before both the allowlist and `ALLOW_PRIVATE`, so neither can open them
- Redirects and meta refreshes followed manually, revalidated at each hop
- Cookie counts logged, never names, values or domains

## Known limitations

- **No JavaScript.** The hard boundary above.
- **DNS rebinding.** The hostname is resolved for validation, then resolved again
  on connect. Closing that gap needs the connection pinned to the validated IP.
  Worth doing before enabling `ALLOW_PRIVATE` on an untrusted network.
- **Challenge detection is signature-based** and will drift as vendors change
  their markup. Run `inspect_response.py` if a verdict looks wrong.
- **HTML-to-text extraction is dependency-free and crude.** `trafilatura` would
  be markedly better for serious article extraction.

## Files

| File | Purpose |
|---|---|
| `local_web_mcp.py` | The server |
| `cookies.py` | Tolerant cookie loader. **Required**, imported by the server |
| `inspect_response.py` | Diagnostic: dumps what the fetcher actually receives |
| `unit_test.py` | Verdicts, target validation, cookie loading and scoping |
| `smoke_test.py` | Blocked targets and one live fetch, over stdio |
| `cookies.txt.example` | Annotated template for a cookie jar |
| `pyproject.toml` | Package metadata and the `local-web-mcp` entry point |
| `uv.lock` | Pinned dependency versions for reproducible installs |
| `requirements.txt` | Runtime dependencies for the plain-venv route |
| `.gitignore` | Excludes the venv, caches, and any cookie file |
| `LICENSE` | Full AGPL-3.0 text |

## Diagnosing a wrong verdict

```bash
uv run python inspect_response.py https://example.com/article
```

Prints the status, the interesting headers, the start of the body, and then runs
the server's own detection against it. **Always check this before changing
detection logic.** A browser and this fetcher are frequently served entirely
different responses, so what you see on screen is not evidence about what the
tool received.

## Tests

```bash
uv run python unit_test.py
uv run python smoke_test.py
```

`unit_test.py` is offline and exits non-zero on failure, so it works as a
pre-commit gate. `smoke_test.py` drives the server over stdio and makes one live
request, so it needs a network connection. Run both from the repository root.

On the plain-venv route, substitute `.venv/bin/python` for `uv run python`.

## Licence

Copyright (C) 2026 David Lindholm.

GNU Affero General Public License v3.0 or later. See [LICENSE](LICENSE).

This program is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.