mcp-fetch

A minimal MCP (Model Context Protocol) server that exposes a single tool: fetch_page.

It supports:

• Browser automation via Playwright (dynamic content, scrolling, waiting for selectors)

• Basic anti-bot shaping (User-Agent, mouse movement, randomized request delays, proxy rotation)

• Large responses via chunking with a transfer_id cursor

• On-disk chunk cache so agents can request more chunks later

Run (uvx)

From this directory:

This is an MCP stdio server: it typically does not print anything until an MCP client connects and sends requests. If you run it directly in a terminal, it should stay running and wait for stdin from an MCP host.

For interactive/manual testing, it will auto-run an HTTP server when stdin is a TTY:

• Open: http://127.0.0.1:8000/mcp

• Override: MCP_FETCH_TRANSPORT=stdio|http, MCP_FETCH_PORT=8000

• Or flags: mcp-fetch --transport http --port 8000

Client config note

If you use uvx, you must include the provided command name at the end. This works:

Tool: fetch_page

Typical flow:

1. Call with url (gets chunk 0 and transfer_id)

2. If you need more, call again with transfer_id and offset=next_offset

Notes:

• This server allows private/internal network access by default (your requested behavior).

• Only http:// and https:// URLs are allowed.

• Use headers for request headers.

• Set to_markdown=false to return raw HTML instead of Markdown.

Tests

Run unit tests:

There is a Playwright integration test (dynamic DOM). It is disabled by default:

• Run with: MCP_FETCH_RUN_PLAYWRIGHT_TESTS=1 python -m unittest discover -s tests -v

Configuration (env)

• MCP_FETCH_CACHE_DIR (default: ./.mcp-fetch-cache)

• MCP_FETCH_TTL_SECONDS (default: 1800)

• MCP_FETCH_MAX_CACHE_BYTES_TOTAL (default: 536870912) (512 MiB)

• MCP_FETCH_MAX_SINGLE_TRANSFER_BYTES (default: 209715200) (200 MiB)

• MCP_FETCH_WAIT_CHUNK_TIMEOUT_SECONDS (default: 30)

• MCP_FETCH_EXPOSE_SHUTDOWN (default: unset) Set to 1 to expose the shutdown tool.