Oxylabs Web API — MCP Server

mcp-name: io.oxylabs/web-api-mcp

A self-hostable Model Context Protocol server that gives any MCP-capable agent live web access through the Oxylabs Web API.

All six are annotated readOnlyHint — nothing here writes anything — so clients can run them without prompting.

The skill ships with the server

Connecting the server is the whole install. The agent skill is bundled in the package and served as an MCP resource and a prompt, so a client that never adds the web-api-skills repo still gets the judgment for using these tools well — search to find and scrape to read, when JavaScript rendering earns its cost, what to do with an empty page, how to cite.

scripts/sync-skill.sh refreshes the bundled copy from the skills repo — the canonical copy lives there, and the two must not drift.

Related MCP server: searxNcrawl

Fitting the client's context

Oversized content is measured in tokens, not characters: 40 000 characters of English is about 10 000 tokens, but 40 000 characters of Chinese is about 40 000, and Claude Code, Claude Desktop and Cursor all reject a tool result over 25 000. The estimate is script-aware for that reason. The budget is OXYLABS_MAX_INLINE_TOKENS (default 10 000), and a client that knows its own limit can override it per request with an X-MCP-Max-Tokens header — 0 opts out entirely.

JavaScript rendering is a job, not a wait

run_js pages take 30-150 seconds and routinely outlive a tool call. So scrape(url, run_js=True) (and screenshots, and scrape_target with run_js in its params or an async/... endpoint) is sent to the API's own queue at /v1/async/..., which answers straight away:

{ "status": "pending", "request_id": "7504857924934611969", "note": "Queued. … call check_scrape('7504857924934611969') …" }

The agent polls check_scrape(request_id) — after ~30s, then every ~10s — and does other work in between. A pending reply says a slow render is normal, so it doesn't read as a stuck one. The API holds the result, so nothing about a job lives in this process: restarts and extra replicas do not lose it.

It says when a page needed rendering

The agent shouldn't have to guess whether an empty page is empty or just unrendered, and it shouldn't pay for a render on every page to find out. So a plain scrape that comes back with almost no visible text — or with a "please enable JavaScript" notice — is flagged:

{
  "content": "# Loading…",
  "content_thin": {
    "visible_chars": 9,
    "reason": "almost no text",
    "note": "…renders client-side. Retry the same call with run_js=True…"
  }
}

HTML is measured on its text, not its markup, so a 3 KB shell of <meta> tags still reads as thin. The flag is a hint, not a retry: rendering is slow and billed, and a genuinely short page would pay for it on every fetch. The threshold is 500 visible characters.

Structured extraction costs extra

scrape(url, extract="…") returns the fields you name as JSON instead of a page to read. The page is parsed by a model per call, which is billed above a plain scrape, so the server asks the user to approve each run over MCP elicitation. Clients that can't elicit get an error explaining why rather than a silent charge; OXYLABS_EXTRACT_APPROVAL=0 waives the prompt once the user has agreed to the cost.

Large pages

Web pages routinely exceed what is sensible to hand an agent in one response, so scrape does not return oversized content inline:

• Running locally (stdio): the page is written to a temp file. The agent gets the first 2 000 characters plus a path, and pulls the rest through read_scraped(path, offset) — reading only as far as it needs instead of paying for the whole page up front.

• Running remotely (HTTP): there is no shared filesystem, so a path would be useless. The content is truncated with a note stating the full length.

The threshold is OXYLABS_MAX_INLINE_TOKENS (default 10 000). read_scraped can only read files in the spill directory — it is deliberately not a general file reader.

Full API documentation: Oxylabs Web API docs

Requirements

• Python 3.10+ (built on FastMCP, installed with the package)

• An Oxylabs Web API key — in the Oxylabs dashboard, create a Web API instance and generate a key for it

Install

uv tool install git+https://github.com/thevastas/oxy_mcp

That puts oxylabs-web-api-mcp on your PATH in its own environment. pipx install git+https://github.com/thevastas/oxy_mcp does the same. Installing into a system Python usually fails — most are marked externally managed and refuse. To work on the server itself:

git clone https://github.com/thevastas/oxy_mcp.git
cd oxy_mcp
pip install -e .

server.json is the MCP registry manifest. It has no packages block yet — add one once the server is published somewhere installable, since a registry entry pointing at nothing is worse than no entry.

Use it locally (stdio)

Point your client at the installed command. Claude Code:

claude mcp add oxylabs-web-api \
  --env OXYLABS_WEB_API_KEY=your_api_key_here \
  -- oxylabs-web-api-mcp

Claude Desktop / Cursor / any client that reads a JSON config:

{
  "mcpServers": {
    "oxylabs-web-api": {
      "command": "oxylabs-web-api-mcp",
      "env": { "OXYLABS_WEB_API_KEY": "your_api_key_here" }
    }
  }
}

Self-host it (HTTP)

The HTTP transport is for running one shared server for a team or for agents that can't spawn local processes.

export OXYLABS_WEB_API_KEY=your_api_key_here
export MCP_ALLOWED_HOSTS='mcp.internal.example.com,localhost:*'
oxylabs-web-api-mcp --transport http --host 0.0.0.0 --port 8080

The endpoint is then http://<host>:8080/mcp.

Docker

docker build -t oxylabs-web-api-mcp .
docker run --rm -p 8080:8080 \
  -e OXYLABS_WEB_API_KEY=your_api_key_here \
  -e MCP_ALLOWED_HOSTS='localhost:*,mcp.internal.example.com' \
  oxylabs-web-api-mcp

MCP_ALLOWED_HOSTS is not optional. The HTTP transport turns on DNS-rebinding protection, so a server that doesn't declare its own hostname rejects every request with a Host header it doesn't recognise (421 Misdirected Request). List the hostname clients actually connect to. host:* matches any port on that host.

Configuration

Copy .env.example to .env for local use. On stdio the server reads that .env from its working directory: any OXYLABS_* name not already set in the environment is filled from there, so a project that keeps its key in .env needs no launcher wrapper. Real environment variables always win, and only OXYLABS_* names are read. Point it elsewhere with OXYLABS_ENV_FILE=/path/to/.env.

The key is read at request time and is never written to disk or logged.

Security notes

• Callers can bring their own key over HTTP. Send Authorization: Bearer <key> with each request and the server uses it for that call, so one deployment serves several callers on their own quota. OXYLABS_WEB_API_KEY in the server environment is the fallback when no header arrives.

• Cap the spend. OXYLABS_RATE_LIMIT=100/1h refuses tool calls past a sliding window, so a runaway agent loop cannot drain the key. Off by default.

• If you rely on that fallback, treat the endpoint as privileged. The server does no authentication of its own — anyone who can reach it spends the key it holds. Put it behind your VPN, an ingress with auth, or a service mesh. Don't expose it to the public internet.

• scrape fetches whatever URL it is given. If you expose this server to untrusted prompts, restrict egress at the network layer rather than trusting the caller.

• read_scraped is restricted to the spill directory. Don't point OXYLABS_SPILL_DIR at a directory holding anything else — it would make those files readable by the agent.

Development

pip install -e '.[dev]'
ruff check . && ruff format --check .
pytest                         # or: python tests/test_server.py
./scripts/sync-skill.sh        # refresh the bundled skill from web-api-skills

The tests are offline: error parsing, input validation, envelope trimming, endpoint-name handling, queueing and polling, and the extract approval gate. CI runs the same on 3.10, 3.12 and 3.13.

License

MIT