Skip to main content
Glama
thevastas

Oxylabs Web API MCP Server

by thevastas

Oxylabs Web API — MCP Server

mcp-name: io.oxylabs/web-api-mcp

A self-hostable Model Context Protocol server that gives any MCP-capable agent live web access through the Oxylabs Web API.

Tool

What it does

search

Search the live web, returns ranked organic results (title, description, URL)

scrape

Read a single URL as Markdown by default, including JS-heavy and bot-protected pages

extract

Pull named fields off a page as JSON, no selectors. Billed above a scrape, so the user approves each run

check_scrape

Collect the result of a JavaScript-rendering job

read_scraped

Read a large page that was offloaded to disk, in chunks

list_scrapers

List the scrape endpoints the API implements, or describe one's parameters

scrape_target

Call a target-specific scrape endpoint with its own parameters

All seven are annotated readOnlyHint — nothing here writes anything — so clients can run them without prompting.

The skill ships with the server

Connecting the server is the whole install. The agent skill is bundled in the package and served as an MCP resource and a prompt, so a client that never adds the web-api-skills repo still gets the judgment for using these tools well — search to find and scrape to read, when JavaScript rendering earns its cost, what to do with an empty page, how to cite.

oxylabs://skill/web-api

The skill itself, as Markdown

prompt web_research

Takes a question, hands the agent the task plus the skill

scripts/sync-skill.sh refreshes the bundled copy from the skills repo — the canonical copy lives there, and the two must not drift.

Related MCP server: grounder-mcp

Fitting the client's context

Oversized content is measured in tokens, not characters: 40 000 characters of English is about 10 000 tokens, but 40 000 characters of Chinese is about 40 000, and Claude Code, Claude Desktop and Cursor all reject a tool result over 25 000. The estimate is script-aware for that reason. The budget is OXYLABS_MAX_INLINE_TOKENS (default 10 000), and a client that knows its own limit can override it per request with an X-MCP-Max-Tokens header — 0 opts out entirely.

JavaScript rendering is a job, not a wait

run_js pages take 30-150 seconds and routinely outlive a tool call. So scrape(url, run_js=True) (and extract, and scrape_target with run_js in its params) returns a job id straight away:

{ "job_id": "9f3c1a20b7d4", "status": "running", "url": "https://example.com" }

The agent polls check_scrape(job_id) — after ~30s, then every ~10s — and does other work in between. Each reply carries elapsed_seconds and says whether the job is still inside the normal 150s window, so a slow render doesn't read as a stuck one. Results are kept for OXYLABS_JOB_TTL_MINUTES (default 60), long after the job itself has finished.

It says when a page needed rendering

The agent shouldn't have to guess whether an empty page is empty or just unrendered, and it shouldn't pay for a render on every page to find out. So a plain scrape that comes back with almost no visible text — or with a "please enable JavaScript" notice — is flagged:

{
  "content": "# Loading…",
  "content_thin": {
    "visible_chars": 9,
    "reason": "almost no text",
    "note": "…renders client-side. Retry the same call with run_js=True…"
  }
}

HTML is measured on its text, not its markup, so a 3 KB shell of <meta> tags still reads as thin. The flag is a hint, not a retry: rendering is slow and billed, and a genuinely short page would pay for it on every fetch. The threshold is 500 visible characters.

Jobs live in the server process: they do not survive a restart, and they are not shared between HTTP replicas. Run one replica, or give it sticky sessions.

Structured extraction costs extra

extract(url, prompt) returns the fields you name as JSON instead of a page to read. The page is parsed by a model per call, which is billed above a plain scrape, so the server asks the user to approve each run over MCP elicitation. Clients that can't elicit get an error explaining why rather than a silent charge; OXYLABS_EXTRACT_APPROVAL=0 waives the prompt once the user has agreed to the cost.

Large pages

Web pages routinely exceed what is sensible to hand an agent in one response, so scrape does not return oversized content inline:

  • Running locally (stdio): the page is written to a temp file. The agent gets the first 2 000 characters plus a path, and pulls the rest through read_scraped(path, offset) — reading only as far as it needs instead of paying for the whole page up front.

  • Running remotely (HTTP): there is no shared filesystem, so a path would be useless. The content is truncated with a note stating the full length.

The threshold is OXYLABS_MAX_INLINE_TOKENS (default 10 000). read_scraped can only read files in the spill directory — it is deliberately not a general file reader.

Full API documentation: Oxylabs Web API docs

Requirements

  • Python 3.10+ (built on FastMCP, installed with the package)

  • An Oxylabs Web API key — in the Oxylabs dashboard, create a Web API instance and generate a key for it

Install

uv tool install git+https://github.com/thevastas/oxy_mcp

That puts oxylabs-web-api-mcp on your PATH in its own environment. pipx install git+https://github.com/thevastas/oxy_mcp does the same. Installing into a system Python usually fails — most are marked externally managed and refuse. To work on the server itself:

git clone https://github.com/thevastas/oxy_mcp.git
cd oxy_mcp
pip install -e .

server.json is the MCP registry manifest. It has no packages block yet — add one once the server is published somewhere installable, since a registry entry pointing at nothing is worse than no entry.

Use it locally (stdio)

Point your client at the installed command. Claude Code:

claude mcp add oxylabs-web-api \
  --env OXYLABS_WEB_API_KEY=your_api_key_here \
  -- oxylabs-web-api-mcp

Claude Desktop / Cursor / any client that reads a JSON config:

{
  "mcpServers": {
    "oxylabs-web-api": {
      "command": "oxylabs-web-api-mcp",
      "env": { "OXYLABS_WEB_API_KEY": "your_api_key_here" }
    }
  }
}

Self-host it (HTTP)

The HTTP transport is for running one shared server for a team or for agents that can't spawn local processes.

export OXYLABS_WEB_API_KEY=your_api_key_here
export MCP_ALLOWED_HOSTS='mcp.internal.example.com,localhost:*'
oxylabs-web-api-mcp --transport http --host 0.0.0.0 --port 8080

The endpoint is then http://<host>:8080/mcp.

Docker

docker build -t oxylabs-web-api-mcp .
docker run --rm -p 8080:8080 \
  -e OXYLABS_WEB_API_KEY=your_api_key_here \
  -e MCP_ALLOWED_HOSTS='localhost:*,mcp.internal.example.com' \
  oxylabs-web-api-mcp

MCP_ALLOWED_HOSTS is not optional. The HTTP transport turns on DNS-rebinding protection, so a server that doesn't declare its own hostname rejects every request with a Host header it doesn't recognise (421 Misdirected Request). List the hostname clients actually connect to. host:* matches any port on that host.

Configuration

Variable

Default

Purpose

OXYLABS_WEB_API_KEY

(required on stdio)

Web API key, sent as Authorization: Bearer <key>. Over HTTP a per-request Authorization: Bearer header takes precedence

OXYLABS_BASE_URL

https://webapi.oxylabs.io

Override for staging or a proxy

OXYLABS_TIMEOUT

120

Per-request timeout in seconds

OXYLABS_RETRIES

2

Retries on a transient 429/500/502/503/504

OXYLABS_RATE_LIMIT

(off)

Cap this server's own spend, e.g. 100/1h, 50/30m

OXYLABS_JOB_TTL_MINUTES

60

How long a finished job's result stays pollable

OXYLABS_EXTRACT_APPROVAL

1

Set to 0 to skip the user prompt on extract

OXYLABS_MAX_INLINE_TOKENS

10000

Above this, content is offloaded or truncated

OXYLABS_SPILL_DIR

system temp

Where offloaded pages are written (stdio only)

OXYLABS_SPILL

1

Set to 0 to keep everything inline even on stdio

MCP_TRANSPORT

stdio

stdio or http

HOST / PORT

127.0.0.1 / 8080

HTTP transport bind address

MCP_ALLOWED_HOSTS

localhost:*,127.0.0.1:*

Comma-separated Host allowlist (HTTP only)

MCP_ALLOWED_ORIGINS

(empty)

Comma-separated Origin allowlist (browser clients)

Copy .env.example to .env for local use. On stdio the server reads that .env from its working directory: any OXYLABS_* name not already set in the environment is filled from there, so a project that keeps its key in .env needs no launcher wrapper. Real environment variables always win, and only OXYLABS_* names are read. Point it elsewhere with OXYLABS_ENV_FILE=/path/to/.env.

The key is read at request time and is never written to disk or logged.

Security notes

  • Callers can bring their own key over HTTP. Send Authorization: Bearer <key> with each request and the server uses it for that call, so one deployment serves several callers on their own quota. OXYLABS_WEB_API_KEY in the server environment is the fallback when no header arrives.

  • Cap the spend. OXYLABS_RATE_LIMIT=100/1h refuses tool calls past a sliding window, so a runaway agent loop cannot drain the key. Off by default.

  • If you rely on that fallback, treat the endpoint as privileged. The server does no authentication of its own — anyone who can reach it spends the key it holds. Put it behind your VPN, an ingress with auth, or a service mesh. Don't expose it to the public internet.

  • scrape fetches whatever URL it is given. If you expose this server to untrusted prompts, restrict egress at the network layer rather than trusting the caller.

  • read_scraped is restricted to the spill directory. Don't point OXYLABS_SPILL_DIR at a directory holding anything else — it would make those files readable by the agent.

Development

pip install -e '.[dev]'
ruff check . && ruff format --check .
pytest                         # or: python tests/test_server.py
./scripts/sync-skill.sh        # refresh the bundled skill from web-api-skills

The tests are offline: error parsing, input validation, envelope trimming, endpoint-name handling, the job lifecycle and the extract approval gate. CI runs the same on 3.10, 3.12 and 3.13.

License

MIT

Available Tools

7 tools
check_scrapeCheck a render jobA
Read-only

Check a JavaScript-rendering job started by scrape or extract.

While it says running, poll again in ~10 seconds — and do other work between polls. A render runs 30-150s, so a dozen polls are normal and a job still running at 100s is not stuck. The finished result is returned in full.

ParametersJSON Schema
NameRequiredDescriptionDefault
job_idYesThe `job_id` returned by a `run_js` call.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only declare readOnlyHint and openWorldHint, so the description carries the polling/latency story and does it well, including the 'a job still running at 100s is not stuck' reassurance and the completeness of the finished result. It stops short of describing failure or timeout states, which would matter for a long-running async job.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences, front-loaded with the purpose before the operational guidance. Every sentence carries distinct information (what it checks, poll cadence, expected duration band) with no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an async job-status tool with an output schema, the description covers the poll loop, timing expectations, and result completeness, which is what an agent needs to call it correctly. Missing only edge-case behavior such as failed or expired jobs.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There is only one parameter and schema description coverage is 100%, so the schema already fully documents job_id. The description adds no format, source, or syntax detail beyond the schema, making the baseline 3 appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Check a JavaScript-rendering job') and names the tools that create such jobs (`scrape`, `extract`), which helps separate it from sibling operations like search and read_scraped. The only wrinkle is that the input schema attributes job_id to a `run_js` call, which does not appear in the sibling list, slightly muddying which upstream tool it complements.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit operational guidance: poll again in ~10 seconds while status is 'running', do other work between polls, and no need to fetch results elsewhere because the finished result is returned in full. It also bounds normal job duration (30-150s) so the agent knows when polling is still expected rather than pathological.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extractExtract fields as JSONA
Read-only

Pull named fields off a page as JSON, without writing selectors.

Costs more than scrape — the page is parsed by a model, per call — so the user is asked to approve each run. Scrape the page and read it yourself when a page you were going to read anyway would answer the question; use this when you want the fields themselves, in a shape you can compute on.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesAbsolute http(s) URL of the page to read.
promptYesThe fields to pull out of the page, in plain words. Name them and say what shape you want.
run_jsNoExecute the page's JavaScript. Needed for pages that render client-side and arrive empty otherwise. Slow: this returns a job id to poll with `check_scrape` instead of the content. Try without it first.
locationNoTwo-letter country code to fetch the page from. Use it when the page varies by country — pricing, availability, language.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only declare readOnlyHint=true and openWorldHint=true, so the description carries real extra weight: it discloses that the page is parsed by a model per call, that it costs more than 'scrape', and that the user must approve each run. That approval/cost gate is meaningful context an agent needs before invoking. It stops short of describing failure modes or what happens if selectors/fields aren't found.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tightly written sentences, front-loaded with the verb+resource, then the cost/approval constraint, then the routing rule. No filler sentences; every clause earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, the description needn't explain return values, and the schema covers params fully. It covers purpose, routing, and the key behavioral gate (cost/approval). The only remaining gap is that it doesn't describe how the returned JSON is shaped or what happens on parse failure, but that is largely covered by the output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all four parameters (including the run_js polling caveat and the location country code). The description adds no parameter-level detail beyond what the schema provides, so the baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource — 'pull named fields off a page as JSON' — and frames the distinguishing mechanism ('without writing selectors'). It contrasts cleanly against the sibling 'scrape', so an agent can tell the two apart without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Names the alternative ('scrape the page and read it yourself') and gives the exact condition that selects it ('when a page you were going to read anyway would answer the question'), plus the condition for this tool ('when you want the fields themselves, in a shape you can compute on'). This is close to the ideal when/when-not formulation.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_scrapersList scrape endpointsA
Read-only

List the scrape endpoints this API implements, or describe one of them.

Call this before assuming a dedicated scraper does or does not exist for a target, then call it again with the endpoint name to see what that endpoint accepts. That is the authoritative parameter list — more current than any documentation. Run the endpoint itself with scrape_target.

ParametersJSON Schema
NameRequiredDescriptionDefault
endpointNoName a scrape endpoint to get its parameters and their types instead of the list. Omit it to list what exists.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnlyHint=true, openWorldHint=true), so the bar is lower. The description still adds non-obvious context: the returned parameter list is 'authoritative' and 'more current than any documentation', which tells the agent to trust this call over cached knowledge — a real behavioral trait beyond the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with purpose, then the two-step usage pattern, then the hand-off to scrape_target. Every sentence carries a distinct instruction and there is no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return-value detail is unnecessary. Given a single optional parameter, full schema coverage, and clear routing to the sibling that executes scrapes, the description supplies everything an agent needs to call this correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the single `endpoint` parameter is already documented in the schema with an example ('scrape/amazon/search') and default null. The description restates the omit-vs-supply behavior, which reinforces but does not extend the schema, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Starts with a specific verb+resource ('List the scrape endpoints') and immediately adds the second mode ('or describe one of them'). It also names the sibling that performs the actual work ('run the endpoint itself with scrape_target'), so an agent can separate discovery from execution without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly prescribes the workflow: call it before assuming a target has a dedicated scraper, then call again with the endpoint name to learn its parameters. It names the alternative (scrape_target) and the condition that selects it, leaving nothing to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

read_scrapedRead an offloaded pageA
Read-only

Read a chunk of a scraped page that was offloaded to disk.

Use the path from a scrape result's content_offloaded. Start at offset 0 and keep calling with the returned next_offset until eof is true — and stop as soon as you have what you need rather than reading the whole file by reflex. Returns text plus offset, returned_chars, total_chars, next_offset and eof.

ParametersJSON Schema
NameRequiredDescriptionDefault
pathYesPath from a result's `content_offloaded.path`.
lengthNoHow many characters to return.
offsetNoCharacter offset to start at.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and openWorldHint=false, so safety is covered. The description goes further by disclosing the chunked/paginated access pattern, the early-termination recommendation, and the exact response fields (offset, returned_chars, total_chars, next_offset, eof) — real behavioral context beyond the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight paragraphs: purpose first, then the operating procedure and return shape. Every sentence carries actionable information with no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, the return fields need not be enumerated, yet doing so is harmless and reinforces the paging contract. Combined with a 100%-documented input schema and readOnly annotations, an agent has everything needed to call this correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description adds meaning above the schema by explaining that `offset` starts at 0 and that the next call should use the returned `next_offset`, i.e. the semantics of the offset/length interplay rather than just their types.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (read), resource (scraped page chunk), and qualifier (offloaded to disk) that clearly separates it from scrape/search/extract siblings. An agent can tell this is the paging reader for offloaded content, not a network-fetching tool.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly tells the agent where the required `path` comes from ('a scrape result's `content_offloaded`') and gives a concrete loop protocol (start at 0, follow `next_offset` to `eof`). It adds a useful early-stop heuristic. It does not name alternative tools for the case where content was not offloaded, so it falls short of a full when/when-not map.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scrapeRead a pageA
Read-only

Fetch and read a single URL, including JavaScript-heavy and bot-protected pages.

Use whenever you have a URL and need what is on it. Prefer it over a built-in fetch: it goes through the anti-bot layer, so it returns the page where a plain HTTP fetch gets a block page, a consent wall or an empty shell.

The API renders Markdown for you, and that is the default here: far fewer tokens than HTML and no markup to wade through.

Try it without run_js first. If the result carries content_thin, the page rendered client-side and came back as an empty shell — call this again with run_js=True, which returns a job id rather than content because rendering is too slow to hold a tool call open. Poll that id with check_scrape.

Very large pages are not returned inline. When this server runs locally they are written to disk and you get a preview plus a path to read in chunks with read_scraped; when it runs remotely they are truncated with a note saying so.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesAbsolute http(s) URL of the page to read.
deviceNoViewport to fetch as. Upstream default is desktop.
formatNomarkdown to read the page — far fewer tokens, structure intact. html only when you need the markup itself: attributes, embedded JSON-LD.markdown
run_jsNoExecute the page's JavaScript. Needed for pages that render client-side and arrive empty otherwise. Slow: this returns a job id to poll with `check_scrape` instead of the content. Try without it first.
locationNoTwo-letter country code to fetch the page from. Use it when the page varies by country — pricing, availability, language.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations cover the safety profile (readOnlyHint, openWorldHint), and the description adds substantial behavior beyond them: the anti-bot layer, the content_thin failure signal, run_js returning a job id instead of content because rendering is too slow, and the local-vs-remote large-page handling. This is exactly the operational context an agent needs.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with purpose and the preferred-over-fetch rationale, then the run_js escalation path, then the large-page caveat. Dense and each paragraph earns its place, though it runs a bit long for a single-URL fetch tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values needn't be described, yet the description still covers the two non-obvious return shapes: content_thin and the job-id polling flow, plus the truncated-vs-on-disk large-page outcome. Nothing needed to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% so the baseline is 3, but the description adds real meaning: it explains why run_js exists (client-side rendering), why format defaults to markdown (token cost), and the consequence of setting run_js. device and location are left to the schema, which is acceptable given full coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Fetch and read a single URL') with explicit scope including JavaScript-heavy and bot-protected pages. Distinguishes itself from a plain HTTP fetch and from siblings like check_scrape and read_scraped by name.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when-to-use ('whenever you have a URL and need what is on it'), when to prefer it over alternatives ('prefer over a built-in fetch'), and a sequenced workflow ('try it without run_js first'). Routing to check_scrape and read_scraped is spelled out with the triggering conditions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scrape_targetCall a target scraperA
Read-only

Call a target-specific scrape endpoint with its own parameters.

The generic scrape tool reads any URL. This runs the dedicated scrapers instead — the ones with pagination, store context, sort order and the rest. Look the endpoint up with list_scrapers(), read its parameters with list_scrapers(endpoint), then call it here. A run_js in params returns a job id to poll, same as scrape.

ParametersJSON Schema
NameRequiredDescriptionDefault
paramsYesThe endpoint's own request body. Get the accepted keys and types from `list_scrapers(endpoint)` rather than guessing them.
endpointYesA scrape endpoint path from `list_scrapers`, without the /v1/ prefix, e.g. 'scrape/amazon/search'.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and openWorldHint=true, so safety is covered. The description adds real behavioral context beyond that: a `run_js` in `params` returns a job id that must be polled, same as `scrape`. It does not cover error modes or how the job id maps to `check_scrape`/`read_scraped`, but the key async trait is disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four short sentences, front-loaded with the core action and the sibling distinction, then the workflow, then the async caveat. No filler; every sentence carries operational information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be explained, and the description still flags the job-id polling case. Endpoint sourcing, parameter sourcing, and the alternative tool are all covered, leaving nothing an agent needs before calling.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3, and the description goes further by telling the agent not to guess `params` keys and to source them from `list_scrapers(endpoint)`. That discovery guidance is genuine added value for an open `additionalProperties: true` object.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource — 'call a target-specific scrape endpoint with its own parameters' — and explicitly contrasts itself with the generic `scrape` sibling. An agent can distinguish this from `scrape`, `search`, or `extract` without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit ordered workflow: look the endpoint up with `list_scrapers()`, read its parameters with `list_scrapers(endpoint)`, then call it here. It also names the alternative (`scrape` for arbitrary URLs) and the condition that selects it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 7 tool updatesv0.1.0
    • First observedcheck_scrape
    • First observedextract
    • First observedlist_scrapers
    • First observedread_scraped
    • First observedscrape
    • First observedscrape_target
    • First observedsearch

TDQS

A4.4/5.0

Scored across 7 tools

Disambiguation5/5

Each tool has a clearly distinct role: scrape (generic URL fetch), search (web search), extract (structured field extraction), check_scrape (async job polling), read_scraped (chunked disk reads), list_scrapers (endpoint discovery), and scrape_target (dedicated parametric scrapers). The scrape vs. scrape_target overlap is explicitly resolved in descriptions, and the async/scrape workflow is cleanly separated from synchronous reads.

Naming Consistency4/5

All names use snake_case and mostly follow a verb_noun pattern (check_scrape, read_scraped, list_scrapers, scrape_target). A few core tools are bare verbs (scrape, search, extract) without an object noun, a minor deviation from the otherwise predictable convention.

Tool Count5/5

Seven tools is a tight, well-scoped set for a web scraping API. Each tool covers a necessary capability (fetch, search, extract, async polling, disk read, discovery, parametric scrape) with no filler.

Completeness4/5

The surface covers the full scraping lifecycle: discovery (list_scrapers), generic and target-specific fetching, structured extraction, async rendering with polling, and offloaded-content reading. Minor gaps like batch/multi-URL operations or site crawling aren't represented, but core workflows are complete.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    D
    maintenance
    Web search, page fetching, and research from the terminal or any MCP client — no API key required.
    1
    MIT
  • A
    license
    A
    quality
    B
    maintenance
    Gives local and cloud LLMs live web grounding as four MCP tools: web_search, fetch, deep_search (a token-capped, cited evidence pack sized to a small context window), and research (an agentic search-and-read loop). Flat monthly pricing, no query content stored.
    4
    MIT
  • A
    license
    Not graded
    quality
    B
    maintenance
    An MCP server that gives AI agents controlled access to the web, offering unified tools for fetching pages as Markdown, searching across multiple providers, and extracting structured data via CSS selectors.
    Apache 2.0
  • F
    license
    Not graded
    quality
    C
    maintenance
    Enables AI agents to perform unified web research through a single MCP server, including search, page fetching, recursive crawling, document parsing, YouTube transcript extraction, and deep multi-query research.
    2
    -