Oxylabs Web API MCP Server
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Oxylabs Web API MCP Serversearch the web for the latest AI news and scrape the top result"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Oxylabs Web API — MCP Server
mcp-name: io.oxylabs/web-api-mcp
A self-hostable Model Context Protocol server that gives any MCP-capable agent live web access through the Oxylabs Web API.
Tool | What it does |
| Search the live web, returns ranked organic results (title, description, URL) |
| Read a single URL as Markdown by default, including JS-heavy and bot-protected pages |
| Pull named fields off a page as JSON, no selectors. Billed above a scrape, so the user approves each run |
| Collect the result of a JavaScript-rendering job |
| Read a large page that was offloaded to disk, in chunks |
| List the scrape endpoints the API implements, or describe one's parameters |
| Call a target-specific scrape endpoint with its own parameters |
All seven are annotated readOnlyHint — nothing here writes anything — so clients can run
them without prompting.
The skill ships with the server
Connecting the server is the whole install. The agent skill is bundled in the package and served as an MCP resource and a prompt, so a client that never adds the web-api-skills repo still gets the judgment for using these tools well — search to find and scrape to read, when JavaScript rendering earns its cost, what to do with an empty page, how to cite.
| The skill itself, as Markdown |
prompt | Takes a |
scripts/sync-skill.sh refreshes the bundled copy from the skills repo — the canonical
copy lives there, and the two must not drift.
Related MCP server: grounder-mcp
Fitting the client's context
Oversized content is measured in tokens, not characters: 40 000 characters of English
is about 10 000 tokens, but 40 000 characters of Chinese is about 40 000, and Claude Code,
Claude Desktop and Cursor all reject a tool result over 25 000. The estimate is script-aware
for that reason. The budget is OXYLABS_MAX_INLINE_TOKENS (default 10 000), and a client
that knows its own limit can override it per request with an X-MCP-Max-Tokens header —
0 opts out entirely.
JavaScript rendering is a job, not a wait
run_js pages take 30-150 seconds and routinely outlive a tool call. So scrape(url, run_js=True) (and extract, and scrape_target with run_js in its params) returns a job
id straight away:
{ "job_id": "9f3c1a20b7d4", "status": "running", "url": "https://example.com" }The agent polls check_scrape(job_id) — after ~30s, then every ~10s — and does other work
in between. Each reply carries elapsed_seconds and says whether the job is still inside
the normal 150s window, so a slow render doesn't read as a stuck one. Results are kept for
OXYLABS_JOB_TTL_MINUTES (default 60), long after the job itself has finished.
It says when a page needed rendering
The agent shouldn't have to guess whether an empty page is empty or just unrendered, and it
shouldn't pay for a render on every page to find out. So a plain scrape that comes back
with almost no visible text — or with a "please enable JavaScript" notice — is flagged:
{
"content": "# Loading…",
"content_thin": {
"visible_chars": 9,
"reason": "almost no text",
"note": "…renders client-side. Retry the same call with run_js=True…"
}
}HTML is measured on its text, not its markup, so a 3 KB shell of <meta> tags still reads
as thin. The flag is a hint, not a retry: rendering is slow and billed, and a genuinely
short page would pay for it on every fetch. The threshold is 500 visible characters.
Jobs live in the server process: they do not survive a restart, and they are not shared between HTTP replicas. Run one replica, or give it sticky sessions.
Structured extraction costs extra
extract(url, prompt) returns the fields you name as JSON instead of a page to read. The
page is parsed by a model per call, which is billed above a plain scrape, so the server
asks the user to approve each run over MCP elicitation. Clients that can't elicit get an
error explaining why rather than a silent charge; OXYLABS_EXTRACT_APPROVAL=0 waives the
prompt once the user has agreed to the cost.
Large pages
Web pages routinely exceed what is sensible to hand an agent in one response, so scrape
does not return oversized content inline:
Running locally (stdio): the page is written to a temp file. The agent gets the first 2 000 characters plus a path, and pulls the rest through
read_scraped(path, offset)— reading only as far as it needs instead of paying for the whole page up front.Running remotely (HTTP): there is no shared filesystem, so a path would be useless. The content is truncated with a note stating the full length.
The threshold is OXYLABS_MAX_INLINE_TOKENS (default 10 000). read_scraped can only read
files in the spill directory — it is deliberately not a general file reader.
Full API documentation: Oxylabs Web API docs
Requirements
Python 3.10+ (built on FastMCP, installed with the package)
An Oxylabs Web API key — in the Oxylabs dashboard, create a Web API instance and generate a key for it
Install
uv tool install git+https://github.com/thevastas/oxy_mcpThat puts oxylabs-web-api-mcp on your PATH in its own environment. pipx install git+https://github.com/thevastas/oxy_mcp does the same. Installing into a system Python
usually fails — most are marked externally managed and refuse. To work on the server
itself:
git clone https://github.com/thevastas/oxy_mcp.git
cd oxy_mcp
pip install -e .server.json is the MCP registry
manifest. It has no packages block yet — add one once the server is published somewhere
installable, since a registry entry pointing at nothing is worse than no entry.
Use it locally (stdio)
Point your client at the installed command. Claude Code:
claude mcp add oxylabs-web-api \
--env OXYLABS_WEB_API_KEY=your_api_key_here \
-- oxylabs-web-api-mcpClaude Desktop / Cursor / any client that reads a JSON config:
{
"mcpServers": {
"oxylabs-web-api": {
"command": "oxylabs-web-api-mcp",
"env": { "OXYLABS_WEB_API_KEY": "your_api_key_here" }
}
}
}Self-host it (HTTP)
The HTTP transport is for running one shared server for a team or for agents that can't spawn local processes.
export OXYLABS_WEB_API_KEY=your_api_key_here
export MCP_ALLOWED_HOSTS='mcp.internal.example.com,localhost:*'
oxylabs-web-api-mcp --transport http --host 0.0.0.0 --port 8080The endpoint is then http://<host>:8080/mcp.
Docker
docker build -t oxylabs-web-api-mcp .
docker run --rm -p 8080:8080 \
-e OXYLABS_WEB_API_KEY=your_api_key_here \
-e MCP_ALLOWED_HOSTS='localhost:*,mcp.internal.example.com' \
oxylabs-web-api-mcp
MCP_ALLOWED_HOSTSis not optional. The HTTP transport turns on DNS-rebinding protection, so a server that doesn't declare its own hostname rejects every request with aHostheader it doesn't recognise (421 Misdirected Request). List the hostname clients actually connect to.host:*matches any port on that host.
Configuration
Variable | Default | Purpose |
| (required on stdio) | Web API key, sent as |
|
| Override for staging or a proxy |
|
| Per-request timeout in seconds |
|
| Retries on a transient 429/500/502/503/504 |
| (off) | Cap this server's own spend, e.g. |
|
| How long a finished job's result stays pollable |
|
| Set to |
|
| Above this, content is offloaded or truncated |
| system temp | Where offloaded pages are written (stdio only) |
|
| Set to |
|
|
|
|
| HTTP transport bind address |
|
| Comma-separated |
| (empty) | Comma-separated |
Copy .env.example to .env for local use. On stdio the server reads that .env from
its working directory: any OXYLABS_* name not already set in the environment is filled
from there, so a project that keeps its key in .env needs no launcher wrapper. Real environment variables always win, and
only OXYLABS_* names are read. Point it elsewhere with OXYLABS_ENV_FILE=/path/to/.env.
The key is read at request time and is never written to disk or logged.
Security notes
Callers can bring their own key over HTTP. Send
Authorization: Bearer <key>with each request and the server uses it for that call, so one deployment serves several callers on their own quota.OXYLABS_WEB_API_KEYin the server environment is the fallback when no header arrives.Cap the spend.
OXYLABS_RATE_LIMIT=100/1hrefuses tool calls past a sliding window, so a runaway agent loop cannot drain the key. Off by default.If you rely on that fallback, treat the endpoint as privileged. The server does no authentication of its own — anyone who can reach it spends the key it holds. Put it behind your VPN, an ingress with auth, or a service mesh. Don't expose it to the public internet.
scrapefetches whatever URL it is given. If you expose this server to untrusted prompts, restrict egress at the network layer rather than trusting the caller.read_scrapedis restricted to the spill directory. Don't pointOXYLABS_SPILL_DIRat a directory holding anything else — it would make those files readable by the agent.
Development
pip install -e '.[dev]'
ruff check . && ruff format --check .
pytest # or: python tests/test_server.py
./scripts/sync-skill.sh # refresh the bundled skill from web-api-skillsThe tests are offline: error parsing, input validation, envelope trimming, endpoint-name
handling, the job lifecycle and the extract approval gate. CI runs the same on 3.10,
3.12 and 3.13.
License
MIT
Available Tools
7 toolscheck_scrapeCheck a render jobARead-only
Check a JavaScript-rendering job started by scrape or extract.
While it says running, poll again in ~10 seconds — and do other work between polls. A render runs 30-150s, so a dozen polls are normal and a job still running at 100s is not stuck. The finished result is returned in full.
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes | The `job_id` returned by a `run_js` call. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare readOnlyHint and openWorldHint, so the description carries the polling/latency story and does it well, including the 'a job still running at 100s is not stuck' reassurance and the completeness of the finished result. It stops short of describing failure or timeout states, which would matter for a long-running async job.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, front-loaded with the purpose before the operational guidance. Every sentence carries distinct information (what it checks, poll cadence, expected duration band) with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For an async job-status tool with an output schema, the description covers the poll loop, timing expectations, and result completeness, which is what an agent needs to call it correctly. Missing only edge-case behavior such as failed or expired jobs.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There is only one parameter and schema description coverage is 100%, so the schema already fully documents job_id. The description adds no format, source, or syntax detail beyond the schema, making the baseline 3 appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Check a JavaScript-rendering job') and names the tools that create such jobs (`scrape`, `extract`), which helps separate it from sibling operations like search and read_scraped. The only wrinkle is that the input schema attributes job_id to a `run_js` call, which does not appear in the sibling list, slightly muddying which upstream tool it complements.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit operational guidance: poll again in ~10 seconds while status is 'running', do other work between polls, and no need to fetch results elsewhere because the finished result is returned in full. It also bounds normal job duration (30-150s) so the agent knows when polling is still expected rather than pathological.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extractExtract fields as JSONARead-only
Pull named fields off a page as JSON, without writing selectors.
Costs more than scrape — the page is parsed by a model, per call — so the user is
asked to approve each run. Scrape the page and read it yourself when a page you were
going to read anyway would answer the question; use this when you want the fields
themselves, in a shape you can compute on.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Absolute http(s) URL of the page to read. | |
| prompt | Yes | The fields to pull out of the page, in plain words. Name them and say what shape you want. | |
| run_js | No | Execute the page's JavaScript. Needed for pages that render client-side and arrive empty otherwise. Slow: this returns a job id to poll with `check_scrape` instead of the content. Try without it first. | |
| location | No | Two-letter country code to fetch the page from. Use it when the page varies by country — pricing, availability, language. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare readOnlyHint=true and openWorldHint=true, so the description carries real extra weight: it discloses that the page is parsed by a model per call, that it costs more than 'scrape', and that the user must approve each run. That approval/cost gate is meaningful context an agent needs before invoking. It stops short of describing failure modes or what happens if selectors/fields aren't found.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tightly written sentences, front-loaded with the verb+resource, then the cost/approval constraint, then the routing rule. No filler sentences; every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present, the description needn't explain return values, and the schema covers params fully. It covers purpose, routing, and the key behavioral gate (cost/approval). The only remaining gap is that it doesn't describe how the returned JSON is shaped or what happens on parse failure, but that is largely covered by the output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all four parameters (including the run_js polling caveat and the location country code). The description adds no parameter-level detail beyond what the schema provides, so the baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource — 'pull named fields off a page as JSON' — and frames the distinguishing mechanism ('without writing selectors'). It contrasts cleanly against the sibling 'scrape', so an agent can tell the two apart without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Names the alternative ('scrape the page and read it yourself') and gives the exact condition that selects it ('when a page you were going to read anyway would answer the question'), plus the condition for this tool ('when you want the fields themselves, in a shape you can compute on'). This is close to the ideal when/when-not formulation.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_scrapersList scrape endpointsARead-only
List the scrape endpoints this API implements, or describe one of them.
Call this before assuming a dedicated scraper does or does not exist for a target,
then call it again with the endpoint name to see what that endpoint accepts. That is
the authoritative parameter list — more current than any documentation. Run the
endpoint itself with scrape_target.
| Name | Required | Description | Default |
|---|---|---|---|
| endpoint | No | Name a scrape endpoint to get its parameters and their types instead of the list. Omit it to list what exists. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover the safety profile (readOnlyHint=true, openWorldHint=true), so the bar is lower. The description still adds non-obvious context: the returned parameter list is 'authoritative' and 'more current than any documentation', which tells the agent to trust this call over cached knowledge — a real behavioral trait beyond the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with purpose, then the two-step usage pattern, then the hand-off to scrape_target. Every sentence carries a distinct instruction and there is no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return-value detail is unnecessary. Given a single optional parameter, full schema coverage, and clear routing to the sibling that executes scrapes, the description supplies everything an agent needs to call this correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the single `endpoint` parameter is already documented in the schema with an example ('scrape/amazon/search') and default null. The description restates the omit-vs-supply behavior, which reinforces but does not extend the schema, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Starts with a specific verb+resource ('List the scrape endpoints') and immediately adds the second mode ('or describe one of them'). It also names the sibling that performs the actual work ('run the endpoint itself with scrape_target'), so an agent can separate discovery from execution without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly prescribes the workflow: call it before assuming a target has a dedicated scraper, then call again with the endpoint name to learn its parameters. It names the alternative (scrape_target) and the condition that selects it, leaving nothing to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
read_scrapedRead an offloaded pageARead-only
Read a chunk of a scraped page that was offloaded to disk.
Use the path from a scrape result's content_offloaded. Start at offset 0 and keep
calling with the returned next_offset until eof is true — and stop as soon as you
have what you need rather than reading the whole file by reflex. Returns text plus
offset, returned_chars, total_chars, next_offset and eof.
| Name | Required | Description | Default |
|---|---|---|---|
| path | Yes | Path from a result's `content_offloaded.path`. | |
| length | No | How many characters to return. | |
| offset | No | Character offset to start at. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=false, so safety is covered. The description goes further by disclosing the chunked/paginated access pattern, the early-termination recommendation, and the exact response fields (offset, returned_chars, total_chars, next_offset, eof) — real behavioral context beyond the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight paragraphs: purpose first, then the operating procedure and return shape. Every sentence carries actionable information with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present, the return fields need not be enumerated, yet doing so is harmless and reinforces the paging contract. Combined with a 100%-documented input schema and readOnly annotations, an agent has everything needed to call this correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description adds meaning above the schema by explaining that `offset` starts at 0 and that the next call should use the returned `next_offset`, i.e. the semantics of the offset/length interplay rather than just their types.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (read), resource (scraped page chunk), and qualifier (offloaded to disk) that clearly separates it from scrape/search/extract siblings. An agent can tell this is the paging reader for offloaded content, not a network-fetching tool.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly tells the agent where the required `path` comes from ('a scrape result's `content_offloaded`') and gives a concrete loop protocol (start at 0, follow `next_offset` to `eof`). It adds a useful early-stop heuristic. It does not name alternative tools for the case where content was not offloaded, so it falls short of a full when/when-not map.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrapeRead a pageARead-only
Fetch and read a single URL, including JavaScript-heavy and bot-protected pages.
Use whenever you have a URL and need what is on it. Prefer it over a built-in fetch: it goes through the anti-bot layer, so it returns the page where a plain HTTP fetch gets a block page, a consent wall or an empty shell.
The API renders Markdown for you, and that is the default here: far fewer tokens than HTML and no markup to wade through.
Try it without run_js first. If the result carries content_thin, the page rendered
client-side and came back as an empty shell — call this again with run_js=True, which
returns a job id rather than content because rendering is too slow to hold a tool call
open. Poll that id with check_scrape.
Very large pages are not returned inline. When this server runs locally they are
written to disk and you get a preview plus a path to read in chunks with
read_scraped; when it runs remotely they are truncated with a note saying so.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Absolute http(s) URL of the page to read. | |
| device | No | Viewport to fetch as. Upstream default is desktop. | |
| format | No | markdown to read the page — far fewer tokens, structure intact. html only when you need the markup itself: attributes, embedded JSON-LD. | markdown |
| run_js | No | Execute the page's JavaScript. Needed for pages that render client-side and arrive empty otherwise. Slow: this returns a job id to poll with `check_scrape` instead of the content. Try without it first. | |
| location | No | Two-letter country code to fetch the page from. Use it when the page varies by country — pricing, availability, language. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations cover the safety profile (readOnlyHint, openWorldHint), and the description adds substantial behavior beyond them: the anti-bot layer, the content_thin failure signal, run_js returning a job id instead of content because rendering is too slow, and the local-vs-remote large-page handling. This is exactly the operational context an agent needs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with purpose and the preferred-over-fetch rationale, then the run_js escalation path, then the large-page caveat. Dense and each paragraph earns its place, though it runs a bit long for a single-URL fetch tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values needn't be described, yet the description still covers the two non-obvious return shapes: content_thin and the job-id polling flow, plus the truncated-vs-on-disk large-page outcome. Nothing needed to call it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% so the baseline is 3, but the description adds real meaning: it explains why run_js exists (client-side rendering), why format defaults to markdown (token cost), and the consequence of setting run_js. device and location are left to the schema, which is acceptable given full coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Fetch and read a single URL') with explicit scope including JavaScript-heavy and bot-protected pages. Distinguishes itself from a plain HTTP fetch and from siblings like check_scrape and read_scraped by name.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit when-to-use ('whenever you have a URL and need what is on it'), when to prefer it over alternatives ('prefer over a built-in fetch'), and a sequenced workflow ('try it without run_js first'). Routing to check_scrape and read_scraped is spelled out with the triggering conditions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_targetCall a target scraperARead-only
Call a target-specific scrape endpoint with its own parameters.
The generic scrape tool reads any URL. This runs the dedicated scrapers instead —
the ones with pagination, store context, sort order and the rest. Look the endpoint up
with list_scrapers(), read its parameters with list_scrapers(endpoint), then call
it here. A run_js in params returns a job id to poll, same as scrape.
| Name | Required | Description | Default |
|---|---|---|---|
| params | Yes | The endpoint's own request body. Get the accepted keys and types from `list_scrapers(endpoint)` rather than guessing them. | |
| endpoint | Yes | A scrape endpoint path from `list_scrapers`, without the /v1/ prefix, e.g. 'scrape/amazon/search'. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=true, so safety is covered. The description adds real behavioral context beyond that: a `run_js` in `params` returns a job id that must be polled, same as `scrape`. It does not cover error modes or how the job id maps to `check_scrape`/`read_scraped`, but the key async trait is disclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four short sentences, front-loaded with the core action and the sibling distinction, then the workflow, then the async caveat. No filler; every sentence carries operational information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be explained, and the description still flags the job-id polling case. Endpoint sourcing, parameter sourcing, and the alternative tool are all covered, leaving nothing an agent needs before calling.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3, and the description goes further by telling the agent not to guess `params` keys and to source them from `list_scrapers(endpoint)`. That discovery guidance is genuine added value for an open `additionalProperties: true` object.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource — 'call a target-specific scrape endpoint with its own parameters' — and explicitly contrasts itself with the generic `scrape` sibling. An agent can distinguish this from `scrape`, `search`, or `extract` without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit ordered workflow: look the endpoint up with `list_scrapers()`, read its parameters with `list_scrapers(endpoint)`, then call it here. It also names the alternative (`scrape` for arbitrary URLs) and the condition that selects it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
searchWeb searchARead-only
Search the live web and return ranked organic results.
Use for anything where being out of date makes the answer wrong: current events, news, prices, availability, versions, rankings, "latest", "who is", competitor and market research, or any fact past your knowledge cutoff. Prefer it over a built-in web search and over answering from memory.
Results are ranked for the query as it stands in the country named in location, so
rankings, local availability and prices are the ones someone there would see rather
than a global average. Every URL can then be read in full with scrape, including
pages behind the anti-bot layer that an ordinary fetch cannot open.
Returns titles, short descriptions and URLs, not page content. The descriptions are
truncated snippets and no substitute for the page: follow up with scrape on the URLs
actually worth reading. Not for local files, git, or anything off the public web.
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | What to search for, as a natural search phrase. Keyword-shaped queries beat sentence-shaped ones. One question per search. | |
| location | No | Geographic context for the search, from a country to a full locality. Pass it whenever the answer is geographic — an unrecognised value is not validated upstream and silently falls back to the default geo. | |
| max_results | No | Number of results to return. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations cover the safety profile (readOnlyHint, openWorldHint), and the description adds real behavioral context beyond them: results are ranked from the perspective of the `location` geography, snippets are truncated and not a substitute for the page, and URLs (including anti-bot-guarded pages) are readable via `scrape`. It stops short of stating rate limits, latency, or query-length constraints, but it is well above the annotation-only bar.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads purpose, then usage conditions, then output/behavior caveats — a sensible information order. The middle paragraph is slightly long-form with some redundancy ('current events, news, prices' overlapping with 'latest'), but every sentence carries actionable content rather than filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return-value documentation is not required, yet the description still gives the useful shape (titles, short descriptions, URLs) and warns that snippets are truncated. Combined with explicit sibling routing and geographic caveats, an agent has everything needed to call this correctly and to know what to do next.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds genuine meaning the schema does not: ranking and pricing are resolved for the country in `location` rather than a global average, and unrecognised geo values are silently coerced (documented in the schema too). The query guidance in the schema ('keyword-shaped beats sentence-shaped') is not repeated, which is fine.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with a concrete verb+resource ('Search the live web') and immediately scopes the output ('ranked organic results'). It also draws the line against siblings by naming `scrape` as the follow-up tool for reading page content, so the agent can separate retrieval from reading without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly enumerates when to use it (current events, prices, availability, versions, 'latest', past-cutoff facts, competitor research) and states the preference over a built-in web search or answering from memory. It also carries a clear negative case: 'Not for local files, git, or anything off the public web.'
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
7 tool updates
v0.1.0- First observed
check_scrape - First observed
extract - First observed
list_scrapers - First observed
read_scraped - First observed
scrape - First observed
scrape_target - First observed
search
TDQS
Scored across 7 tools
Each tool has a clearly distinct role: scrape (generic URL fetch), search (web search), extract (structured field extraction), check_scrape (async job polling), read_scraped (chunked disk reads), list_scrapers (endpoint discovery), and scrape_target (dedicated parametric scrapers). The scrape vs. scrape_target overlap is explicitly resolved in descriptions, and the async/scrape workflow is cleanly separated from synchronous reads.
All names use snake_case and mostly follow a verb_noun pattern (check_scrape, read_scraped, list_scrapers, scrape_target). A few core tools are bare verbs (scrape, search, extract) without an object noun, a minor deviation from the otherwise predictable convention.
Seven tools is a tight, well-scoped set for a web scraping API. Each tool covers a necessary capability (fetch, search, extract, async polling, disk read, discovery, parametric scrape) with no filler.
The surface covers the full scraping lifecycle: discovery (list_scrapers), generic and target-specific fetching, structured extraction, async rendering with polling, and offloaded-content reading. Minor gaps like batch/multi-URL operations or site crawling aren't represented, but core workflows are complete.
Maintenance
Related MCP Connectors
Scrape, crawl and search the web for AI agents via MCP.
Live AI-native web search with citations. One tool for every MCP client. Flat per-request pricing.
Docs: https://docs.keenable.ai/mcp-server Keenable is a free, remote MCP server that gives agents access to the web index. Search the web with ranked results and date/site filters, then fetch any indexed page as clean markdown. Works out of the box with no account or API key.
One MCP for the Web. Easily search, crawl, navigate, and extract websites without getting blocked.…
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceWeb search, page fetching, and research from the terminal or any MCP client — no API key required.1MIT
- AlicenseAqualityBmaintenanceGives local and cloud LLMs live web grounding as four MCP tools: web_search, fetch, deep_search (a token-capped, cited evidence pack sized to a small context window), and research (an agentic search-and-read loop). Flat monthly pricing, no query content stored.4MIT
- AlicenseNot gradedqualityBmaintenanceAn MCP server that gives AI agents controlled access to the web, offering unified tools for fetching pages as Markdown, searching across multiple providers, and extracting structured data via CSS selectors.Apache 2.0
- FlicenseNot gradedqualityCmaintenanceEnables AI agents to perform unified web research through a single MCP server, including search, page fetching, recursive crawling, document parsing, YouTube transcript extraction, and deep multi-query research.2-