Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@@crawlbrulee/mcp Scrape the product page at https://crawlbrulee.com and return markdown with links"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
EU-native web scraping for AI agents & developers.
plug crawlbrulee into your agent. the official mcp server for crawlbrulee gives mcp-aware agents — Claude Code, Codex, Cursor, Claude Desktop — native tools to scrape pages, map sites, run background jobs, and check usage. one call turns any url into clean markdown, screenshots, metadata and links.
everything runs in the EU. the fetch, the render, the cache and your result never leave EU servers. the proxy exit is the one hop you choose: pick an EU exit and nothing leaves at all. gdpr-aligned, with a data processing agreement.
output made for models. markdown with the page chrome stripped and the links kept, ready for the prompt. full-page screenshots can come back as tiles sized for an image model.
the hard parts, handled. headless Chrome when a page needs it, rotating proxies with country selection, automatic retries, ad and cookie-banner removal, caching, background jobs and signed webhooks.
start free. 750 credits, no credit card.
get a free api key → dashboard.crawlbrulee.com
the server:
npx-runnable — zero install.
wraps the @crawlbrulee/sdk under the hood; this mcp is just a thin protocol adapter.
stdio transport for terminal-based agents.
strict, fully-described tool schemas — agents see what every parameter does without reading docs.
this readme covers the mcp server itself — its tools and how to wire it into a host. for how the api behaves — endpoints, parameters, and error semantics — please see our api docs.
# Claude Code
claude mcp add crawlbrulee \
--env CRAWLBRULEE_API_KEY=cwbl_... \
-- npx -y @crawlbrulee/mcp
# Cursor — add to ~/.cursor/mcp.json:
{
"mcpServers": {
"crawlbrulee": {
"command": "npx",
"args": ["-y", "@crawlbrulee/mcp"],
"env": { "CRAWLBRULEE_API_KEY": "cwbl_..." }
}
}
}the same pattern works for Codex, Claude Desktop, and any other host that
accepts a stdio mcp launch command — set command: npx, args: ["-y", "@crawlbrulee/mcp"], and forward CRAWLBRULEE_API_KEY via the env block.
Related MCP server: Anybrowse
env var | required | description |
| yes | api key sent as |
the mcp reads the env var on first tool invocation — not at startup — so a typo in your config surfaces as a clear tool-error message rather than the server failing to come up. see authentication for how the api consumes keys.
scrapefetch a single url and return the requested content (markdown, cleaned html, raw html, links, images, screenshot, page metadata).
input — only url is required; everything else has sane defaults.
{
"url": "https://example.com",
"extract": {
"markdown": true,
"links": true,
"screenshot": { "type": "full_page", "device_mode": "desktop" },
},
"require_js": false,
"proxy": "basic",
"cleanup": { "ads_and_popups": true, "exclude_selectors": ["nav", "footer"] },
"cache": { "max_age": 3600 },
"location": { "locale": "en-US", "country": "US" },
}output — full scrape result. page metadata (title, OG tags, etc.) is returned under metadata. extracted images are returned as absolute urls — query strings are preserved, and relative srcs are resolved against the page url. screenshots are returned as signed download urls the agent can fetch separately. in rare cases a screenshot can't be captured: when you requested other outputs too, the screenshot field is simply left out while the rest is still returned — but a screenshot-only call that can't deliver errors instead (unsupported_screenshot_output, HTTP 422, when the content type can't be screenshotted) and isn't billed. the result also carries a top-level response_meta.usage block:
{
"url": "https://example.com",
"markdown": "...",
"metadata": { "title": "Example Domain" },
"response_meta": {
"usage": {
"credits": 1,
"engine": "http", // "http" | "browser" | "screenshot" | "cache"
"proxy": "basic", // resolved tier actually used: "basic" | "advanced" (never "auto")
"screenshot_slices": 0, // 1 when the screenshot-split add-on was billed, otherwise 0
},
},
}alongside response_meta.usage, the result surfaces any non-fatal warnings — stable string codes an agent can switch on. an outsized page is truncated rather than refused, and the code names which part was cut:
code | what it means for the payload |
| the page was taller than the scrolling-capture height cap; the screenshot covers the top of the page. |
| the page had more than 30,000 links; the |
| the page had more than 10,000 inline images; the |
| the page body exceeded 10,000,000 characters; |
| the page head exceeded 2,000,000 characters; |
and if you requested an extract that doesn't apply to the content type (e.g. markdown of a pdf), the field name comes back in an unsupported_fields list — with the rest of the payload still returned.
every input field, its default, and its constraints are documented under the scrape endpoint — with extraction, screenshots, proxies & location, and caching covering the individual blocks.
scrape_asyncsubmit a scrape job to run asynchronously and get back a job_id immediately, instead of holding the connection open. use this for long-running scrapes (heavy js rendering, full-page screenshots of long pages); for a quick one-shot fetch prefer the synchronous scrape tool. then poll scrape_status until the job is done and fetch the page with scrape_result.
takes the same input as scrape plus an optional per-job completion webhook:
{
"url": "https://example.com",
"extract": { "markdown": true },
"webhook": {
// Endpoint that receives one signed `scrape.complete` POST when the job
// finishes. http/https (HTTPS required in production), max 2048 chars.
"url": "https://hooks.example.com/cwbl",
// Opaque correlation object echoed back verbatim in the delivery's
// `data.metadata`. Serializes to at most 2048 bytes.
"metadata": { "ref": "order-42" },
},
}output — { "job_id": "..." }.
when a webhook is attached, we deliver a single signed scrape.complete POST to your endpoint once the job reaches a terminal state, with your metadata echoed under data.metadata and the job's usage under data.response_meta.usage — so you can react to completion (and track cost) without polling. verify the X-Cwbl-Signature header with the sdk's verifyWebhookSignature (configure the signing secret in the dashboard under account → webhooks).
the job lifecycle is documented under async scrape; the delivery contract and payload shape under webhooks, with the signature scheme in webhook verification.
scrape_statuslook up the current lifecycle status of an async job: pending, running, done, or failed (with an error message when failed). once the job is done the response also carries a response_meta.usage block (credits, billed engine, resolved proxy tier, screenshot_slices). a cache hit is represented by engine: "cache". poll until done, then call scrape_result.
{ "job_id": "..." }scrape_resultfetch the extracted content of a completed async job — the same result shape as the synchronous scrape tool (including metadata and response_meta.usage). errors if the job is still pending/running, so check scrape_status first.
{ "job_id": "..." }mapbuild (or fetch a cached) link-map for a website. combines sitemap discovery with homepage link extraction. use this to enumerate a site before scraping selected pages. each link is just { url }.
max_urls (default 5000, max 100000) is a discovery budget, not a trim at the end — discovery stops as soon as that many urls are found, so a smaller value is a faster, cheaper crawl. limit (default 5000, max 10000) only pages the answer.
returned urls are normalized the same way scrape normalizes its returned url, so map-then-scrape stays on one host. results are ordered with the most useful links first.
the response's response_meta carries pagination, truncation, and a usage block (credits, billed engine, resolved proxy tier). map responses do not include screenshot-slice accounting.
{
"url": "https://example.com",
"sitemap_only": false,
"types": { "internal": true, "external": false, "internal_subdomains": true },
"max_urls": 5000,
"page": 1,
"limit": 1000,
}a map stopped by your own max_urls returns exactly that many links with response_capped: false — the signal that the site has more is truncation.discovery_cap_reason:
{
"truncation": {
"storage_capped": false,
"response_capped": false,
"total_before_max_urls": 5000,
"total_detected_before_storage_cap": 5000,
"discovery_capped": true, // discovery stopped before reading every sitemap file
"sitemaps_skipped": 3, // files skipped or only partly read
"discovery_cap_reason": "max_urls", // retry with a higher max_urls
},
}discovery_cap_reason is one of max_urls, time, file_budget, depth, file_size, unread_files, or null when nothing stopped discovery. only max_urls is a limit you can raise from the request. unread_files means a sitemap file the site publishes could not be read at all this time — often temporary, so asking again later can return more. time, file_budget, depth and file_size mean the site itself is big, slow or deep, and a retry will not help.
see the map endpoint for discovery rules and pagination semantics.
usagereturns the current billing-cycle snapshot: total / used / available credits, used quota percent, max concurrency, and cycle reset timestamp. takes no arguments. what a call costs, and how credits are counted, is documented under credits & pricing.
whoamireturns the organization name, token name, and truncated token preview for the configured api key. useful for confirming which account is in use before credit-consuming operations.
every tool returns an mcp error result (isError: true) when the api call fails. the error text follows a stable format:
[<errorName>] <message> (HTTP <status>)agents can branch on the errorName code. the set comes from the sdk's ApiErrorName union plus two synthetic codes added by this mcp (missing_api_key, internal_error):
code | meaning |
|
|
| server rejected the api key (revoked, wrong env, etc.). |
| temporary backend failure (HTTP 503). your key is fine — retry with backoff. |
| rate limit hit — back off and retry. |
| plan credit / concurrency cap exceeded. show |
| input failed server validation. |
| target url was rejected before fetching. |
| target url is on the blocklist. |
| origin's anti-bot defenses blocked the fetch. |
| origin redirected the fetch in a loop (HTTP 422). the target's doing — don't retry blindly. |
| the page's html was too large to process (HTTP 422). terminal — never retry it. |
| origin returned an error during scraping. |
| screenshot-only request on a content type that can't be screenshotted (HTTP 422). not billed. |
| async job ID unknown (e.g. bad |
| network / read timeout. safe to retry. |
| caller cancelled before completion. |
| unhandled server-side failure. |
| sdk error without a typed name. |
| bug in this mcp — please open an issue. |
the api docs carry the canonical error reference — every error name, what causes it, and how to recover.
pnpm install
pnpm typecheck # tsc --noEmit
pnpm lint # eslint
pnpm test # vitest run
pnpm build # tsup → dist/index.js with shebang
pnpm verify # all of the aboverun the built mcp locally:
CRAWLBRULEE_API_KEY=cwbl_... node ./dist/index.jsit will block waiting for an mcp client on stdio. combine with the MCP Inspector for interactive debugging.
this readme covers the mcp server itself — installing it, wiring it into a host, and the tools it exposes. for how the api behaves — endpoints, parameters, and error semantics — the api docs are canonical. the mcp guide covers host setup in more depth.
one api, many ways to call it:
js/ts sdk — @crawlbrulee/sdk (the sdk this mcp wraps)
python sdk — crawlbrulee on pypi
cli — npx crawlbrulee
mcp server — @crawlbrulee/mcp (this one)
agent skills — for skills-aware coding agents
docs: crawlbrulee.com/docs · dashboard: dashboard.crawlbrulee.com
Build (or fetch a cached) link-map for a website by combining sitemap discovery with homepage link extraction. Returns paginated lists of discovered URLs, each item just { url }. Filterable by link type (internal / external / subdomain). Use this to enumerate a site before scraping selected pages. max_urls (default 5000, max 100000) is a discovery budget, not a trim at the end: discovery stops as soon as that many URLs are found, so a smaller value is a faster, cheaper crawl. A map stopped that way returns exactly max_urls links with response_capped false — the signal that the site has more is response_meta.truncation.discovery_cap_reason. When that is "max_urls", ask again with a higher max_urls to get more; "unread_files" means a sitemap file could not be read this time and is often temporary, so asking again later can return more; "time", "file_budget", "depth" and "file_size" mean the site itself is big, slow or deep and a retry will not help. discovery_capped says discovery stopped early, sitemaps_skipped how many sitemap files were skipped or only partly read. limit (default 5000, max 10000) only pages the answer. Returned URLs are normalized the same way scrape normalizes its returned url, so map-then-scrape stays on one host. Results are ordered with the most useful links first. The response carries response_meta.usage = { credits, engine, proxy } — the resolved proxy tier is never auto; map responses do not include screenshot-slice accounting.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The website URL to map. Mapping always targets the site root, so the path, query string and fragment are dropped; known tracking parameters are removed before the request is processed. | |
| page | No | Page number for paginated results | |
| cache | No | Cache settings for this request | |
| limit | No | Number of URLs to return per page. Default 5000, maximum 10000. | |
| proxy | No | Proxy tier to use for fetching | auto |
| types | No | Filter which link types to include | |
| location | No | Optional country emulation for the map | |
| max_urls | No | Maximum number of URLs to discover and store in the map. Default 5000, maximum 100000. Sitemap discovery stops as soon as this many URLs have been found, so a smaller value is a faster and lighter crawl, not just a smaller answer. | |
| sitemap_only | No | Only use sitemap.xml — skip homepage link extraction |
| Name | Required | Description |
|---|---|---|
| links | Yes | List of discovered URLs for the current page |
| response_meta | Yes | Response metadata including pagination, truncation, and usage info |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does so thoroughly. It discloses discovery-budget stopping semantics, `response_capped` false behavior, all truncation reasons, URL normalization relative to `scrape`, ordering by usefulness, and response details like `response_meta.usage` and that `proxy` is never `auto`. This goes well beyond the schema and gives agents a reliable mental model of non-obvious behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but every sentence earns its place. It is front-loaded with the core function and use case, then progresses to parameters, then to response/error nuances. No filler or redundancy; the density is justified by the tool's complexity and the absence of annotations.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex tool with 9 parameters, nested objects, and an output schema, the description is complete. It covers the return format, pagination, cache semantics, filtering, discovery stopping, truncation signals, retry guidance, and response metadata. An agent has everything needed to decide when to call it and how to interpret results.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description adds meaningful nuance beyond the schema, particularly for `max_urls` (a discovery budget, not a trim), `limit` (only pages the answer), and the behavior of `types` filtering. It does not add supplementary meaning for every parameter, but the extra context is valuable and improves correct usage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Build (or fetch a cached) link-map for a website' via sitemap discovery and homepage link extraction. It explicitly states what the tool returns (paginated lists of discovered URLs) and positions it as an enumeration tool distinct from the scraping siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Use this to enumerate a site before scraping selected pages', giving a clear context. It also explains when retrying is useful based on `discovery_cap_reason` (e.g., 'max_urls' warrants a higher budget, 'unread_files' may be temporary). It does not explicitly name an alternative tool or provide a when-not-to-use statement, but the use case is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fetch a single URL via the crawlbrulee scraping API and return the requested content (markdown, cleaned HTML, raw HTML, links, images, screenshot, page metadata in metadata). Use this for one-shot page extraction. For full-site discovery use the map tool first. Screenshot URLs in the response are signed download links — the agent can fetch them when needed. The response also carries response_meta.usage = { credits, engine, proxy (the resolved tier — never auto), screenshot_slices }. engine is the billed engine (http, browser, screenshot, cache); a cache hit is represented by engine: "cache", so you can see what the request cost. Extraction is capped per page: 30,000 links, 10,000 inline images and 10,000,000 characters of HTML. A page past a cap is truncated rather than refused, and the response warnings array names which one (links_truncated, inline_images_truncated, raw_html_truncated) — so treat that output as incomplete. Use cleanup to control what is removed before any output is built: ads_and_popups (on by default) drops ads, cookie banners and chat widgets, and exclude_selectors removes anything else. It shapes markdown, cleaned_html, links, images and the screenshot, and NEVER raw_html — so request raw_html when you need the page exactly as it arrived. warnings also reports a section whose extraction failed outright (links_unavailable, inline_images_unavailable, metadata_unavailable): that field comes back omitted or empty while the rest of the scrape succeeded, so do NOT conclude the page had no links/images/metadata — re-run the scrape instead. An empty field with no such warning does mean the page had none.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to scrape. Known tracking parameters are removed before the page is fetched, so they are neither sent to the target site nor part of the cache key. Every other query parameter is kept verbatim and is part of the cache key. | |
| cache | No | Cache settings for this request | |
| proxy | No | Proxy tier to use for fetching | auto |
| cleanup | No | What is removed from the page before any output is built. Applies to markdown, cleaned_html, links and images on every engine, and to the screenshot. Never applies to raw_html, which is always the page before we removed anything. | |
| extract | No | Which content formats to extract. Defaults to metadata + cleaned_html. | |
| location | No | Optional locale + country emulation for the scrape | |
| require_js | No | Use a headless browser to render JavaScript before scraping |
| Name | Required | Description |
|---|---|---|
| url | Yes | The URL that was actually scraped, after any redirects, in normalized form — the base that links, images, and internal labels are computed against |
| links | No | Links found on the page |
| images | No | Inline images found on the page |
| markdown | No | Page content converted to clean Markdown |
| metadata | No | Extracted page metadata (title, OG tags, etc.) |
| raw_html | No | Raw, unprocessed HTML of the page |
| warnings | No | Non-error notices about the scrape. Truncation codes — `screenshot_truncated` (long page exceeded the scrolling-screenshot height cap), `links_truncated` / `inline_images_truncated` (page had more links/images than the per-page extraction caps), `raw_html_truncated` / `metadata_truncated` (rendered HTML exceeded the per-page size budget) — mean the field is present but capped. Unavailability codes — `links_unavailable` / `inline_images_unavailable` / `metadata_unavailable` — mean that optional field could not be extracted and was omitted (null/empty) while the rest of the scrape succeeded, so an empty field carrying one of these does NOT mean the page had none. Stable string codes — clients can switch on them. Warnings are stored with the result: async result fetches and cache hits carry them too, filtered to the fields the request asked for. |
| screenshot | No | Screenshot of the page, if requested |
| cleaned_html | No | Cleaned HTML of the main page content |
| content_type | No | Content-Type header returned by the server |
| requested_url | Yes | The URL you requested, echoed verbatim — before any redirects |
| response_meta | Yes | Request-level metadata. `response_meta.usage` reports credits charged, the billed engine, the resolved proxy tier, and any screenshot-slice add-on. |
| unsupported_fields | No | Extract fields that were requested but are not supported for this content type |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations present, the description fully carries the behavioral burden and does so exceptionally: it discloses cache-hit behavior via engine:'cache', signed screenshot URLs, per-page truncation caps and their warnings, partial-failure semantics (links_unavailable etc.), and the meaning of empty fields. This is far beyond any structured metadata.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long, but the tool is complex (7 parameters, nested screenshot/cleanup/cache objects) and the prose is dense with non-redundant operational details. It is front-loaded with the purpose statement and only sacrifices brevity where the behavior of the API genuinely needs explanation.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the absence of annotations and the complexity of the tool, the description is complete: it covers selection, output formats, cleanup effects, limits, warnings, cache semantics, and response metadata. An agent has enough context to invoke the tool and interpret partial results correctly; the output schema covers the rest.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the schema texts are already detailed, so the baseline is 3. The description adds genuine value by clarifying that cleanup shapes all outputs except raw_html, that a cache hit is distinguishable in response_meta, and that truncated output must be treated as incomplete (warnings list). It does not need to re-document every parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb and resource ('Fetch a single URL') and lists every output format it can return. It explicitly distinguishes itself from the `map` sibling ('For full-site discovery use the map tool first') and frames itself as one-shot page extraction, so an agent cannot confuse it with the crawl tool.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It states exactly when to use this tool ('one-shot page extraction') and steers the agent to `map` for full-site discovery. Additional operational guidance (request raw_html when the untouched page is needed, use cleanup for ad/popup removal) helps the agent choose output modes correctly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Submit a scrape job to run ASYNCHRONOUSLY and return a job_id immediately, instead of holding the connection open. Use this for long-running scrapes (heavy JS rendering, full-page screenshots of long pages) — for a quick one-shot fetch prefer the synchronous scrape tool, which blocks and returns the page directly. Poll the job with scrape_status and fetch the page with scrape_result once done. Optionally attach a per-job completion webhook: we deliver a single signed scrape.complete POST to your endpoint when the job finishes (HTTPS required in production), and echoes your opaque webhook.metadata back in the delivery (under data.metadata, alongside data.response_meta.usage).
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to scrape. Known tracking parameters are removed before the page is fetched, so they are neither sent to the target site nor part of the cache key. Every other query parameter is kept verbatim and is part of the cache key. | |
| cache | No | Cache settings for this request | |
| proxy | No | Proxy tier to use for fetching | auto |
| cleanup | No | What is removed from the page before any output is built. Applies to markdown, cleaned_html, links and images on every engine, and to the screenshot. Never applies to raw_html, which is always the page before we removed anything. | |
| extract | No | Which content formats to extract. Defaults to metadata + cleaned_html. | |
| webhook | No | Completion webhook delivered when this async job finishes. Async-only: the synchronous `scrape` tool does not accept it. | |
| location | No | Optional locale + country emulation for the scrape | |
| require_js | No | Use a headless browser to render JavaScript before scraping |
| Name | Required | Description |
|---|---|---|
| job_id | Yes | Job identifier — pass it to the `scrape_status` / `scrape_result` tools |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses async behavior, immediate return of job_id, webhook delivery with signature, HTTPS production requirement, and metadata echo. It doesn't cover failure modes or retries, but the core behavioral profile is well conveyed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two dense sentences front-load the async behavior WF and workflow guidance, then cover the webhook option. Every clause adds value; no repetition or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For an 8-param async tool with no annotations but a rich schema and output schema, the description covers the essential decision factors (async vs sync, polling, webhook). It doesn't mention error handling or limits, but the existing coverage is strong.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description does not add much beyond the schema's rich parameter descriptions, except tying the webhook metadata echo to the job lifecycle. It relies on the schema for parameter-level detail, which is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: submitting a scrape job that runs asynchronously and returns a job_id immediately. It also distinguishes itself from the synchronous `scrape` tool, making its purpose unambiguous even among siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly states when to use this tool (long-running scrapes) and when to prefer the synchronous alternative, and it outlines the full workflow: poll with scrape_status, fetch with scrape_result. No guesses needed.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fetch the extracted content of a completed async scrape job (the same result shape as the synchronous scrape tool: markdown, cleaned HTML, raw HTML, links, images, screenshot, page metadata in metadata, and response_meta.usage). Errors if the job is still pending/running — check scrape_status first (status done) before calling this. Screenshot URLs are signed download links.
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes | Job identifier returned by the `scrape_async` tool |
| Name | Required | Description |
|---|---|---|
| url | Yes | The URL that was actually scraped, after any redirects, in normalized form — the base that links, images, and internal labels are computed against |
| links | No | Links found on the page |
| images | No | Inline images found on the page |
| markdown | No | Page content converted to clean Markdown |
| metadata | No | Extracted page metadata (title, OG tags, etc.) |
| raw_html | No | Raw, unprocessed HTML of the page |
| warnings | No | Non-error notices about the scrape. Truncation codes — `screenshot_truncated` (long page exceeded the scrolling-screenshot height cap), `links_truncated` / `inline_images_truncated` (page had more links/images than the per-page extraction caps), `raw_html_truncated` / `metadata_truncated` (rendered HTML exceeded the per-page size budget) — mean the field is present but capped. Unavailability codes — `links_unavailable` / `inline_images_unavailable` / `metadata_unavailable` — mean that optional field could not be extracted and was omitted (null/empty) while the rest of the scrape succeeded, so an empty field carrying one of these does NOT mean the page had none. Stable string codes — clients can switch on them. Warnings are stored with the result: async result fetches and cache hits carry them too, filtered to the fields the request asked for. |
| screenshot | No | Screenshot of the page, if requested |
| cleaned_html | No | Cleaned HTML of the main page content |
| content_type | No | Content-Type header returned by the server |
| requested_url | Yes | The URL you requested, echoed verbatim — before any redirects |
| response_meta | Yes | Request-level metadata. `response_meta.usage` reports credits charged, the billed engine, the resolved proxy tier, and any screenshot-slice add-on. |
| unsupported_fields | No | Extract fields that were requested but are not supported for this content type |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden and delivers: it discloses the error-on-pending behavior, the required status precondition, that screenshot URLs are signed download links, and that the payload mirrors the sync `scrape` output. This goes well beyond the bare schema. Slight gap: no mention of rate limits or whether results are retrievable more than once, but the key operational behaviors are covered.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three compact clauses, purpose first, then error/precondition guidance, then the signed-link caveat. Dense but not padded; every sentence earns its place. Slightly long relative to the simplest tools, but the content justifies it given the cross-tool preconditions.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Complete for a one-parameter result-fetching tool: purpose, return shape, failure conditions, and prerequisite workflow are all stated, and an output schema exists to document the return values. Little an agent needs to call it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%—`job_id` is described as the identifier returned by `scrape_async`—so the baseline of 3 applies. The description adds no further parameter-level detail, but none is needed since the schema already ties the parameter to its origin tool.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource—'Fetch the extracted content of a completed async scrape job'—and explicitly contrasts the result shape with the synchronous `scrape` tool. The async-scope distinction clearly separates it from siblings `scrape_async` (submission) and `scrape` (sync execution) without needing to inspect either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit when-to-use guidance: call only after the job is `done`, and names the prerequisite tool `scrape_status` to check first. Also states the exclusion condition—errors if the job is `pending`/`running`—and references `scrape_async`'s return of `job_id`. This routes the agent unambiguously against its siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Look up the current lifecycle status of an async scrape job submitted via scrape_async. Returns the job state (pending, running, done, failed), with an error message when it failed and a response_meta.usage block (credits, billed engine, resolved proxy tier, screenshot_slices) once it is done. Poll this until the status is done, then call scrape_result to fetch the page. If you registered a completion webhook on submit you can skip polling and react to the delivery instead.
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes | Job identifier returned by the `scrape_async` tool |
| Name | Required | Description |
|---|---|---|
| error | No | Error message if the job ended in `failed` |
| job_id | Yes | The job identifier |
| status | Yes | Current state of the job (pending, running, done, failed) |
| created_at | Yes | ISO-8601 UTC timestamp when the job was created |
| response_meta | No | Usage accounting for the finished job. Present only once the job is `done`; `response_meta.usage` reports credits charged, the billed engine, the resolved proxy tier, and any screenshot-slice add-on. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It richly covers the job states, failure error message, the usage block after completion, and the polling/webhook behavior. It doesn't mention auth, rate limits, or invalid job_id behavior, but for a simple status-read tool the disclosed behavior is quite strong.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, no waste: it states the action, lists the valuable return data, gives the polling workflow, and offers the webhook shortcut. The core purpose is front-loaded and every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter, read-only status tool with an output schema, the description covers the complete workflow: what to poll for, how to know when done, what to do next, and how to avoid polling entirely. Nothing an agent needs to correctly invoke this tool is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and job_id is already described as the identifier returned by scrape_async in the schema. The description adds provenance ('returned by scrape_async'), reinforcing where the agent obtains the value, which is useful beyond the schema's basic type and description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb ('Look up'), a distinct resource ('async scrape job'), and the lifecycle scope ('status'), immediately differentiating it from scrape_result and scrape_async. The state list (pending/running/done/failed) and the note to call scrape_result after done make the tool's specific role unmistakable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly instructs to poll until done and then call scrape_result, and names the webhook alternative that removes the need for polling. This gives the agent a clear when-to-use / when-not-to-use decision relative to its siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return the current billing-cycle usage for the authenticated organization: total / used / available credits, used quota percent, max concurrency, and cycle reset time. Call this before launching large scrape jobs to confirm available credit.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| usage_reset | Yes | ISO 8601 UTC timestamp when the current billing cycle ends and used_credits resets. |
| used_credits | Yes | Credits spent so far in the current billing cycle. May exceed total_credits on plans that allow overages. |
| total_credits | Yes | Total credits available for the current billing cycle (plan base + purchased + gifted). |
| max_concurrency | Yes | Maximum number of concurrent jobs allowed for this organization (plan base + purchased + gifted extras). |
| available_credits | Yes | Remaining credits (max(0, total_credits - used_credits)). Clamped to 0 when in overage. |
| used_quota_percent | Yes | Percentage of total_credits used in the current cycle, rounded to 1 decimal. Not capped — values >100 indicate overage. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full behavioral burden. It clearly states that the tool returns usage data for the authenticated organization, lists the specific metrics, and implies a read-only operation. It does not discuss rate limits or authentication requirements, but the described behavior is transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loads the return content, and closes with a practical usage directive. Every sentence earns its place with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given zero parameters, an output schema that documents return values, and a straightforward read operation, the description covers what the tool does, what it returns, and when to use it. Nothing essential is missing for an agent to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the schema is trivially complete. The description adds no parameter-level detail because none is needed; the baseline of 4 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Return') and names the exact resource ('current billing-cycle usage for the authenticated organization') with a detailed list of returned fields. It is clearly distinct from the sibling scraping tools and whoami.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly tells the agent when to call this tool ('before launching large scrape jobs to confirm available credit'). It does not explicitly list exclusions or alternatives, but the context is clear enough for correct selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return the organization name, token name, and truncated token preview for the API key currently configured on the MCP server. Useful for confirming which account is in use before performing credit-consuming operations.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| token_name | Yes | User-assigned name of the API token authenticating the request. |
| token_preview | Yes | Truncated preview of the API token (e.g. `cwbl_…xyz`). Safe to display; does not authenticate. |
| organization_name | Yes | Display name of the organization that owns the token. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It explicitly states the operation is a return (read-only) and discloses that the token preview is truncated, which is a useful security-related behavioral trait. It does not explicitly mention side effects or credit consumption, but for a whoami-style identity check the behavior is simple and adequately conveyed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no wasted words. The first sentence front-loads the action and return values; the second adds a practical usage note. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter tool with an output schema, the description is fully complete. It states what information is returned and provides a clear reason to call the tool. No additional context is needed for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the baseline is 4. The description adds no parameter detail, but none is needed since the input schema is empty and there is nothing to explain.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description starts with the verb 'Return' and specifies a concrete resource: the organization name, token name, and truncated token preview for the currently configured API key. This clearly identifies the tool's purpose and differentiates it from the scraping, mapping, and usage sibling tools without ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives an explicit when-to-use situation: confirming which account is in use before credit-consuming operations. It does not name alternative tools or state when not to use it, but the siblings are so distinct that the context is sufficient for an agent to route correctly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v1.0.3mapscrapescrape_asyncscrape_resultscrape_statususagewhoamiScored across 7 tools
Each tool has a clearly distinct role: synchronous scraping, async submission/status/result retrieval, site mapping, and account/billing info. There is no overlap between scraping, mapping, or the utility check tools, and the async lifecycle is cleanly split into submit, poll, and fetch.
The scrape family follows a predictable pattern: scrape, scrape_async, scrape_status, scrape_result. However, map, usage, and whoami deviate from the verb_noun convention, using single-word command-style names, though the overall naming remains readable and intuitive.
7 tools is well-scoped for a scraping API: core sync and async operations, lifecycle management, site discovery, and account/usage utilities. Each tool fills a necessary role without unnecessary redundancy or bloat.
The set covers the essential scraping workflow well: discovery via map, sync scraping, async submission with polling and result retrieval, plus usage and identity checks. A minor gap is the lack of an async job cancellation or listing tool, but this does not break the primary workflows.
Web scraping for agents. Point it at a URL and it returns the page as clean markdown, JavaScript-rendered pages included. Point it at a site and it maps the URLs or crawls the section you need in the background, a few pages at a time so results fit in the conversation. Search the web and read full pages, extract fields with a JSON schema you define (validated, never invented), read a store's catalogue or a blog's posts from the platform's own feed, and check whether a page has changed. Failed requests cost nothing. The free plan includes 1,500 credits a month.
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Scrape, crawl and search the web for AI agents via MCP.
Screenshot, diff, audit and sitemap-capture any web page — 5 MCP tools for AI agents.
