Skip to main content
Glama

Scrape pages by URL

scrape_urls

Fetch pages whose URLs you already have and return their content. No project is created. Use crawl_site when the URLs must be found by following links, and map_site to list a site's URLs without fetching. list_pages with url reads a page a project already stored, for free. Inputs: urls come from the request or from map_site. formats adds bodies, e.g. 'markdown,text'. config overrides fetch settings, e.g. {"render_js": "always"}; get_project with no project_id lists the keys. full=true reads one URL in full and runs config.actions first, e.g. [{"type": "click", "selector": "#load-more", "repeat": "until_gone"}] or [{"type": "scroll", "repeat": 5}] (each scroll waits for the page to grow); a step's unknown key is refused. formats="markdown,screenshot" also takes a picture of the page; for one URL it is given as screenshotUrl. Cost: 1 credit for a plain fetch, 4 with a browser, +1 for a screenshot. A page the site refuses is reported as blocked and is free. A short page is rendered once to see what its scripts add (example.com adds five languages), and kept at 4 when they add text; config {"render_js": "never"} reads only what the server sent, at 1. Lists: a list runs as one batch. If it outlasts the connection's time limit, a job comes back for get_job, and sending the same list again within ten minutes returns that batch, marked reused with its ageSeconds; after that, or with force_refresh=true, the URLs are read again. Needs write access; a read-only connection is refused with code 'read_only'.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "urls": "<urls>"
    },
    "name": "scrape_urls"
  }
}

Input Schema

NameRequiredDescriptionDefault
fullNotrue reads the one URL in full -- raw html, head and extracted fields, images, the engine used -- running config's browser actions first. Default false.
urlsYesAbsolute http(s) URLs, 1 to 500; exactly one when full is true.
configNoFetch settings to override, as {key: value}; omit for the defaults.
formatsNoBodies per page, comma-separated from markdown (default), text, cleanHtml and rawHtml, plus screenshot for a picture of the page (+1 credit; one URL gives it as screenshotUrl). With full=true, config's formats list chooses them instead.markdown
force_refreshNotrue reads the URLs again now, even when the same call was made in the last ten minutes (which otherwise returns that read, marked reused). Default false.
parse_documentsNoRead PDF (a scanned one by OCR) and Word (.docx) URLs as text (default true). A parse_documents key in config wins.

Output Schema

NameRequiredDescriptionDefault
jobNoWhile a batch is still going: {kind: 'batch', id} for get_job and cancel_job.
codeNoOnly on a failure: a machine-readable reason, e.g. read_only, validation, plan_limit or not_found.
noteNoWhat was left out and how to get it.
pageNofull=true only: the page in full -- markdown, html, head {title, description, canonical, h1}, fields, images, engine, pageStatus -- with each body capped at 12,000 characters.
restNoA list past its index: {count, byStatus, notOk} for the urls not shown.
errorNoOnly when the call failed: what went wrong, in words an assistant can act on.
indexNoA list only: every url as {url, title, words, status}, so none is lost when excerpts are cut. A url that did not come back ok also has reason and errorCode, e.g. status 'unreachable' with errorCode DNS_FAILED is a name that does not resolve ('missing' is a 404).
pagesNoThe pages read. A list: up to 50, each {url, title, words, status, links (a count), markdown excerpt}, the excerpts sharing 60,000 characters. One URL: that page in full, each body asked for capped at 12,000 characters.
detailNoOnly on a failure: the API's own wording, when the error was rephrased.
noticeNoOnly when a limit, not the site, held pages back: one line saying what they needed and why (the plan, the stealth trial or the credits left). Tell the person; it comes first in the answer.
reusedNotrue when this is the read an identical call made in the last ten minutes, not a new one; force_refresh=true reads again.
statusNo'done' once read; 'running' with a `job` while a batch goes on; with full=true, 'blocked' when the site refused and 'error' when the read failed. On an error, the HTTP status code.
noticesNoWith `notice`: one per reason, {reason, pages, needs, examples, unlockedBy, upgradeUrl}, `engine` on a search's and `outlook` {tier, likely, why} on a page's; unlockedBy is the plan that reads them, null for credits or a cap the workspace set itself.
ageSecondsNoWith reused: how many seconds ago that read finished.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • changedInput schema / properties / formats / description
      Previous value: -"Bodies per page, comma-separated from markdown (default), text, cleanHtml and rawHtml. With full=true, config's formats list chooses them instead."New value: +"Bodies per page, comma-separated from markdown (default), text, cleanHtml and rawHtml, plus screenshot for a picture of the page (+1 credit; one URL gives it as screenshotUrl). With full=true, config's formats list chooses them instead."
  2. Changed1 schema field changed
    • changedOutput schema / properties / notices / description
      Previous value: -"With `notice`: one per reason, {reason, pages, needs, examples, unlockedBy, upgradeUrl}, and `engine` on a search's; unlockedBy is the plan that reads them, null for credits or a cap the workspace set itself."New value: +"With `notice`: one per reason, {reason, pages, needs, examples, unlockedBy, upgradeUrl}, `engine` on a search's and `outlook` {tier, likely, why} on a page's; unlockedBy is the plan that reads them, null for credits or a cap the workspace set itself."
  3. Changed7 schema fields changed
    • addedInput schema / properties / force_refresh
      Added value: +{
      +  "default": false,
      +  "description": "true reads the URLs again now, even when the same call was made in the last ten minutes (which otherwise returns that read, marked reused). Default false.",
      +  "title": "Force Refresh",
      +  "type": "boolean"
      +}
    • changedInput schema / properties / parse_documents / description
      Previous value: -"Read PDF, Word and spreadsheet URLs as text (default true). A parse_documents key in config wins."New value: +"Read PDF (a scanned one by OCR) and Word (.docx) URLs as text (default true). A parse_documents key in config wins."
    • addedOutput schema / properties / ageSeconds
      Added value: +{
      +  "description": "With reused: how many seconds ago that read finished.",
      +  "title": "Ageseconds"
      +}
    • changedOutput schema / properties / index / description
      Previous value: -"A list only: every url as {url, title, words, status}, so none is lost when excerpts are cut."New value: +"A list only: every url as {url, title, words, status}, so none is lost when excerpts are cut. A url that did not come back ok also has reason and errorCode, e.g. status 'unreachable' with errorCode DNS_FAILED is a name that does not resolve ('missing' is a 404)."
    • addedOutput schema / properties / notice
      Added value: +{
      +  "description": "Only when a limit, not the site, held pages back: one line saying what they needed and why (the plan, the stealth trial or the credits left). Tell the person; it comes first in the answer.",
      +  "title": "Notice"
      +}
    • addedOutput schema / properties / notices
      Added value: +{
      +  "description": "With `notice`: one per reason, {reason, pages, needs, examples, unlockedBy, upgradeUrl}, and `engine` on a search's; unlockedBy is the plan that reads them, null for credits or a cap the workspace set itself.",
      +  "title": "Notices"
      +}
    • addedOutput schema / properties / reused
      Added value: +{
      +  "description": "true when this is the read an identical call made in the last ten minutes, not a new one; force_refresh=true reads again.",
      +  "title": "Reused"
      +}
  4. Changed6 schema fields changed
    • changedInput schema / properties / config / description
      Previous value: -"Optional fetch settings: any subset of the keys describe_project_config lists, e.g. {\"render_js\": \"always\", \"only_main_content\": true}. Omit for the defaults."New value: +"Fetch settings to override, as {key: value}; omit for the defaults."
    • changedInput schema / properties / formats / description
      Previous value: -"Comma-separated bodies to return per page, from markdown (default), text, cleanHtml and rawHtml, e.g. 'markdown,text'."New value: +"Bodies per page, comma-separated from markdown (default), text, cleanHtml and rawHtml. With full=true, config's formats list chooses them instead."
    • addedInput schema / properties / full
      Added value: +{
      +  "default": false,
      +  "description": "true reads the one URL in full -- raw html, head and extracted fields, images, the engine used -- running config's browser actions first. Default false.",
      +  "title": "Full",
      +  "type": "boolean"
      +}
    • changedInput schema / properties / parse_documents / description
      Previous value: -"Read PDFs, Word files and spreadsheets as text (default true); false leaves their text out. A parse_documents key in config wins."New value: +"Read PDF, Word and spreadsheet URLs as text (default true). A parse_documents key in config wins."
    • changedInput schema / properties / urls / description
      Previous value: -"The pages to fetch: 1 to 500 absolute http(s) URLs."New value: +"Absolute http(s) URLs, 1 to 500; exactly one when full is true."
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
      +  "additionalProperties": true,
      +  "description": "A scrape. A list: status, the pages with excerpts and an index of every url. One URL: that page in\nfull. full=true: the extraction, with the page under `page`. A batch still going: status 'running' and\na job.",
      +  "properties": {
      +    "code": {
      +      "description": "Only on a failure: a machine-readable reason, e.g. read_only, validation, plan_limit or not_found.",
      +      "title": "Code"
      +    },
      +    "detail": {
      +      "description": "Only on a failure: the API's own wording, when the error was rephrased.",
      +      "title": "Detail"
      +    },
      +    "error": {
      +      "description": "Only when the call failed: what went wrong, in words an assistant can act on.",
      +      "title": "Error"
      +    },
      +    "index": {
      +      "description": "A list only: every url as {url, title, words, status}, so none is lost when excerpts are cut.",
      +      "title": "Index"
      +    },
      +    "job": {
      +      "description": "While a batch is still going: {kind: 'batch', id} for get_job and cancel_job.",
      +      "title": "Job"
      +    },
      +    "note": {
      +      "description": "What was left out and how to get it.",
      +      "title": "Note"
      +    },
      +    "page": {
      +      "description": "full=true only: the page in full -- markdown, html, head {title, description, canonical, h1}, fields, images, engine, pageStatus -- with each body capped at 12,000 characters.",
      +      "title": "Page"
      +    },
      +    "pages": {
      +      "description": "The pages read. A list: up to 50, each {url, title, words, status, links (a count), markdown excerpt}, the excerpts sharing 60,000 characters. One URL: that page in full, each body asked for capped at 12,000 characters.",
      +      "title": "Pages"
      +    },
      +    "rest": {
      +      "description": "A list past its index: {count, byStatus, notOk} for the urls not shown.",
      +      "title": "Rest"
      +    },
      +    "status": {
      +      "description": "'done' once read; 'running' with a `job` while a batch goes on; with full=true, 'blocked' when the site refused and 'error' when the read failed. On an error, the HTTP status code.",
      +      "title": "Status"
      +    }
      +  },
      +  "title": "ScrapeOut",
      +  "type": "object"
      +}
  5. Changed4 schema fields changed
    • addedInput schema / properties / config / description
      Added value: +"Optional fetch settings: any subset of the keys describe_project_config lists, e.g. {\"render_js\": \"always\", \"only_main_content\": true}. Omit for the defaults."
    • addedInput schema / properties / formats / description
      Added value: +"Comma-separated bodies to return per page, from markdown (default), text, cleanHtml and rawHtml, e.g. 'markdown,text'."
    • addedInput schema / properties / parse_documents / description
      Added value: +"Read PDFs, Word files and spreadsheets as text (default true); false leaves their text out. A parse_documents key in config wins."
    • addedInput schema / properties / urls / description
      Added value: +"The pages to fetch: 1 to 500 absolute http(s) URLs."
  6. First observed

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (which only say non-read-only, open-world, non-destructive), it discloses credit costs per mode, that blocked pages are free, the one-render probe for short pages and how render_js changes cost, batch/timeout behaviour with job fallback and a ten-minute reuse window with ageSeconds, and that write access is required with a 'read_only' error code.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

It is long but well front-loaded and segmented (purpose/alternatives, then inputs, cost, list behaviour, auth), and every block answers a distinct question an agent would ask. A few clauses repeat schema content, e.g. the formats enumeration, which could be trimmed.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a six-parameter, credit-metered, network-facing tool, everything an agent must know to call it correctly and interpret the outcome is present: cost model, caching/reuse semantics, long-run job fallback, rendering overrides, and the write-access precondition. Return values are covered by the output schema, so their absence from the description is appropriate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds genuinely new meaning: config accepts keys discoverable via get_project with no project_id, and config.actions carries a step DSL with worked examples (click with repeat 'until_gone', scroll with repeat) plus the rule that an unknown step key is refused — none of which is in the schema. Some formats text ('markdown,text') merely restates the schema, keeping it from a 5.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence names the specific verb and resource ('Fetch pages ... return their content') and states the defining boundary ('No project is created'), which immediately separates it from crawl_site, map_site and list_pages.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It routes three alternatives explicitly with the condition that selects each: crawl_site when URLs must be discovered by following links, map_site to list URLs without fetching, list_pages when a project already stored the page (and is free). This is exactly the when-to-use/when-not guidance an agent needs.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.