Skip to main content
Glama

Scrape pages by URL

scrape_urls

Fetch pages whose URLs you already have and return their content. No project is created. Use crawl_site when the URLs must be found by following links, and map_site to list a site's URLs without fetching. list_pages with url reads a page a project already stored, for free. Inputs: urls come from the request or from map_site. formats adds bodies, e.g. 'markdown,text'. config overrides fetch settings, e.g. {"render_js": "always"}; get_project with no project_id lists the keys. full=true reads one URL in full and runs config.actions first, e.g. [{"type": "click", "selector": "#load-more", "repeat": "until_gone"}]. Cost: 1 credit for a plain fetch, 4 with a browser, +1 for a screenshot. A page the site refuses is reported as blocked and is free. Lists: a list runs as one batch. If it outlasts the connection's time limit, a job comes back for get_job, and sending the same list again returns that batch. Needs write access; a read-only connection is refused with code 'read_only'.

Input Schema

NameRequiredDescriptionDefault
fullNotrue reads the one URL in full -- raw html, head and extracted fields, images, the engine used -- running config's browser actions first. Default false.
urlsYesAbsolute http(s) URLs, 1 to 500; exactly one when full is true.
configNoFetch settings to override, as {key: value}; omit for the defaults.
formatsNoBodies per page, comma-separated from markdown (default), text, cleanHtml and rawHtml. With full=true, config's formats list chooses them instead.markdown
parse_documentsNoRead PDF, Word and spreadsheet URLs as text (default true). A parse_documents key in config wins.

Output Schema

NameRequiredDescriptionDefault
jobNoWhile a batch is still going: {kind: 'batch', id} for get_job and cancel_job.
codeNoOnly on a failure: a machine-readable reason, e.g. read_only, validation, plan_limit or not_found.
noteNoWhat was left out and how to get it.
pageNofull=true only: the page in full -- markdown, html, head {title, description, canonical, h1}, fields, images, engine, pageStatus -- with each body capped at 12,000 characters.
restNoA list past its index: {count, byStatus, notOk} for the urls not shown.
errorNoOnly when the call failed: what went wrong, in words an assistant can act on.
indexNoA list only: every url as {url, title, words, status}, so none is lost when excerpts are cut.
pagesNoThe pages read. A list: up to 50, each {url, title, words, status, links (a count), markdown excerpt}, the excerpts sharing 60,000 characters. One URL: that page in full, each body asked for capped at 12,000 characters.
detailNoOnly on a failure: the API's own wording, when the error was rephrased.
statusNo'done' once read; 'running' with a `job` while a batch goes on; with full=true, 'blocked' when the site refused and 'error' when the read failed. On an error, the HTTP status code.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed6 schema fields changed
    • changedInput schema / properties / config / description
      Previous value: -"Optional fetch settings: any subset of the keys describe_project_config lists, e.g. {\"render_js\": \"always\", \"only_main_content\": true}. Omit for the defaults."New value: +"Fetch settings to override, as {key: value}; omit for the defaults."
    • changedInput schema / properties / formats / description
      Previous value: -"Comma-separated bodies to return per page, from markdown (default), text, cleanHtml and rawHtml, e.g. 'markdown,text'."New value: +"Bodies per page, comma-separated from markdown (default), text, cleanHtml and rawHtml. With full=true, config's formats list chooses them instead."
    • addedInput schema / properties / full
      Added value: +{
      +  "default": false,
      +  "description": "true reads the one URL in full -- raw html, head and extracted fields, images, the engine used -- running config's browser actions first. Default false.",
      +  "title": "Full",
      +  "type": "boolean"
      +}
    • changedInput schema / properties / parse_documents / description
      Previous value: -"Read PDFs, Word files and spreadsheets as text (default true); false leaves their text out. A parse_documents key in config wins."New value: +"Read PDF, Word and spreadsheet URLs as text (default true). A parse_documents key in config wins."
    • changedInput schema / properties / urls / description
      Previous value: -"The pages to fetch: 1 to 500 absolute http(s) URLs."New value: +"Absolute http(s) URLs, 1 to 500; exactly one when full is true."
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
      +  "additionalProperties": true,
      +  "description": "A scrape. A list: status, the pages with excerpts and an index of every url. One URL: that page in\nfull. full=true: the extraction, with the page under `page`. A batch still going: status 'running' and\na job.",
      +  "properties": {
      +    "code": {
      +      "description": "Only on a failure: a machine-readable reason, e.g. read_only, validation, plan_limit or not_found.",
      +      "title": "Code"
      +    },
      +    "detail": {
      +      "description": "Only on a failure: the API's own wording, when the error was rephrased.",
      +      "title": "Detail"
      +    },
      +    "error": {
      +      "description": "Only when the call failed: what went wrong, in words an assistant can act on.",
      +      "title": "Error"
      +    },
      +    "index": {
      +      "description": "A list only: every url as {url, title, words, status}, so none is lost when excerpts are cut.",
      +      "title": "Index"
      +    },
      +    "job": {
      +      "description": "While a batch is still going: {kind: 'batch', id} for get_job and cancel_job.",
      +      "title": "Job"
      +    },
      +    "note": {
      +      "description": "What was left out and how to get it.",
      +      "title": "Note"
      +    },
      +    "page": {
      +      "description": "full=true only: the page in full -- markdown, html, head {title, description, canonical, h1}, fields, images, engine, pageStatus -- with each body capped at 12,000 characters.",
      +      "title": "Page"
      +    },
      +    "pages": {
      +      "description": "The pages read. A list: up to 50, each {url, title, words, status, links (a count), markdown excerpt}, the excerpts sharing 60,000 characters. One URL: that page in full, each body asked for capped at 12,000 characters.",
      +      "title": "Pages"
      +    },
      +    "rest": {
      +      "description": "A list past its index: {count, byStatus, notOk} for the urls not shown.",
      +      "title": "Rest"
      +    },
      +    "status": {
      +      "description": "'done' once read; 'running' with a `job` while a batch goes on; with full=true, 'blocked' when the site refused and 'error' when the read failed. On an error, the HTTP status code.",
      +      "title": "Status"
      +    }
      +  },
      +  "title": "ScrapeOut",
      +  "type": "object"
      +}
  2. Changed4 schema fields changed
    • addedInput schema / properties / config / description
      Added value: +"Optional fetch settings: any subset of the keys describe_project_config lists, e.g. {\"render_js\": \"always\", \"only_main_content\": true}. Omit for the defaults."
    • addedInput schema / properties / formats / description
      Added value: +"Comma-separated bodies to return per page, from markdown (default), text, cleanHtml and rawHtml, e.g. 'markdown,text'."
    • addedInput schema / properties / parse_documents / description
      Added value: +"Read PDFs, Word files and spreadsheets as text (default true); false leaves their text out. A parse_documents key in config wins."
    • addedInput schema / properties / urls / description
      Added value: +"The pages to fetch: 1 to 500 absolute http(s) URLs."
  3. First observed

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well beyond the annotations: it discloses the credit cost model (1 plain, 4 browser, +1 screenshot), that blocked pages are free, that a list runs as one batch and may return a job for get_job if it outlasts the connection limit, and that write access is required (read-only connections refused with code 'read_only'). The annotations' readOnlyHint=false is consistent with and enriched by this.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Dense but front-loaded and labelled ('Inputs:', 'Cost:', 'Lists:', 'Needs write access'), with the core action and its scope in the first sentence. Every subsequent sentence carries distinct operational information with no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 5-parameter, open-world, non-idempotent fetch tool with an output schema already present, the description covers everything an agent needs before calling: alternatives, input provenance, cost, batch/job behavior and auth requirements. Nothing material is left out.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% so the baseline is 3, and the description earns above that by supplying concrete examples: formats 'markdown,text', config {"render_js": "always"}, get_project with no project_id for valid keys, and a full=true config.actions example with a click/until_gone selector. It does overlap the schema on formats and full, so it is not maximally additive.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Fetch pages whose URLs you already have and return their content') plus an explicit scoping constraint ('No project is created'). It immediately differentiates itself from crawl_site, map_site and list_pages by naming each.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit routing rules: crawl_site when URLs must be discovered by link-following, map_site to list URLs without fetching, list_pages with url for already-stored pages 'for free'. Each alternative is paired with the condition that selects it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.