What can you do with this server?

This server provides read-only web scraping and page-analysis tools: it extracts declared structured data, learns and replays drift-checked extractors, fetches pages, audits markup, reads feeds, maps/crawls sites, and tests selectors, all without LLM calls. * Extract a page's declared data (JSON-LD, microdata, RDFa, OpenGraph, HTML meta, etc.) with per-value provenance, conflicts, summary answers, optional induced rows, visible-text guesses, and Wayback Machine captures. * Read a page's main content as Markdown, with optional YAML front matter and paged offsets. * Fetch a page's raw HTML in slices, reporting the fetch method used. * Compile an extractor from example values or from CSS/XPath selectors, run it on new pages, and detect when a site's layout has drifted. * Heal an extractor after a redesign, proposing where moved fields now live with evidence, and reporting lost fields. * Audit a page's structured data against Google's documented rich-result requirements, plus robots.txt, llms.txt, and TDMRep checks. * Read RSS, Atom, and JSON Feed items, with normalized dates and enclosure metadata. * Map a site's URLs from sitemaps or start-page links. * Crawl a site politely, following same-site links up to configurable depth/page limits, and summarize each page. * Batch-extract summaries/records from multiple URLs in one polite pass. * Test CSS or XPath selectors against a page to see the values and locations they match.

Which integrations are available for this server?

Allows reading RSS feeds and retrieving their items, along with Atom and JSON Feed support.

sluicer

by Gi0tto

Overview Schema Source Related Servers Discussions Admin

Python

Hybrid

Most scrapers break without a sound. The site changes its markup, and the scraper keeps running and returns nulls, or the wrong column, for weeks before anyone notices.

Sluicer works the other way round. Show it a value on a few pages of a site (a price, a title, a date) and it learns where that value lives. On every page it reads after that, it checks the page against what it learnt. If the layout has changed, the run fails with exit code 3 and names the check that broke. sluicer heal then proposes where each field went, when the new page still shows values the extractor was learnt from.

It also reads everything a page already declares about itself: JSON-LD, microdata, RDFa, OpenGraph and four more vocabularies, merged into one record per thing, with every value pointing to the exact place on the page it came from. No model reads any page, so the same page always gives the same answer.

Quick start

uv tool install sluicer   # or: pipx install sluicer

# Learn where a book's title and price sit, from two pages of one template.
sluicer compile https://books.toscrape.com/catalogue/page-1.html \
  https://books.toscrape.com/catalogue/page-2.html \
  --want title="A Light in the Attic" --want price=51.77 -o books.json

# Replay it on another page of that template: 20 rows of title and price, exit 0.
sluicer run books.json https://books.toscrape.com/catalogue/page-3.html

# A page it was not learnt for: exit 3, naming the check that failed.
sluicer run books.json https://quotes.toscrape.com/

books.toscrape.com and quotes.toscrape.com are public sandboxes made for trying scrapers on. The last command prints FAILED https://quotes.toscrape.com/: expected the listing at html>body>…>ol.row, got not found, its path shortened here, and exits 3.

After a redesign, sluicer heal looks on the new page for the values the extractor was learnt from, and proposes a new place for each field it finds them in. In a clone of this repository, examples/shop/ holds a made-up shop before and after a redesign that renamed every class:

sluicer compile examples/shop/before-1.html examples/shop/before-2.html \
  --want title="A Light in the Attic" --want price=51.77 -o shop.json
sluicer run shop.json examples/shop/after.html   # exit 3: the listing is not found
sluicer heal shop.json examples/shop/after.html -o shop-healed.json

container: html>body>div.page>ol.row -> html>body>main.content>div.page>section.grid
member: li.product -> div.card
moved: title -> h2.name>a (5 of 5 learnt values found there; the next best place had 0)
moved: price -> div.cost (5 of 5 learnt values found there; the next best place had 0)
Wrote shop-healed.json.

Each move says how many of the field's learnt values were found in its new place, and how many the next best place held. heal proposes; it does not repair. It can only move a field whose old values the new page still shows, so run it on a page you learnt from, or one listing the same items. On the drift benchmark's 21 real redesigns, 18 of the new pages shared no item with the old ones, and heal was fully right on none and partly right on 2. When a field or the listing is lost, it exits 3 and writes nothing unless given --force.

Exit codes follow grep: 0 found -- a record or a summary answer, a <title> alone included -- 1 found nothing, 2 could not read the page, and 3 when a page broke its extractor's checks, a heal lost a field or left a move undecided, or an audit found a documented rule broken. diff exits 1 when something changed.

The checks are about structure, not truth: a run fails when a field is no longer where it was learnt, no longer reads the way it did, or no longer has its shape. A change that keeps all three, such as a different number in the price's place, passes. On SWDE, the checks flagged 18% of the extractors' wrong answers; the other 82% passed.

Reading what a page declares needs no example at all. The product page read here is examples/brake-pads.html, in a clone of this repository; without one, fetch it first with mkdir -p examples && curl -o examples/brake-pads.html https://raw.githubusercontent.com/Gi0tto/sluicer/main/examples/brake-pads.html. The library is imported from the Python you ran pip install sluicer in:

>>> import sluicer
>>> page = open("examples/brake-pads.html", "rb").read()
>>> result = sluicer.extract(page, url="https://example.com/p/bp-2210")
>>> price = result.summary["price"]
>>> price.value, price.source, price.key
('41.90', 'jsonld', 'Product.offers.price')
>>> price.where
'/html/head/script[1]#/offers/price'
>>> result.normalised
{'price': '41.90', 'currency': 'EUR', 'gtin': '4006381333931'}
>>> [(answer.value, answer.source) for answer in result.conflicts[0].answers]
[('41.90', 'jsonld'), ('39.90', 'opengraph')]

The page states one price in its JSON-LD and another in its OpenGraph tags; Sluicer reports the conflict instead of picking one in silence. sluicer inspect page.html shows the same reading laid out for a person.

Related MCP server: mcp-untun

Install

uv tool install sluicer   # the sluicer command, in an environment of its own
uvx sluicer --version     # or run it once, installing nothing
pip install sluicer       # the library, in your project's virtual environment

pipx install sluicer works as uv tool install does, and uv add sluicer adds the library to a uv project. Extras go in brackets, as in uv tool install "sluicer[browser,markdown,mcp]". The base install reads HTML you already have and fetches pages over plain HTTP, with lxml, click, cssselect (CSS selectors) and protego (robots.txt) alone, and tomli on Python 3.10 to read a configuration file: the HTTP client is Python's own.

extra	adds
`browser`	a browser, Playwright's Chromium, for a page plain HTTP brings back as an empty shell
`stealth`	the stealth rung, by scrapling: one page, only when asked with `--stealth`, never in a crawl
`markdown`	a page's main content as Markdown, by trafilatura
`mcp`	the MCP server, with `markdown`; add `browser` for pages that need one
`api`	the HTTP API, with `mcp`
`microformats`	microformats2, which is off by default
`fetch`	deprecated since 0.8: `browser` and `stealth` together, what it installed before

To let Sluicer use a browser, install one once: uvx --from "sluicer[browser]" playwright install chromium. Without it, plain HTTP still works, and a page that needed a browser says so.

What you can give it

you have	run	and get
a page, as HTML or a URL	`sluicer extract page.html`	every record it declares, a summary that answers 25 questions, its conflicts, each value with where it came from
many pages of one template	`sluicer compile ... --want price=41.90`, then `sluicer run`	the fields you gave an example of, from every page, checked
the selectors you already know	`sluicer compile --select price='span.price::text' ...`, then `sluicer run`	the fields you named, held to the same checks: a selector a redesign broke fails the run
a page that declares nothing	`sluicer extract page.html --induce`	the rows its markup repeats: a listing's cards, a table's lines
a whole site	`sluicer map URL`, `sluicer crawl URL -o site.jsonl`	its addresses from its sitemaps, or every page it links to, read politely
a list of URLs, a web archive	`sluicer batch urls.txt`, `sluicer warc crawl.warc.gz`	one JSON line per page
a feed	`sluicer feed URL`	its items, as one JSON document
an article	`sluicer markdown URL`	its main text as Markdown

Every request names Sluicer, and robots.txt and Crawl-delay are obeyed; a crawl waits when a site asks it to. Only a command that reads single pages or learns an extractor can be told otherwise, with --stealth or --no-robots; map, crawl and batch cannot. sluicer --help lists every command, and the command line reference explains each one.

In your agent

claude mcp add sluicer -- uvx --with "sluicer[mcp]" sluicer mcp   # Claude Code
codex mcp add sluicer -- uvx --with "sluicer[mcp]" sluicer mcp    # Codex

The MCP server has twelve read-only tools, among them extract_declared, select_values, compile_extractor, run_extractor and heal_extractor. Every answer carries ok, true only when it can be used as it is. The server does not fetch localhost, private networks or cloud metadata addresses unless it is started with SLUICER_ALLOW_PRIVATE=1. In your agent covers Cursor, VS Code, Gemini CLI, Claude Desktop, Zed, LangChain, the OpenAI Agents SDK and Pydantic AI. For any other language, sluicer serve offers the same tools over HTTP (HTTP API), and at /mcp the MCP server itself over streamable HTTP, for n8n, Dify and any client that does not start servers over stdio (Over HTTP).

Measured, losses included

Every number below comes from a public test set, and every scoreboard gives the command that produces it again. Where another tool does better on what a row measures, the row shows it.

scoreboard	what is measured	Sluicer	beside it
SWDE, 80 sites	extractors learnt from three pages, run on the other 124,051	F1 0.850, 11,156 wrong answers	Scrapling 0.671, 56,058 wrong
Drift, 44 before/after pairs on 25 sites	a change noticed: 21 changed, 23 did not	0 failed silently, 0 false alarms	--
Products, 140 pages	price, availability (F1)	0.750, 0.907	Zyte's paid API 0.918, 0.957
WCXB, 511 pages	title, author, date found; dates invented	0.727, 0.532, 0.581; 8 invented	trafilatura 0.745, 0.750, 0.838; 216 invented
As served, 360 pages	dates found; right when it answers	0.780; 0.734	trafilatura 0.855; 0.393
News, 21 languages	title, author, date found	0.871, 0.829, 0.970	trafilatura 0.852, 0.879, 0.970
trafilatura's set, 851 annotated pages	title, author, date found	0.776, 0.468, 0.585	trafilatura 0.738, 0.669, 0.865

Sluicer reads only what a page states in its markup, so on titles, authors and dates it answers less often than tools that also read the visible text, and it invents far fewer dates. Its rules were written while reading the pages of these scoreboards, so the numbers show how it does on pages it was tuned on. The one held-out test is the half of SWDE's sites whose pages and errors were not read while making the rules; its numbers were read at each release and a few times besides, each listed there, once to decide whether to keep a rule. There it scores 0.845, including five camera sites that were read before the split and so are not a clean test. bench/PREREG.md records which pages each rule was made on.

When not to use Sluicer

You need authors or dates that pages do not declare. trafilatura reads them from the visible text and finds more of them. Sluicer's --visible option guesses them too: on the scoreboards it finds more of them and invents some, and on two of them its dates are right less often when it answers.
You need an article's full text. sluicer markdown uses trafilatura for it; if you need trafilatura's options or other output formats, use it directly.
The site blocks bots. Sluicer is not built to get past bot protection: every request names it, unless a single-page command is given --stealth.

Why Sluicer compares it with extruct, trafilatura, Scrapling, Crawl4AI and Firecrawl, and says when each is the better choice.

Learn more

Getting started, a ten-minute tour, and Extractors, how learning, checking and healing work.
FAQ, Known limits and What is stable before 1.0.
Moving from extruct: from sluicer.compat import extruct answers extruct's own calls.
The full documentation is at https://gi0tto.github.io/sluicer/.

Community

Questions, ideas and what you built go to Discussions. A page Sluicer read wrong is an issue: attach the page, so the fix comes with a test. Report a vulnerability privately, as SECURITY.md explains, and see CONTRIBUTING.md to set up a checkout.

Sluicer is built and maintained by one person. If it saves you time or money, sponsoring it pays for keeping the scoreboards measured and the extractors working as the web changes.

Licence

MIT, except two data files under their own licences: schema.org's type names (CC BY-SA 3.0) and CLDR's month and weekday names (Unicode License v3); the package's licence expression is MIT AND CC-BY-SA-3.0 AND Unicode-3.0. The base install needs lxml, click, cssselect and protego, all BSD-3-Clause, and on Python 3.10 tomli, MIT. The extras pull a wider tree that is not all permissive: tld is MPL-1.1, GPL-2.0-only or LGPL-2.1-or-later, orjson is MPL-2.0 alongside Apache-2.0 or MIT, and certifi is MPL-2.0. CI lists every licence in that tree and fails on one nobody has read; NOTICE says more.

Available Tools

12 tools

audit_pageAudit a page's markupA

Read-onlyIdempotent

Check a page's structured data against what Google documents for it.

html_or_url: an http(s) URL to fetch, or the HTML itself. site: for a URL, also read the site's robots.txt, llms.txt and llms-full.txt.

Returns {"ok", "url", "records", "page", "not_checked", "errors", "warnings", "notes"}, and for a URL read with site "crawlers", "robots_txt", "other_agents", "llms_txt", "llms_full_txt" and "fetch". Every JSON-LD, microdata and RDFa record lists the rich-result features its type is documented for, each with requirements_met and the required and recommended properties it lacks, and findings that each name a severity, the record's source, the property path and the URL of the rule. crawlers says, per AI agent from its vendor's own page, whether robots.txt admits the page. ok is true whenever the audit ran: a page with errors is an answer; "not_checked" says what was not. errors, warnings and notes count everything found; past 75,000 bytes the last records, page findings and other_agents are left out, counted in records_left_out, page_left_out and other_agents_left_out.

ParametersJSON Schema

Name	Required	Description	Default
`site`	No	for a URL, also read the site's robots.txt, llms.txt and llms-full.txt.
`html_or_url`	Yes	an http(s) URL to fetch, or the HTML itself.

Output Schema

ParametersJSON Schema

Name	Required	Description
`ok`	Yes
`tdm`	No
`url`	No
`page`	No
`error`	No
`fetch`	No
`notes`	No
`errors`	No
`records`	No
`crawlers`	No
`llms_txt`	No
`warnings`	No
`robots_txt`	No
`not_checked`	No
`other_agents`	No
`llms_full_txt`	No
`page_left_out`	No
`records_left_out`	No
`other_agents_left_out`	No

TDQS

A3.7/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description goes well beyond the readOnly/idempotent annotations, disclosing nuanced behavior: ok is true even when the audit finds errors, not_checked records skipped items, and output is truncated past 75,000 bytes with counts in records_left_out, page_left_out, and other_agents_left_out. This is exemplary transparency about limits and semantics.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense and technically heavy, but every sentence adds information: input modes, the site flag, return fields, rule details, and truncation limits. It is front-loaded with the core purpose before diving into output specifics, so it earns a high mark despite its length.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a two-parameter tool with a rich output contract, the description explains everything an agent needs: how to pass a URL versus raw HTML, what site adds, what each output section means, how errors affect ok, and what happens when the result is too large. Nothing critical is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, and the description essentially restates the schema descriptions for html_or_url and site rather than adding new semantic value. The baseline of 3 applies because the schema already documents both parameters adequately.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action and resource: "Check a page's structured data against what Google documents for it." This clearly distinguishes the audit/validation purpose from sibling tools like fetch_page or page_markdown. It does not name an alternative sibling, so it falls just short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explains parameter behavior and output shape, but it gives no guidance on when to choose this tool over the listed siblings, nor does it define exclusions or prerequisites. An agent must infer usage from the phrase 'Check a page's structured data against what Google documents'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

compile_extractorLearn an extractorA

Read-onlyIdempotent

Learn an extractor from pages of one template, to replay later for free.

pages: http(s) URLs, or the HTML itself, of pages built from one template -- two or three pages of one listing, or of one kind of product page. listing: learn the rows the pages repeat; by default only where they declare nothing about a thing. want: example values one row of the listing holds, by the name each column is to have, as {"price": "41.90", "title": "Brake pad set"}: they choose the listing and the columns, and only those columns are kept. A value that no row holds is an error that names it. select: instead of want, each field by a CSS or XPath selector you write, as {"title": "h1", "price": "span.price::text"}; pages may then be empty. Where the fields are is not learnt; what the pages show of them is, and a run fails when a selector finds nothing. Try a selector first with select_values. rows: with select, the selector of a listing's rows ("li.product"), each field then read inside each row.

Returns {"ok", "extractor"}: keep that object and hand it to run_extractor. It holds what the pages declared, the listing's place, its fields, and what every field looked like. An extractor heavier than one answer may be, 75,000 bytes, is too_large: sluicer compile writes it to a file.

ParametersJSON Schema

Name	Required	Description
`rows`	No	with select, the selector of a listing's rows ("li.product"), each field then read inside each row.
`want`	No	example values one row of the listing holds, by the name each column is to have, as {"price": "41.90", "title": "Brake pad set"}: they choose the listing and the columns, and only those columns are kept. A value that no row holds is an error that names it.
`pages`	Yes	http(s) URLs, or the HTML itself, of pages built from one template -- two or three pages of one listing, or of one kind of product page.
`select`	No	instead of want, each field by a CSS or XPath selector you write, as {"title": "h1", "price": "span.price::text"}; pages may then be empty. Where the fields are is not learnt; what the pages show of them is, and a run fails when a selector finds nothing. Try a selector first with select_values.
`listing`	No	learn the rows the pages repeat; by default only where they declare nothing about a thing.

Output Schema

ParametersJSON Schema

Name	Required	Description
`ok`	Yes
`error`	No
`extractor`	No

TDQS

A4.6/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is covered. The description adds valuable behavioral context: it explains that a value no row holds is an error that names it, that a run fails when a selector finds nothing, that pages may be empty with select, and that an extractor heavier than 75,000 bytes is too_large and sluicer compile writes it to a file. It also discloses what the returned extractor contains. This goes beyond the annotations without contradicting them.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but well-structured, with each parameter name bolded and followed by a concise explanation. It front-loads the core purpose in the first sentence and then systematically covers each parameter. Some sentences are long and packed with multiple clauses, which slightly reduces scannability, but every sentence earns its place and there is no fluff.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description is complete for a tool with 5 parameters, 100% schema coverage, and an output schema. It explains the input requirements (two or three pages of one template), the two modes (want vs select), the error conditions, the output format, and the next step (hand to run_extractor). The output schema exists, so the description needn't detail return values, but it still summarizes what the extractor holds. The only minor gap is not explaining what 'declared' means in the listing context, but the schema and examples make it understandable.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all five parameters. The description adds meaning by explaining the relationship between parameters: want and select are alternatives ('instead of want'), rows is used 'with select', and listing's default behavior is clarified ('by default only where they declare nothing about a thing'). It also gives concrete examples of want and select values. This is more than the schema alone provides, though the schema already carries the parameter names and basic descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Learn an extractor from pages of one template, to replay later for free.' It clearly distinguishes this from siblings like run_extractor (replay) and select_values (try a selector first). The title 'Learn an extractor' is expanded with concrete detail about what learning means and what the output is for.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit when-to-use guidance: use it to learn an extractor from pages of one template, and explicitly says 'Try a selector first with select_values' when using select. It also names run_extractor as the tool to hand the returned extractor to. The distinction between want (example values) and select (CSS/XPath selectors) is clearly explained, and the 'listing' parameter's default behavior is stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_siteCrawl a siteA

Read-onlyIdempotent

Crawl a site from url, following its links, and summarise every page.

url: an http(s) address where the crawl starts; only links on its site are followed. max_pages: the most pages taken, 1 to 25. max_depth: the most links from url, 0 to 3; 0 reads url alone. respect_tdm: give a page whose site reserves its text and data mining rights (TDMRep) as a tdm_reserved error, never its summary. include: text an address must contain for its link to be followed (any one of them); plain text, not a pattern. exclude: text that stops a link being followed when its address contains it.

Returns {"ok", "url", "pages", "stopped"}. Pages come breadth first, each {"ok", "url", "depth", "found_on", "landed", "fetch", "canonical", "summary", "sources", "types", "links"} -- the summary and the types declared, not the records; call extract_declared on a page for those -- or, when it has nothing, {"ok": false, "error"} with the page's reason. stopped is "done", "max_pages" (links were left unfollowed) or "time_budget" (a minute passed). One request at a time, a second apart or the site's Crawl-delay, robots.txt obeyed; a page asked again after a request that may succeed later says so in "retries". ok is false only when no page could be read, and error then says why. Past 75,000 bytes the heaviest summary answers of any page go first, named in that page's summary_left_out, then the last pages, counted in pages_left_out.

ParametersJSON Schema

Name	Required	Description
`url`	Yes	an http(s) address where the crawl starts; only links on its site are followed.
`exclude`	No	text that stops a link being followed when its address contains it.
`include`	No	text an address must contain for its link to be followed (any one of them); plain text, not a pattern.
`max_depth`	No	the most links from url, 0 to 3; 0 reads url alone.
`max_pages`	No	the most pages taken, 1 to 25.
`respect_tdm`	No	give a page whose site reserves its text and data mining rights (TDMRep) as a tdm_reserved error, never its summary.

Output Schema

ParametersJSON Schema

Name	Required	Description
`ok`	Yes
`url`	No
`error`	No
`pages`	No
`stopped`	No
`pages_left_out`	No

TDQS

A4.5/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate read-only, idempotent, non-destructive behavior, and the description substantially expands on this with robots.txt compliance, rate limiting, one-request-at-a-time, retries, TDM handling, breadth-first order, time budget, and size-limit behavior. No contradictions with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but every sentence adds necessary detail, organized as purpose, parameters, return shape, then operational constraints. The structure is front-loaded and scannable despite its density.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity and six parameters, the description fully covers input, output shape, error cases, stopping conditions, rate limiting, robots.txt, and size limits. It even routes the agent to extract_declared for page records, so nothing needed to call the tool correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all six parameters. The description mostly restates the schema's parameter descriptions; it adds useful behavioral context around respect_tdm and include/exclude, but no major new parameter-level semantics beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific action ('Crawl a site from url, following its links, and summarise every page') with a clear resource and scope. This differentiates it from siblings like fetch_page (single page) and extract_declared (records extraction).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description clearly conveys when crawling is appropriate and even points to extract_declared as the alternative for extracting declared records. It does not explicitly contrast with map_site or fetch_page, but the crawl/summarize purpose and max_depth=0 behavior provide clear context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_declaredExtract a page's declared dataA

Read-onlyIdempotent

Read the structured data a page declares, with where each value came from.

html_or_url: an http(s) URL to fetch, or the HTML itself. induce: also read repeated rows (a listing, a feed) from a page that declares nothing about them; those fields say source "induced". at: a date (2024, 2024-06, 2024-06-01): read the URL as the Wayback Machine captured it nearest to then; "fetch" says which capture. respect_tdm: answer tdm_reserved instead of the page when the site reserves its text and data mining rights (TDMRep: its tdmrep.json, headers or meta tags). records: also return every record, not only the summary and what was normalised; false keeps the answer small. Records that would make the answer larger than 75,000 bytes are left out and counted in records_left_out. visible: also guess the title, author, publication and update dates the page shows a reader, in "visible", each {"value", "where", "rule"}; guesses, never part of the summary, which holds only what the page declares.

Returns {"ok", "url", "summary", "records", "sources"}, and "fetch" for a URL. records are typed fields, each {"value", "source", "where"}: source is the vocabulary that declared it (jsonld, microdata, opengraph, html, ...), where the XPath of the element that did -- for JSON-LD the block's, with a JSON pointer after "#" -- or null for a meta tag, whose key is its place. A nested value such as a price inside "offers" arrives whole. summary answers title, author, date, price and the rest, one value each, naming its source, key and where. conflicts lists each question the page answers two ways that mean different things -- a price of 41.90 in JSON-LD and 39.90 in OpenGraph -- the summary's answer first: say so rather than trusting either. On failure ok is false and "error" says why; there is never a record. An answer weighs at most 75,000 bytes: past it the records go first, counted in records_left_out, then the conflicts, counted in conflicts_left_out, then the heaviest summary, visible, normalised and links entries, each named in summary_left_out, visible_left_out, normalised_left_out or links_left_out.

ParametersJSON Schema

Name	Required	Description
`at`	No	a date (2024, 2024-06, 2024-06-01): read the URL as the Wayback Machine captured it nearest to then; "fetch" says which capture.
`induce`	No	also read repeated rows (a listing, a feed) from a page that declares nothing about them; those fields say source "induced".
`records`	No	also return every record, not only the summary and what was normalised; false keeps the answer small. Records that would make the answer larger than 75,000 bytes are left out and counted in records_left_out.
`visible`	No	also guess the title, author, publication and update dates the page shows a reader, in "visible", each {"value", "where", "rule"}; guesses, never part of the summary, which holds only what the page declares.
`html_or_url`	Yes	an http(s) URL to fetch, or the HTML itself.
`respect_tdm`	No	answer tdm_reserved instead of the page when the site reserves its text and data mining rights (TDMRep: its tdmrep.json, headers or meta tags).

Output Schema

ParametersJSON Schema

Name	Required	Description
`ok`	Yes
`url`	No
`error`	No
`fetch`	No
`links`	No
`rights`	No
`records`	No
`sources`	No
`summary`	No
`visible`	No
`conflicts`	No
`normalised`	No
`links_left_out`	No
`records_left_out`	No
`summary_left_out`	No
`visible_left_out`	No
`conflicts_left_out`	No
`normalised_left_out`	No

TDQS

A4.2/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description goes far beyond the annotations (readOnlyHint, idempotentHint, destructiveHint) by disclosing detailed behaviors: the exact return shape, truncation at 75,000 bytes with counters for left-out items, conflict reporting, failure semantics, and TDM handling. This provides critical operational transparency the annotations do not cover.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but well-structured: purpose first, then per-parameter details, then the return format and edge cases. It front-loads the core intent and uses a consistent layout, though the extensive prose about truncation and conflicts could be tightened. Overall it earns its place, but the length prevents a perfect score.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity, the description is highly complete: it covers all six parameters, the return object (even though an output schema exists), failure modes, size limits, special options (Wayback, TDM), and conflict behavior. An agent has enough context to call the tool correctly even without examining the output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is 100% and the tool description repeats the parameter descriptions verbatim without adding new parameter-specific meaning. The return-format discussion indirectly clarifies the effect of parameters like records and visible, but no additional parameter semantics are introduced beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource: 'Read the structured data a page declares, with where each value came from.' This clearly distinguishes the tool from siblings like run_extractor or read_feed by focusing on declared data with provenance. The title 'Extract a page's declared data' is consistent and informative.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description does not explicitly state when to use this tool versus alternatives such as run_extractor, read_feed, or extract_many. It implies usage through its purpose and parameter descriptions (e.g., at for Wayback, respect_tdm for TDM), but offers no direct guidance on sibling selection or when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_manyExtract several pagesA

Read-onlyIdempotent

Read several pages' declared data, politely, in the order given.

urls: 1 to 25 http(s) addresses, on one site or several; one given twice is read once. records: also return each page's records, not only its summary; the heaviest pages' records are left out first to keep the answer under 75,000 bytes, each counted in that page's records_left_out. induce: also read repeated rows from a page that declares nothing about them; those fields say source "induced". respect_tdm: give a page whose site reserves its text and data mining rights (TDMRep) as a tdm_reserved error, never its data.

Returns {"ok", "pages", "stopped"}. Pages come in the order given, each {"ok", "url", "landed", "fetch", "canonical", "summary", "sources", "types", "links"}, and "records" when asked -- or, when it has nothing, {"ok": false, "error"} with the page's reason. A page asked again after a request that may succeed later says so in "retries". Each site is asked one request at a time, a second apart or its Crawl-delay, robots.txt obeyed; several sites at once. stopped is "done", or "time_budget" when a minute passed first and the pages after are left out. ok is false only when no page could be read, and error then says why. Past 75,000 bytes the heaviest pages' records go first, then the heaviest summary answers, named in summary_left_out, then the last pages, counted in pages_left_out. For many more addresses, or a whole site, the command line's sluicer batch has no such bounds.

ParametersJSON Schema

Name	Required	Description
`urls`	Yes	1 to 25 http(s) addresses, on one site or several; one given twice is read once.
`induce`	No	also read repeated rows from a page that declares nothing about them; those fields say source "induced".
`records`	No	also return each page's records, not only its summary; the heaviest pages' records are left out first to keep the answer under 75,000 bytes, each counted in that page's records_left_out.
`respect_tdm`	No	give a page whose site reserves its text and data mining rights (TDMRep) as a tdm_reserved error, never its data.

Output Schema

ParametersJSON Schema

Name	Required	Description
`ok`	Yes
`error`	No
`pages`	No
`stopped`	No
`pages_left_out`	No

TDQS

A4.6/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the readOnly/idempotent annotations, the description reveals rate-limiting behavior, robots.txt compliance, TDM handling, retries, time-budget stopping, duplicate reads, and truncation mechanics. These are significant behavioral details the annotations alone do not provide.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but every section earns its place: purpose, flag semantics, output shape, failure modes, and limits. It is front-loaded with the core action and then structured into readable chunks, though the density is high enough that it could lose some less essential details.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity of the tool and the output schema, the description is remarkably complete: it covers exact output keys, page-level success/failure shapes, ordering guarantees, truncation thresholds, rate limits, robots/TDM behavior, retries, and global failure conditions. Nothing essential is missing for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description repeats and elaborates on the same flag semantics found in the schema but does not add meaning beyond what the input schema already documents.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Read several pages' declared data, politely, in the order given.' It clearly distinguishes this batch read tool from single-page or site-crawling siblings by emphasizing multiple URLs, deduplication, and bounded scope.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly states the valid input range (1 to 25 http(s) addresses), deduplication behavior, and gives a clear alternative for larger jobs: 'the command line's sluicer batch has no such bounds.' This tells an agent when to choose this tool and when to look elsewhere.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

fetch_pageFetch a pageA

Read-onlyIdempotent

Fetch a page's HTML, and say what it cost: plain HTTP or a browser.

url: an http(s) URL. Literal HTML is refused, since nothing would be fetched. offset: where in the HTML this answer starts, 0 for the beginning; the answer before gives the next as next_offset. max_chars: the most characters this answer carries, 1 to 60,000; fewer when more would weigh over 75,000 bytes, as 60,000 characters of Chinese do.

Returns {"ok", "html", "url", "fetch", "truncated", "length", "next_offset"}: html is one slice of the page, length the whole page's, and next_offset where the next slice starts, or null when this one reaches the end. Prefer extract_declared or page_markdown, which return what is in the page rather than all of it.

ParametersJSON Schema

Name	Required	Description
`url`	Yes	an http(s) URL. Literal HTML is refused, since nothing would be fetched.
`offset`	No	where in the HTML this answer starts, 0 for the beginning; the answer before gives the next as next_offset.
`max_chars`	No	the most characters this answer carries, 1 to 60,000; fewer when more would weigh over 75,000 bytes, as 60,000 characters of Chinese do.

Output Schema

ParametersJSON Schema

Name	Required	Description
`ok`	Yes
`url`	No
`html`	No
`error`	No
`fetch`	No
`length`	No
`truncated`	No
`next_offset`	No

TDQS

A4.7/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotations already mark this read-only, idempotent, and non-destructive, and the description adds valuable behavioral details beyond that: the returned 'fetch' field reveals whether a browser was used, results are sliced with next_offset pagination, and max_chars is subject to a byte-weight cap with an illustrative example. Nothing contradicts the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description front-loads the core behavior in the first sentence, uses a compact keyed parameter layout, and clearly separates the return contract and the alternative-tool guidance. Every sentence contributes: parameter rules, return shape, and routing advice. It is dense but not bloated.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given there is an output schema, the description goes beyond what is structurally required by explaining pagination semantics, truncation behavior, the meaning of next_offset, and the literal-HTML rejection. It is complete enough for an agent to call the tool correctly and interpret the response, without missing critical usage constraints.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, and the schema property descriptions are essentially identical to the description text for url, offset, and max_chars. The baseline is therefore 3; the description restates semantics rather than adding meaning beyond the schema, though it does frame the parameters coherently as a pagination mechanism.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Fetch a page's HTML' and immediately reveals a distinguishing trait—reporting whether the fetch used 'plain HTTP or a browser.' This differentiates it from siblings like page_markdown and extract_declared, which return processed content rather than raw HTML.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly says 'Prefer extract_declared or page_markdown, which return what is in the page rather than all of it,' giving a clear when-not-to-use rule and naming alternatives. It also warns that literal HTML is refused, so an agent knows not to pass page content directly as the URL.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

heal_extractorHeal an extractorA

Read-onlyIdempotent

Learn pages again after a redesign, and say what moved where.

extractor: the object compile_extractor returned. pages: http(s) URLs, or the HTML itself, of the redesigned pages.

Returns {"ok", "extractor", "changes", "lost"}. A field that moved keeps its old name, so rows read with the healed extractor keep their columns, and its change carries the evidence: how many of the values it was learnt with were found in the new place. "lost" is true when a change is data the page no longer has -- vanished, summary-lost, type-lost, listing-lost, or broken: a selector written by hand that the pages no longer bear out, which heal never rewrites -- and then ok is false: the old extractor, which keeps failing, is the safer one to keep until a person looks. A healed extractor heavier than 75,000 bytes is too_large.

ParametersJSON Schema

Name	Required	Description	Default
`pages`	Yes	http(s) URLs, or the HTML itself, of the redesigned pages.
`extractor`	Yes	the object compile_extractor returned.

Output Schema

ParametersJSON Schema

Name	Required	Description
`ok`	Yes
`lost`	No
`error`	No
`changes`	No
`extractor`	No

TDQS

A4.1/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description thoroughly discloses behavior beyond the annotations: it details the return structure (ok, extractor, changes, lost), explains the semantics of 'lost' and 'ok' being false, notes the 75,000-byte size limit, and clarifies that hand-written selectors are never rewritten. This significantly enriches the read-only and non-destructive annotations. No contradiction.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but efficiently packed; every sentence contributes to understanding the tool's purpose, inputs, outputs, and edge cases. It front-loads the core purpose and then systematically explains parameters and return behavior. No filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description is comprehensive for the tool's complexity. It covers prerequisites (extractor from compile_extractor), input types, output fields and their meanings, error conditions (lost data, ok false), and the size limit. With an output schema present, the description fully equips an agent to call the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, with both parameters already described in the input schema. The description repeats the same information (extractor is from compile_extractor, pages are URLs or HTML) without adding new semantics. Baseline 3 is appropriate since the schema carries the full burden.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the purpose: heal an extractor by learning pages after a redesign and reporting changes. It specifies the resource (extractor) and action (learn/heal) precisely. While it doesn't explicitly differentiate from sibling tools like run_extractor or compile_extractor, the purpose is distinct and unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives clear context: use this tool after a redesign to relearn pages and identify what moved. It doesn't mention alternatives or exclusions, but the trigger condition ('after a redesign') is explicit, providing sufficient guidance for an agent to decide when to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

map_siteMap a siteA

Read-onlyIdempotent

List a site's addresses, from its sitemaps or its start page's links.

url: an http(s) address on the site; its links stand in when the site has no sitemap. limit: the most addresses returned, 1 to 1,000.

Returns {"ok", "url", "source", "urls", "sitemaps", "truncated"}. source is "sitemaps" or "links"; each of urls is {"url", "lastmod", "sitemap"}, only addresses on the site, in the order the sitemaps list them; sitemaps says what became of each one tried. At most ten sitemaps are read, politely, within a minute; truncated is true when a bound cut the map short, urls_left_out counting the addresses left out to keep the answer under 75,000 bytes. Hand the addresses worth reading to extract_declared, or crawl_site to follow links from one.

ParametersJSON Schema

Name	Required	Description	Default
`url`	Yes	an http(s) address on the site; its links stand in when the site has no sitemap.
`limit`	No	the most addresses returned, 1 to 1,000.

Output Schema

ParametersJSON Schema

Name	Required	Description
`ok`	Yes
`url`	No
`urls`	No
`error`	No
`source`	No
`sitemaps`	No
`truncated`	No
`urls_left_out`	No

TDQS

A4.7/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only, idempotent, and non-destructive behavior, but the description discloses additional operational details: only ten sitemaps read 'politely, within a minute', a 75,000-byte response cap, truncation behavior with urls_left_out, and that urls include only on-site addresses. These specifics are not derivable from the annotations and materially affect how the agent interprets and uses results.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is densely informative without waste. It leads with the core purpose, then parameters, output structure, operational limits, and follow-up guidance. Every sentence carries a distinct piece of information; nothing is redundant. The structure naturally guides the reader from what, to how, to what next.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with this complexity (source fallback, sitemap handling, truncation, follow-up routing), the description fully covers the invocation context. It explains both possible sources, output schema (including fields like source and sitemaps), edge cases (truncation), and downstream usage. The presence of an output schema in the tool definition supplements the textual description, so nothing essential is omitted.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and both parameters have descriptions identical to the tool description, so the schema already explains them. The description adds value by connecting the limit parameter to output behavior ('truncated is true when a bound cut the map short' and 'urls_left_out counting the addresses left out'), which goes beyond the schema's simple 'most addresses returned' phrasing.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific and unambiguous purpose: 'List a site's addresses, from its sitemaps or its start page's links.' It clearly identifies the resource (site addresses) and the sources (sitemaps or links). It differentiates itself from siblings by explicitly routing the output to extract_declared or crawl_site, making it clear this is the discovery step, not a content fetch or extraction tool.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explains the tool's role in a pipeline: 'Hand the addresses worth reading to extract_declared, or crawl_site to follow links from one.' This tells an agent when to use it (when needing to enumerate site addresses) and what to do next. However, it does not explicitly compare against all siblings (e.g., fetch_page, page_markdown) or state conditions when those would be preferred, leaving some room for inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

page_markdownRead a page as markdownA

Read-onlyIdempotent

Return a page's main content as markdown, without navigation or footer.

html_or_url: an http(s) URL to fetch, or the HTML itself. front_matter: open the markdown with a YAML block of what the page declares about itself (title, author, dates, url...) and each answer's source. at: a date: read the URL as the Wayback Machine captured it then. respect_tdm: answer tdm_reserved when the site reserves its text and data mining rights (TDMRep). offset: where in the markdown this answer starts, 0 for the beginning; the answer before gives the next as next_offset. max_chars: the most characters this answer carries, 1 to 60,000; fewer when more would weigh over 75,000 bytes, as 60,000 characters of Chinese do.

Returns {"ok", "markdown", "url", "length", "next_offset"}, and "fetch" for a URL: markdown is one slice, length the whole markdown's, next_offset where the next slice starts or null at the end. The markdown is always the page's own content: a failure is ok false with "error", never text that could be mistaken for the page.

ParametersJSON Schema

Name	Required	Description
`at`	No	a date: read the URL as the Wayback Machine captured it then.
`offset`	No	where in the markdown this answer starts, 0 for the beginning; the answer before gives the next as next_offset.
`max_chars`	No	the most characters this answer carries, 1 to 60,000; fewer when more would weigh over 75,000 bytes, as 60,000 characters of Chinese do.
`html_or_url`	Yes	an http(s) URL to fetch, or the HTML itself.
`respect_tdm`	No	answer tdm_reserved when the site reserves its text and data mining rights (TDMRep).
`front_matter`	No	open the markdown with a YAML block of what the page declares about itself (title, author, dates, url...) and each answer's source.

Output Schema

ParametersJSON Schema

Name	Required	Description
`ok`	Yes
`url`	No
`error`	No
`fetch`	No
`length`	No
`markdown`	No
`next_offset`	No

TDQS

A4.1/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (readOnlyHint, idempotentHint, etc.), the description discloses return format, pagination via offset/next_offset, max_chars byte-weight limits, TDMRep behavior, Wayback Machine use with 'at', and a strong failure guarantee ('never text that could be mistaken for the page'). This substantially exceeds what annotations provide.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose is front-loaded, but the parameter list largely repeats the schema descriptions verbatim, introducing redundancy. The return-format explanation is valuable, but the duplicated parameter block makes the description longer than necessary.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 6 parameters and complex behavior (slicing, pagination, TDM, Wayback), the description covers invocation details, error behavior, and return values. An agent has everything needed to call it correctly without guessing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, and the description's parameter explanations are near-identical to the schema's property descriptions. It adds no new semantic meaning beyond what the schema already communicates, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a precise verb and resource: 'Return a page's main content as markdown, without navigation or footer.' This clearly distinguishes it from siblings like fetch_page (likely raw content) or extraction tools. The scope ('main content') and format ('markdown') are explicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage ('read a page as markdown') but provides no explicit guidance on when to choose this tool over alternatives, no exclusions, and no mention of sibling tools. An agent can infer when it is appropriate, but it is not stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

read_feedRead a feedA

Read-onlyIdempotent

Read a feed's items: RSS, Atom or JSON Feed.

url_or_text: an http(s) URL of a feed -- or of a page that declares one, which is followed to it -- or the feed itself. limit: the most items answered, 1 to 500; items_total says how many the feed holds.

Returns {"ok", "url", "format", "title", "link", "description", "items", "items_total"}, each item {"title", "link", "id", "published", "updated", "summary", "content", "authors", "categories", "enclosures", "normalised"}, dates in normalised as ISO 8601. What is not a feed, and declares none, is bad_input. Items that would make the answer weigh over 75,000 bytes are left out, counted in items_left_out; fetch_page reads the whole feed in slices.

ParametersJSON Schema

Name	Required	Description	Default
`limit`	No	the most items answered, 1 to 500; items_total says how many the feed holds.
`url_or_text`	Yes	an http(s) URL of a feed -- or of a page that declares one, which is followed to it -- or the feed itself.

Output Schema

ParametersJSON Schema

Name	Required	Description
`ok`	Yes
`url`	No
`link`	No
`error`	No
`fetch`	No
`items`	No
`title`	No
`format`	No
`updated`	No
`language`	No
`description`	No
`items_total`	No
`items_left_out`	No

TDQS

A4.2/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already mark the tool readOnly, idempotent, and non-destructive, and the description adds substantial behavior beyond those: what happens for non-feed input (bad_input), the 75,000-byte response cap, items_left_out counting, ISO 8601 normalisation, and the distinction from fetch_page. No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The summary sentence is front-loaded and useful, but the description repeats the schema's parameter descriptions and enumerates return fields even though an output schema exists. It is readable and compact enough, yet not every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers accepted URL/text forms, response format, error behavior, truncation, and a relevant sibling alternative. One minor gap is that the returned object list omits items_left_out even though the prose says items left out are counted in it; this is a small internal inconsistency rather than a missing major behavior.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, and the description's parameter sentences are nearly verbatim duplicates of the schema text. The byte-size truncation behavior adds useful context around limit, but it does not add meaningfully finer parameter-level detail beyond what the schema already provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with the specific action and resource: "Read a feed's items" and names the supported formats (RSS, Atom, JSON Feed). It also distinguishes itself from the sibling fetch_page by noting that fetch_page reads the whole feed in slices, so an agent can tell this tool focuses on bounded feed-item reading.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives clear context for accepted inputs: an http(s) URL of a feed, a page that declares a feed, or the feed text itself. It also hints at the alternative fetch_page for reading the whole feed without byte-based truncation, though it stops short of an explicit 'use this instead when...' rule.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_extractorRun an extractorA

Read-onlyIdempotent

Replay an extractor on one page, and check the page still keeps to it.

extractor: the object compile_extractor returned. html_or_url: an http(s) URL to fetch, or the HTML itself.

Returns {"ok", "rows", "fields", "summary", "failed"}; "fields" holds the page's own values an extractor learnt from examples. ok is false when the page drifted -- the listing moved, rows or a field vanished, a price no longer looks like a price -- and "failed" says which expectation broke. Never read rows from an answer whose ok is false as if nothing happened. Past 75,000 bytes the last rows are left out, counted in rows_left_out, then the heaviest summary and fields entries, named in summary_left_out and fields_left_out.

ParametersJSON Schema

Name	Required	Description	Default
`extractor`	Yes	the object compile_extractor returned.
`html_or_url`	Yes	an http(s) URL to fetch, or the HTML itself.

Output Schema

ParametersJSON Schema

Name	Required	Description
`ok`	Yes
`rows`	No
`error`	No
`failed`	No
`fields`	No
`summary`	No
`rows_left_out`	No
`fields_left_out`	No
`summary_left_out`	No

TDQS

A4.5/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds significant behavioral detail beyond the readOnly/idempotent hints: ok=false means page drift, 'failed' names the broken expectation, and the warning 'Never read rows from an answer whose ok is false as if nothing happened' is a crucial safety caveat. It also discloses the 75,000-byte truncation behavior and the exact fields that report left-out data.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with the core purpose, then parameter definitions, return-value semantics, a safety warning, and truncation behavior. Every sentence earns its place; there is no filler or redundant restatement of the tool name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a non-trivial verification tool, the description covers the call pattern, the meaning of ok/failed, the danger of trusting failed rows, and the truncation edge case. The presence of an output schema reduces the need to fully specify return structure, and the prose covers the important semantic and edge-case behavior.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, and the description restates the same meaning for both parameters: extractor is the compile_extractor return value and html_or_url is either an http(s) URL or raw HTML. This is useful but adds no information beyond the structured schema, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb-plus-resource framing: 'Replay an extractor on one page, and check the page still keeps to it.' The 'on one page' scope and the drift-checking return semantics clearly distinguish it from siblings like extract_many, crawl_site, and heal_extractor. The title, name, and description align without tautology.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The context is clear: use this after compile_extractor to validate a single page against an extractor and detect drift. It does not explicitly name alternatives or state when not to use it, but the 'on one page' scope and the focus on checking expectations imply the correct use case strongly enough.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

select_valuesSelect values on a pageA

Read-onlyIdempotent

Say what a CSS or XPath selector gives on a page, and where each value is.

html_or_url: an http(s) URL to fetch, or the HTML itself. selector: CSS, with ::text for an element's own text and ::attr(name) for an attribute ("span.price::text", "a::attr(href)"), or XPath ("//h1", "//a/@href"), told apart by how it begins: an XPath begins with /, ./, ( or @, or is written after "xpath:". respect_tdm: answer tdm_reserved when the site reserves its text and data mining rights (TDMRep).

Returns {"ok", "url", "values", "count"}, and "fetch" for a URL: values are {"value", "where"}, the text or attribute read, spaces collapsed, links resolved, and the XPath of its element; count is how many the selector gave. A selector that cannot be read is bad_input naming it; one that gives nothing is ok with no values. Past 75,000 bytes the last values are left out, counted in values_left_out. Use it to try the selectors compile_extractor's select takes.

ParametersJSON Schema

Name	Required	Description
`selector`	Yes	CSS, with ::text for an element's own text and ::attr(name) for an attribute ("span.price::text", "a::attr(href)"), or XPath ("//h1", "//a/@href"), told apart by how it begins: an XPath begins with /, ./, ( or @, or is written after "xpath:".
`html_or_url`	Yes	an http(s) URL to fetch, or the HTML itself.
`respect_tdm`	No	answer tdm_reserved when the site reserves its text and data mining rights (TDMRep).

Output Schema

ParametersJSON Schema

Name	Required	Description
`ok`	Yes
`url`	No
`count`	No
`error`	No
`fetch`	No
`values`	No
`values_left_out`	No

TDQS

A4.4/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description goes far beyond the annotations (readOnlyHint=true, idempotentHint=true, destructiveHint=false) by detailing the exact return format (ok, url, values, count), error handling (bad_input for unreadable selectors, ok with no values for no matches), truncation behavior beyond 75,000 bytes with values_left_out, and respect_tdm behavior. This is rich contextual information that helps an agent anticipate edge cases.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured and front-loaded with a clear purpose. It then systematically covers parameter syntax, return format, error cases, and truncation. Each sentence contributes value, but it is somewhat dense and lengthy. It is concise for the complexity, earning a 4 rather than a 5.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity and the presence of an output schema (implied by the description), the description is complete. It explains the return structure, error scenarios, truncation, and TDM handling. The only minor ambiguity is the 'fetch' phrasing, but overall an agent has sufficient information to call the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, and the description largely repeats the parameter details already in the schema (e.g., selector syntax, html_or_url meaning, respect_tdm effect). It adds marginal new meaning, such as the output semantics tied to parameters, but for the parameters themselves it does not significantly enhance the schema's explanations. The baseline of 3 is appropriate since the schema carries the load.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a clear purpose: 'Say what a CSS or XPath selector gives on a page, and where each value is.' It specifies the verb (select values), resource (page), and the scope (via selectors). It also differentiates from siblings by explicitly referencing compile_extractor for testing selectors, which helps distinguish it in the tool list.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives a clear usage context: 'Use it to try the selectors compile_extractor's select takes.' This tells the agent when to use it for selector verification. However, it does not explicitly state when not to use it or mention alternatives like page_markdown or extract_declared, so exclusions are absent. It provides clear context but not exhaustive when/when-not guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

12 tool updatesv0.9.0Sep 26, 2026

Changedaudit_page5 fields changed

changedOutput schema / $defs / ErrorDetail / description

Previous value: -"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. The others need something\nto change first -- an install, an input, or the caller's mind about a site\nthat said no."New value: +"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. Not every ``fetch_failed``\nis: a redirect loop, or an encoding this install cannot read, would be\nmet again. The others need something to change first -- an install, an\ninput, or the caller's mind about a site that said no."

changedOutput schema / $defs / ErrorDetail / properties / code / enum

Previous value: -[
-  "missing_extra",
-  "refused_by_robots",
-  "refused_address",
-  "fetch_failed",
-  "too_large",
-  "bad_input",
-  "tdm_reserved"
-]New value: +[
+  "missing_extra",
+  "refused_by_robots",
+  "refused_by_site",
+  "payment_required",
+  "refused_address",
+  "fetch_failed",
+  "too_large",
+  "bad_input",
+  "tdm_reserved"
+]

addedOutput schema / properties / other_agents_left_out

Added value: +{
+  "title": "Other Agents Left Out",
+  "type": "integer"
+}

addedOutput schema / properties / page_left_out

Added value: +{
+  "title": "Page Left Out",
+  "type": "integer"
+}

addedOutput schema / properties / records_left_out

Added value: +{
+  "title": "Records Left Out",
+  "type": "integer"
+}

Changedcompile_extractor4 fields changed

addedInput schema / properties / rows

Added value: +{
+  "anyOf": [
+    {
+      "type": "string"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null,
+  "description": "with select, the selector of a listing's rows (\"li.product\"), each field then read inside each row.",
+  "title": "Rows"
+}

addedInput schema / properties / select

Added value: +{
+  "anyOf": [
+    {
+      "additionalProperties": {
+        "type": "string"
+      },
+      "type": "object"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null,
+  "description": "instead of want, each field by a CSS or XPath selector you write, as {\"title\": \"h1\", \"price\": \"span.price::text\"}; pages may then be empty. Where the fields are is not learnt; what the pages show of them is, and a run fails when a selector finds nothing. Try a selector first with select_values.",
+  "title": "Select"
+}

changedOutput schema / $defs / ErrorDetail / description

Previous value: -"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. The others need something\nto change first -- an install, an input, or the caller's mind about a site\nthat said no."New value: +"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. Not every ``fetch_failed``\nis: a redirect loop, or an encoding this install cannot read, would be\nmet again. The others need something to change first -- an install, an\ninput, or the caller's mind about a site that said no."

changedOutput schema / $defs / ErrorDetail / properties / code / enum

Previous value: -[
-  "missing_extra",
-  "refused_by_robots",
-  "refused_address",
-  "fetch_failed",
-  "too_large",
-  "bad_input",
-  "tdm_reserved"
-]New value: +[
+  "missing_extra",
+  "refused_by_robots",
+  "refused_by_site",
+  "payment_required",
+  "refused_address",
+  "fetch_failed",
+  "too_large",
+  "bad_input",
+  "tdm_reserved"
+]

Changedcrawl_site7 fields changed

addedOutput schema / $defs / CrawledPage / properties / retries

Added value: +{
+  "items": {
+    "$ref": "#/$defs/RetryAnswer"
+  },
+  "title": "Retries",
+  "type": "array"
+}

addedOutput schema / $defs / CrawledPage / properties / summary_left_out

Added value: +{
+  "items": {
+    "type": "string"
+  },
+  "title": "Summary Left Out",
+  "type": "array"
+}

changedOutput schema / $defs / ErrorDetail / description

Previous value: -"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. The others need something\nto change first -- an install, an input, or the caller's mind about a site\nthat said no."New value: +"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. Not every ``fetch_failed``\nis: a redirect loop, or an encoding this install cannot read, would be\nmet again. The others need something to change first -- an install, an\ninput, or the caller's mind about a site that said no."

changedOutput schema / $defs / ErrorDetail / properties / code / enum

Previous value: -[
-  "missing_extra",
-  "refused_by_robots",
-  "refused_address",
-  "fetch_failed",
-  "too_large",
-  "bad_input",
-  "tdm_reserved"
-]New value: +[
+  "missing_extra",
+  "refused_by_robots",
+  "refused_by_site",
+  "payment_required",
+  "refused_address",
+  "fetch_failed",
+  "too_large",
+  "bad_input",
+  "tdm_reserved"
+]

changedOutput schema / $defs / PageError / properties / code / enum

Previous value: -[
-  "missing_extra",
-  "refused_by_robots",
-  "refused_address",
-  "fetch_failed",
-  "too_large",
-  "bad_input",
-  "tdm_reserved",
-  "redirected_off_site",
-  "crawl_delay_too_long",
-  "rate_limited"
-]New value: +[
+  "missing_extra",
+  "refused_by_robots",
+  "refused_by_site",
+  "payment_required",
+  "refused_address",
+  "fetch_failed",
+  "too_large",
+  "bad_input",
+  "tdm_reserved",
+  "redirected_off_site",
+  "crawl_delay_too_long",
+  "rate_limited"
+]

addedOutput schema / $defs / RetryAnswer

Added value: +{
+  "description": "One time a crawled page was asked again: what the request before came\nto, and the seconds from its end to this one's start.",
+  "properties": {
+    "after": {
+      "title": "After",
+      "type": "number"
+    },
+    "reason": {
+      "title": "Reason",
+      "type": "string"
+    }
+  },
+  "required": [
+    "reason",
+    "after"
+  ],
+  "title": "RetryAnswer",
+  "type": "object"
+}

addedOutput schema / properties / pages_left_out

Added value: +{
+  "title": "Pages Left Out",
+  "type": "integer"
+}

Changedextract_declared7 fields changed

changedOutput schema / $defs / ErrorDetail / description

Previous value: -"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. The others need something\nto change first -- an install, an input, or the caller's mind about a site\nthat said no."New value: +"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. Not every ``fetch_failed``\nis: a redirect loop, or an encoding this install cannot read, would be\nmet again. The others need something to change first -- an install, an\ninput, or the caller's mind about a site that said no."

changedOutput schema / $defs / ErrorDetail / properties / code / enum

Previous value: -[
-  "missing_extra",
-  "refused_by_robots",
-  "refused_address",
-  "fetch_failed",
-  "too_large",
-  "bad_input",
-  "tdm_reserved"
-]New value: +[
+  "missing_extra",
+  "refused_by_robots",
+  "refused_by_site",
+  "payment_required",
+  "refused_address",
+  "fetch_failed",
+  "too_large",
+  "bad_input",
+  "tdm_reserved"
+]

addedOutput schema / properties / conflicts_left_out

Added value: +{
+  "title": "Conflicts Left Out",
+  "type": "integer"
+}

addedOutput schema / properties / links_left_out

Added value: +{
+  "items": {
+    "type": "string"
+  },
+  "title": "Links Left Out",
+  "type": "array"
+}

addedOutput schema / properties / normalised_left_out

Added value: +{
+  "items": {
+    "type": "string"
+  },
+  "title": "Normalised Left Out",
+  "type": "array"
+}

addedOutput schema / properties / summary_left_out

Added value: +{
+  "items": {
+    "type": "string"
+  },
+  "title": "Summary Left Out",
+  "type": "array"
+}

addedOutput schema / properties / visible_left_out

Added value: +{
+  "items": {
+    "type": "string"
+  },
+  "title": "Visible Left Out",
+  "type": "array"
+}

Addedextract_many

Changedfetch_page3 fields changed

changedInput schema / properties / max_chars / description

Previous value: -"how many characters this answer carries, 1 to 60,000."New value: +"the most characters this answer carries, 1 to 60,000; fewer when more would weigh over 75,000 bytes, as 60,000 characters of Chinese do."

changedOutput schema / $defs / ErrorDetail / description

Previous value: -"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. The others need something\nto change first -- an install, an input, or the caller's mind about a site\nthat said no."New value: +"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. Not every ``fetch_failed``\nis: a redirect loop, or an encoding this install cannot read, would be\nmet again. The others need something to change first -- an install, an\ninput, or the caller's mind about a site that said no."

changedOutput schema / $defs / ErrorDetail / properties / code / enum

Previous value: -[
-  "missing_extra",
-  "refused_by_robots",
-  "refused_address",
-  "fetch_failed",
-  "too_large",
-  "bad_input",
-  "tdm_reserved"
-]New value: +[
+  "missing_extra",
+  "refused_by_robots",
+  "refused_by_site",
+  "payment_required",
+  "refused_address",
+  "fetch_failed",
+  "too_large",
+  "bad_input",
+  "tdm_reserved"
+]

Changedheal_extractor2 fields changed

changedOutput schema / $defs / ErrorDetail / description

Previous value: -"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. The others need something\nto change first -- an install, an input, or the caller's mind about a site\nthat said no."New value: +"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. Not every ``fetch_failed``\nis: a redirect loop, or an encoding this install cannot read, would be\nmet again. The others need something to change first -- an install, an\ninput, or the caller's mind about a site that said no."

changedOutput schema / $defs / ErrorDetail / properties / code / enum

Previous value: -[
-  "missing_extra",
-  "refused_by_robots",
-  "refused_address",
-  "fetch_failed",
-  "too_large",
-  "bad_input",
-  "tdm_reserved"
-]New value: +[
+  "missing_extra",
+  "refused_by_robots",
+  "refused_by_site",
+  "payment_required",
+  "refused_address",
+  "fetch_failed",
+  "too_large",
+  "bad_input",
+  "tdm_reserved"
+]

Changedmap_site3 fields changed

changedOutput schema / $defs / ErrorDetail / description

Previous value: -"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. The others need something\nto change first -- an install, an input, or the caller's mind about a site\nthat said no."New value: +"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. Not every ``fetch_failed``\nis: a redirect loop, or an encoding this install cannot read, would be\nmet again. The others need something to change first -- an install, an\ninput, or the caller's mind about a site that said no."

changedOutput schema / $defs / ErrorDetail / properties / code / enum

Previous value: -[
-  "missing_extra",
-  "refused_by_robots",
-  "refused_address",
-  "fetch_failed",
-  "too_large",
-  "bad_input",
-  "tdm_reserved"
-]New value: +[
+  "missing_extra",
+  "refused_by_robots",
+  "refused_by_site",
+  "payment_required",
+  "refused_address",
+  "fetch_failed",
+  "too_large",
+  "bad_input",
+  "tdm_reserved"
+]

addedOutput schema / properties / urls_left_out

Added value: +{
+  "title": "Urls Left Out",
+  "type": "integer"
+}

Changedpage_markdown3 fields changed

changedInput schema / properties / max_chars / description

Previous value: -"how many characters this answer carries, 1 to 60,000."New value: +"the most characters this answer carries, 1 to 60,000; fewer when more would weigh over 75,000 bytes, as 60,000 characters of Chinese do."

changedOutput schema / $defs / ErrorDetail / description

Previous value: -"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. The others need something\nto change first -- an install, an input, or the caller's mind about a site\nthat said no."New value: +"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. Not every ``fetch_failed``\nis: a redirect loop, or an encoding this install cannot read, would be\nmet again. The others need something to change first -- an install, an\ninput, or the caller's mind about a site that said no."

changedOutput schema / $defs / ErrorDetail / properties / code / enum

Previous value: -[
-  "missing_extra",
-  "refused_by_robots",
-  "refused_address",
-  "fetch_failed",
-  "too_large",
-  "bad_input",
-  "tdm_reserved"
-]New value: +[
+  "missing_extra",
+  "refused_by_robots",
+  "refused_by_site",
+  "payment_required",
+  "refused_address",
+  "fetch_failed",
+  "too_large",
+  "bad_input",
+  "tdm_reserved"
+]

Changedread_feed3 fields changed

changedOutput schema / $defs / ErrorDetail / description

Previous value: -"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. The others need something\nto change first -- an install, an input, or the caller's mind about a site\nthat said no."New value: +"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. Not every ``fetch_failed``\nis: a redirect loop, or an encoding this install cannot read, would be\nmet again. The others need something to change first -- an install, an\ninput, or the caller's mind about a site that said no."

changedOutput schema / $defs / ErrorDetail / properties / code / enum

Previous value: -[
-  "missing_extra",
-  "refused_by_robots",
-  "refused_address",
-  "fetch_failed",
-  "too_large",
-  "bad_input",
-  "tdm_reserved"
-]New value: +[
+  "missing_extra",
+  "refused_by_robots",
+  "refused_by_site",
+  "payment_required",
+  "refused_address",
+  "fetch_failed",
+  "too_large",
+  "bad_input",
+  "tdm_reserved"
+]

addedOutput schema / properties / items_left_out

Added value: +{
+  "title": "Items Left Out",
+  "type": "integer"
+}

Changedrun_extractor5 fields changed

changedOutput schema / $defs / ErrorDetail / description

Previous value: -"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. The others need something\nto change first -- an install, an input, or the caller's mind about a site\nthat said no."New value: +"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. Not every ``fetch_failed``\nis: a redirect loop, or an encoding this install cannot read, would be\nmet again. The others need something to change first -- an install, an\ninput, or the caller's mind about a site that said no."

changedOutput schema / $defs / ErrorDetail / properties / code / enum

Previous value: -[
-  "missing_extra",
-  "refused_by_robots",
-  "refused_address",
-  "fetch_failed",
-  "too_large",
-  "bad_input",
-  "tdm_reserved"
-]New value: +[
+  "missing_extra",
+  "refused_by_robots",
+  "refused_by_site",
+  "payment_required",
+  "refused_address",
+  "fetch_failed",
+  "too_large",
+  "bad_input",
+  "tdm_reserved"
+]

addedOutput schema / properties / fields_left_out

Added value: +{
+  "items": {
+    "type": "string"
+  },
+  "title": "Fields Left Out",
+  "type": "array"
+}

addedOutput schema / properties / rows_left_out

Added value: +{
+  "title": "Rows Left Out",
+  "type": "integer"
+}

addedOutput schema / properties / summary_left_out

Added value: +{
+  "items": {
+    "type": "string"
+  },
+  "title": "Summary Left Out",
+  "type": "array"
+}

Addedselect_values

10 tool updatesv0.7.0Sep 24, 2026

Changedaudit_page2 fields changed

addedInput schema / properties / html_or_url / description

Added value: +"an http(s) URL to fetch, or the HTML itself."

addedInput schema / properties / site / description

Added value: +"for a URL, also read the site's robots.txt, llms.txt and llms-full.txt."

Changedcompile_extractor3 fields changed

addedInput schema / properties / listing / description

Added value: +"learn the rows the pages repeat; by default only where they declare nothing about a thing."

addedInput schema / properties / pages / description

Added value: +"http(s) URLs, or the HTML itself, of pages built from one template -- two or three pages of one listing, or of one kind of product page."

addedInput schema / properties / want / description

Added value: +"example values one row of the listing holds, by the name each column is to have, as {\"price\": \"41.90\", \"title\": \"Brake pad set\"}: they choose the listing and the columns, and only those columns are kept. A value that no row holds is an error that names it."

Changedcrawl_site6 fields changed

addedInput schema / properties / exclude / description

Added value: +"text that stops a link being followed when its address contains it."

addedInput schema / properties / include / description

Added value: +"text an address must contain for its link to be followed (any one of them); plain text, not a pattern."

addedInput schema / properties / max_depth / description

Added value: +"the most links from url, 0 to 3; 0 reads url alone."

addedInput schema / properties / max_pages / description
```
Added value: +"the most pages taken, 1 to 25."
```

addedInput schema / properties / respect_tdm / description

Added value: +"give a page whose site reserves its text and data mining rights (TDMRep) as a tdm_reserved error, never its summary."

addedInput schema / properties / url / description

Added value: +"an http(s) address where the crawl starts; only links on its site are followed."

Changedextract_declared9 fields changed

addedInput schema / properties / at / description

Added value: +"a date (2024, 2024-06, 2024-06-01): read the URL as the Wayback Machine captured it nearest to then; \"fetch\" says which capture."

addedInput schema / properties / html_or_url / description

Added value: +"an http(s) URL to fetch, or the HTML itself."

addedInput schema / properties / induce / description

Added value: +"also read repeated rows (a listing, a feed) from a page that declares nothing about them; those fields say source \"induced\"."

addedInput schema / properties / records

Added value: +{
+  "default": true,
+  "description": "also return every record, not only the summary and what was normalised; false keeps the answer small. Records that would make the answer larger than 75,000 bytes are left out and counted in records_left_out.",
+  "title": "Records",
+  "type": "boolean"
+}

addedInput schema / properties / respect_tdm / description

Added value: +"answer tdm_reserved instead of the page when the site reserves its text and data mining rights (TDMRep: its tdmrep.json, headers or meta tags)."

addedInput schema / properties / visible

Added value: +{
+  "default": false,
+  "description": "also guess the title, author, publication and update dates the page shows a reader, in \"visible\", each {\"value\", \"where\", \"rule\"}; guesses, never part of the summary, which holds only what the page declares.",
+  "title": "Visible",
+  "type": "boolean"
+}

addedOutput schema / $defs / GuessAnswer

Added value: +{
+  "description": "What a page shows and may not declare: a guess, its element and rule.",
+  "properties": {
+    "rule": {
+      "title": "Rule",
+      "type": "string"
+    },
+    "value": {
+      "title": "Value",
+      "type": "string"
+    },
+    "where": {
+      "title": "Where",
+      "type": "string"
+    }
+  },
+  "required": [
+    "value",
+    "where",
+    "rule"
+  ],
+  "title": "GuessAnswer",
+  "type": "object"
+}

addedOutput schema / properties / records_left_out

Added value: +{
+  "title": "Records Left Out",
+  "type": "integer"
+}

addedOutput schema / properties / visible

Added value: +{
+  "additionalProperties": {
+    "$ref": "#/$defs/GuessAnswer"
+  },
+  "title": "Visible",
+  "type": "object"
+}

Changedfetch_page4 fields changed

addedInput schema / properties / max_chars

Added value: +{
+  "default": 30000,
+  "description": "how many characters this answer carries, 1 to 60,000.",
+  "title": "Max Chars",
+  "type": "integer"
+}

addedInput schema / properties / offset

Added value: +{
+  "default": 0,
+  "description": "where in the HTML this answer starts, 0 for the beginning; the answer before gives the next as next_offset.",
+  "title": "Offset",
+  "type": "integer"
+}

addedInput schema / properties / url / description

Added value: +"an http(s) URL. Literal HTML is refused, since nothing would be fetched."

addedOutput schema / properties / next_offset

Added value: +{
+  "anyOf": [
+    {
+      "type": "integer"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "title": "Next Offset"
+}

Changedheal_extractor2 fields changed
- addedInput schema / properties / extractor / description
```
Added value: +"the object compile_extractor returned."
```
- addedInput schema / properties / pages / description
```
Added value: +"http(s) URLs, or the HTML itself, of the redesigned pages."
```

Changedmap_site2 fields changed

addedInput schema / properties / limit / description

Added value: +"the most addresses returned, 1 to 1,000."

addedInput schema / properties / url / description

Added value: +"an http(s) address on the site; its links stand in when the site has no sitemap."

Changedpage_markdown8 fields changed

addedInput schema / properties / at / description

Added value: +"a date: read the URL as the Wayback Machine captured it then."

addedInput schema / properties / front_matter / description

Added value: +"open the markdown with a YAML block of what the page declares about itself (title, author, dates, url...) and each answer's source."

addedInput schema / properties / html_or_url / description

Added value: +"an http(s) URL to fetch, or the HTML itself."

addedInput schema / properties / max_chars

Added value: +{
+  "default": 30000,
+  "description": "how many characters this answer carries, 1 to 60,000.",
+  "title": "Max Chars",
+  "type": "integer"
+}

addedInput schema / properties / offset

Added value: +{
+  "default": 0,
+  "description": "where in the markdown this answer starts, 0 for the beginning; the answer before gives the next as next_offset.",
+  "title": "Offset",
+  "type": "integer"
+}

addedInput schema / properties / respect_tdm / description

Added value: +"answer tdm_reserved when the site reserves its text and data mining rights (TDMRep)."

addedOutput schema / properties / length

Added value: +{
+  "title": "Length",
+  "type": "integer"
+}

addedOutput schema / properties / next_offset

Added value: +{
+  "anyOf": [
+    {
+      "type": "integer"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "title": "Next Offset"
+}

Changedread_feed2 fields changed

addedInput schema / properties / limit / description

Added value: +"the most items answered, 1 to 500; items_total says how many the feed holds."

addedInput schema / properties / url_or_text / description

Added value: +"an http(s) URL of a feed -- or of a page that declares one, which is followed to it -- or the feed itself."

Changedrun_extractor2 fields changed
- addedInput schema / properties / extractor / description
```
Added value: +"the object compile_extractor returned."
```
- addedInput schema / properties / html_or_url / description
```
Added value: +"an http(s) URL to fetch, or the HTML itself."
```

10 tool updatesv0.1.0Sep 24, 2026
- First observedaudit_page
- First observedcompile_extractor
- First observedcrawl_site
- First observedextract_declared
- First observedfetch_page
- First observedheal_extractor
- First observedmap_site
- First observedpage_markdown
- First observedread_feed
- First observedrun_extractor

Deploy Server

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A4.3/5.0

Scored 2026-09-26 07:30 across 12 tools

Disambiguation5/5

Each tool has a clearly distinct purpose: fetching raw HTML, extracting declared data, converting to markdown, testing selectors, compiling/replaying/healing extractors, auditing structured data, reading feeds, mapping sites, crawling, and batch extraction. Even overlapping tools like extract_declared and extract_many are separated by single vs. multiple pages, and crawl_site vs. map_site by following links vs. listing sitemap URLs. No two tools appear to do the same thing.

Naming Consistency4/5

Most tools follow a verb_noun pattern (fetch_page, compile_extractor, run_extractor, audit_page, read_feed, map_site, crawl_site, select_values), but a few deviate: 'extract_declared' and 'extract_many' use verb+modifier, and 'page_markdown' is noun-like, breaking the pattern. Still, the naming is generally predictable and readable.

Tool Count5/5

12 tools is well within the typical 3-15 range for a web data extraction server. Each tool addresses a distinct capability—fetching, extracting, testing, learning, replaying, healing, auditing, reading feeds, mapping, crawling, and batch processing—without redundancy or bloat.

Completeness5/5

The toolset covers the full lifecycle of web data extraction: raw fetch, structured data extraction, markdown conversion, selector testing, extractor creation/replay/healing, structured data auditing, feed reading, sitemap mapping, site crawling, and batched extraction. It also handles TDM rights, Wayback Machine, and pagination, leaving no obvious gaps for its stated purpose.

Maintenance

ActivityMaintained

ResponsivenessNo issues

Resources

Need Help?

Related Servers

Related MCP Connectors

Related MCP Servers

firecrawl-mcp-server
RAG Systems Web Scraping
firecrawl
A
license
A
quality
C
maintenance
A Model Context Protocol (MCP) server implementation that integrates with FireCrawl for advanced web scraping capabilities.
Updated a day ago (2026-09-26 02:01 UTC)
26
91,722 npm
7,514
MIT
mcp-untun
Developer Tools App Automation Command Line
minte-app
F
license
A
quality
D
maintenance
Unofficial untun MCP
Updated a year ago (2025-06-23 06:34 UTC)
3
3 npm
-
MasterMCP
HC010602
-
license
Not graded
quality
D
maintenance
MCP server
Updated a year ago (2025-05-30 09:28 UTC)
-
clickup-mcp
Project Management App Automation
hauptsacheNet
A
license
Not graded
quality
A
maintenance
clickup-mcp
Updated 16 days ago (2026-09-11 10:10 UTC)
859 npm
49
MIT