Skip to main content
Glama

LLM Documentation Scraper (doc-scraper)

CI Go Version Go Reference License Glama score

A configurable, concurrent, and resumable web crawler written in Go. Specifically designed to scrape technical documentation websites, extract core content, convert it cleanly to Markdown format suitable for ingestion by Large Language Models (LLMs), and save the results locally.

doc-scraper crawling a docs site and answering search queries offline

doc-scraper crawls documentation sites into clean Markdown and serves them to coding agents over MCP. Your agent searches and reads the docs you chose, offline, instead of guessing from training data or a live web search. Point doc-scraper add at any docs page and it detects the framework (30+ docs generators, including Docusaurus, Starlight, Sphinx, VitePress, Material for MkDocs, and mdBook), validates the content selector, and drafts the config for you.

Documentation: https://sriram-pr.github.io/doc-scraper/

Features

  • Offline search for agents: BM25 full-text search (search_docs) and page reads (read_page) over the stored corpus, with no network access

  • Framework detection: 30+ docs generators recognized, each selector validated on the page, Readability fallback, JavaScript-only shells reported instead of crawled empty

  • Clean output: GitHub-Flavored Markdown per page with frontmatter, plus pages.jsonl, llms.txt, and llms-full.txt

  • Polite and resumable: robots.txt and sitemaps, per-host rate limits, retries with backoff, resumable crawls, and staged output that never clobbers the last good copy

  • Stays fresh: incremental re-crawls, watch mode, and crawl history (get_freshness, diff_crawl)

  • Many sites at once: parallel crawls with shared limits

Related MCP server: go-docs-mcp

Install

Download a release binary (Linux, macOS, Windows; amd64 and arm64) from the Releases page, or:

go install github.com/Sriram-PR/doc-scraper/v2/cmd/doc-scraper@latest
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD":/data ghcr.io/sriram-pr/doc-scraper:latest crawl -site rust_cli_book

Claude Desktop users can install doc-scraper.mcpb from the latest release instead. Details: Install.

Quick start

doc-scraper add -site rust_cli_book https://rust-cli.github.io/book/index.html   # detect, preview, confirm
doc-scraper crawl -site rust_cli_book             # Markdown lands in ./crawled_docs/rust_cli_book/
doc-scraper search -site rust_cli_book "error handling"

Then give it to Claude Code:

claude mcp add --transport stdio doc-scraper -- /path/to/doc-scraper mcp-server -config /path/to/config.yaml

See Claude Code setup for scopes and a first prompt.

Docs

Contributing

Contributions are welcome. Read CONTRIBUTING.md before starting: it covers setup, the testing rules CI enforces, how to claim an issue, and the policy on AI-assisted contributions. Issues labeled good first issue are a good place to begin.

Report security problems privately as described in SECURITY.md. Everyone taking part follows the Code of Conduct.

Privacy Policy

doc-scraper collects nothing: no telemetry, no analytics, no accounts. All output and state stays on your machine, and the only network requests it makes are the crawls and fetches you explicitly ask for. Full policy: PRIVACY.md.

License

This project is licensed under the Apache-2.0 License.

Acknowledgements

Available Tools

13 tools
cancel_crawlA
Idempotent

Cancel a running or pending crawl job by job ID. The previously stored corpus and crawl history are left untouched. Has no effect on jobs already in a terminal state.

NameRequiredDescriptionDefault
job_idYesThe job ID returned by crawl_site

Output Schema

NameRequiredDescription
job_idYes
statusYes
messageYes
site_keyYes
cancelledYesfalse when the job was already in a terminal state

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare non-read-only, non-destructive, idempotent behavior. The description adds valuable specifics: stored corpus and crawl history are untouched, and terminal-state jobs are unaffected. It does not cover auth or rate limits, but those are not critical here.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences, front-loaded with the core action and immediately followed by the key side-effect and state constraints. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With a complete input schema, rich annotations, and an output schema, the description covers purpose, side effects, and state limitations. It could add a note about checking job status before cancelling, but otherwise is sufficient for a simple cancel operation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The single job_id parameter has 100% schema coverage and the schema already documents it as the ID returned by crawl_site. The description adds no semantic detail beyond 'by job ID', so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (cancel) and resource (running or pending crawl job) with scope. Though it doesn't name a sibling, its purpose is unmistakable against crawl_site and get_job_status in the sibling list.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Clearly indicates the applicable job states (running or pending) and an exclusion (terminal state jobs have no effect). It does not name an alternative tool or say when to prefer checking status first.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_siteA

Start a background crawl for a configured site. Returns immediately with a job ID. The crawl builds in a staging area and replaces the stored corpus only when it completes, so a cancelled or failed crawl leaves the previous corpus intact. A crawl that lost a large share of its pages is kept back instead (the job fails with refused set), and the stored corpus is kept.

NameRequiredDescriptionDefault
site_keyYesSite key from config (use list_sites to discover available keys)
incrementalNoEnable incremental mode (skip unchanged pages)
allow_shrinkNoPublish the crawl even if it lost many pages compared to the last one (use with incremental:true to publish a kept-back crawl)

Output Schema

NameRequiredDescription
job_idYes
statusYesstarted, or already_running when a crawl for the site is in progress
messageYes
site_keyYes
incrementalYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Discloses behaviour well beyond the annotations: builds in a staging area, atomic replacement of the corpus on success, previous corpus preserved on cancel/failure, and a kept-back crawl that fails with 'refused' when pages are lost. This is consistent with destructiveHint=false and non-idempotent semantics and clarifies the failure mode precisely.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four tight sentences, front-loaded with purpose and return behaviour, then failure semantics. No filler; each sentence carries distinct information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return structure need not be spelled out, and the description still notes the job ID. For a 3-parameter mutation with annotations, the failure and atomicity story is fully covered.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% so the baseline is 3, but the description adds real meaning to allow_shrink by explaining the shrink guardrail and the 'refused' failure that it overrides. Incremental mode itself is left to the schema text.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Start a background crawl for a configured site') plus the immediate return ('job ID'), which distinguishes it from siblings like get_job_status and cancel_crawl that operate on the resulting job.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Clear operating context: asynchronous start, staging-then-publish semantics, and the shrink guardrail with the workaround implied by the allow_shrink schema text. It does not explicitly name alternatives (e.g. why to prefer this over re-drafting a site), so it stops short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

describe_serverA
Read-onlyIdempotent

Returns server identity, configured sites, and recent crawl jobs in one call. Call this first to orient yourself: it consolidates what would otherwise require list_sites plus several get_job_status calls. The MCP tool list is already advertised by the protocol so it is not duplicated here.

NameRequiredDescriptionDefault

No parameters

Output Schema

NameRequiredDescription
sitesYes
serverYes
total_jobsYes
jobs_cappedYes
recent_jobsYesNewest first, at most 20
total_sitesYes
next_actionsYes
config_writesYesenabled or disabled; save_site needs enabled (mcp-server -allow-config-writes)

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish that this is a read-only, idempotent, non-destructive operation. The description adds useful context that the call returns consolidated identity, sites, and recent job data in one call, though it does not discuss pagination, caching, or response limits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with what the tool returns, followed by when to call it and why. All three sentences contribute directly to selecting and understanding the tool, with no wasted wording.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the rich annotations, no parameters, and an output schema, the description is sufficiently complete. It explains the tool's purpose, usage, consolidation value, and even clarifies why the MCP tool list is not included.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There are zero parameters, so parameter semantics are inherently simple. The description does not need to explain parameters, and the baseline for a no-parameter tool applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resources: returns server identity, configured sites, and recent crawl jobs. It clearly distinguishes itself from siblings by explaining that it consolidates what would otherwise require list_sites plus several get_job_status calls.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives explicit usage guidance: 'Call this first to orient yourself.' It also names the alternatives it replaces (list_sites and several get_job_status calls), leaving little ambiguity about when to use this tool versus its siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

diff_crawlA
Read-onlyIdempotent

Return added/removed/changed pages between the latest crawl and a baseline crawl: the crawl before it by default, or with since, the most recent crawl whose crawl_ended_at <= since. Hash-based verdicts from the SQLite history index. To see what a crawl changed, run crawl_site, wait for it to finish, then call diff_crawl without since.

NameRequiredDescriptionDefault
sinceNoRFC3339 timestamp; the most recent crawl whose crawl_ended_at <= since, compared to the second, is the baseline (e.g. 2026-05-23T22:00:00Z). Omit to compare with the previous crawl.
offsetNoPagination offset, 0-based
site_keyYesSite key from config
max_resultsNoMaximum diff entries to return (default: 100, max: 1000; larger values are capped, 0 or less means the default)

Output Schema

NameRequiredDescription
noteNoWhy the diff is empty or partial, when it is
addedYes
sinceNoPresent when since was given
totalYes
offsetYes
changedYes
removedYes
returnedYes
site_keyYes
max_resultsYes
current_crawlNo
baseline_crawlNo
unchanged_countYes

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive/openWorld=false, so the safety profile is covered. The description adds genuinely non-obvious behavior: verdicts are hash-based from a SQLite history index (no content re-fetch), and the baseline is data-dependent, requiring a prior crawl. It does not state what happens with zero or one prior crawl (empty result vs. error), which is the remaining gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core verb+resource, then baseline semantics, then a short workflow recipe; every sentence contributes. There is mild duplication between "the crawl before it by default" and the schema's "Omit to compare with the previous crawl."

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return shape needn't be described; the description covers baseline resolution and the prerequisite crawl workflow, which is what an agent needs to call it correctly. The only unaddressed case is degenerate history (no prior crawl) where the agent doesn't know whether to expect an error or empty diff.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so `since`, `offset`, `max_results`, and `site_key` are already documented in structured fields, and the description's `since` wording largely restates the schema's RFC3339 baseline rule. It adds the useful default ("the crawl before it by default") but nothing about pagination or `max_results` interaction. Baseline 3 applies when the schema carries the detail.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

"Return added/removed/changed pages between the latest crawl and a baseline crawl" gives a specific verb (return) plus the exact resource (page-level diff between two crawls). It also implicitly separates the tool from list_pages/get_page (content access) and get_freshness (staleness), so an agent can route correctly from names alone.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description spells out the operational sequence explicitly: "run crawl_site, wait for it to finish, then call diff_crawl without since," naming the sibling tool and the required precondition. It also states the default baseline and the alternative (pass `since`) with the exact selection rule, leaving nothing to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

draft_siteA
Read-onlyIdempotent

Draft a site entry and preview one page extracted with it. Never writes the config. With url, probes a new docs site the way doc-scraper add does (robots.txt, llms.txt, sitemap, framework detection). With only site_key, starts from that configured site's entry without probing. overrides sets entry fields by their config names (content_selector, allowed_path_prefix, disallowed_path_patterns, max_depth, ...); durations are strings such as "1s". Judge the preview (code blocks kept, nav_leak, chars), draft again with overrides until it looks right, then pass entry to save_site.

NameRequiredDescriptionDefault
urlNoA docs URL to probe as a new site
site_keyNoWith url: the key for the new site. Without url: the configured site to start from
overridesNoEntry fields to set, by config field name
preview_urlNoThe page to preview (default: the sampled docs page, else the seed or first start URL)

Output Schema

NameRequiredDescription
entryYesThe site entry by config field name; pass it, changed as needed, to save_site
existsYesWhether site_key is configured now; save_site then needs replace: true
previewYes
evidenceYes
site_keyYes
warningsYes
frameworkNo
generatorNo
confidenceNo
entry_yamlYes
next_actionsYes
config_writesYesenabled or disabled; save_site needs enabled
signal_sourceNo
sitemap_page_countNo

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/openWorld/idempotent/non-destructive, and the description reinforces this with 'Never writes the config' while disclosing what probing actually does (robots.txt, llms.txt, sitemap, framework detection) — real behavioral context beyond the hints. It stops short of covering failure modes or network cost/rate limits for the probe.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads purpose and the no-write guarantee, then moves through the two modes, the overrides format, and the recommended workflow. Dense but zero filler sentences; every clause carries operational information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 4-param tool with a nested free-form object and an output schema present, the description supplies the missing pieces: overrides field naming/format, the meaning of the preview, and the explicit handoff to save_site. Return-value details are correctly delegated to the output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3. The description does add value for `overrides` — naming concrete config field names (content_selector, allowed_path_prefix, disallowed_path_patterns, max_depth) and the duration string format ('1s') — which compensates for the nested object having no defined properties, but url/site_key/preview_url semantics are already fully covered by the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Draft a site entry and preview one page extracted with it') plus a negative scope clause ('Never writes the config'). This clearly separates it from the sibling save_site, which is named explicitly as the follow-up step.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit branching rule: with `url` it probes a new docs site the way `doc-scraper add` does; with only `site_key` it starts from an existing configured entry without probing. It also states the full iteration loop (judge preview → re-draft with overrides → pass entry to save_site), naming the alternative tool that commits.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_freshnessA
Read-onlyIdempotent

Return the most recent crawl summary for a site (last_crawl with its start and end times, total_pages and mode, plus age_seconds), output/state dir presence and any running job. Use this to decide whether to query the existing crawl or run crawl_site first.

NameRequiredDescriptionDefault
site_keyYesSite key from config (use list_sites to discover available keys)

Output Schema

NameRequiredDescription
site_keyYes
last_crawlNoLatest recorded crawl; absent if none
output_dirYes
age_secondsNoSeconds since last_crawl ended; present with last_crawl
running_jobNo
next_actionsYes
output_existsYes
state_dir_existsYes

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover read-only, idempotent, non-destructive, closed-world behavior, so the description's added value is the decision-relevant content: age_seconds, dir presence, and any running job. It goes beyond restating annotations by describing what state the agent will see, though it doesn't say how freshness should be interpreted.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the return payload and closing with the decision use case. Dense but nothing is wasted; the parenthetical field list is long yet earned.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Complete for a single-parameter read tool with annotations and an output schema: the agent knows what it returns and why to call it. The only gap is guidance on interpreting age_seconds to judge staleness, which the agent must infer.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the single site_key parameter is fully documented in the schema, including a pointer to list_sites. The description adds no parameter-level detail, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('return the most recent crawl summary for a site') and enumerates the payload (last_crawl start/end, total_pages, mode, age_seconds, dir presence, running job). This is clearly distinguishable from crawl_site, get_job_status, and other siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly frames usage as a decision step: 'Use this to decide whether to query the existing crawl or run crawl_site first,' naming the alternative tool. It never states when not to use it or what threshold makes a crawl stale, but the routing intent is unambiguous.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_job_statusA
Read-onlyIdempotent

Get the status and page counts of a crawl job. Set wait_seconds to wait for the job to finish instead of polling: the call returns as soon as the job completes, fails or is cancelled, or when the wait runs out, with the status at that moment.

NameRequiredDescriptionDefault
job_idYesThe job ID returned by crawl_site
wait_secondsNoSeconds to wait for the job to finish before returning (default: 0, max: 60; larger values are capped)

Output Schema

NameRequiredDescription
job_idYes
statusYespending, running, completed, failed or cancelled
refusedNoSet when a finished crawl was kept back instead of published: empty, temporary_failures or shrink
failuresNoFailed pages by class (temporary, gone, selector, policy, other), each with a count and one sample
site_keyYes
started_atYes
incrementalYes
pages_savedYesPages written by this crawl
completed_atNo
pages_failedYesPages that failed; noindex pages are neither saved nor failed
pages_queuedYesPages waiting to be crawled; for a cancelled job, the pages left uncrawled
error_messageNo
pages_processedYes
pages_unchangedYesIncremental re-checks whose content had not changed; not counted in pages_processed
duration_secondsNoPresent once completed_at is
pages_carried_forwardYesPages that failed temporarily and kept their copy from the previous crawl

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive and closed-world, so the safety profile is covered. The description adds genuine behavioral detail beyond that: the call returns as soon as the job completes, fails, or is cancelled, or when the wait expires, and reports status at that moment. It doesn't discuss error behavior for unknown/expired job IDs.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with what the tool does, then the non-obvious wait semantics. No padding or restatement of the title.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

A two-parameter read tool with an output schema, full schema coverage, and complete annotations; the description covers the one non-obvious behavior (blocking until terminal state or timeout). Nothing an agent needs to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description goes further by explaining what wait_seconds actually does semantically — it blocks and returns early on terminal states rather than simply sleeping. That meaning is not obvious from the schema's 'Seconds to wait' wording.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Get the status and page counts of a crawl job') and names the object precisely enough to separate it from siblings like cancel_crawl, get_freshness, and list_pages. An agent knows exactly what it gets back conceptually without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly contrasts the long-poll mode with the polling alternative ('Set wait_seconds to wait for the job to finish instead of polling'), which is real when-to-use guidance. It stops short of naming sibling tools for adjacent needs (e.g., cancel_crawl to abort, list_pages to inspect results).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_pageA
Read-onlyIdempotent

Fetch an HTML page live over the network and return its content as markdown. This is an on-demand fetch independent of any crawl: it does not read the stored crawl output, and it ignores the site's configured content_selector and scope. Large pages are truncated at max_bytes; follow next_offset to read the rest (each call fetches the page again).

NameRequiredDescriptionDefault
urlYesThe URL to fetch
offsetNoByte offset into the page content, for reading a truncated page in parts (default: 0)
max_bytesNoMaximum content bytes to return (default: 102400, max: 1048576; larger values are capped, 0 or less means the default)
content_selectorNoCSS selector for main content (defaults to 'body')

Output Schema

NameRequiredDescription
urlYes
titleYes
offsetYes
contentYesPage content converted to markdown
truncatedYes
next_offsetNoPass as offset to read the next part; present only when truncated
total_lengthYesBytes of the whole converted page
fetch_time_msYes
content_lengthYesBytes of content in this response

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare it read-only and open-world, but the description adds materially: bypassing crawl storage and site config, truncation at max_bytes, and the non-obvious cost that each paginated call re-fetches the page. The only blemish is the reference to a 'next_offset' field that does not exist in the schema (the parameter is 'offset'), which could momentarily confuse an agent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences, front-loaded with the action and the crawl-independence caveat before the truncation mechanics. Nearly every clause earns its place; the 'next_offset' wording is the one piece of noise that should have been 'offset'.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 4-parameter fetch tool with full schema coverage and an output schema, the description covers what an agent must know: it is live, it ignores stored/config state, and large pages are truncated with a re-fetch cost. Nothing essential is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3 and all four parameters already carry descriptions. The description adds real meaning for offset/max_bytes by tying them to truncation and the refetch requirement, but it does so under a mismatched name ('next_offset'), so the added value is partial rather than clean.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Fetch an HTML page live over the network and return its content as markdown') and immediately scopes it against the crawl-based siblings by saying it does not read stored crawl output. An agent can distinguish this from read_page/list_pages/crawl_site without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly frames the tool as an on-demand live fetch independent of any crawl and notes it ignores the site's content_selector and scope, which is exactly the condition that decides between this and the stored-crawl siblings. It stops short of naming the concrete alternative (read_page) or stating when a live fetch is wasteful, so it is clear context rather than full routing guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_pagesA
Read-onlyIdempotent

List crawled pages for a site, paginated and sorted by URL. Returns metadata only (URL, title, depth, crawled_at, content_length). Pass any URL returned here to read_page to get its stored markdown; get_page re-fetches a URL live rather than returning the crawled copy.

NameRequiredDescriptionDefault
offsetNoPagination offset, 0-based (default: 0)
site_keyYesSite key from config (use list_sites to discover available keys)
max_resultsNoMaximum pages to return (default: 100, max: 1000; larger values are capped, 0 or less means the default)

Output Schema

NameRequiredDescription
pagesYes
totalYes
offsetYes
messageNoPresent when the site has no crawl output yet
returnedYes
site_keyYes
max_resultsYes

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, closed-world, non-destructive, so the safety profile is covered. The description adds behavior beyond them: results are sorted by URL, only metadata is returned (URL, title, depth, crawled_at, content_length), and the stored copy vs live fetch distinction between read_page and get_page. It doesn't discuss pagination termination or total counts, but the added context is substantial.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three dense sentences with zero filler: purpose and return shape first, then the routing rule, then the sibling disambiguation. Every sentence earns its place and nothing is buried.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists so return values needn't be spelled out, yet the description still names the key metadata fields. For a simple, annotation-covered read tool with fully documented parameters, nothing an agent needs in order to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, and the schema already documents offset, site_key provenance (list_sites), and max_results defaults/caps. The description only implies pagination ('paginated') and adds no parameter-level syntax or bounds beyond the schema, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb and resource ('List crawled pages for a site') plus scope details (paginated, sorted by URL) and return shape (metadata only). It also explicitly distinguishes itself from both read_page and get_page, so an agent can tell the three apart without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives explicit routing: 'Pass any URL returned here to read_page to get its stored markdown' and clarifies that 'get_page re-fetches a URL live rather than returning the crawled copy.' That's a when-to-use plus a named alternative with its selection condition.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_sitesA
Read-onlyIdempotent

List all configured sites available for crawling

NameRequiredDescriptionDefault

No parameters

Output Schema

NameRequiredDescription
sitesYes
config_pathYes
total_sitesYes

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, destructiveHint=false and a closed-world scope, so the safety and side-effect profile is fully covered elsewhere. The description adds no behavioral context beyond that (e.g., empty-list behavior, scope of 'configured'), so it neither contradicts nor enriches the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with no filler; the verb and scope come first and nothing is wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a nullary list tool, the annotations cover the safety profile and the output schema covers the return shape, so the description need not explain result structure. Nothing an agent needs to call this correctly is absent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, so per the baseline there is nothing for the description to disambiguate. No parameter-related value is expected or missing.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource (list sites) plus scope ('all configured', 'available for crawling'), which implicitly separates it from mutation siblings like crawl_site. It does not explicitly distinguish itself from the other list-style sibling list_pages, so it stops short of a top score.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: 'available for crawling' hints this is a discovery step before crawl_site, but there is no explicit when-to-use, when-not-to-use, or mention of an alternative tool. Adequate but leaves the agent to infer the workflow.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

read_pageA
Read-onlyIdempotent

Return a page's markdown from the stored crawl output, without any network access. This is the counterpart to get_page: read_page serves the crawled copy that already had the site's content_selector applied, while get_page re-fetches the URL live. Use list_pages to discover URLs, then read_page to read them. Pass a search_docs result's anchor to read only that section. Large pages are truncated at max_bytes; follow next_offset to read the rest.

NameRequiredDescriptionDefault
urlYesURL of a crawled page, as reported by list_pages
anchorNoHeading anchor, as in a search_docs result; returns only that section, subsections included
offsetNoByte offset into the page content, for reading a truncated page in parts (default: 0)
site_keyYesSite key from config (use list_sites to discover available keys)
max_bytesNoMaximum content bytes to return (default: 102400, max: 1048576; larger values are capped, 0 or less means the default)

Output Schema

NameRequiredDescription
urlYes
depthYes
titleYes
offsetYes
sourceYes
contentYes
headingsYes
site_keyYes
truncatedYesMore of the page, or of the anchor's section, follows
crawled_atYes
next_offsetNoPass as offset to read the next part; present only when truncated
section_endNoByte offset where the anchor's section ends; present when anchor was given
content_hashYes
total_lengthYesBytes of the whole stored page
content_lengthYesBytes of content in this response

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, closed-world, non-destructive, so the safety profile is carried elsewhere. The description nevertheless adds real behavioral context: no network access, content_selector already applied, truncation at max_bytes, and continuation via the next_offset returned in the response. Minor gap: it does not mention that offset values are byte-based here, though the schema covers that.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Five short sentences, each carrying distinct information: what it returns, how it differs from get_page, the discovery workflow, the anchor shortcut, and pagination. No filler and the differentiation is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, return values need not be explained, yet the description still covers truncation and continuation. For a 5-parameter read tool with full schema coverage and annotations, nothing an agent needs to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3, but the description adds meaning beyond the schema: it links anchor to search_docs results, explains offset as continuation of a truncated page, and explains max_bytes truncation behavior. It also references a next_offset return field, which pairs with the offset parameter rather than duplicating it.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource (return a page's markdown from stored crawl output) and immediately contrasts it with the sibling get_page, explaining the exact difference (crawled copy with content_selector applied vs live re-fetch). An agent can distinguish the two without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly routes the agent: list_pages to discover URLs, read_page to read them, get_page for live fetching, and passing a search_docs anchor to read only a section. This covers both when to use it and which sibling to pick instead.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

save_siteA
DestructiveIdempotent

Write a site entry to the config file and load it into the running server, so crawl_site, search_docs and read_page can use it at once. entry is a full entry by config field name, normally draft_site's entry with changes; fields left out are unset. Overwriting a configured site needs replace: true. Only the entry's lines change in the file. Refused while the site is crawling, and on servers started without -allow-config-writes.

NameRequiredDescriptionDefault
entryYesThe full site entry, by config field name
replaceNoOverwrite an existing site (default: false; must be false for a new site)
site_keyYesThe key to save the entry under

Output Schema

NameRequiredDescription
writtenYes
replacedYes
site_keyYes
next_actionsYes

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes beyond the annotations by disclosing scoping ('only the entry's lines change in the file'), the partial-merge semantics ('fields left out are unset'), and the two refusal conditions (active crawl, config-writes flag). These details substantially reduce the risk of an incorrect destructive call, and the safety annotations already flag the mutation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences, front-loaded with the action and immediate effect, then behavioral constraints, then failure conditions. No filler and no restated schema text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be described. Given a destructive tool with nested parameters, the description supplies everything else an agent needs: the write's scope, merge semantics, replace requirement, and both refusal conditions.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and each parameter has a description, but the description adds meaning the schema cannot: the entry object is empty in the schema, yet the description explains it is keyed by config field name, is normally derived from draft_site, and leaves omitted fields unset. It could state the field names explicitly, which is why this is not a top score.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource (write a site entry to the config file and load it into the running server) and names what becomes possible afterward (crawl_site, search_docs, read_page). A reader can distinguish it from draft_site, which only drafts an entry.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says the entry normally comes from draft_site with modifications, when replace:true is required for overwrites, when it is refused (while crawling, or servers started without -allow-config-writes), and that omitted fields are unset. When-to-use, prerequisites, and failure conditions are all covered.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_docsA
Read-onlyIdempotent

Full-text search across all crawled documentation, ranked by relevance (BM25 with stemming), with zero network access. Results carry the page URL, its section heading path, and a snippet with match terms marked [like this]; follow up with read_page for the full page. Supports FTS5 syntax: quoted phrases, OR, and trailing * for prefix matching. Searches the stored index only; run crawl_site first for uncrawled sites.

NameRequiredDescriptionDefault
limitNoMaximum results to return (default: 10, max: 50; larger values are capped, 0 or less means the default)
queryYesSearch terms. Plain words are ANDed; when too few results contain every word, the rest are filled with partial matches (match: partial) that contain only some words or built-in synonyms of them. "quoted phrases", OR, and prefix* work too and are never relaxed.
site_keyNoLimit results to one site (use list_sites to discover keys). Omit to search all sites.

Output Schema

NameRequiredDescription
countYes
limitYes
queryYes
resultsYes
site_keyNoPresent when the search was limited to one site
next_actionsYes

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds context beyond annotations: BM25/stemming ranking, zero-network guarantee, stored-index-only scoping, and result shape (URL, heading path, [marked] snippet). Annotations already declare readOnly/idempotent/openWorld=false; the description reinforces and enriches rather than contradicting. Query relaxation behavior lives in the schema, so a small gap remains.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads purpose and ranking, then follows with return format, syntax, and prerequisites. Every sentence carries information; slightly dense but no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite an output schema existing, the description still explains result contents and how to continue to full pages. Combined with the schema, an agent has everything needed to call and follow up correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so parameters are already fully documented including ANDing, relaxation, and limit capping. The description's FTS5 syntax note largely duplicates what the schema states, so it adds little beyond the structured baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Full-text search across all crawled documentation') plus ranking mechanism (BM25 with stemming). Clearly distinguishes itself from read_page and crawl_site, which are named as follow-up/prerequisite tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly routes the agent: 'follow up with read_page for the full page' and 'run crawl_site first for uncrawled sites.' Both alternatives and the conditions selecting them are named, so the agent needn't infer anything.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 4 tool updatesv2.24.1
    • Changedcrawl_site1 field changed
      • addedInput schema / properties / allow_shrink
        Added value: +{
        +  "description": "Publish the crawl even if it lost many pages compared to the last one (use with incremental:true to publish a kept-back crawl)",
        +  "type": "boolean"
        +}
    • Changeddescribe_server6 fields changed
      • addedOutput schema / properties / recent_jobs / items / properties / failures
        Added value: +{
        +  "description": "Failed pages by class (temporary, gone, selector, policy, other), each with a count and one sample",
        +  "items": {
        +    "properties": {
        +      "class": {
        +        "type": "string"
        +      },
        +      "count": {
        +        "type": "integer"
        +      },
        +      "sample_error": {
        +        "type": "string"
        +      },
        +      "sample_url": {
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "class",
        +      "count",
        +      "sample_url",
        +      "sample_error"
        +    ],
        +    "type": "object"
        +  },
        +  "type": [
        +    "null",
        +    "array"
        +  ]
        +}
      • addedOutput schema / properties / recent_jobs / items / properties / pages_carried_forward
        Added value: +{
        +  "description": "Pages that failed temporarily and kept their copy from the previous crawl",
        +  "type": "integer"
        +}
      • addedOutput schema / properties / recent_jobs / items / properties / pages_failed
        Added value: +{
        +  "description": "Pages that failed; noindex pages are neither saved nor failed",
        +  "type": "integer"
        +}
      • addedOutput schema / properties / recent_jobs / items / properties / pages_saved
        Added value: +{
        +  "description": "Pages written by this crawl",
        +  "type": "integer"
        +}
      • addedOutput schema / properties / recent_jobs / items / properties / refused
        Added value: +{
        +  "description": "Set when a finished crawl was kept back instead of published: empty, temporary_failures or shrink",
        +  "type": "string"
        +}
      • changedOutput schema / properties / recent_jobs / items / required
        Previous value: -[
        -  "job_id",
        -  "site_key",
        -  "status",
        -  "started_at",
        -  "pages_processed",
        -  "pages_queued",
        -  "pages_unchanged",
        -  "incremental"
        -]New value: +[
        +  "job_id",
        +  "site_key",
        +  "status",
        +  "started_at",
        +  "pages_processed",
        +  "pages_queued",
        +  "pages_unchanged",
        +  "pages_saved",
        +  "pages_failed",
        +  "pages_carried_forward",
        +  "incremental"
        +]
    • Changedget_freshness6 fields changed
      • addedOutput schema / properties / running_job / properties / failures
        Added value: +{
        +  "description": "Failed pages by class (temporary, gone, selector, policy, other), each with a count and one sample",
        +  "items": {
        +    "properties": {
        +      "class": {
        +        "type": "string"
        +      },
        +      "count": {
        +        "type": "integer"
        +      },
        +      "sample_error": {
        +        "type": "string"
        +      },
        +      "sample_url": {
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "class",
        +      "count",
        +      "sample_url",
        +      "sample_error"
        +    ],
        +    "type": "object"
        +  },
        +  "type": [
        +    "null",
        +    "array"
        +  ]
        +}
      • addedOutput schema / properties / running_job / properties / pages_carried_forward
        Added value: +{
        +  "description": "Pages that failed temporarily and kept their copy from the previous crawl",
        +  "type": "integer"
        +}
      • addedOutput schema / properties / running_job / properties / pages_failed
        Added value: +{
        +  "description": "Pages that failed; noindex pages are neither saved nor failed",
        +  "type": "integer"
        +}
      • addedOutput schema / properties / running_job / properties / pages_saved
        Added value: +{
        +  "description": "Pages written by this crawl",
        +  "type": "integer"
        +}
      • addedOutput schema / properties / running_job / properties / refused
        Added value: +{
        +  "description": "Set when a finished crawl was kept back instead of published: empty, temporary_failures or shrink",
        +  "type": "string"
        +}
      • changedOutput schema / properties / running_job / required
        Previous value: -[
        -  "job_id",
        -  "site_key",
        -  "status",
        -  "started_at",
        -  "pages_processed",
        -  "pages_queued",
        -  "pages_unchanged",
        -  "incremental"
        -]New value: +[
        +  "job_id",
        +  "site_key",
        +  "status",
        +  "started_at",
        +  "pages_processed",
        +  "pages_queued",
        +  "pages_unchanged",
        +  "pages_saved",
        +  "pages_failed",
        +  "pages_carried_forward",
        +  "incremental"
        +]
    • Changedget_job_status6 fields changed
      • addedOutput schema / properties / failures
        Added value: +{
        +  "description": "Failed pages by class (temporary, gone, selector, policy, other), each with a count and one sample",
        +  "items": {
        +    "properties": {
        +      "class": {
        +        "type": "string"
        +      },
        +      "count": {
        +        "type": "integer"
        +      },
        +      "sample_error": {
        +        "type": "string"
        +      },
        +      "sample_url": {
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "class",
        +      "count",
        +      "sample_url",
        +      "sample_error"
        +    ],
        +    "type": "object"
        +  },
        +  "type": [
        +    "null",
        +    "array"
        +  ]
        +}
      • addedOutput schema / properties / pages_carried_forward
        Added value: +{
        +  "description": "Pages that failed temporarily and kept their copy from the previous crawl",
        +  "type": "integer"
        +}
      • addedOutput schema / properties / pages_failed
        Added value: +{
        +  "description": "Pages that failed; noindex pages are neither saved nor failed",
        +  "type": "integer"
        +}
      • addedOutput schema / properties / pages_saved
        Added value: +{
        +  "description": "Pages written by this crawl",
        +  "type": "integer"
        +}
      • addedOutput schema / properties / refused
        Added value: +{
        +  "description": "Set when a finished crawl was kept back instead of published: empty, temporary_failures or shrink",
        +  "type": "string"
        +}
      • changedOutput schema / required
        Previous value: -[
        -  "job_id",
        -  "site_key",
        -  "status",
        -  "started_at",
        -  "pages_processed",
        -  "pages_queued",
        -  "pages_unchanged",
        -  "incremental"
        -]New value: +[
        +  "job_id",
        +  "site_key",
        +  "status",
        +  "started_at",
        +  "pages_processed",
        +  "pages_queued",
        +  "pages_unchanged",
        +  "pages_saved",
        +  "pages_failed",
        +  "pages_carried_forward",
        +  "incremental"
        +]
  2. 3 tool updatesv2.24.0
    • Changeddescribe_server2 fields changed
      • addedOutput schema / properties / config_writes
        Added value: +{
        +  "description": "enabled or disabled; save_site needs enabled (mcp-server -allow-config-writes)",
        +  "type": "string"
        +}
      • changedOutput schema / required
        Previous value: -[
        -  "server",
        -  "sites",
        -  "recent_jobs",
        -  "total_sites",
        -  "total_jobs",
        -  "jobs_capped",
        -  "next_actions"
        -]New value: +[
        +  "server",
        +  "sites",
        +  "recent_jobs",
        +  "total_sites",
        +  "total_jobs",
        +  "jobs_capped",
        +  "config_writes",
        +  "next_actions"
        +]
    • Addeddraft_site
    • Addedsave_site
  3. 9 tool updatesv2.21.0
    • Changedcrawl_site1 field changed
      • changedInput schema / properties / site_key / description
        Previous value: -"Site key from config file (e.g., 'langchain_py', 'rust_docs')"New value: +"Site key from config (use list_sites to discover available keys)"
    • Changeddescribe_server1 field changed
      • addedOutput schema / properties / recent_jobs / items / properties / pages_queued / description
        Added value: +"Pages waiting to be crawled; for a cancelled job, the pages left uncrawled"
    • Changeddiff_crawl7 fields changed
      • changedInput schema / properties / max_results / description
        Previous value: -"Maximum diff entries to return (default 100, max 1000)"New value: +"Maximum diff entries to return (default: 100, max: 1000; larger values are capped, 0 or less means the default)"
      • changedInput schema / properties / max_results / type
        Previous value: -"number"New value: +"integer"
      • changedInput schema / properties / offset / type
        Previous value: -"number"New value: +"integer"
      • changedInput schema / properties / since / description
        Previous value: -"RFC3339 timestamp; the most recent crawl whose crawl_ended_at <= since, compared to the second, is the baseline (e.g. 2026-05-23T22:00:00Z)"New value: +"RFC3339 timestamp; the most recent crawl whose crawl_ended_at <= since, compared to the second, is the baseline (e.g. 2026-05-23T22:00:00Z). Omit to compare with the previous crawl."
      • changedInput schema / required
        Previous value: -[
        -  "site_key",
        -  "since"
        -]New value: +[
        +  "site_key"
        +]
      • addedOutput schema / properties / since / description
        Added value: +"Present when since was given"
      • changedOutput schema / required
        Previous value: -[
        -  "site_key",
        -  "since",
        -  "added",
        -  "removed",
        -  "changed",
        -  "unchanged_count",
        -  "total",
        -  "returned",
        -  "offset",
        -  "max_results"
        -]New value: +[
        +  "site_key",
        +  "added",
        +  "removed",
        +  "changed",
        +  "unchanged_count",
        +  "total",
        +  "returned",
        +  "offset",
        +  "max_results"
        +]
    • Changedget_freshness1 field changed
      • addedOutput schema / properties / running_job / properties / pages_queued / description
        Added value: +"Pages waiting to be crawled; for a cancelled job, the pages left uncrawled"
    • Changedget_job_status2 fields changed
      • addedInput schema / properties / wait_seconds
        Added value: +{
        +  "description": "Seconds to wait for the job to finish before returning (default: 0, max: 60; larger values are capped)",
        +  "type": "integer"
        +}
      • addedOutput schema / properties / pages_queued / description
        Added value: +"Pages waiting to be crawled; for a cancelled job, the pages left uncrawled"
    • Changedget_page8 fields changed
      • addedInput schema / properties / max_bytes
        Added value: +{
        +  "description": "Maximum content bytes to return (default: 102400, max: 1048576; larger values are capped, 0 or less means the default)",
        +  "type": "integer"
        +}
      • addedInput schema / properties / offset
        Added value: +{
        +  "description": "Byte offset into the page content, for reading a truncated page in parts (default: 0)",
        +  "type": "integer"
        +}
      • addedOutput schema / properties / content_length / description
        Added value: +"Bytes of content in this response"
      • addedOutput schema / properties / next_offset
        Added value: +{
        +  "description": "Pass as offset to read the next part; present only when truncated",
        +  "type": [
        +    "null",
        +    "integer"
        +  ]
        +}
      • addedOutput schema / properties / offset
        Added value: +{
        +  "type": "integer"
        +}
      • addedOutput schema / properties / total_length
        Added value: +{
        +  "description": "Bytes of the whole converted page",
        +  "type": "integer"
        +}
      • addedOutput schema / properties / truncated
        Added value: +{
        +  "type": "boolean"
        +}
      • changedOutput schema / required
        Previous value: -[
        -  "url",
        -  "title",
        -  "content",
        -  "content_length",
        -  "fetch_time_ms"
        -]New value: +[
        +  "url",
        +  "title",
        +  "content",
        +  "content_length",
        +  "total_length",
        +  "offset",
        +  "truncated",
        +  "fetch_time_ms"
        +]
    • Changedlist_pages3 fields changed
      • changedInput schema / properties / max_results / description
        Previous value: -"Maximum pages to return (default: 100, max: 1000)"New value: +"Maximum pages to return (default: 100, max: 1000; larger values are capped, 0 or less means the default)"
      • changedInput schema / properties / max_results / type
        Previous value: -"number"New value: +"integer"
      • changedInput schema / properties / offset / type
        Previous value: -"number"New value: +"integer"
    • Changedread_page6 fields changed
      • addedInput schema / properties / anchor
        Added value: +{
        +  "description": "Heading anchor, as in a search_docs result; returns only that section, subsections included",
        +  "type": "string"
        +}
      • changedInput schema / properties / max_bytes / description
        Previous value: -"Maximum content bytes to return (default: 102400, max: 1048576)"New value: +"Maximum content bytes to return (default: 102400, max: 1048576; larger values are capped, 0 or less means the default)"
      • changedInput schema / properties / max_bytes / type
        Previous value: -"number"New value: +"integer"
      • changedInput schema / properties / offset / type
        Previous value: -"number"New value: +"integer"
      • addedOutput schema / properties / section_end
        Added value: +{
        +  "description": "Byte offset where the anchor's section ends; present when anchor was given",
        +  "type": [
        +    "null",
        +    "integer"
        +  ]
        +}
      • addedOutput schema / properties / truncated / description
        Added value: +"More of the page, or of the anchor's section, follows"
    • Changedsearch_docs4 fields changed
      • changedInput schema / properties / limit / description
        Previous value: -"Maximum results to return (default: 10, max: 50)"New value: +"Maximum results to return (default: 10, max: 50; larger values are capped, 0 or less means the default)"
      • changedInput schema / properties / limit / type
        Previous value: -"number"New value: +"integer"
      • addedOutput schema / properties / limit
        Added value: +{
        +  "type": "integer"
        +}
      • changedOutput schema / required
        Previous value: -[
        -  "query",
        -  "count",
        -  "results",
        -  "next_actions"
        -]New value: +[
        +  "query",
        +  "count",
        +  "limit",
        +  "results",
        +  "next_actions"
        +]
  4. 11 tool updatesv2.15.0
    • Changedcancel_crawl1 field changed
      • changedOutput schema / (root)
        Previous value: -nullNew value: +{
        +  "properties": {
        +    "cancelled": {
        +      "description": "false when the job was already in a terminal state",
        +      "type": "boolean"
        +    },
        +    "job_id": {
        +      "type": "string"
        +    },
        +    "message": {
        +      "type": "string"
        +    },
        +    "site_key": {
        +      "type": "string"
        +    },
        +    "status": {
        +      "type": "string"
        +    }
        +  },
        +  "required": [
        +    "job_id",
        +    "cancelled",
        +    "site_key",
        +    "status",
        +    "message"
        +  ],
        +  "type": "object"
        +}
    • Changedcrawl_site1 field changed
      • changedOutput schema / (root)
        Previous value: -nullNew value: +{
        +  "properties": {
        +    "incremental": {
        +      "type": "boolean"
        +    },
        +    "job_id": {
        +      "type": "string"
        +    },
        +    "message": {
        +      "type": "string"
        +    },
        +    "site_key": {
        +      "type": "string"
        +    },
        +    "status": {
        +      "description": "started, or already_running when a crawl for the site is in progress",
        +      "type": "string"
        +    }
        +  },
        +  "required": [
        +    "status",
        +    "message",
        +    "job_id",
        +    "site_key",
        +    "incremental"
        +  ],
        +  "type": "object"
        +}
    • Changeddescribe_server1 field changed
      • changedOutput schema / (root)
        Previous value: -nullNew value: +{
        +  "properties": {
        +    "jobs_capped": {
        +      "type": "boolean"
        +    },
        +    "next_actions": {
        +      "type": "string"
        +    },
        +    "recent_jobs": {
        +      "description": "Newest first, at most 20",
        +      "items": {
        +        "properties": {
        +          "completed_at": {
        +            "type": "string"
        +          },
        +          "duration_seconds": {
        +            "description": "Present once completed_at is",
        +            "type": [
        +              "null",
        +              "number"
        +            ]
        +          },
        +          "error_message": {
        +            "type": "string"
        +          },
        +          "incremental": {
        +            "type": "boolean"
        +          },
        +          "job_id": {
        +            "type": "string"
        +          },
        +          "pages_processed": {
        +            "type": "integer"
        +          },
        +          "pages_queued": {
        +            "type": "integer"
        +          },
        +          "pages_unchanged": {
        +            "description": "Incremental re-checks whose content had not changed; not counted in pages_processed",
        +            "type": "integer"
        +          },
        +          "site_key": {
        +            "type": "string"
        +          },
        +          "started_at": {
        +            "type": "string"
        +          },
        +          "status": {
        +            "description": "pending, running, completed, failed or cancelled",
        +            "type": "string"
        +          }
        +        },
        +        "required": [
        +          "job_id",
        +          "site_key",
        +          "status",
        +          "started_at",
        +          "pages_processed",
        +          "pages_queued",
        +          "pages_unchanged",
        +          "incremental"
        +        ],
        +        "type": "object"
        +      },
        +      "type": [
        +        "null",
        +        "array"
        +      ]
        +    },
        +    "server": {
        +      "properties": {
        +        "config_path": {
        +          "type": "string"
        +        },
        +        "name": {
        +          "type": "string"
        +        },
        +        "version": {
        +          "type": "string"
        +        }
        +      },
        +      "required": [
        +        "name",
        +        "version",
        +        "config_path"
        +      ],
        +      "type": "object"
        +    },
        +    "sites": {
        +      "items": {
        +        "properties": {
        +          "domain": {
        +            "type": "string"
        +          },
        +          "key": {
        +            "type": "string"
        +          },
        +          "last_crawled": {
        +            "type": "string"
        +          },
        +          "status": {
        +            "description": "\"running\" while a crawl is active; absent otherwise",
        +            "type": "string"
        +          }
        +        },
        +        "required": [
        +          "key",
        +          "domain"
        +        ],
        +        "type": "object"
        +      },
        +      "type": [
        +        "null",
        +        "array"
        +      ]
        +    },
        +    "total_jobs": {
        +      "type": "integer"
        +    },
        +    "total_sites": {
        +      "type": "integer"
        +    }
        +  },
        +  "required": [
        +    "server",
        +    "sites",
        +    "recent_jobs",
        +    "total_sites",
        +    "total_jobs",
        +    "jobs_capped",
        +    "next_actions"
        +  ],
        +  "type": "object"
        +}
    • Changeddiff_crawl1 field changed
      • changedOutput schema / (root)
        Previous value: -nullNew value: +{
        +  "properties": {
        +    "added": {
        +      "items": {
        +        "properties": {
        +          "content_hash": {
        +            "type": "string"
        +          },
        +          "depth": {
        +            "type": "integer"
        +          },
        +          "prior_hash": {
        +            "description": "Baseline hash; changed entries only",
        +            "type": "string"
        +          },
        +          "title": {
        +            "type": "string"
        +          },
        +          "url": {
        +            "type": "string"
        +          }
        +        },
        +        "required": [
        +          "url",
        +          "title",
        +          "content_hash",
        +          "depth"
        +        ],
        +        "type": "object"
        +      },
        +      "type": [
        +        "null",
        +        "array"
        +      ]
        +    },
        +    "baseline_crawl": {
        +      "properties": {
        +        "crawl_ended_at": {
        +          "type": "string"
        +        },
        +        "crawl_id": {
        +          "type": "integer"
        +        },
        +        "crawl_started_at": {
        +          "type": "string"
        +        },
        +        "mode": {
        +          "description": "full or incremental",
        +          "type": "string"
        +        },
        +        "total_pages": {
        +          "type": "integer"
        +        }
        +      },
        +      "required": [
        +        "crawl_id",
        +        "crawl_started_at",
        +        "crawl_ended_at",
        +        "total_pages",
        +        "mode"
        +      ],
        +      "type": [
        +        "null",
        +        "object"
        +      ]
        +    },
        +    "changed": {
        +      "items": {
        +        "properties": {
        +          "content_hash": {
        +            "type": "string"
        +          },
        +          "depth": {
        +            "type": "integer"
        +          },
        +          "prior_hash": {
        +            "description": "Baseline hash; changed entries only",
        +            "type": "string"
        +          },
        +          "title": {
        +            "type": "string"
        +          },
        +          "url": {
        +            "type": "string"
        +          }
        +        },
        +        "required": [
        +          "url",
        +          "title",
        +          "content_hash",
        +          "depth"
        +        ],
        +        "type": "object"
        +      },
        +      "type": [
        +        "null",
        +        "array"
        +      ]
        +    },
        +    "current_crawl": {
        +      "properties": {
        +        "crawl_ended_at": {
        +          "type": "string"
        +        },
        +        "crawl_id": {
        +          "type": "integer"
        +        },
        +        "crawl_started_at": {
        +          "type": "string"
        +        },
        +        "mode": {
        +          "description": "full or incremental",
        +          "type": "string"
        +        },
        +        "total_pages": {
        +          "type": "integer"
        +        }
        +      },
        +      "required": [
        +        "crawl_id",
        +        "crawl_started_at",
        +        "crawl_ended_at",
        +        "total_pages",
        +        "mode"
        +      ],
        +      "type": [
        +        "null",
        +        "object"
        +      ]
        +    },
        +    "max_results": {
        +      "type": "integer"
        +    },
        +    "note": {
        +      "description": "Why the diff is empty or partial, when it is",
        +      "type": "string"
        +    },
        +    "offset": {
        +      "type": "integer"
        +    },
        +    "removed": {
        +      "items": {
        +        "properties": {
        +          "content_hash": {
        +            "type": "string"
        +          },
        +          "depth": {
        +            "type": "integer"
        +          },
        +          "prior_hash": {
        +            "description": "Baseline hash; changed entries only",
        +            "type": "string"
        +          },
        +          "title": {
        +            "type": "string"
        +          },
        +          "url": {
        +            "type": "string"
        +          }
        +        },
        +        "required": [
        +          "url",
        +          "title",
        +          "content_hash",
        +          "depth"
        +        ],
        +        "type": "object"
        +      },
        +      "type": [
        +        "null",
        +        "array"
        +      ]
        +    },
        +    "returned": {
        +      "type": "integer"
        +    },
        +    "since": {
        +      "type": "string"
        +    },
        +    "site_key": {
        +      "type": "string"
        +    },
        +    "total": {
        +      "type": "integer"
        +    },
        +    "unchanged_count": {
        +      "type": "integer"
        +    }
        +  },
        +  "required": [
        +    "site_key",
        +    "since",
        +    "added",
        +    "removed",
        +    "changed",
        +    "unchanged_count",
        +    "total",
        +    "returned",
        +    "offset",
        +    "max_results"
        +  ],
        +  "type": "object"
        +}
    • Changedget_freshness1 field changed
      • changedOutput schema / (root)
        Previous value: -nullNew value: +{
        +  "properties": {
        +    "age_seconds": {
        +      "description": "Seconds since last_crawl ended; present with last_crawl",
        +      "type": [
        +        "null",
        +        "integer"
        +      ]
        +    },
        +    "last_crawl": {
        +      "description": "Latest recorded crawl; absent if none",
        +      "properties": {
        +        "crawl_ended_at": {
        +          "type": "string"
        +        },
        +        "crawl_id": {
        +          "type": "integer"
        +        },
        +        "crawl_started_at": {
        +          "type": "string"
        +        },
        +        "mode": {
        +          "description": "full or incremental",
        +          "type": "string"
        +        },
        +        "total_pages": {
        +          "type": "integer"
        +        }
        +      },
        +      "required": [
        +        "crawl_id",
        +        "crawl_started_at",
        +        "crawl_ended_at",
        +        "total_pages",
        +        "mode"
        +      ],
        +      "type": [
        +        "null",
        +        "object"
        +      ]
        +    },
        +    "next_actions": {
        +      "type": "string"
        +    },
        +    "output_dir": {
        +      "type": "string"
        +    },
        +    "output_exists": {
        +      "type": "boolean"
        +    },
        +    "running_job": {
        +      "properties": {
        +        "completed_at": {
        +          "type": "string"
        +        },
        +        "duration_seconds": {
        +          "description": "Present once completed_at is",
        +          "type": [
        +            "null",
        +            "number"
        +          ]
        +        },
        +        "error_message": {
        +          "type": "string"
        +        },
        +        "incremental": {
        +          "type": "boolean"
        +        },
        +        "job_id": {
        +          "type": "string"
        +        },
        +        "pages_processed": {
        +          "type": "integer"
        +        },
        +        "pages_queued": {
        +          "type": "integer"
        +        },
        +        "pages_unchanged": {
        +          "description": "Incremental re-checks whose content had not changed; not counted in pages_processed",
        +          "type": "integer"
        +        },
        +        "site_key": {
        +          "type": "string"
        +        },
        +        "started_at": {
        +          "type": "string"
        +        },
        +        "status": {
        +          "description": "pending, running, completed, failed or cancelled",
        +          "type": "string"
        +        }
        +      },
        +      "required": [
        +        "job_id",
        +        "site_key",
        +        "status",
        +        "started_at",
        +        "pages_processed",
        +        "pages_queued",
        +        "pages_unchanged",
        +        "incremental"
        +      ],
        +      "type": [
        +        "null",
        +        "object"
        +      ]
        +    },
        +    "site_key": {
        +      "type": "string"
        +    },
        +    "state_dir_exists": {
        +      "type": "boolean"
        +    }
        +  },
        +  "required": [
        +    "site_key",
        +    "output_dir",
        +    "output_exists",
        +    "state_dir_exists",
        +    "next_actions"
        +  ],
        +  "type": "object"
        +}
    • Changedget_job_status1 field changed
      • changedOutput schema / (root)
        Previous value: -nullNew value: +{
        +  "properties": {
        +    "completed_at": {
        +      "type": "string"
        +    },
        +    "duration_seconds": {
        +      "description": "Present once completed_at is",
        +      "type": [
        +        "null",
        +        "number"
        +      ]
        +    },
        +    "error_message": {
        +      "type": "string"
        +    },
        +    "incremental": {
        +      "type": "boolean"
        +    },
        +    "job_id": {
        +      "type": "string"
        +    },
        +    "pages_processed": {
        +      "type": "integer"
        +    },
        +    "pages_queued": {
        +      "type": "integer"
        +    },
        +    "pages_unchanged": {
        +      "description": "Incremental re-checks whose content had not changed; not counted in pages_processed",
        +      "type": "integer"
        +    },
        +    "site_key": {
        +      "type": "string"
        +    },
        +    "started_at": {
        +      "type": "string"
        +    },
        +    "status": {
        +      "description": "pending, running, completed, failed or cancelled",
        +      "type": "string"
        +    }
        +  },
        +  "required": [
        +    "job_id",
        +    "site_key",
        +    "status",
        +    "started_at",
        +    "pages_processed",
        +    "pages_queued",
        +    "pages_unchanged",
        +    "incremental"
        +  ],
        +  "type": "object"
        +}
    • Changedget_page1 field changed
      • changedOutput schema / (root)
        Previous value: -nullNew value: +{
        +  "properties": {
        +    "content": {
        +      "description": "Page content converted to markdown",
        +      "type": "string"
        +    },
        +    "content_length": {
        +      "type": "integer"
        +    },
        +    "fetch_time_ms": {
        +      "type": "integer"
        +    },
        +    "title": {
        +      "type": "string"
        +    },
        +    "url": {
        +      "type": "string"
        +    }
        +  },
        +  "required": [
        +    "url",
        +    "title",
        +    "content",
        +    "content_length",
        +    "fetch_time_ms"
        +  ],
        +  "type": "object"
        +}
    • Changedlist_pages1 field changed
      • changedOutput schema / (root)
        Previous value: -nullNew value: +{
        +  "properties": {
        +    "max_results": {
        +      "type": "integer"
        +    },
        +    "message": {
        +      "description": "Present when the site has no crawl output yet",
        +      "type": "string"
        +    },
        +    "offset": {
        +      "type": "integer"
        +    },
        +    "pages": {
        +      "items": {
        +        "properties": {
        +          "content_length": {
        +            "type": "integer"
        +          },
        +          "crawled_at": {
        +            "type": "string"
        +          },
        +          "depth": {
        +            "type": "integer"
        +          },
        +          "title": {
        +            "type": "string"
        +          },
        +          "url": {
        +            "type": "string"
        +          }
        +        },
        +        "required": [
        +          "url",
        +          "title",
        +          "depth",
        +          "crawled_at",
        +          "content_length"
        +        ],
        +        "type": "object"
        +      },
        +      "type": [
        +        "null",
        +        "array"
        +      ]
        +    },
        +    "returned": {
        +      "type": "integer"
        +    },
        +    "site_key": {
        +      "type": "string"
        +    },
        +    "total": {
        +      "type": "integer"
        +    }
        +  },
        +  "required": [
        +    "site_key",
        +    "pages",
        +    "total",
        +    "offset",
        +    "max_results",
        +    "returned"
        +  ],
        +  "type": "object"
        +}
    • Changedlist_sites1 field changed
      • changedOutput schema / (root)
        Previous value: -nullNew value: +{
        +  "properties": {
        +    "config_path": {
        +      "type": "string"
        +    },
        +    "sites": {
        +      "items": {
        +        "properties": {
        +          "domain": {
        +            "type": "string"
        +          },
        +          "key": {
        +            "type": "string"
        +          },
        +          "last_crawled": {
        +            "type": "string"
        +          },
        +          "max_depth": {
        +            "type": "integer"
        +          },
        +          "path_prefix": {
        +            "type": "string"
        +          },
        +          "start_urls_count": {
        +            "type": "integer"
        +          },
        +          "status": {
        +            "description": "\"running\" while a crawl is active; absent otherwise",
        +            "type": "string"
        +          }
        +        },
        +        "required": [
        +          "key",
        +          "domain",
        +          "path_prefix",
        +          "start_urls_count",
        +          "max_depth"
        +        ],
        +        "type": "object"
        +      },
        +      "type": [
        +        "null",
        +        "array"
        +      ]
        +    },
        +    "total_sites": {
        +      "type": "integer"
        +    }
        +  },
        +  "required": [
        +    "sites",
        +    "config_path",
        +    "total_sites"
        +  ],
        +  "type": "object"
        +}
    • Changedread_page1 field changed
      • changedOutput schema / (root)
        Previous value: -nullNew value: +{
        +  "properties": {
        +    "content": {
        +      "type": "string"
        +    },
        +    "content_hash": {
        +      "type": "string"
        +    },
        +    "content_length": {
        +      "description": "Bytes of content in this response",
        +      "type": "integer"
        +    },
        +    "crawled_at": {
        +      "type": "string"
        +    },
        +    "depth": {
        +      "type": "integer"
        +    },
        +    "headings": {
        +      "items": {
        +        "type": "string"
        +      },
        +      "type": [
        +        "null",
        +        "array"
        +      ]
        +    },
        +    "next_offset": {
        +      "description": "Pass as offset to read the next part; present only when truncated",
        +      "type": [
        +        "null",
        +        "integer"
        +      ]
        +    },
        +    "offset": {
        +      "type": "integer"
        +    },
        +    "site_key": {
        +      "type": "string"
        +    },
        +    "source": {
        +      "type": "string"
        +    },
        +    "title": {
        +      "type": "string"
        +    },
        +    "total_length": {
        +      "description": "Bytes of the whole stored page",
        +      "type": "integer"
        +    },
        +    "truncated": {
        +      "type": "boolean"
        +    },
        +    "url": {
        +      "type": "string"
        +    }
        +  },
        +  "required": [
        +    "site_key",
        +    "url",
        +    "title",
        +    "content",
        +    "content_length",
        +    "total_length",
        +    "offset",
        +    "truncated",
        +    "depth",
        +    "crawled_at",
        +    "content_hash",
        +    "headings",
        +    "source"
        +  ],
        +  "type": "object"
        +}
    • Changedsearch_docs2 fields changed
      • changedInput schema / properties / query / description
        Previous value: -"Search terms. Plain words are ANDed; \"quoted phrases\", OR, and prefix* work too."New value: +"Search terms. Plain words are ANDed; when too few results contain every word, the rest are filled with partial matches (match: partial) that contain only some words or built-in synonyms of them. \"quoted phrases\", OR, and prefix* work too and are never relaxed."
      • changedOutput schema / (root)
        Previous value: -nullNew value: +{
        +  "properties": {
        +    "count": {
        +      "type": "integer"
        +    },
        +    "next_actions": {
        +      "type": "string"
        +    },
        +    "query": {
        +      "type": "string"
        +    },
        +    "results": {
        +      "items": {
        +        "properties": {
        +          "anchor": {
        +            "type": "string"
        +          },
        +          "heading_path": {
        +            "type": "string"
        +          },
        +          "match": {
        +            "type": "string"
        +          },
        +          "rank": {
        +            "type": "number"
        +          },
        +          "site_key": {
        +            "type": "string"
        +          },
        +          "snippet": {
        +            "type": "string"
        +          },
        +          "title": {
        +            "type": "string"
        +          },
        +          "url": {
        +            "type": "string"
        +          }
        +        },
        +        "required": [
        +          "site_key",
        +          "url",
        +          "title",
        +          "snippet",
        +          "rank",
        +          "match"
        +        ],
        +        "type": "object"
        +      },
        +      "type": [
        +        "null",
        +        "array"
        +      ]
        +    },
        +    "site_key": {
        +      "description": "Present when the search was limited to one site",
        +      "type": "string"
        +    }
        +  },
        +  "required": [
        +    "query",
        +    "count",
        +    "results",
        +    "next_actions"
        +  ],
        +  "type": "object"
        +}
  5. 1 tool updatev2.12.0
    • Changeddiff_crawl1 field changed
      • changedInput schema / properties / since / description
        Previous value: -"RFC3339 timestamp; the most recent crawl whose crawl_ended_at <= since is the baseline (e.g. 2026-05-23T22:00:00Z)"New value: +"RFC3339 timestamp; the most recent crawl whose crawl_ended_at <= since, compared to the second, is the baseline (e.g. 2026-05-23T22:00:00Z)"
  6. 2 tool updatesv2.10.1
    • Changeddescribe_server2 fields changed
      • addedInput schema / properties
        Added value: +{}
      • addedInput schema / required
        Added value: +[]
    • Changedlist_sites2 fields changed
      • addedInput schema / properties
        Added value: +{}
      • addedInput schema / required
        Added value: +[]
  7. 11 tool updates
    • First observedcancel_crawl
    • First observedcrawl_site
    • First observeddescribe_server
    • First observeddiff_crawl
    • First observedget_freshness
    • First observedget_job_status
    • First observedget_page
    • First observedlist_pages
    • First observedlist_sites
    • First observedread_page
    • First observedsearch_docs

TDQS

A4.3/5.0

Scored across 13 tools

Disambiguation4/5

Most tools target clearly distinct operations (crawl lifecycle, page access, site config, search). A few boundaries blur: get_freshness vs describe_server vs get_job_status all surface crawl/job state, and get_page vs read_page are only separable by carefully reading the descriptions. The descriptions do explicitly call out these distinctions, which mitigates the overlap.

Naming Consistency5/5

Every tool is snake_case verb_noun (crawl_site, get_job_status, read_page, save_site, search_docs), with only the conventional noun-only list_sites/list_pages as natural exceptions. No mixed conventions or camelCase.

Tool Count5/5

13 tools sits comfortably in the well-scoped range, and each maps to a real workflow step (crawl, wait, inspect, diff, search, read, configure). Nothing appears redundant or padding.

Completeness4/5

The crawl lifecycle, search, page reads, and site onboarding are all covered, including diff and cancellation. The main gap is site lifecycle management: there is no delete_site/remove_site and no explicit update beyond overwriting via save_site with replace:true, so removing a configured site is a dead end.

Maintenance

ActivityActive
ResponsivenessSyncing

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    C
    maintenance
    Local-first MCP server providing semantic search over library docs, fully offline. Single Go binary speaks MCP over stdio against a vector index pinned to the binary version. Like Context7 with the internet turned off. Apache 2.0. Linux + macOS, also available as a container image.
    2
    4
    Apache 2.0
  • A
    license
    A
    quality
    B
    maintenance
    Go MCP server for multi-format document access — PDF, TXT, MD, DOCX, CSV, images. 12 tools including OCR, search, table extraction, and URL fetch. Single binary, no runtime.
    13
    11
    MIT
  • F
    license
    Not graded
    quality
    D
    maintenance
    Documentation crawler MCP server that crawls and indexes documentation sites so that any MCP-compatible AI can search, read, and expand on the content.
    1
    -
  • A
    license
    Not graded
    quality
    C
    maintenance
    A documentation MCP server that crawls websites and Git repositories, stores them as Markdown, and provides tools to search and retrieve documentation for local LLMs and AI agents.
    Apache 2.0