doc-scraper
doc-scraper MCP server — crawl, search, and read technical documentation sites offline for LLM/RAG use.
Orient:
describe_serverreturns server identity, configured sites, and recent jobs in one call;list_siteslists configured site keys and metadata.Crawl:
crawl_sitestarts a background crawl (optionally incremental) and returns a job ID;get_job_statustracks it;cancel_crawlcancels a running or pending job.Read crawled content:
list_pagesenumerates a site's crawled pages (paginated, metadata only);read_pagereturns a page's stored markdown offline, supporting byte-offset paging for large pages.Search offline:
search_docsruns BM25 full-text search over the crawled corpus with snippets and FTS5 syntax (phrases, OR, prefix*), optionally scoped to one site.Fetch live:
get_pagefetches any URL over the network on demand and returns markdown, independent of crawl scope or stored output.Track change over time:
get_freshnessreports the latest crawl summary and staleness;diff_crawlreports pages added, removed, or changed since a given timestamp.
Automatically detects and crawls Docusaurus-based documentation sites, extracting core content and converting it to Markdown for LLM ingestion.
Automatically detects and crawls GitBook documentation sites, extracting core content and converting it to Markdown for LLM ingestion.
Automatically detects and crawls Sphinx-based documentation sites, extracting core content and converting it to Markdown for LLM ingestion.
LLM Documentation Scraper (doc-scraper)
A configurable, concurrent, and resumable web crawler written in Go. Specifically designed to scrape technical documentation websites, extract core content, convert it cleanly to Markdown format suitable for ingestion by Large Language Models (LLMs), and save the results locally.

doc-scraper crawls documentation sites into clean Markdown and serves them to coding agents over MCP. Your agent searches and reads the docs you chose, offline, instead of guessing from training data or a live web search. Point doc-scraper add at any docs page and it detects the framework (30+ docs generators, including Docusaurus, Starlight, Sphinx, VitePress, Material for MkDocs, and mdBook), validates the content selector, and drafts the config for you.
Documentation: https://sriram-pr.github.io/doc-scraper/
Features
Offline search for agents: BM25 full-text search (
search_docs) and page reads (read_page) over the stored corpus, with no network accessFramework detection: 30+ docs generators recognized, each selector validated on the page, Readability fallback, JavaScript-only shells reported instead of crawled empty
Clean output: GitHub-Flavored Markdown per page with frontmatter, plus
pages.jsonl,llms.txt, andllms-full.txtPolite and resumable: robots.txt and sitemaps, per-host rate limits, retries with backoff, resumable crawls, and staged output that never clobbers the last good copy
Stays fresh: incremental re-crawls, watch mode, and crawl history (
get_freshness,diff_crawl)Many sites at once: parallel crawls with shared limits
Related MCP server: go-docs-mcp
Install
Download a release binary (Linux, macOS, Windows; amd64 and arm64) from the Releases page, or:
go install github.com/Sriram-PR/doc-scraper/v2/cmd/doc-scraper@latestdocker run --rm --user "$(id -u):$(id -g)" -v "$PWD":/data ghcr.io/sriram-pr/doc-scraper:latest crawl -site rust_cli_bookClaude Desktop users can install doc-scraper.mcpb from the latest release instead. Details: Install.
Quick start
doc-scraper add -site rust_cli_book https://rust-cli.github.io/book/index.html # detect, preview, confirm
doc-scraper crawl -site rust_cli_book # Markdown lands in ./crawled_docs/rust_cli_book/
doc-scraper search -site rust_cli_book "error handling"Then give it to Claude Code:
claude mcp add --transport stdio doc-scraper -- /path/to/doc-scraper mcp-server -config /path/to/config.yamlSee Claude Code setup for scopes and a first prompt.
Docs
Getting started: install, quick start, adding a site
Use with agents (MCP): Claude Code, Claude Desktop, any stdio client, and the 11 tools
Docs frameworks: how detection works, with guides for Docusaurus, Starlight, Sphinx, VitePress, Material for MkDocs, and mdBook
Configuration and CLI reference
Contributing
Contributions are welcome. Read CONTRIBUTING.md before starting: it covers setup, the testing rules CI enforces, how to claim an issue, and the policy on AI-assisted contributions. Issues labeled good first issue are a good place to begin.
Report security problems privately as described in SECURITY.md. Everyone taking part follows the Code of Conduct.
Privacy Policy
doc-scraper collects nothing: no telemetry, no analytics, no accounts. All output and state stays on your machine, and the only network requests it makes are the crawls and fetches you explicitly ask for. Full policy: PRIVACY.md.
License
This project is licensed under the Apache-2.0 License.
Acknowledgements
GoQuery for HTML parsing
html-to-markdown for conversion
BadgerDB for state persistence
mcp-go for MCP server implementation
go-readability for content extraction fallback
modernc.org/sqlite for the pure-Go crawl-history index
robotstxt for
robots.txtparsingyaml.v3 for configuration parsing
google/uuid for crawl job IDs
x/sync for the weighted semaphores that cap concurrency
Available Tools
11 toolscancel_crawlAIdempotent
Cancel a running or pending crawl job by job ID. The previously stored corpus and crawl history are left untouched. Has no effect on jobs already in a terminal state.
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes | The job ID returned by crawl_site |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations (destructiveHint=false, idempotentHint=true), the description adds useful behavioral facts: the stored corpus and crawl history are left untouched, and terminal-state jobs are unaffected. This reassures the agent about side effects, though it doesn't cover the return value or confirming state after cancellation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, front-loaded with the core action, each contributing a distinct fact (action, preserved data, terminal-state no-op). No waste.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-parameter cancel operation with annotations covering the safety profile and no output schema to explain, the description is nearly complete. Missing only post-cancellation state/return detail, which is minor.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the single job_id is fully documented there (including its origin from crawl_site). The description's 'by job ID' adds nothing beyond the schema, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('Cancel') and resource ('crawl job') plus the qualifying scope ('running or pending'). An agent can immediately distinguish it from siblings like get_job_status or crawl_site.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Clearly states the applicable states (running or pending) and explicitly rules out terminal-state jobs, which is real when-to-use guidance. It does not name an alternative tool, so it falls short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_siteA
Start a background crawl for a configured site. Returns immediately with a job ID. The crawl builds in a staging area and replaces the stored corpus only when it completes, so a cancelled or failed crawl leaves the previous corpus intact.
| Name | Required | Description | Default |
|---|---|---|---|
| site_key | Yes | Site key from config file (e.g., 'langchain_py', 'rust_docs') | |
| incremental | No | Enable incremental mode (skip unchanged pages) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes well beyond the annotations: it discloses that execution is asynchronous (returns a job ID immediately), that writes go to a staging area and replace the corpus atomically, and that cancelled/failed crawls leave the prior corpus intact. That last point meaningfully explains why destructiveHint=false despite this being a mutating crawl.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight sentences, zero filler. The core action is front-loaded, and the async return and atomicity guarantee follow in logical order.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description compensates by stating the return value (job ID) and explains the lifecycle/failure semantics. An agent has everything needed to invoke this correctly and understand the consequences.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so site_key and incremental are already documented in the schema. The description adds no extra semantics about these parameters, which is the expected baseline when the schema carries the full burden.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Start a background crawl for a configured site') and adds the async contract ('Returns immediately with a job ID'). This cleanly separates it from siblings like get_job_status and cancel_crawl without needing to name them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The 'background crawl' framing implies this is the kicking-off action, but there is no explicit when-to-use guidance or mention of alternatives such as diff_crawl or list_sites. Usage is inferable from the verb rather than stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
describe_serverRead-onlyIdempotent
Returns server identity, configured sites, and recent crawl jobs in one call. Call this first to orient yourself: it consolidates what would otherwise require list_sites plus several get_job_status calls. The MCP tool list is already advertised by the protocol so it is not duplicated here.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
diff_crawlARead-onlyIdempotent
Return added/removed/changed pages between the latest crawl and the most recent crawl whose crawl_ended_at <= since. Hash-based verdicts from the SQLite history index. Pair with get_freshness: pass the last_crawl_ended_at value as since after running crawl_site with incremental set to true to see exactly what changed.
| Name | Required | Description | Default |
|---|---|---|---|
| since | Yes | RFC3339 timestamp; the most recent crawl whose crawl_ended_at <= since is the baseline (e.g. 2026-05-23T22:00:00Z) | |
| offset | No | Pagination offset, 0-based | |
| site_key | Yes | Site key from config | |
| max_results | No | Maximum diff entries to return (default 100, max 1000) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, so the safety profile is covered. The description adds behavioral context by noting 'hash-based verdicts from the SQLite history index' and precisely defining the baseline selection rule, which goes beyond the structured annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, with the core behavior first and the workflow example last. Each sentence carries distinct information (what, how, when) with no filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given there is no output schema, the description gives a sufficient high-level contract: added/removed/changed pages with hash-based verdicts. It also reduces ambiguity by naming the exact sequence with crawl_site and get_freshness, so an agent can correctly select and chain the tool; minor gaps like response entry shape remain but are not essential for invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema covers 100% of parameters, so the baseline is 3. The description adds cross-tool value by explaining where `since` comes from ('pass the last_crawl_ended_at value as since'), which is not present in the input schema; the other parameters are already well documented in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a clear action and resource: 'Return added/removed/changed pages between the latest crawl and the most recent crawl whose crawl_ended_at <= since.' This is specific enough to distinguish it from sibling tools like get_freshness, and the title 'Diff crawls over time' reinforces the same function.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides a concrete workflow: 'Pair with get_freshness: pass the last_crawl_ended_at value as since after running crawl_site with incremental set to true.' This tells an agent when to call it and how to chain it with siblings, though it does not explicitly state exclusions or when-not-to-use alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_freshnessARead-onlyIdempotent
Return the most recent crawl summary for a site (last_crawl_started_at/ended_at, total_pages, mode, age_seconds) plus output/state dir presence and any running job. Use this to decide whether to query the existing crawl or run crawl_site first.
| Name | Required | Description | Default |
|---|---|---|---|
| site_key | Yes | Site key from config (use list_sites to discover available keys) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, so the safety profile is covered. The description goes further by disclosing what the tool returns (crawl summary fields, output/state dir presence, running job), which is useful behavioral detail beyond the structured annotations. It accurately describes a read-only inspection operation without contradicting any annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two dense sentences: the first lists the return content, the second gives the decision context. No filler and the key return information is front-loaded. The description achieves high information density in a compact format.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter read-only tool with no output schema, this description is complete. It states what is returned, how to decide to use the tool, and the annotations cover idempotence and safety. The provided information fully supports correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema documents site_key at 100% coverage, including the hint to use list_sites to discover available keys. The description adds no parameter-level details, but with full schema coverage the baseline score of 3 is appropriate. No extra semantics are needed beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource ('Return the most recent crawl summary for a site') and enumerates the returned fields, making the operation unmistakable. It also injects a decision cue ('Use this to decide whether to query the existing crawl or run crawl_site first') that distinguishes it from the crawl_site sibling. This is a clear, specific purpose statement that differentiates it from other tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly states when to use the tool: before deciding between querying an existing crawl and running crawl_site. This names the alternative and the condition, satisfying the usage guidance dimension. The tool's role as a freshness check is directly tied to a decision, giving clear context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_job_statusBRead-onlyIdempotent
Get the status of a crawl job
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes | The job ID returned by crawl_site |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is established. The description adds no behavioral context beyond the word 'Get', such as what status values may appear or whether the call reflects updated progress. No contradiction exists.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, focused sentence with no filler. The verb and object are front-loaded, making it immediately scannable for an agent.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter, read-only status lookup with strong annotations, the definition is mostly sufficient. However, with no output schema, it does not indicate what statuses or fields the response contains, and the absence of usage guidance leaves minor but real ambiguity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the single job_id parameter is adequately documented as the ID returned by crawl_site. The description itself adds no parameter-level meaning beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Get') and resource ('status of a crawl job'), making the tool's purpose clear. It is naturally distinct from siblings like crawl_site or cancel_crawl, though it does not explicitly name or contrast alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: checking the status of a crawl job makes sense after calling crawl_site, and the schema's job_id description references crawl_site as the source of the ID. However, the description itself gives no explicit when-to-use or when-not-to-use guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_pageARead-onlyIdempotent
Fetch a URL live over the network and return its content as markdown. This is an on-demand fetch independent of any crawl: it does not read the stored crawl output, and it ignores the site's configured content_selector and scope.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to fetch | |
| content_selector | No | CSS selector for main content (defaults to 'body') |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations already provide readOnlyHint, openWorldHint, idempotentHint, and destructiveHint=false. The description adds substantial behavioral detail beyond these: live network access, markdown output, independence from crawl output, and intentional disregard for site-configured content_selector/scope. There is no contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with action and output, then a compact contrast that disambiguates from crawl-based tools. Every sentence earns its place with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter read-only tool with rich annotations, 100% schema coverage, and no output schema, the description is sufficient. It explains what, when, and one key quirk, leaving no ambiguity about invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema covers 100% of the parameters, including the content_selector default of 'body'. The description adds no meaningful parameter-level detail; its mention of content_selector refers to site configuration rather than the argument itself. Baseline 3 is appropriate because the schema carries the parameter documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: 'Fetch a URL live over the network and return its content as markdown.' It also distinguishes itself from siblings by explicitly noting it does not read stored crawl output, which clearly separates it from read_page and other crawl-related tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides a clear context: use this for an on-demand, live fetch independent of any crawl, as opposed to reading stored crawl output. It doesn't name an explicit alternative such as read_page, but the contrast with crawl-based reads gives an agent enough guidance to choose correctly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_pagesARead-onlyIdempotent
List crawled pages for a site, paginated and sorted by URL. Returns metadata only (URL, title, depth, crawled_at, content_length). Pass any URL returned here to read_page to get its stored markdown; get_page re-fetches a URL live rather than returning the crawled copy.
| Name | Required | Description | Default |
|---|---|---|---|
| offset | No | Pagination offset, 0-based (default: 0) | |
| site_key | Yes | Site key from config (use list_sites to discover available keys) | |
| max_results | No | Maximum pages to return (default: 100, max: 1000) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover read-only, idempotent, and non-destructive behavior. The description adds useful behavioral context: it returns metadata only (URL, title, depth, crawled_at, content_length), is paginated and sorted by URL, and reflects stored crawled pages rather than live fetches. It doesn't discuss errors or partial failures, so it is strong but not exhaustive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three focused sentences front-load the core action, state output scope and shape, and route to sibling tools. Every sentence earns its place with no wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a read-only listing tool with fully documented parameters and safety annotations, the description covers action, scope, output fields, pagination ordering, and downstream tool usage. The schema supplies the remaining parameter details, so nothing essential is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline applies. The description's mention of pagination and 'for a site' mildly reinforces site_key, offset, and max_results, but it does not add parameter meaning beyond what the schema already documents.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description uses a specific verb 'List' with a clear resource 'crawled pages for a site' and adds pagination and sort order. It also distinguishes itself from read_page and get_page by explicitly stating that it returns metadata only and that the pages are crawled, not live.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly routes downstream usage: URLs returned here should be passed to read_page for stored markdown, while get_page is for live re-fetching rather than crawled copies. This gives an agent clear when-to-use and when-not-to-use guidance for the relevant siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_sitesRead-onlyIdempotent
List all configured sites available for crawling
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
read_pageARead-onlyIdempotent
Return a page's markdown from the stored crawl output, without any network access. This is the counterpart to get_page: read_page serves the crawled copy that already had the site's content_selector applied, while get_page re-fetches the URL live. Use list_pages to discover URLs, then read_page to read them. Large pages are truncated at max_bytes; follow next_offset to read the rest.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL of a crawled page, as reported by list_pages | |
| offset | No | Byte offset into the page content, for reading a truncated page in parts (default: 0) | |
| site_key | Yes | Site key from config (use list_sites to discover available keys) | |
| max_bytes | No | Maximum content bytes to return (default: 102400, max: 1048576) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnly/idempotent annotations, the description reveals important behavior: no network access, content_selector already applied, truncation at max_bytes, and a next_offset mechanism for reading the rest. These are not derivable from the annotations or schema alone.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four sentences, each earning its place: core function, sibling distinction, discovery workflow, and pagination behavior. The most important information is front-loaded, and there is no filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool without an output schema, the description adequately covers return format, truncation, and continuation via next_offset. It also provides the surrounding workflow with list_pages and clarifies the live-crawl tradeoff, leaving no critical gap for an agent to call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and the schema already explains each parameter's meaning. The description's mention of max_bytes truncation and next_offset adds behavioral context, but it does not add new semantic detail about the parameters themselves.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Return a page's markdown from the stored crawl output, without any network access.' It explicitly contrasts itself with get_page, making the tool's role unambiguous and differentiating it from the most similar sibling.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives clear workflow guidance: 'Use list_pages to discover URLs, then read_page to read them.' It also explains the exact tradeoff versus get_page (crawled copy vs live re-fetch), so an agent knows when to select this tool over the alternative.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_docsARead-onlyIdempotent
Full-text search across all crawled documentation, ranked by relevance (BM25 with stemming), with zero network access. Results carry the page URL, its section heading path, and a snippet with match terms marked [like this]; follow up with read_page for the full page. Supports FTS5 syntax: quoted phrases, OR, and trailing * for prefix matching. Searches the stored index only; run crawl_site first for uncrawled sites.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Maximum results to return (default: 10, max: 50) | |
| query | Yes | Search terms. Plain words are ANDed; "quoted phrases", OR, and prefix* work too. | |
| site_key | No | Limit results to one site (use list_sites to discover keys). Omit to search all sites. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations (readOnlyHint, idempotentHint, destructiveHint), the description reveals important behavioral traits: zero network access, BM25 ranking with stemming, result format including URL and section path, FTS5 syntax support, and the stored-index-only scope. This adds substantial context about what happens when calling the tool and what the output contains, with no contradiction to annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four dense sentences, each earning its place: purpose and ranking, result format and follow-up, query syntax, and index caveat. The most important scoping constraints are front-loaded, and there is no fluff or repetition of schema content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given there is no output schema, the description properly explains what results contain (URL, heading path, snippet with highlighted matches) and how to get the full content. It also covers prerequisites (crawl_site), querying syntax, and scope limitations. The agent has enough context to invoke the tool correctly without missing behavioral details.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already describes all three parameters at 100% coverage, including the query syntax, limit bounds, and site_key usage. The description reiterates the FTS5 syntax but does not meaningfully extend parameter-level understanding beyond the schema. Baseline 3 is appropriate because the schema carries the parameter-heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Full-text search across all crawled documentation'. It further distinguishes the tool from siblings by stating it is ranked by relevance, works offline, and searches only the stored index, which clearly differentiates it from list_pages, get_page, and read_page.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context for when to use the tool: when you need to search crawled documentation. It also provides practical guidance by telling the agent to run crawl_site first for uncrawled sites and to follow up with read_page for the full page. However, it does not explicitly mention when not to use this tool in favor of a sibling like list_pages or get_page, so it falls just short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
2 tool updates
v2.10.1- Changed
describe_server2 fields changed- added
Input schema / propertiesAdded value: +{} - added
Input schema / requiredAdded value: +[]
- Changed
list_sites2 fields changed- added
Input schema / propertiesAdded value: +{} - added
Input schema / requiredAdded value: +[]
11 tool updates
- First observed
cancel_crawl - First observed
crawl_site - First observed
describe_server - First observed
diff_crawl - First observed
get_freshness - First observed
get_job_status - First observed
get_page - First observed
list_pages - First observed
list_sites - First observed
read_page - First observed
search_docs
TDQS
Scored across 11 tools
Each tool has a clearly distinct role, and the descriptions explicitly resolve the two potentially overlapping pairs: read_page (stored crawl copy) versus get_page (live fetch), and list_pages (metadata) versus search_docs (full-text). crawl management tools (crawl_site, cancel_crawl, get_job_status, get_freshness, diff_crawl) target different stages of the crawl lifecycle, not the same operation.
All tool names use consistent snake_case with a verb_noun pattern (list_pages, crawl_site, get_job_status, search_docs, etc.). No camelCase or vague naming deviations appear.
With 11 tools, the set is well-scoped for a documentation scraper: it covers crawl control, page reading, search, and freshness without feeling bloated or thin. Each tool has a distinct and necessary function.
The surface covers the core lifecycle for crawling, reading, searching, and diffing documentation, leaving no dead ends for those workflows. Minor gaps exist around managing site configuration itself (only list_sites is provided) and no explicit standalone job-list operation, though describe_server partially fills the latter.
Maintenance
Related MCP Connectors
Agent-native MCP server over the public saagarpatel.dev corpus. Read-only, stateless.
- fastCRWOAuthio.github.us
Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
Document-to-Markdown MCP server — convert PDF, Office and HTML into LLM-ready Markdown.
Turn any public website into an MCP server for agents to search, read and navigate.
Related MCP Servers
- AlicenseAqualityCmaintenanceLocal-first MCP server providing semantic search over library docs, fully offline. Single Go binary speaks MCP over stdio against a vector index pinned to the binary version. Like Context7 with the internet turned off. Apache 2.0. Linux + macOS, also available as a container image.24Apache 2.0
- AlicenseAqualityBmaintenanceGo MCP server for multi-format document access — PDF, TXT, MD, DOCX, CSV, images. 12 tools including OCR, search, table extraction, and URL fetch. Single binary, no runtime.1311MIT
- FlicenseNot gradedqualityDmaintenanceDocumentation crawler MCP server that crawls and indexes documentation sites so that any MCP-compatible AI can search, read, and expand on the content.1-
- AlicenseNot gradedqualityCmaintenanceA documentation MCP server that crawls websites and Git repositories, stores them as Markdown, and provides tools to search and retrieve documentation for local LLMs and AI agents.Apache 2.0