doc-scraper
This MCP server lets you crawl documentation sites into local Markdown and serve the stored corpus to agents for offline search, reading, freshness checks, and crawl management.
List configured sites and get a consolidated server/site/recent-job overview.
Start full or incremental background crawls and return a job ID.
Check crawl job status, optionally waiting up to 60 seconds, and cancel jobs.
Check site crawl freshness and compare latest versus baseline crawls for added, removed, or changed pages.
List crawled pages and read stored Markdown by URL and optional heading anchor with no network access.
Run BM25 full-text search across crawled docs, optionally scoped to one site.
Fetch arbitrary HTML pages live over the network and convert them to Markdown with truncation/pagination controls.
Preserve previous crawl output/state while staging new crawls, so failed or cancelled crawls do not clobber the last good corpus.
Automatically detects and crawls Docusaurus-based documentation sites, extracting core content and converting it to Markdown for LLM ingestion.
Automatically detects and crawls GitBook documentation sites, extracting core content and converting it to Markdown for LLM ingestion.
Automatically detects and crawls Sphinx-based documentation sites, extracting core content and converting it to Markdown for LLM ingestion.
LLM Documentation Scraper (doc-scraper)
A configurable, concurrent, and resumable web crawler written in Go. Specifically designed to scrape technical documentation websites, extract core content, convert it cleanly to Markdown format suitable for ingestion by Large Language Models (LLMs), and save the results locally.

doc-scraper crawls documentation sites into clean Markdown and serves them to coding agents over MCP. Your agent searches and reads the docs you chose, offline, instead of guessing from training data or a live web search. Point doc-scraper add at any docs page and it detects the framework (30+ docs generators, including Docusaurus, Starlight, Sphinx, VitePress, Material for MkDocs, and mdBook), validates the content selector, and drafts the config for you.
Documentation: https://sriram-pr.github.io/doc-scraper/
Features
Offline search for agents: BM25 full-text search (
search_docs) and page reads (read_page) over the stored corpus, with no network accessFramework detection: 30+ docs generators recognized, each selector validated on the page, Readability fallback, JavaScript-only shells reported instead of crawled empty
Clean output: GitHub-Flavored Markdown per page with frontmatter, plus
pages.jsonl,llms.txt, andllms-full.txtPolite and resumable: robots.txt and sitemaps, per-host rate limits, retries with backoff, resumable crawls, and staged output that never clobbers the last good copy
Stays fresh: incremental re-crawls, watch mode, and crawl history (
get_freshness,diff_crawl)Many sites at once: parallel crawls with shared limits
Related MCP server: go-docs-mcp
Install
Download a release binary (Linux, macOS, Windows; amd64 and arm64) from the Releases page, or:
go install github.com/Sriram-PR/doc-scraper/v2/cmd/doc-scraper@latestdocker run --rm --user "$(id -u):$(id -g)" -v "$PWD":/data ghcr.io/sriram-pr/doc-scraper:latest crawl -site rust_cli_bookClaude Desktop users can install doc-scraper.mcpb from the latest release instead. Details: Install.
Quick start
doc-scraper add -site rust_cli_book https://rust-cli.github.io/book/index.html # detect, preview, confirm
doc-scraper crawl -site rust_cli_book # Markdown lands in ./crawled_docs/rust_cli_book/
doc-scraper search -site rust_cli_book "error handling"Then give it to Claude Code:
claude mcp add --transport stdio doc-scraper -- /path/to/doc-scraper mcp-server -config /path/to/config.yamlSee Claude Code setup for scopes and a first prompt.
Docs
Getting started: install, quick start, adding a site
Use with agents (MCP): Claude Code, Claude Desktop, any stdio client, and the 13 tools
Docs frameworks: how detection works, with guides for Docusaurus, Starlight, Sphinx, VitePress, Material for MkDocs, and mdBook
Configuration and CLI reference
Contributing
Contributions are welcome. Read CONTRIBUTING.md before starting: it covers setup, the testing rules CI enforces, how to claim an issue, and the policy on AI-assisted contributions. Issues labeled good first issue are a good place to begin.
Report security problems privately as described in SECURITY.md. Everyone taking part follows the Code of Conduct.
Privacy Policy
doc-scraper collects nothing: no telemetry, no analytics, no accounts. All output and state stays on your machine, and the only network requests it makes are the crawls and fetches you explicitly ask for. Full policy: PRIVACY.md.
License
This project is licensed under the Apache-2.0 License.
Acknowledgements
GoQuery for HTML parsing
html-to-markdown for conversion
BadgerDB for state persistence
mcp-go for MCP server implementation
go-readability for content extraction fallback
modernc.org/sqlite for the pure-Go crawl-history index
robotstxt for
robots.txtparsingyaml.v3 for configuration parsing
google/uuid for crawl job IDs
x/sync for the weighted semaphores that cap concurrency
Available Tools
13 toolscancel_crawlAIdempotent
Cancel a running or pending crawl job by job ID. The previously stored corpus and crawl history are left untouched. Has no effect on jobs already in a terminal state.
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes | The job ID returned by crawl_site |
Output Schema
| Name | Required | Description |
|---|---|---|
| job_id | Yes | |
| status | Yes | |
| message | Yes | |
| site_key | Yes | |
| cancelled | Yes | false when the job was already in a terminal state |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare non-read-only, non-destructive, idempotent behavior. The description adds valuable specifics: stored corpus and crawl history are untouched, and terminal-state jobs are unaffected. It does not cover auth or rate limits, but those are not critical here.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, front-loaded with the core action and immediately followed by the key side-effect and state constraints. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With a complete input schema, rich annotations, and an output schema, the description covers purpose, side effects, and state limitations. It could add a note about checking job status before cancelling, but otherwise is sufficient for a simple cancel operation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The single job_id parameter has 100% schema coverage and the schema already documents it as the ID returned by crawl_site. The description adds no semantic detail beyond 'by job ID', so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (cancel) and resource (running or pending crawl job) with scope. Though it doesn't name a sibling, its purpose is unmistakable against crawl_site and get_job_status in the sibling list.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Clearly indicates the applicable job states (running or pending) and an exclusion (terminal state jobs have no effect). It does not name an alternative tool or say when to prefer checking status first.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_siteA
Start a background crawl for a configured site. Returns immediately with a job ID. The crawl builds in a staging area and replaces the stored corpus only when it completes, so a cancelled or failed crawl leaves the previous corpus intact. A crawl that lost a large share of its pages is kept back instead (the job fails with refused set), and the stored corpus is kept.
| Name | Required | Description | Default |
|---|---|---|---|
| site_key | Yes | Site key from config (use list_sites to discover available keys) | |
| incremental | No | Enable incremental mode (skip unchanged pages) | |
| allow_shrink | No | Publish the crawl even if it lost many pages compared to the last one (use with incremental:true to publish a kept-back crawl) |
Output Schema
| Name | Required | Description |
|---|---|---|
| job_id | Yes | |
| status | Yes | started, or already_running when a crawl for the site is in progress |
| message | Yes | |
| site_key | Yes | |
| incremental | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Discloses behaviour well beyond the annotations: builds in a staging area, atomic replacement of the corpus on success, previous corpus preserved on cancel/failure, and a kept-back crawl that fails with 'refused' when pages are lost. This is consistent with destructiveHint=false and non-idempotent semantics and clarifies the failure mode precisely.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four tight sentences, front-loaded with purpose and return behaviour, then failure semantics. No filler; each sentence carries distinct information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return structure need not be spelled out, and the description still notes the job ID. For a 3-parameter mutation with annotations, the failure and atomicity story is fully covered.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% so the baseline is 3, but the description adds real meaning to allow_shrink by explaining the shrink guardrail and the 'refused' failure that it overrides. Incremental mode itself is left to the schema text.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Start a background crawl for a configured site') plus the immediate return ('job ID'), which distinguishes it from siblings like get_job_status and cancel_crawl that operate on the resulting job.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Clear operating context: asynchronous start, staging-then-publish semantics, and the shrink guardrail with the workaround implied by the allow_shrink schema text. It does not explicitly name alternatives (e.g. why to prefer this over re-drafting a site), so it stops short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
describe_serverARead-onlyIdempotent
Returns server identity, configured sites, and recent crawl jobs in one call. Call this first to orient yourself: it consolidates what would otherwise require list_sites plus several get_job_status calls. The MCP tool list is already advertised by the protocol so it is not duplicated here.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
| sites | Yes | |
| server | Yes | |
| total_jobs | Yes | |
| jobs_capped | Yes | |
| recent_jobs | Yes | Newest first, at most 20 |
| total_sites | Yes | |
| next_actions | Yes | |
| config_writes | Yes | enabled or disabled; save_site needs enabled (mcp-server -allow-config-writes) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already establish that this is a read-only, idempotent, non-destructive operation. The description adds useful context that the call returns consolidated identity, sites, and recent job data in one call, though it does not discuss pagination, caching, or response limits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with what the tool returns, followed by when to call it and why. All three sentences contribute directly to selecting and understanding the tool, with no wasted wording.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the rich annotations, no parameters, and an output schema, the description is sufficiently complete. It explains the tool's purpose, usage, consolidation value, and even clarifies why the MCP tool list is not included.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are zero parameters, so parameter semantics are inherently simple. The description does not need to explain parameters, and the baseline for a no-parameter tool applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resources: returns server identity, configured sites, and recent crawl jobs. It clearly distinguishes itself from siblings by explaining that it consolidates what would otherwise require list_sites plus several get_job_status calls.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives explicit usage guidance: 'Call this first to orient yourself.' It also names the alternatives it replaces (list_sites and several get_job_status calls), leaving little ambiguity about when to use this tool versus its siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
diff_crawlARead-onlyIdempotent
Return added/removed/changed pages between the latest crawl and a baseline crawl: the crawl before it by default, or with since, the most recent crawl whose crawl_ended_at <= since. Hash-based verdicts from the SQLite history index. To see what a crawl changed, run crawl_site, wait for it to finish, then call diff_crawl without since.
| Name | Required | Description | Default |
|---|---|---|---|
| since | No | RFC3339 timestamp; the most recent crawl whose crawl_ended_at <= since, compared to the second, is the baseline (e.g. 2026-05-23T22:00:00Z). Omit to compare with the previous crawl. | |
| offset | No | Pagination offset, 0-based | |
| site_key | Yes | Site key from config | |
| max_results | No | Maximum diff entries to return (default: 100, max: 1000; larger values are capped, 0 or less means the default) |
Output Schema
| Name | Required | Description |
|---|---|---|
| note | No | Why the diff is empty or partial, when it is |
| added | Yes | |
| since | No | Present when since was given |
| total | Yes | |
| offset | Yes | |
| changed | Yes | |
| removed | Yes | |
| returned | Yes | |
| site_key | Yes | |
| max_results | Yes | |
| current_crawl | No | |
| baseline_crawl | No | |
| unchanged_count | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/non-destructive/openWorld=false, so the safety profile is covered. The description adds genuinely non-obvious behavior: verdicts are hash-based from a SQLite history index (no content re-fetch), and the baseline is data-dependent, requiring a prior crawl. It does not state what happens with zero or one prior crawl (empty result vs. error), which is the remaining gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core verb+resource, then baseline semantics, then a short workflow recipe; every sentence contributes. There is mild duplication between "the crawl before it by default" and the schema's "Omit to compare with the previous crawl."
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return shape needn't be described; the description covers baseline resolution and the prerequisite crawl workflow, which is what an agent needs to call it correctly. The only unaddressed case is degenerate history (no prior crawl) where the agent doesn't know whether to expect an error or empty diff.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so `since`, `offset`, `max_results`, and `site_key` are already documented in structured fields, and the description's `since` wording largely restates the schema's RFC3339 baseline rule. It adds the useful default ("the crawl before it by default") but nothing about pagination or `max_results` interaction. Baseline 3 applies when the schema carries the detail.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
"Return added/removed/changed pages between the latest crawl and a baseline crawl" gives a specific verb (return) plus the exact resource (page-level diff between two crawls). It also implicitly separates the tool from list_pages/get_page (content access) and get_freshness (staleness), so an agent can route correctly from names alone.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description spells out the operational sequence explicitly: "run crawl_site, wait for it to finish, then call diff_crawl without since," naming the sibling tool and the required precondition. It also states the default baseline and the alternative (pass `since`) with the exact selection rule, leaving nothing to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
draft_siteARead-onlyIdempotent
Draft a site entry and preview one page extracted with it. Never writes the config. With url, probes a new docs site the way doc-scraper add does (robots.txt, llms.txt, sitemap, framework detection). With only site_key, starts from that configured site's entry without probing. overrides sets entry fields by their config names (content_selector, allowed_path_prefix, disallowed_path_patterns, max_depth, ...); durations are strings such as "1s". Judge the preview (code blocks kept, nav_leak, chars), draft again with overrides until it looks right, then pass entry to save_site.
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | A docs URL to probe as a new site | |
| site_key | No | With url: the key for the new site. Without url: the configured site to start from | |
| overrides | No | Entry fields to set, by config field name | |
| preview_url | No | The page to preview (default: the sampled docs page, else the seed or first start URL) |
Output Schema
| Name | Required | Description |
|---|---|---|
| entry | Yes | The site entry by config field name; pass it, changed as needed, to save_site |
| exists | Yes | Whether site_key is configured now; save_site then needs replace: true |
| preview | Yes | |
| evidence | Yes | |
| site_key | Yes | |
| warnings | Yes | |
| framework | No | |
| generator | No | |
| confidence | No | |
| entry_yaml | Yes | |
| next_actions | Yes | |
| config_writes | Yes | enabled or disabled; save_site needs enabled |
| signal_source | No | |
| sitemap_page_count | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/openWorld/idempotent/non-destructive, and the description reinforces this with 'Never writes the config' while disclosing what probing actually does (robots.txt, llms.txt, sitemap, framework detection) — real behavioral context beyond the hints. It stops short of covering failure modes or network cost/rate limits for the probe.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads purpose and the no-write guarantee, then moves through the two modes, the overrides format, and the recommended workflow. Dense but zero filler sentences; every clause carries operational information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 4-param tool with a nested free-form object and an output schema present, the description supplies the missing pieces: overrides field naming/format, the meaning of the preview, and the explicit handoff to save_site. Return-value details are correctly delegated to the output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description does add value for `overrides` — naming concrete config field names (content_selector, allowed_path_prefix, disallowed_path_patterns, max_depth) and the duration string format ('1s') — which compensates for the nested object having no defined properties, but url/site_key/preview_url semantics are already fully covered by the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Draft a site entry and preview one page extracted with it') plus a negative scope clause ('Never writes the config'). This clearly separates it from the sibling save_site, which is named explicitly as the follow-up step.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit branching rule: with `url` it probes a new docs site the way `doc-scraper add` does; with only `site_key` it starts from an existing configured entry without probing. It also states the full iteration loop (judge preview → re-draft with overrides → pass entry to save_site), naming the alternative tool that commits.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_freshnessARead-onlyIdempotent
Return the most recent crawl summary for a site (last_crawl with its start and end times, total_pages and mode, plus age_seconds), output/state dir presence and any running job. Use this to decide whether to query the existing crawl or run crawl_site first.
| Name | Required | Description | Default |
|---|---|---|---|
| site_key | Yes | Site key from config (use list_sites to discover available keys) |
Output Schema
| Name | Required | Description |
|---|---|---|
| site_key | Yes | |
| last_crawl | No | Latest recorded crawl; absent if none |
| output_dir | Yes | |
| age_seconds | No | Seconds since last_crawl ended; present with last_crawl |
| running_job | No | |
| next_actions | Yes | |
| output_exists | Yes | |
| state_dir_exists | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover read-only, idempotent, non-destructive, closed-world behavior, so the description's added value is the decision-relevant content: age_seconds, dir presence, and any running job. It goes beyond restating annotations by describing what state the agent will see, though it doesn't say how freshness should be interpreted.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the return payload and closing with the decision use case. Dense but nothing is wasted; the parenthetical field list is long yet earned.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Complete for a single-parameter read tool with annotations and an output schema: the agent knows what it returns and why to call it. The only gap is guidance on interpreting age_seconds to judge staleness, which the agent must infer.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the single site_key parameter is fully documented in the schema, including a pointer to list_sites. The description adds no parameter-level detail, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('return the most recent crawl summary for a site') and enumerates the payload (last_crawl start/end, total_pages, mode, age_seconds, dir presence, running job). This is clearly distinguishable from crawl_site, get_job_status, and other siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly frames usage as a decision step: 'Use this to decide whether to query the existing crawl or run crawl_site first,' naming the alternative tool. It never states when not to use it or what threshold makes a crawl stale, but the routing intent is unambiguous.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_job_statusARead-onlyIdempotent
Get the status and page counts of a crawl job. Set wait_seconds to wait for the job to finish instead of polling: the call returns as soon as the job completes, fails or is cancelled, or when the wait runs out, with the status at that moment.
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes | The job ID returned by crawl_site | |
| wait_seconds | No | Seconds to wait for the job to finish before returning (default: 0, max: 60; larger values are capped) |
Output Schema
| Name | Required | Description |
|---|---|---|
| job_id | Yes | |
| status | Yes | pending, running, completed, failed or cancelled |
| refused | No | Set when a finished crawl was kept back instead of published: empty, temporary_failures or shrink |
| failures | No | Failed pages by class (temporary, gone, selector, policy, other), each with a count and one sample |
| site_key | Yes | |
| started_at | Yes | |
| incremental | Yes | |
| pages_saved | Yes | Pages written by this crawl |
| completed_at | No | |
| pages_failed | Yes | Pages that failed; noindex pages are neither saved nor failed |
| pages_queued | Yes | Pages waiting to be crawled; for a cancelled job, the pages left uncrawled |
| error_message | No | |
| pages_processed | Yes | |
| pages_unchanged | Yes | Incremental re-checks whose content had not changed; not counted in pages_processed |
| duration_seconds | No | Present once completed_at is |
| pages_carried_forward | Yes | Pages that failed temporarily and kept their copy from the previous crawl |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, non-destructive and closed-world, so the safety profile is covered. The description adds genuine behavioral detail beyond that: the call returns as soon as the job completes, fails, or is cancelled, or when the wait expires, and reports status at that moment. It doesn't discuss error behavior for unknown/expired job IDs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with what the tool does, then the non-obvious wait semantics. No padding or restatement of the title.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
A two-parameter read tool with an output schema, full schema coverage, and complete annotations; the description covers the one non-obvious behavior (blocking until terminal state or timeout). Nothing an agent needs to call it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description goes further by explaining what wait_seconds actually does semantically — it blocks and returns early on terminal states rather than simply sleeping. That meaning is not obvious from the schema's 'Seconds to wait' wording.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Get the status and page counts of a crawl job') and names the object precisely enough to separate it from siblings like cancel_crawl, get_freshness, and list_pages. An agent knows exactly what it gets back conceptually without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly contrasts the long-poll mode with the polling alternative ('Set wait_seconds to wait for the job to finish instead of polling'), which is real when-to-use guidance. It stops short of naming sibling tools for adjacent needs (e.g., cancel_crawl to abort, list_pages to inspect results).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_pageARead-onlyIdempotent
Fetch an HTML page live over the network and return its content as markdown. This is an on-demand fetch independent of any crawl: it does not read the stored crawl output, and it ignores the site's configured content_selector and scope. Large pages are truncated at max_bytes; follow next_offset to read the rest (each call fetches the page again).
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to fetch | |
| offset | No | Byte offset into the page content, for reading a truncated page in parts (default: 0) | |
| max_bytes | No | Maximum content bytes to return (default: 102400, max: 1048576; larger values are capped, 0 or less means the default) | |
| content_selector | No | CSS selector for main content (defaults to 'body') |
Output Schema
| Name | Required | Description |
|---|---|---|
| url | Yes | |
| title | Yes | |
| offset | Yes | |
| content | Yes | Page content converted to markdown |
| truncated | Yes | |
| next_offset | No | Pass as offset to read the next part; present only when truncated |
| total_length | Yes | Bytes of the whole converted page |
| fetch_time_ms | Yes | |
| content_length | Yes | Bytes of content in this response |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare it read-only and open-world, but the description adds materially: bypassing crawl storage and site config, truncation at max_bytes, and the non-obvious cost that each paginated call re-fetches the page. The only blemish is the reference to a 'next_offset' field that does not exist in the schema (the parameter is 'offset'), which could momentarily confuse an agent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight sentences, front-loaded with the action and the crawl-independence caveat before the truncation mechanics. Nearly every clause earns its place; the 'next_offset' wording is the one piece of noise that should have been 'offset'.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 4-parameter fetch tool with full schema coverage and an output schema, the description covers what an agent must know: it is live, it ignores stored/config state, and large pages are truncated with a re-fetch cost. Nothing essential is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3 and all four parameters already carry descriptions. The description adds real meaning for offset/max_bytes by tying them to truncation and the refetch requirement, but it does so under a mismatched name ('next_offset'), so the added value is partial rather than clean.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Fetch an HTML page live over the network and return its content as markdown') and immediately scopes it against the crawl-based siblings by saying it does not read stored crawl output. An agent can distinguish this from read_page/list_pages/crawl_site without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly frames the tool as an on-demand live fetch independent of any crawl and notes it ignores the site's content_selector and scope, which is exactly the condition that decides between this and the stored-crawl siblings. It stops short of naming the concrete alternative (read_page) or stating when a live fetch is wasteful, so it is clear context rather than full routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_pagesARead-onlyIdempotent
List crawled pages for a site, paginated and sorted by URL. Returns metadata only (URL, title, depth, crawled_at, content_length). Pass any URL returned here to read_page to get its stored markdown; get_page re-fetches a URL live rather than returning the crawled copy.
| Name | Required | Description | Default |
|---|---|---|---|
| offset | No | Pagination offset, 0-based (default: 0) | |
| site_key | Yes | Site key from config (use list_sites to discover available keys) | |
| max_results | No | Maximum pages to return (default: 100, max: 1000; larger values are capped, 0 or less means the default) |
Output Schema
| Name | Required | Description |
|---|---|---|
| pages | Yes | |
| total | Yes | |
| offset | Yes | |
| message | No | Present when the site has no crawl output yet |
| returned | Yes | |
| site_key | Yes | |
| max_results | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, closed-world, non-destructive, so the safety profile is covered. The description adds behavior beyond them: results are sorted by URL, only metadata is returned (URL, title, depth, crawled_at, content_length), and the stored copy vs live fetch distinction between read_page and get_page. It doesn't discuss pagination termination or total counts, but the added context is substantial.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three dense sentences with zero filler: purpose and return shape first, then the routing rule, then the sibling disambiguation. Every sentence earns its place and nothing is buried.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists so return values needn't be spelled out, yet the description still names the key metadata fields. For a simple, annotation-covered read tool with fully documented parameters, nothing an agent needs in order to call it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and the schema already documents offset, site_key provenance (list_sites), and max_results defaults/caps. The description only implies pagination ('paginated') and adds no parameter-level syntax or bounds beyond the schema, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb and resource ('List crawled pages for a site') plus scope details (paginated, sorted by URL) and return shape (metadata only). It also explicitly distinguishes itself from both read_page and get_page, so an agent can tell the three apart without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives explicit routing: 'Pass any URL returned here to read_page to get its stored markdown' and clarifies that 'get_page re-fetches a URL live rather than returning the crawled copy.' That's a when-to-use plus a named alternative with its selection condition.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_sitesARead-onlyIdempotent
List all configured sites available for crawling
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
| sites | Yes | |
| config_path | Yes | |
| total_sites | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, destructiveHint=false and a closed-world scope, so the safety and side-effect profile is fully covered elsewhere. The description adds no behavioral context beyond that (e.g., empty-list behavior, scope of 'configured'), so it neither contradicts nor enriches the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no filler; the verb and scope come first and nothing is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a nullary list tool, the annotations cover the safety profile and the output schema covers the return shape, so the description need not explain result structure. Nothing an agent needs to call this correctly is absent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so per the baseline there is nothing for the description to disambiguate. No parameter-related value is expected or missing.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource (list sites) plus scope ('all configured', 'available for crawling'), which implicitly separates it from mutation siblings like crawl_site. It does not explicitly distinguish itself from the other list-style sibling list_pages, so it stops short of a top score.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: 'available for crawling' hints this is a discovery step before crawl_site, but there is no explicit when-to-use, when-not-to-use, or mention of an alternative tool. Adequate but leaves the agent to infer the workflow.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
read_pageARead-onlyIdempotent
Return a page's markdown from the stored crawl output, without any network access. This is the counterpart to get_page: read_page serves the crawled copy that already had the site's content_selector applied, while get_page re-fetches the URL live. Use list_pages to discover URLs, then read_page to read them. Pass a search_docs result's anchor to read only that section. Large pages are truncated at max_bytes; follow next_offset to read the rest.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL of a crawled page, as reported by list_pages | |
| anchor | No | Heading anchor, as in a search_docs result; returns only that section, subsections included | |
| offset | No | Byte offset into the page content, for reading a truncated page in parts (default: 0) | |
| site_key | Yes | Site key from config (use list_sites to discover available keys) | |
| max_bytes | No | Maximum content bytes to return (default: 102400, max: 1048576; larger values are capped, 0 or less means the default) |
Output Schema
| Name | Required | Description |
|---|---|---|
| url | Yes | |
| depth | Yes | |
| title | Yes | |
| offset | Yes | |
| source | Yes | |
| content | Yes | |
| headings | Yes | |
| site_key | Yes | |
| truncated | Yes | More of the page, or of the anchor's section, follows |
| crawled_at | Yes | |
| next_offset | No | Pass as offset to read the next part; present only when truncated |
| section_end | No | Byte offset where the anchor's section ends; present when anchor was given |
| content_hash | Yes | |
| total_length | Yes | Bytes of the whole stored page |
| content_length | Yes | Bytes of content in this response |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, closed-world, non-destructive, so the safety profile is carried elsewhere. The description nevertheless adds real behavioral context: no network access, content_selector already applied, truncation at max_bytes, and continuation via the next_offset returned in the response. Minor gap: it does not mention that offset values are byte-based here, though the schema covers that.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Five short sentences, each carrying distinct information: what it returns, how it differs from get_page, the discovery workflow, the anchor shortcut, and pagination. No filler and the differentiation is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present, return values need not be explained, yet the description still covers truncation and continuation. For a 5-parameter read tool with full schema coverage and annotations, nothing an agent needs to call it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3, but the description adds meaning beyond the schema: it links anchor to search_docs results, explains offset as continuation of a truncated page, and explains max_bytes truncation behavior. It also references a next_offset return field, which pairs with the offset parameter rather than duplicating it.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource (return a page's markdown from stored crawl output) and immediately contrasts it with the sibling get_page, explaining the exact difference (crawled copy with content_selector applied vs live re-fetch). An agent can distinguish the two without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly routes the agent: list_pages to discover URLs, read_page to read them, get_page for live fetching, and passing a search_docs anchor to read only a section. This covers both when to use it and which sibling to pick instead.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
save_siteADestructiveIdempotent
Write a site entry to the config file and load it into the running server, so crawl_site, search_docs and read_page can use it at once. entry is a full entry by config field name, normally draft_site's entry with changes; fields left out are unset. Overwriting a configured site needs replace: true. Only the entry's lines change in the file. Refused while the site is crawling, and on servers started without -allow-config-writes.
| Name | Required | Description | Default |
|---|---|---|---|
| entry | Yes | The full site entry, by config field name | |
| replace | No | Overwrite an existing site (default: false; must be false for a new site) | |
| site_key | Yes | The key to save the entry under |
Output Schema
| Name | Required | Description |
|---|---|---|
| written | Yes | |
| replaced | Yes | |
| site_key | Yes | |
| next_actions | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes beyond the annotations by disclosing scoping ('only the entry's lines change in the file'), the partial-merge semantics ('fields left out are unset'), and the two refusal conditions (active crawl, config-writes flag). These details substantially reduce the risk of an incorrect destructive call, and the safety annotations already flag the mutation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight sentences, front-loaded with the action and immediate effect, then behavioral constraints, then failure conditions. No filler and no restated schema text.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be described. Given a destructive tool with nested parameters, the description supplies everything else an agent needs: the write's scope, merge semantics, replace requirement, and both refusal conditions.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and each parameter has a description, but the description adds meaning the schema cannot: the entry object is empty in the schema, yet the description explains it is keyed by config field name, is normally derived from draft_site, and leaves omitted fields unset. It could state the field names explicitly, which is why this is not a top score.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource (write a site entry to the config file and load it into the running server) and names what becomes possible afterward (crawl_site, search_docs, read_page). A reader can distinguish it from draft_site, which only drafts an entry.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says the entry normally comes from draft_site with modifications, when replace:true is required for overwrites, when it is refused (while crawling, or servers started without -allow-config-writes), and that omitted fields are unset. When-to-use, prerequisites, and failure conditions are all covered.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_docsARead-onlyIdempotent
Full-text search across all crawled documentation, ranked by relevance (BM25 with stemming), with zero network access. Results carry the page URL, its section heading path, and a snippet with match terms marked [like this]; follow up with read_page for the full page. Supports FTS5 syntax: quoted phrases, OR, and trailing * for prefix matching. Searches the stored index only; run crawl_site first for uncrawled sites.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Maximum results to return (default: 10, max: 50; larger values are capped, 0 or less means the default) | |
| query | Yes | Search terms. Plain words are ANDed; when too few results contain every word, the rest are filled with partial matches (match: partial) that contain only some words or built-in synonyms of them. "quoted phrases", OR, and prefix* work too and are never relaxed. | |
| site_key | No | Limit results to one site (use list_sites to discover keys). Omit to search all sites. |
Output Schema
| Name | Required | Description |
|---|---|---|
| count | Yes | |
| limit | Yes | |
| query | Yes | |
| results | Yes | |
| site_key | No | Present when the search was limited to one site |
| next_actions | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Adds context beyond annotations: BM25/stemming ranking, zero-network guarantee, stored-index-only scoping, and result shape (URL, heading path, [marked] snippet). Annotations already declare readOnly/idempotent/openWorld=false; the description reinforces and enriches rather than contradicting. Query relaxation behavior lives in the schema, so a small gap remains.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads purpose and ranking, then follows with return format, syntax, and prerequisites. Every sentence carries information; slightly dense but no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite an output schema existing, the description still explains result contents and how to continue to full pages. Combined with the schema, an agent has everything needed to call and follow up correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so parameters are already fully documented including ANDing, relaxation, and limit capping. The description's FTS5 syntax note largely duplicates what the schema states, so it adds little beyond the structured baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Full-text search across all crawled documentation') plus ranking mechanism (BM25 with stemming). Clearly distinguishes itself from read_page and crawl_site, which are named as follow-up/prerequisite tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly routes the agent: 'follow up with read_page for the full page' and 'run crawl_site first for uncrawled sites.' Both alternatives and the conditions selecting them are named, so the agent needn't infer anything.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
4 tool updates
v2.24.1- Changed
crawl_site1 field changed- added
Input schema / properties / allow_shrinkAdded value: +{ + "description": "Publish the crawl even if it lost many pages compared to the last one (use with incremental:true to publish a kept-back crawl)", + "type": "boolean" +}
- Changed
describe_server6 fields changed- added
Output schema / properties / recent_jobs / items / properties / failuresAdded value: +{ + "description": "Failed pages by class (temporary, gone, selector, policy, other), each with a count and one sample", + "items": { + "properties": { + "class": { + "type": "string" + }, + "count": { + "type": "integer" + }, + "sample_error": { + "type": "string" + }, + "sample_url": { + "type": "string" + } + }, + "required": [ + "class", + "count", + "sample_url", + "sample_error" + ], + "type": "object" + }, + "type": [ + "null", + "array" + ] +} - added
Output schema / properties / recent_jobs / items / properties / pages_carried_forwardAdded value: +{ + "description": "Pages that failed temporarily and kept their copy from the previous crawl", + "type": "integer" +} - added
Output schema / properties / recent_jobs / items / properties / pages_failedAdded value: +{ + "description": "Pages that failed; noindex pages are neither saved nor failed", + "type": "integer" +} - added
Output schema / properties / recent_jobs / items / properties / pages_savedAdded value: +{ + "description": "Pages written by this crawl", + "type": "integer" +} - added
Output schema / properties / recent_jobs / items / properties / refusedAdded value: +{ + "description": "Set when a finished crawl was kept back instead of published: empty, temporary_failures or shrink", + "type": "string" +} - changed
Output schema / properties / recent_jobs / items / requiredPrevious value: -[ - "job_id", - "site_key", - "status", - "started_at", - "pages_processed", - "pages_queued", - "pages_unchanged", - "incremental" -]New value: +[ + "job_id", + "site_key", + "status", + "started_at", + "pages_processed", + "pages_queued", + "pages_unchanged", + "pages_saved", + "pages_failed", + "pages_carried_forward", + "incremental" +]
- Changed
get_freshness6 fields changed- added
Output schema / properties / running_job / properties / failuresAdded value: +{ + "description": "Failed pages by class (temporary, gone, selector, policy, other), each with a count and one sample", + "items": { + "properties": { + "class": { + "type": "string" + }, + "count": { + "type": "integer" + }, + "sample_error": { + "type": "string" + }, + "sample_url": { + "type": "string" + } + }, + "required": [ + "class", + "count", + "sample_url", + "sample_error" + ], + "type": "object" + }, + "type": [ + "null", + "array" + ] +} - added
Output schema / properties / running_job / properties / pages_carried_forwardAdded value: +{ + "description": "Pages that failed temporarily and kept their copy from the previous crawl", + "type": "integer" +} - added
Output schema / properties / running_job / properties / pages_failedAdded value: +{ + "description": "Pages that failed; noindex pages are neither saved nor failed", + "type": "integer" +} - added
Output schema / properties / running_job / properties / pages_savedAdded value: +{ + "description": "Pages written by this crawl", + "type": "integer" +} - added
Output schema / properties / running_job / properties / refusedAdded value: +{ + "description": "Set when a finished crawl was kept back instead of published: empty, temporary_failures or shrink", + "type": "string" +} - changed
Output schema / properties / running_job / requiredPrevious value: -[ - "job_id", - "site_key", - "status", - "started_at", - "pages_processed", - "pages_queued", - "pages_unchanged", - "incremental" -]New value: +[ + "job_id", + "site_key", + "status", + "started_at", + "pages_processed", + "pages_queued", + "pages_unchanged", + "pages_saved", + "pages_failed", + "pages_carried_forward", + "incremental" +]
- Changed
get_job_status6 fields changed- added
Output schema / properties / failuresAdded value: +{ + "description": "Failed pages by class (temporary, gone, selector, policy, other), each with a count and one sample", + "items": { + "properties": { + "class": { + "type": "string" + }, + "count": { + "type": "integer" + }, + "sample_error": { + "type": "string" + }, + "sample_url": { + "type": "string" + } + }, + "required": [ + "class", + "count", + "sample_url", + "sample_error" + ], + "type": "object" + }, + "type": [ + "null", + "array" + ] +} - added
Output schema / properties / pages_carried_forwardAdded value: +{ + "description": "Pages that failed temporarily and kept their copy from the previous crawl", + "type": "integer" +} - added
Output schema / properties / pages_failedAdded value: +{ + "description": "Pages that failed; noindex pages are neither saved nor failed", + "type": "integer" +} - added
Output schema / properties / pages_savedAdded value: +{ + "description": "Pages written by this crawl", + "type": "integer" +} - added
Output schema / properties / refusedAdded value: +{ + "description": "Set when a finished crawl was kept back instead of published: empty, temporary_failures or shrink", + "type": "string" +} - changed
Output schema / requiredPrevious value: -[ - "job_id", - "site_key", - "status", - "started_at", - "pages_processed", - "pages_queued", - "pages_unchanged", - "incremental" -]New value: +[ + "job_id", + "site_key", + "status", + "started_at", + "pages_processed", + "pages_queued", + "pages_unchanged", + "pages_saved", + "pages_failed", + "pages_carried_forward", + "incremental" +]
3 tool updates
v2.24.0- Changed
describe_server2 fields changed- added
Output schema / properties / config_writesAdded value: +{ + "description": "enabled or disabled; save_site needs enabled (mcp-server -allow-config-writes)", + "type": "string" +} - changed
Output schema / requiredPrevious value: -[ - "server", - "sites", - "recent_jobs", - "total_sites", - "total_jobs", - "jobs_capped", - "next_actions" -]New value: +[ + "server", + "sites", + "recent_jobs", + "total_sites", + "total_jobs", + "jobs_capped", + "config_writes", + "next_actions" +]
- Added
draft_site - Added
save_site
9 tool updates
v2.21.0- Changed
crawl_site1 field changed- changed
Input schema / properties / site_key / descriptionPrevious value: -"Site key from config file (e.g., 'langchain_py', 'rust_docs')"New value: +"Site key from config (use list_sites to discover available keys)"
- Changed
describe_server1 field changed- added
Output schema / properties / recent_jobs / items / properties / pages_queued / descriptionAdded value: +"Pages waiting to be crawled; for a cancelled job, the pages left uncrawled"
- Changed
diff_crawl7 fields changed- changed
Input schema / properties / max_results / descriptionPrevious value: -"Maximum diff entries to return (default 100, max 1000)"New value: +"Maximum diff entries to return (default: 100, max: 1000; larger values are capped, 0 or less means the default)" - changed
Input schema / properties / max_results / typePrevious value: -"number"New value: +"integer" - changed
Input schema / properties / offset / typePrevious value: -"number"New value: +"integer" - changed
Input schema / properties / since / descriptionPrevious value: -"RFC3339 timestamp; the most recent crawl whose crawl_ended_at <= since, compared to the second, is the baseline (e.g. 2026-05-23T22:00:00Z)"New value: +"RFC3339 timestamp; the most recent crawl whose crawl_ended_at <= since, compared to the second, is the baseline (e.g. 2026-05-23T22:00:00Z). Omit to compare with the previous crawl." - changed
Input schema / requiredPrevious value: -[ - "site_key", - "since" -]New value: +[ + "site_key" +] - added
Output schema / properties / since / descriptionAdded value: +"Present when since was given" - changed
Output schema / requiredPrevious value: -[ - "site_key", - "since", - "added", - "removed", - "changed", - "unchanged_count", - "total", - "returned", - "offset", - "max_results" -]New value: +[ + "site_key", + "added", + "removed", + "changed", + "unchanged_count", + "total", + "returned", + "offset", + "max_results" +]
- Changed
get_freshness1 field changed- added
Output schema / properties / running_job / properties / pages_queued / descriptionAdded value: +"Pages waiting to be crawled; for a cancelled job, the pages left uncrawled"
- Changed
get_job_status2 fields changed- added
Input schema / properties / wait_secondsAdded value: +{ + "description": "Seconds to wait for the job to finish before returning (default: 0, max: 60; larger values are capped)", + "type": "integer" +} - added
Output schema / properties / pages_queued / descriptionAdded value: +"Pages waiting to be crawled; for a cancelled job, the pages left uncrawled"
- Changed
get_page8 fields changed- added
Input schema / properties / max_bytesAdded value: +{ + "description": "Maximum content bytes to return (default: 102400, max: 1048576; larger values are capped, 0 or less means the default)", + "type": "integer" +} - added
Input schema / properties / offsetAdded value: +{ + "description": "Byte offset into the page content, for reading a truncated page in parts (default: 0)", + "type": "integer" +} - added
Output schema / properties / content_length / descriptionAdded value: +"Bytes of content in this response" - added
Output schema / properties / next_offsetAdded value: +{ + "description": "Pass as offset to read the next part; present only when truncated", + "type": [ + "null", + "integer" + ] +} - added
Output schema / properties / offsetAdded value: +{ + "type": "integer" +} - added
Output schema / properties / total_lengthAdded value: +{ + "description": "Bytes of the whole converted page", + "type": "integer" +} - added
Output schema / properties / truncatedAdded value: +{ + "type": "boolean" +} - changed
Output schema / requiredPrevious value: -[ - "url", - "title", - "content", - "content_length", - "fetch_time_ms" -]New value: +[ + "url", + "title", + "content", + "content_length", + "total_length", + "offset", + "truncated", + "fetch_time_ms" +]
- Changed
list_pages3 fields changed- changed
Input schema / properties / max_results / descriptionPrevious value: -"Maximum pages to return (default: 100, max: 1000)"New value: +"Maximum pages to return (default: 100, max: 1000; larger values are capped, 0 or less means the default)" - changed
Input schema / properties / max_results / typePrevious value: -"number"New value: +"integer" - changed
Input schema / properties / offset / typePrevious value: -"number"New value: +"integer"
- Changed
read_page6 fields changed- added
Input schema / properties / anchorAdded value: +{ + "description": "Heading anchor, as in a search_docs result; returns only that section, subsections included", + "type": "string" +} - changed
Input schema / properties / max_bytes / descriptionPrevious value: -"Maximum content bytes to return (default: 102400, max: 1048576)"New value: +"Maximum content bytes to return (default: 102400, max: 1048576; larger values are capped, 0 or less means the default)" - changed
Input schema / properties / max_bytes / typePrevious value: -"number"New value: +"integer" - changed
Input schema / properties / offset / typePrevious value: -"number"New value: +"integer" - added
Output schema / properties / section_endAdded value: +{ + "description": "Byte offset where the anchor's section ends; present when anchor was given", + "type": [ + "null", + "integer" + ] +} - added
Output schema / properties / truncated / descriptionAdded value: +"More of the page, or of the anchor's section, follows"
- Changed
search_docs4 fields changed- changed
Input schema / properties / limit / descriptionPrevious value: -"Maximum results to return (default: 10, max: 50)"New value: +"Maximum results to return (default: 10, max: 50; larger values are capped, 0 or less means the default)" - changed
Input schema / properties / limit / typePrevious value: -"number"New value: +"integer" - added
Output schema / properties / limitAdded value: +{ + "type": "integer" +} - changed
Output schema / requiredPrevious value: -[ - "query", - "count", - "results", - "next_actions" -]New value: +[ + "query", + "count", + "limit", + "results", + "next_actions" +]
11 tool updates
v2.15.0- Changed
cancel_crawl1 field changed- changed
Output schema / (root)Previous value: -nullNew value: +{ + "properties": { + "cancelled": { + "description": "false when the job was already in a terminal state", + "type": "boolean" + }, + "job_id": { + "type": "string" + }, + "message": { + "type": "string" + }, + "site_key": { + "type": "string" + }, + "status": { + "type": "string" + } + }, + "required": [ + "job_id", + "cancelled", + "site_key", + "status", + "message" + ], + "type": "object" +}
- Changed
crawl_site1 field changed- changed
Output schema / (root)Previous value: -nullNew value: +{ + "properties": { + "incremental": { + "type": "boolean" + }, + "job_id": { + "type": "string" + }, + "message": { + "type": "string" + }, + "site_key": { + "type": "string" + }, + "status": { + "description": "started, or already_running when a crawl for the site is in progress", + "type": "string" + } + }, + "required": [ + "status", + "message", + "job_id", + "site_key", + "incremental" + ], + "type": "object" +}
- Changed
describe_server1 field changed- changed
Output schema / (root)Previous value: -nullNew value: +{ + "properties": { + "jobs_capped": { + "type": "boolean" + }, + "next_actions": { + "type": "string" + }, + "recent_jobs": { + "description": "Newest first, at most 20", + "items": { + "properties": { + "completed_at": { + "type": "string" + }, + "duration_seconds": { + "description": "Present once completed_at is", + "type": [ + "null", + "number" + ] + }, + "error_message": { + "type": "string" + }, + "incremental": { + "type": "boolean" + }, + "job_id": { + "type": "string" + }, + "pages_processed": { + "type": "integer" + }, + "pages_queued": { + "type": "integer" + }, + "pages_unchanged": { + "description": "Incremental re-checks whose content had not changed; not counted in pages_processed", + "type": "integer" + }, + "site_key": { + "type": "string" + }, + "started_at": { + "type": "string" + }, + "status": { + "description": "pending, running, completed, failed or cancelled", + "type": "string" + } + }, + "required": [ + "job_id", + "site_key", + "status", + "started_at", + "pages_processed", + "pages_queued", + "pages_unchanged", + "incremental" + ], + "type": "object" + }, + "type": [ + "null", + "array" + ] + }, + "server": { + "properties": { + "config_path": { + "type": "string" + }, + "name": { + "type": "string" + }, + "version": { + "type": "string" + } + }, + "required": [ + "name", + "version", + "config_path" + ], + "type": "object" + }, + "sites": { + "items": { + "properties": { + "domain": { + "type": "string" + }, + "key": { + "type": "string" + }, + "last_crawled": { + "type": "string" + }, + "status": { + "description": "\"running\" while a crawl is active; absent otherwise", + "type": "string" + } + }, + "required": [ + "key", + "domain" + ], + "type": "object" + }, + "type": [ + "null", + "array" + ] + }, + "total_jobs": { + "type": "integer" + }, + "total_sites": { + "type": "integer" + } + }, + "required": [ + "server", + "sites", + "recent_jobs", + "total_sites", + "total_jobs", + "jobs_capped", + "next_actions" + ], + "type": "object" +}
- Changed
diff_crawl1 field changed- changed
Output schema / (root)Previous value: -nullNew value: +{ + "properties": { + "added": { + "items": { + "properties": { + "content_hash": { + "type": "string" + }, + "depth": { + "type": "integer" + }, + "prior_hash": { + "description": "Baseline hash; changed entries only", + "type": "string" + }, + "title": { + "type": "string" + }, + "url": { + "type": "string" + } + }, + "required": [ + "url", + "title", + "content_hash", + "depth" + ], + "type": "object" + }, + "type": [ + "null", + "array" + ] + }, + "baseline_crawl": { + "properties": { + "crawl_ended_at": { + "type": "string" + }, + "crawl_id": { + "type": "integer" + }, + "crawl_started_at": { + "type": "string" + }, + "mode": { + "description": "full or incremental", + "type": "string" + }, + "total_pages": { + "type": "integer" + } + }, + "required": [ + "crawl_id", + "crawl_started_at", + "crawl_ended_at", + "total_pages", + "mode" + ], + "type": [ + "null", + "object" + ] + }, + "changed": { + "items": { + "properties": { + "content_hash": { + "type": "string" + }, + "depth": { + "type": "integer" + }, + "prior_hash": { + "description": "Baseline hash; changed entries only", + "type": "string" + }, + "title": { + "type": "string" + }, + "url": { + "type": "string" + } + }, + "required": [ + "url", + "title", + "content_hash", + "depth" + ], + "type": "object" + }, + "type": [ + "null", + "array" + ] + }, + "current_crawl": { + "properties": { + "crawl_ended_at": { + "type": "string" + }, + "crawl_id": { + "type": "integer" + }, + "crawl_started_at": { + "type": "string" + }, + "mode": { + "description": "full or incremental", + "type": "string" + }, + "total_pages": { + "type": "integer" + } + }, + "required": [ + "crawl_id", + "crawl_started_at", + "crawl_ended_at", + "total_pages", + "mode" + ], + "type": [ + "null", + "object" + ] + }, + "max_results": { + "type": "integer" + }, + "note": { + "description": "Why the diff is empty or partial, when it is", + "type": "string" + }, + "offset": { + "type": "integer" + }, + "removed": { + "items": { + "properties": { + "content_hash": { + "type": "string" + }, + "depth": { + "type": "integer" + }, + "prior_hash": { + "description": "Baseline hash; changed entries only", + "type": "string" + }, + "title": { + "type": "string" + }, + "url": { + "type": "string" + } + }, + "required": [ + "url", + "title", + "content_hash", + "depth" + ], + "type": "object" + }, + "type": [ + "null", + "array" + ] + }, + "returned": { + "type": "integer" + }, + "since": { + "type": "string" + }, + "site_key": { + "type": "string" + }, + "total": { + "type": "integer" + }, + "unchanged_count": { + "type": "integer" + } + }, + "required": [ + "site_key", + "since", + "added", + "removed", + "changed", + "unchanged_count", + "total", + "returned", + "offset", + "max_results" + ], + "type": "object" +}
- Changed
get_freshness1 field changed- changed
Output schema / (root)Previous value: -nullNew value: +{ + "properties": { + "age_seconds": { + "description": "Seconds since last_crawl ended; present with last_crawl", + "type": [ + "null", + "integer" + ] + }, + "last_crawl": { + "description": "Latest recorded crawl; absent if none", + "properties": { + "crawl_ended_at": { + "type": "string" + }, + "crawl_id": { + "type": "integer" + }, + "crawl_started_at": { + "type": "string" + }, + "mode": { + "description": "full or incremental", + "type": "string" + }, + "total_pages": { + "type": "integer" + } + }, + "required": [ + "crawl_id", + "crawl_started_at", + "crawl_ended_at", + "total_pages", + "mode" + ], + "type": [ + "null", + "object" + ] + }, + "next_actions": { + "type": "string" + }, + "output_dir": { + "type": "string" + }, + "output_exists": { + "type": "boolean" + }, + "running_job": { + "properties": { + "completed_at": { + "type": "string" + }, + "duration_seconds": { + "description": "Present once completed_at is", + "type": [ + "null", + "number" + ] + }, + "error_message": { + "type": "string" + }, + "incremental": { + "type": "boolean" + }, + "job_id": { + "type": "string" + }, + "pages_processed": { + "type": "integer" + }, + "pages_queued": { + "type": "integer" + }, + "pages_unchanged": { + "description": "Incremental re-checks whose content had not changed; not counted in pages_processed", + "type": "integer" + }, + "site_key": { + "type": "string" + }, + "started_at": { + "type": "string" + }, + "status": { + "description": "pending, running, completed, failed or cancelled", + "type": "string" + } + }, + "required": [ + "job_id", + "site_key", + "status", + "started_at", + "pages_processed", + "pages_queued", + "pages_unchanged", + "incremental" + ], + "type": [ + "null", + "object" + ] + }, + "site_key": { + "type": "string" + }, + "state_dir_exists": { + "type": "boolean" + } + }, + "required": [ + "site_key", + "output_dir", + "output_exists", + "state_dir_exists", + "next_actions" + ], + "type": "object" +}
- Changed
get_job_status1 field changed- changed
Output schema / (root)Previous value: -nullNew value: +{ + "properties": { + "completed_at": { + "type": "string" + }, + "duration_seconds": { + "description": "Present once completed_at is", + "type": [ + "null", + "number" + ] + }, + "error_message": { + "type": "string" + }, + "incremental": { + "type": "boolean" + }, + "job_id": { + "type": "string" + }, + "pages_processed": { + "type": "integer" + }, + "pages_queued": { + "type": "integer" + }, + "pages_unchanged": { + "description": "Incremental re-checks whose content had not changed; not counted in pages_processed", + "type": "integer" + }, + "site_key": { + "type": "string" + }, + "started_at": { + "type": "string" + }, + "status": { + "description": "pending, running, completed, failed or cancelled", + "type": "string" + } + }, + "required": [ + "job_id", + "site_key", + "status", + "started_at", + "pages_processed", + "pages_queued", + "pages_unchanged", + "incremental" + ], + "type": "object" +}
- Changed
get_page1 field changed- changed
Output schema / (root)Previous value: -nullNew value: +{ + "properties": { + "content": { + "description": "Page content converted to markdown", + "type": "string" + }, + "content_length": { + "type": "integer" + }, + "fetch_time_ms": { + "type": "integer" + }, + "title": { + "type": "string" + }, + "url": { + "type": "string" + } + }, + "required": [ + "url", + "title", + "content", + "content_length", + "fetch_time_ms" + ], + "type": "object" +}
- Changed
list_pages1 field changed- changed
Output schema / (root)Previous value: -nullNew value: +{ + "properties": { + "max_results": { + "type": "integer" + }, + "message": { + "description": "Present when the site has no crawl output yet", + "type": "string" + }, + "offset": { + "type": "integer" + }, + "pages": { + "items": { + "properties": { + "content_length": { + "type": "integer" + }, + "crawled_at": { + "type": "string" + }, + "depth": { + "type": "integer" + }, + "title": { + "type": "string" + }, + "url": { + "type": "string" + } + }, + "required": [ + "url", + "title", + "depth", + "crawled_at", + "content_length" + ], + "type": "object" + }, + "type": [ + "null", + "array" + ] + }, + "returned": { + "type": "integer" + }, + "site_key": { + "type": "string" + }, + "total": { + "type": "integer" + } + }, + "required": [ + "site_key", + "pages", + "total", + "offset", + "max_results", + "returned" + ], + "type": "object" +}
- Changed
list_sites1 field changed- changed
Output schema / (root)Previous value: -nullNew value: +{ + "properties": { + "config_path": { + "type": "string" + }, + "sites": { + "items": { + "properties": { + "domain": { + "type": "string" + }, + "key": { + "type": "string" + }, + "last_crawled": { + "type": "string" + }, + "max_depth": { + "type": "integer" + }, + "path_prefix": { + "type": "string" + }, + "start_urls_count": { + "type": "integer" + }, + "status": { + "description": "\"running\" while a crawl is active; absent otherwise", + "type": "string" + } + }, + "required": [ + "key", + "domain", + "path_prefix", + "start_urls_count", + "max_depth" + ], + "type": "object" + }, + "type": [ + "null", + "array" + ] + }, + "total_sites": { + "type": "integer" + } + }, + "required": [ + "sites", + "config_path", + "total_sites" + ], + "type": "object" +}
- Changed
read_page1 field changed- changed
Output schema / (root)Previous value: -nullNew value: +{ + "properties": { + "content": { + "type": "string" + }, + "content_hash": { + "type": "string" + }, + "content_length": { + "description": "Bytes of content in this response", + "type": "integer" + }, + "crawled_at": { + "type": "string" + }, + "depth": { + "type": "integer" + }, + "headings": { + "items": { + "type": "string" + }, + "type": [ + "null", + "array" + ] + }, + "next_offset": { + "description": "Pass as offset to read the next part; present only when truncated", + "type": [ + "null", + "integer" + ] + }, + "offset": { + "type": "integer" + }, + "site_key": { + "type": "string" + }, + "source": { + "type": "string" + }, + "title": { + "type": "string" + }, + "total_length": { + "description": "Bytes of the whole stored page", + "type": "integer" + }, + "truncated": { + "type": "boolean" + }, + "url": { + "type": "string" + } + }, + "required": [ + "site_key", + "url", + "title", + "content", + "content_length", + "total_length", + "offset", + "truncated", + "depth", + "crawled_at", + "content_hash", + "headings", + "source" + ], + "type": "object" +}
- Changed
search_docs2 fields changed- changed
Input schema / properties / query / descriptionPrevious value: -"Search terms. Plain words are ANDed; \"quoted phrases\", OR, and prefix* work too."New value: +"Search terms. Plain words are ANDed; when too few results contain every word, the rest are filled with partial matches (match: partial) that contain only some words or built-in synonyms of them. \"quoted phrases\", OR, and prefix* work too and are never relaxed." - changed
Output schema / (root)Previous value: -nullNew value: +{ + "properties": { + "count": { + "type": "integer" + }, + "next_actions": { + "type": "string" + }, + "query": { + "type": "string" + }, + "results": { + "items": { + "properties": { + "anchor": { + "type": "string" + }, + "heading_path": { + "type": "string" + }, + "match": { + "type": "string" + }, + "rank": { + "type": "number" + }, + "site_key": { + "type": "string" + }, + "snippet": { + "type": "string" + }, + "title": { + "type": "string" + }, + "url": { + "type": "string" + } + }, + "required": [ + "site_key", + "url", + "title", + "snippet", + "rank", + "match" + ], + "type": "object" + }, + "type": [ + "null", + "array" + ] + }, + "site_key": { + "description": "Present when the search was limited to one site", + "type": "string" + } + }, + "required": [ + "query", + "count", + "results", + "next_actions" + ], + "type": "object" +}
1 tool update
v2.12.0- Changed
diff_crawl1 field changed- changed
Input schema / properties / since / descriptionPrevious value: -"RFC3339 timestamp; the most recent crawl whose crawl_ended_at <= since is the baseline (e.g. 2026-05-23T22:00:00Z)"New value: +"RFC3339 timestamp; the most recent crawl whose crawl_ended_at <= since, compared to the second, is the baseline (e.g. 2026-05-23T22:00:00Z)"
2 tool updates
v2.10.1- Changed
describe_server2 fields changed- added
Input schema / propertiesAdded value: +{} - added
Input schema / requiredAdded value: +[]
- Changed
list_sites2 fields changed- added
Input schema / propertiesAdded value: +{} - added
Input schema / requiredAdded value: +[]
11 tool updates
- First observed
cancel_crawl - First observed
crawl_site - First observed
describe_server - First observed
diff_crawl - First observed
get_freshness - First observed
get_job_status - First observed
get_page - First observed
list_pages - First observed
list_sites - First observed
read_page - First observed
search_docs
TDQS
Scored across 13 tools
Most tools target clearly distinct operations (crawl lifecycle, page access, site config, search). A few boundaries blur: get_freshness vs describe_server vs get_job_status all surface crawl/job state, and get_page vs read_page are only separable by carefully reading the descriptions. The descriptions do explicitly call out these distinctions, which mitigates the overlap.
Every tool is snake_case verb_noun (crawl_site, get_job_status, read_page, save_site, search_docs), with only the conventional noun-only list_sites/list_pages as natural exceptions. No mixed conventions or camelCase.
13 tools sits comfortably in the well-scoped range, and each maps to a real workflow step (crawl, wait, inspect, diff, search, read, configure). Nothing appears redundant or padding.
The crawl lifecycle, search, page reads, and site onboarding are all covered, including diff and cancellation. The main gap is site lifecycle management: there is no delete_site/remove_site and no explicit update beyond overwriting via save_site with replace:true, so removing a configured site is a dead end.
Maintenance
Related MCP Connectors
Agent-native MCP server over the public saagarpatel.dev corpus. Read-only, stateless.
Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
Document-to-Markdown MCP server — convert PDF, Office and HTML into LLM-ready Markdown.
One MCP server for all web data. Scrape, search, crawl, and AI research in one install.
Related MCP Servers
- AlicenseAqualityCmaintenanceLocal-first MCP server providing semantic search over library docs, fully offline. Single Go binary speaks MCP over stdio against a vector index pinned to the binary version. Like Context7 with the internet turned off. Apache 2.0. Linux + macOS, also available as a container image.24Apache 2.0
- AlicenseAqualityBmaintenanceGo MCP server for multi-format document access — PDF, TXT, MD, DOCX, CSV, images. 12 tools including OCR, search, table extraction, and URL fetch. Single binary, no runtime.1311MIT
- FlicenseNot gradedqualityDmaintenanceDocumentation crawler MCP server that crawls and indexes documentation sites so that any MCP-compatible AI can search, read, and expand on the content.1-
- AlicenseNot gradedqualityCmaintenanceA documentation MCP server that crawls websites and Git repositories, stores them as Markdown, and provides tools to search and retrieve documentation for local LLMs and AI agents.Apache 2.0