Skip to main content
Glama
KnockOutEZ

wigolo

by KnockOutEZ
OverviewSchemaSourceRelated ServersDiscussionsAdmin
TypeScript
Local

crawl

Read-onlyIdempotent

Extract and cache content from multiple pages starting at a seed URL for indexing docs, wikis, and multi-page references.

Instructions

Crawl a site from a seed URL and return content from many pages. Use for indexing docs, wikis, multi-page references. Built for offline reuse: every page lands in the local cache.

Key parameters:

  • strategy: "bfs" (default) | "dfs" | "sitemap" (fastest for doc sites) | "map" (URL-only discovery).

  • max_depth: link-following depth (default 2).

  • max_pages: page cap (default 20).

  • include_patterns / exclude_patterns: regex filters on URLs.

  • max_tokens_out / include_full_markdown / citation_format: budget + shape controls.

Returns pages[] with title, evidence, depth, and content_completeness (per page — full/partial/shell, so you can skip un-rendered shell pages). Content is deduplicated across pages (anchor-fragment aware). All pages are cached for later cache / find_similar queries.

Input Schema

NameRequiredDescriptionDefault
urlYesSeed URL to start crawling from
strategyNoCrawl strategy: bfs (breadth-first), dfs (depth-first), sitemap (use sitemap.xml), map (URL-only discovery — returns list of URLs without content, faster than full crawl)
use_authNoUse stored auth credentials (default: false)
max_depthNoMaximum link depth from seed (default: 2)
max_pagesNoMaximum pages to crawl (default: 20)
extract_linksNoReturn link graph between pages (default: false)
max_tokens_outNoToken-budget cap on total output. Uses cl100k-base BPE; non-OpenAI tokenizer counts may drift ~5-15%. When both max_tokens_out and max_chars are set, max_tokens_out wins.
citation_formatNoCitation rendering style. 'numbered' (default) inline [N] markers; 'json' returns a citations[] array; 'anthropic_tags' wraps sources in <source id='...'> tags.
max_total_charsNoMax total chars across all pages (default: 100000)
exclude_patternsNoURL regex blacklist — skip matching URLs
include_patternsNoURL regex whitelist — only crawl matching URLs
include_full_markdownNoInclude full markdown body in the response. Default false on multi-result tools (returns evidence excerpts only); set true to restore.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Addedv0.3.0
  2. Removedv0.2.0
  3. First observed

TDQS

A4.1/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well beyond the annotations: discloses that every page is written to the local cache and reusable via cache/find_similar, that content is deduplicated anchor-fragment aware, and that per-page content_completeness flags let the agent skip un-rendered shells. These are meaningful side effects and data-quality traits not captured by readOnly/openWorld hints.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the purpose in one sentence, then uses a compact parameter list and a return-shape paragraph. Some bullets restate defaults already in the schema, which is minor waste but not damaging.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 12-parameter open-world tool with no output schema, it explains the return payload (pages[] with title, evidence, depth, content_completeness) and key controls. It omits mention of use_auth, extract_links, and max_total_chars, but those are documented in the schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 12 parameters and enum meanings. The description adds modest value (marking sitemap as fastest for doc sites, noting map is URL-only discovery, restating defaults) but largely mirrors what is already structured.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource (crawl a site from a seed URL, return content from many pages) and scopes it with concrete use cases like indexing docs, wikis, and multi-page references. The 'many pages' framing implicitly separates it from the single-page fetch sibling, but it never names fetch/search explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Use for indexing docs, wikis, multi-page references' gives clear positive context for selection. It stops short of explicit exclusions or naming alternatives like fetch for single pages or search for targeted lookups.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools