Skip to main content
Glama
Sign In
lucasmartins-ai

LookaCrawler

by lucasmartins-ai
OverviewSchemaSourceRelated ServersDiscussionsAdmin
TypeScript

crawl_website

Recursively crawl an entire website or documentation section with depth limits, URL pattern filtering, and token accounting.

Instructions

Recursively crawl an entire website or documentation section with depth limits, URL pattern filtering, and token accounting.

Input Schema

NameRequiredDescriptionDefault
modeNoCrawl mode ('fast' native fetch or 'deep' Playwright browser).fast
max_depthNoMaximum link traversal depth (default: 2).
max_pagesNoMaximum total pages to crawl (default: 5, max: 20).
start_urlYesRoot URL to start the recursive crawl from.
image_modeNoFormatting of images.ignore
link_formatNoFormatting of extracted hyperlinks.inline
exclude_patternsNoSubstring patterns (`*` = wildcard) for URLs to ignore (e.g. ['/login', '/tags/']). Max 10.
include_patternsNoSubstring patterns (`*` = wildcard) that candidate URLs must match (e.g. ['/docs/']). Max 10.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed6 schema fields changedv1.1.2
    • changedInput schema / properties / exclude_patterns / description
      Previous value: -"Regex patterns for URLs to ignore (e.g. ['/login', '/tags/'])."New value: +"Substring patterns (`*` = wildcard) for URLs to ignore (e.g. ['/login', '/tags/']). Max 10."
    • addedInput schema / properties / exclude_patterns / items / maxLength
      Added value: +200
    • addedInput schema / properties / exclude_patterns / maxItems
      Added value: +10
    • changedInput schema / properties / include_patterns / description
      Previous value: -"Regex patterns that candidate URLs must match (e.g. ['/docs/'])."New value: +"Substring patterns (`*` = wildcard) that candidate URLs must match (e.g. ['/docs/']). Max 10."
    • addedInput schema / properties / include_patterns / items / maxLength
      Added value: +200
    • addedInput schema / properties / include_patterns / maxItems
      Added value: +10
  2. Addedv1.1.1

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral burden. It hints at depth control and 'token accounting' but says nothing about read-only safety, robots.txt/rate-limit behavior, timeouts, or whether the crawl is idempotent, which are the traits an agent needs when firing a recursive multi-page fetch.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single dense sentence with the core action and its three control mechanisms front-loaded, no filler or repetition. It is efficient, though arguably compressed enough that it omits needed detail rather than being maximally informative per word.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an 8-parameter recursive crawler with no annotations and no output schema, the description is thin: it does not explain what the crawl returns, how pagination/max_pages interacts with max_depth, or how token accounting surfaces. An agent has enough to start but not enough to predict behavior or cost.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so every parameter including mode, max_depth, max_pages, and include/exclude patterns is already documented in the schema. The description's mention of 'depth limits' and 'URL pattern filtering' maps to those params but adds no syntax or interaction detail beyond them, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Recursively crawl an entire website or documentation section') and names the mechanisms (depth limits, URL filtering). It does not differentiate itself from siblings like map_website or extract_web_content, which the agent must distinguish without help.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Scope is implied by 'entire website or documentation section', so an agent can infer this is for broad multi-page harvesting. However, there is no explicit when-to-use or when-not-to-use guidance and no mention of the obvious alternatives (map_website for URL discovery, extract_web_content for a single page).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools