Skip to main content
Glama
omgwtfwow

MCP Server for Crawl4AI

by omgwtfwow

MCP Server for Crawl4AI

Note: Tested with Crawl4AI version 0.7.4

npm version License: MIT Node.js CI coverage

TypeScript implementation of an MCP server for Crawl4AI. Provides tools for web crawling, content extraction, and browser automation.

Table of Contents

Related MCP server: mult-fetch-mcp-server

Prerequisites

  • Node.js 18+ and npm

  • A running Crawl4AI server

Quick Start

1. Start the Crawl4AI server (for example, local docker)

docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:0.7.4

2. Add to your MCP client

This MCP server works with any MCP-compatible client (Claude Desktop, Claude Code, Cursor, LMStudio, etc.).

{
  "mcpServers": {
    "crawl4ai": {
      "command": "npx",
      "args": ["mcp-crawl4ai-ts"],
      "env": {
        "CRAWL4AI_BASE_URL": "http://localhost:11235"
      }
    }
  }
}

Using local installation

{
  "mcpServers": {
    "crawl4ai": {
      "command": "node",
      "args": ["/path/to/mcp-crawl4ai-ts/dist/index.js"],
      "env": {
        "CRAWL4AI_BASE_URL": "http://localhost:11235"
      }
    }
  }
}

With all optional variables

{
  "mcpServers": {
    "crawl4ai": {
      "command": "npx",
      "args": ["mcp-crawl4ai-ts"],
      "env": {
        "CRAWL4AI_BASE_URL": "http://localhost:11235",
        "CRAWL4AI_API_KEY": "your-api-key",
        "SERVER_NAME": "custom-name",
        "SERVER_VERSION": "1.0.0"
      }
    }
  }
}

Configuration

Environment Variables

# Required
CRAWL4AI_BASE_URL=http://localhost:11235

# Optional - Server Configuration
CRAWL4AI_API_KEY=          # If your server requires auth
SERVER_NAME=crawl4ai-mcp   # Custom name for the MCP server
SERVER_VERSION=1.0.0       # Custom version

Client-Specific Instructions

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json

Claude Code

claude mcp add crawl4ai -e CRAWL4AI_BASE_URL=http://localhost:11235 -- npx mcp-crawl4ai-ts

Other MCP Clients

Consult your client's documentation for MCP server configuration. The key details:

  • Command: npx mcp-crawl4ai-ts or node /path/to/dist/index.js

  • Required env: CRAWL4AI_BASE_URL

  • Optional env: CRAWL4AI_API_KEY, SERVER_NAME, SERVER_VERSION

Available Tools

1. get_markdown - Extract content as markdown with filtering

{ 
  url: string,                              // Required: URL to extract markdown from
  filter?: 'raw'|'fit'|'bm25'|'llm',       // Filter type (default: 'fit')
  query?: string,                           // Query for bm25/llm filters
  cache?: string                            // Cache-bust parameter (default: '0')
}

Extracts content as markdown with various filtering options. Use 'bm25' or 'llm' filters with a query for specific content extraction.

2. capture_screenshot - Capture webpage screenshot

{ 
  url: string,                   // Required: URL to capture
  screenshot_wait_for?: number   // Seconds to wait before screenshot (default: 2)
}

Returns base64-encoded PNG. Note: This is stateless - for screenshots after JS execution, use crawl with screenshot: true.

3. generate_pdf - Convert webpage to PDF

{ 
  url: string  // Required: URL to convert to PDF
}

Returns base64-encoded PDF. Stateless tool - for PDFs after JS execution, use crawl with pdf: true.

4. execute_js - Execute JavaScript and get return values

{ 
  url: string,                    // Required: URL to load
  scripts: string | string[]      // Required: JavaScript to execute
}

Executes JavaScript and returns results. Each script can use 'return' to get values back. Stateless - for persistent JS execution use crawl with js_code.

5. batch_crawl - Crawl multiple URLs concurrently

{ 
  urls: string[],           // Required: List of URLs to crawl
  max_concurrent?: number,  // Parallel request limit (default: 5)
  remove_images?: boolean,  // Remove images from output (default: false)
  bypass_cache?: boolean,   // Bypass cache for all URLs (default: false)
  configs?: Array<{         // Optional: Per-URL configurations (v3.0.0+)
    url: string,
    [key: string]: any      // Any crawl parameters for this specific URL
  }>
}

Efficiently crawls multiple URLs in parallel. Each URL gets a fresh browser instance. With configs array, you can specify different parameters for each URL.

6. smart_crawl - Auto-detect and handle different content types

{ 
  url: string,            // Required: URL to crawl
  max_depth?: number,     // Maximum depth for recursive crawling (default: 2)
  follow_links?: boolean, // Follow links in content (default: true)
  bypass_cache?: boolean  // Bypass cache (default: false)
}

Intelligently detects content type (HTML/sitemap/RSS) and processes accordingly.

7. get_html - Get sanitized HTML for analysis

{ 
  url: string  // Required: URL to extract HTML from
}

Returns preprocessed HTML optimized for structure analysis. Use for building schemas or analyzing patterns.

8. extract_links - Extract and categorize page links

{ 
  url: string,          // Required: URL to extract links from
  categorize?: boolean  // Group by type (default: true)
}

Extracts all links and groups them by type: internal, external, social media, documents, images.

9. crawl_recursive - Deep crawl website following links

{ 
  url: string,              // Required: Starting URL
  max_depth?: number,       // Maximum depth to crawl (default: 3)
  max_pages?: number,       // Maximum pages to crawl (default: 50)
  include_pattern?: string, // Regex pattern for URLs to include
  exclude_pattern?: string  // Regex pattern for URLs to exclude
}

Crawls a website following internal links up to specified depth. Returns content from all discovered pages.

10. parse_sitemap - Extract URLs from XML sitemaps

{ 
  url: string,              // Required: Sitemap URL (e.g., /sitemap.xml)
  filter_pattern?: string   // Optional: Regex pattern to filter URLs
}

Extracts all URLs from XML sitemaps. Supports regex filtering for specific URL patterns.

11. crawl - Advanced web crawling with full configuration

{
  url: string,                              // URL to crawl
  // Browser Configuration
  browser_type?: 'chromium'|'firefox'|'webkit'|'undetected',  // Browser engine (undetected = stealth mode)
  viewport_width?: number,                  // Browser width (default: 1080)
  viewport_height?: number,                 // Browser height (default: 600)
  user_agent?: string,                      // Custom user agent
  proxy_server?: string | {                 // Proxy URL (string or object format)
    server: string,
    username?: string,
    password?: string
  },
  proxy_username?: string,                  // Proxy auth (if using string format)
  proxy_password?: string,                  // Proxy password (if using string format)
  cookies?: Array<{name, value, domain}>,   // Pre-set cookies
  headers?: Record<string,string>,          // Custom headers
  
  // Crawler Configuration
  word_count_threshold?: number,            // Min words per block (default: 200)
  excluded_tags?: string[],                 // HTML tags to exclude
  remove_overlay_elements?: boolean,        // Remove popups/modals
  js_code?: string | string[],              // JavaScript to execute
  wait_for?: string,                        // Wait condition (selector or JS)
  wait_for_timeout?: number,                // Wait timeout (default: 30000)
  delay_before_scroll?: number,             // Pre-scroll delay
  scroll_delay?: number,                    // Between-scroll delay
  process_iframes?: boolean,                // Include iframe content
  exclude_external_links?: boolean,         // Remove external links
  screenshot?: boolean,                     // Capture screenshot
  pdf?: boolean,                           // Generate PDF
  session_id?: string,                      // Reuse browser session (only works with crawl tool)
  cache_mode?: 'ENABLED'|'BYPASS'|'DISABLED',  // Cache control
  
  // New in v3.0.0 (Crawl4AI 0.7.3/0.7.4)
  css_selector?: string,                    // CSS selector to filter content
  delay_before_return_html?: number,        // Delay in seconds before returning HTML
  include_links?: boolean,                  // Include extracted links in response
  resolve_absolute_urls?: boolean,          // Convert relative URLs to absolute
  
  // LLM Extraction (REST API only supports 'llm' type)
  extraction_type?: 'llm',                  // Only 'llm' extraction is supported via REST API
  extraction_schema?: object,               // Schema for structured extraction
  extraction_instruction?: string,          // Natural language extraction prompt
  extraction_strategy?: {                   // Advanced extraction configuration
    provider?: string,
    api_key?: string,
    model?: string,
    [key: string]: any
  },
  table_extraction_strategy?: {             // Table extraction configuration
    enable_chunking?: boolean,
    thresholds?: object,
    [key: string]: any
  },
  markdown_generator_options?: {            // Markdown generation options
    include_links?: boolean,
    preserve_formatting?: boolean,
    [key: string]: any
  },
  
  timeout?: number,                         // Overall timeout (default: 60000)
  verbose?: boolean                         // Detailed logging
}

12. manage_session - Unified session management

{ 
  action: 'create' | 'clear' | 'list',    // Required: Action to perform
  session_id?: string,                    // For 'create' and 'clear' actions
  initial_url?: string,                   // For 'create' action: URL to load
  browser_type?: 'chromium' | 'firefox' | 'webkit' | 'undetected'  // For 'create' action
}

Unified tool for managing browser sessions. Supports three actions:

  • create: Start a persistent browser session

  • clear: Remove a session from local tracking

  • list: Show all active sessions

Examples:

// Create a new session
{ action: 'create', session_id: 'my-session', initial_url: 'https://example.com' }

// Clear a session
{ action: 'clear', session_id: 'my-session' }

// List all sessions
{ action: 'list' }

13. extract_with_llm - Extract structured data using AI

{ 
  url: string,          // URL to extract data from
  query: string         // Natural language extraction instructions
}

Uses AI to extract structured data from webpages. Returns results immediately without any polling or job management. This is the recommended way to extract specific information since CSS/XPath extraction is not supported via the REST API.

Advanced Configuration

For detailed information about all available configuration options, extraction strategies, and advanced features, please refer to the official Crawl4AI documentation:

Changelog

See CHANGELOG.md for detailed version history and recent updates.

Development

Setup

# 1. Start the Crawl4AI server
docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest

# 2. Install MCP server
git clone https://github.com/omgwtfwow/mcp-crawl4ai-ts.git
cd mcp-crawl4ai-ts
npm install
cp .env.example .env

# 3. Development commands
npm run dev    # Development mode
npm test       # Run tests
npm run lint   # Check code quality
npm run build  # Production build

# 4. Add to your MCP client (See "Using local installation")

Running Integration Tests

Integration tests require a running Crawl4AI server. Configure your environment:

# Required for integration tests
export CRAWL4AI_BASE_URL=http://localhost:11235
export CRAWL4AI_API_KEY=your-api-key  # If authentication is required

# Optional: For LLM extraction tests
export LLM_PROVIDER=openai/gpt-4o-mini
export LLM_API_TOKEN=your-llm-api-key
export LLM_BASE_URL=https://api.openai.com/v1  # If using custom endpoint

# Run integration tests (ALWAYS use the npm script; don't call `jest` directly)
npm run test:integration

# Run a single integration test file
npm run test:integration -- src/__tests__/integration/extract-links.integration.test.ts

> IMPORTANT: Do NOT run `npx jest` directly for integration tests. The npm script injects `NODE_OPTIONS=--experimental-vm-modules` which is required for ESM + ts-jest. Running Jest directly will produce `SyntaxError: Cannot use import statement outside a module` and hang.

Integration tests cover:

  • Dynamic content and JavaScript execution

  • Session management and cookies

  • Content extraction (LLM-based only)

  • Media handling (screenshots, PDFs)

  • Performance and caching

  • Content filtering

  • Bot detection avoidance

  • Error handling

Integration Test Checklist

  1. Docker container healthy:

docker ps --filter name=crawl4ai --format '{{.Names}} {{.Status}}'
curl -sf http://localhost:11235/health || echo "Health check failed"
  1. Env vars loaded (either exported or in .env): CRAWL4AI_BASE_URL (required), optional: CRAWL4AI_API_KEY, LLM_PROVIDER, LLM_API_TOKEN, LLM_BASE_URL.

  2. Use npm run test:integration (never raw jest).

  3. To target one file add it after -- (see example above).

  4. Expect total runtime ~2–3 minutes; longer or immediate hang usually means missing NODE_OPTIONS or wrong Jest version.

Troubleshooting

Symptom

Likely Cause

Fix

SyntaxError: Cannot use import statement outside a module

Ran jest directly without script flags

Re-run with npm run test:integration

Hangs on first test (RUNS ...)

Missing experimental VM modules flag

Use npm script / ensure NODE_OPTIONS=--experimental-vm-modules

Network timeouts

Crawl4AI container not healthy / DNS blocked

Restart container: docker restart <name>

LLM tests skipped

Missing LLM_PROVIDER or LLM_API_TOKEN

Export required LLM vars

New Jest major upgrade breaks tests

Version mismatch with ts-jest

Keep Jest 29.x unless ts-jest upgraded accordingly

Version Compatibility Note

Current stack: jest@29.x + ts-jest@29.x + ESM ("type": "module"). Updating Jest to 30+ requires upgrading ts-jest and revisiting jest.config.cjs. Keep versions aligned to avoid parse errors.

License

MIT

Available Tools

13 tools
batch_crawlA

[STATELESS] Crawl multiple URLs concurrently for efficiency. Use when: processing URL lists, comparing multiple pages, or bulk data extraction. Faster than sequential crawling. Max 5 concurrent by default. Each URL gets a fresh browser. Cannot maintain state between URLs. For persistent operations use create_session + crawl.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlsYesList of URLs to crawl
max_concurrentNoParallel request limit. Higher = faster but more resource intensive. Adjust based on server capacity and rate limits
remove_imagesNoRemove images from output by excluding img, picture, and svg tags
bypass_cacheNoBypass cache for all URLs

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden and does well by disclosing key behavioral traits: stateless operation, fresh browser per URL, concurrency limits (max 5), and performance characteristics ('faster than sequential crawling'). It doesn't mention error handling or output format, keeping it from a perfect score.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is perfectly structured and concise - front-loaded with the core purpose, followed by usage guidelines and behavioral constraints. Every sentence adds value with no wasted words, and it's appropriately sized for the tool's complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a stateless crawling tool with no annotations and no output schema, the description provides excellent context about when to use it, behavioral constraints, and alternatives. It doesn't describe the return format or error handling, which would be helpful given the lack of output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description doesn't add any parameter-specific information beyond what's in the schema, so it meets the baseline but doesn't provide extra value.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose with specific verbs ('crawl multiple URLs concurrently') and distinguishes it from siblings by mentioning it's for efficiency and bulk operations. It explicitly differentiates from sequential crawling and session-based approaches.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit usage guidance with 'Use when:' listing three specific scenarios (processing URL lists, comparing multiple pages, bulk data extraction). It also gives clear alternatives (sequential crawling vs. this tool, and create_session + crawl for persistent operations).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

capture_screenshotA

[STATELESS] Capture webpage screenshot. Returns base64-encoded PNG data. Creates new browser each time. Optionally saves screenshot to local directory. IMPORTANT: Chained calls (execute_js then capture_screenshot) will NOT work - the screenshot won't see JS changes! For JS changes + screenshot use create_session + crawl(session_id, js_code, screenshot:true) in ONE call.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe URL to capture
screenshot_wait_forNoSeconds to wait before taking screenshot (allows page loading/animations)
save_to_directoryNoDirectory path to save screenshot (e.g., ~/Desktop, /tmp). Do NOT include filename - it will be auto-generated. Large screenshots (>800KB) won't be returned inline when saved.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively communicates several key behaviors: stateless operation ('Creates new browser each time'), return format ('Returns base64-encoded PNG data'), performance consideration ('Large screenshots (>800KB) won't be returned inline when saved'), and a critical limitation about JavaScript changes. The only gap is lack of information about error conditions or timeout behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is appropriately sized and front-loaded with the core functionality. The first sentence establishes the main purpose, followed by important behavioral details. The warning about chained calls is crucial but could be slightly more concise. Overall, most sentences earn their place by conveying essential information that isn't in the structured fields.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity of a screenshot tool with browser isolation and JavaScript limitations, and with no annotations or output schema, the description does a good job covering key aspects: stateless operation, return format, sibling tool relationships, and important constraints. The main gap is the lack of information about what happens on failure (timeouts, invalid URLs, etc.), which prevents a perfect score.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all three parameters thoroughly. The description adds minimal parameter-specific information beyond what's in the schema - it mentions the save_to_directory behavior with large screenshots, but doesn't provide additional context about url validation or screenshot_wait_for usage. This meets the baseline expectation when schema coverage is complete.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the specific action ('capture webpage screenshot'), the resource ('webpage'), and distinguishes it from siblings by explicitly mentioning what it doesn't do (chained calls with execute_js) and pointing to the alternative (create_session + crawl). The verb+resource combination is precise and unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit guidance on when NOT to use this tool ('Chained calls... will NOT work') and specifies the alternative approach ('use create_session + crawl... in ONE call'). It also clarifies the stateless nature upfront, which helps set expectations about browser isolation.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawlA

[SUPPORTS SESSIONS] THE ONLY TOOL WITH BROWSER PERSISTENCE

RECOMMENDED PATTERNS: • Inspect first workflow:

  1. get_html(url) → find selectors & verify elements exist

  2. create_session() → "session-123"

  3. crawl({url, session_id: "session-123", js_code: ["action 1"]})

  4. crawl({url: "/page2", session_id: "session-123", js_code: ["action 2"]})

• Multi-step with state:

  1. create_session() → "session-123"

  2. crawl({url, session_id: "session-123"}) → inspect current state

  3. crawl({url, session_id: "session-123", js_code: ["verified actions"]})

WITH session_id: Maintains browser state (cookies, localStorage, page) across calls WITHOUT session_id: Creates fresh browser each time (like other tools)

WHEN TO USE SESSIONS vs STATELESS: • Need state between calls? → create_session + crawl • Just extracting data? → Use stateless tools • Filling forms? → Inspect first, then use sessions • Taking screenshot after JS? → Must use crawl with session • Unsure if elements exist? → Always use get_html first

CRITICAL FOR js_code: RECOMMENDED: Always use screenshot: true when running js_code This avoids server serialization errors and gives visual confirmation

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe URL to crawl
session_idNoENABLES PERSISTENCE: Use SAME ID across all crawl calls to maintain browser state. • First call with ID: Creates persistent browser • Subsequent calls with SAME ID: Reuses browser with all state intact • Different/no ID: Fresh browser (stateless) WARNING: ONLY works with crawl tool - other tools ignore this parameter
browser_typeNoBrowser engine for crawling. Chromium offers best compatibility, Firefox for specific use cases, WebKit for Safari-like behaviorchromium
viewport_widthNoBrowser window width in pixels. Affects responsive layouts and content visibility
viewport_heightNoBrowser window height in pixels. Impacts content loading and screenshot dimensions
user_agentNoCustom browser identity. Use for: mobile sites (include "Mobile"), avoiding bot detection, or specific browser requirements. Example: "Mozilla/5.0 (iPhone...)"
proxy_serverNoProxy server URL (e.g., "http://proxy.example.com:8080")
proxy_usernameNoProxy authentication username
proxy_passwordNoProxy authentication password
cookiesNoPre-set cookies for authentication or personalization
headersNoCustom HTTP headers for API keys, auth tokens, or specific server requirements
word_count_thresholdNoMin words per text block. Filters out menus, footers, and short snippets. Lower = more content but more noise. Higher = only substantial paragraphs
excluded_tagsNoHTML tags to remove completely. Common: ["nav", "footer", "aside", "script", "style"]. Cleans up content before extraction
remove_overlay_elementsNoAutomatically remove popups, modals, and overlays that obscure content
js_codeNoJavaScript to execute. Each string runs separately. Use return to get values. IMPORTANT: Always verify elements exist before acting on them! Use get_html first to find correct selectors, then: GOOD: ["if (document.querySelector('input[name=\"email\"]')) { ... }"] BAD: ["document.querySelector('input[name=\"email\"]').value = '...'"] USAGE PATTERNS: 1. WITH screenshot/pdf: {js_code: [...], screenshot: true} ✓ 2. MULTI-STEP: First {js_code: [...], session_id: "x"}, then {js_only: true, session_id: "x"} 3. AVOID: {js_code: [...], js_only: true} on first call ✗ SELECTOR TIPS: Use get_html first to find: • name="..." (best for forms) • id="..." (if unique) • class="..." (careful, may repeat) FORM EXAMPLE WITH VERIFICATION: [ "const emailInput = document.querySelector('input[name=\"email\"]');", "if (emailInput) emailInput.value = 'user@example.com';", "const submitBtn = document.querySelector('button[type=\"submit\"]');", "if (submitBtn) submitBtn.click();" ]
js_onlyNoFOR SUBSEQUENT CALLS ONLY: Reuse existing session without navigation First call: Use js_code WITHOUT js_only (or with screenshot/pdf) Later calls: Use js_only=true to run more JS in same session ERROR: Using js_only=true on first call causes server errors
wait_forNoWait for element that loads AFTER initial page load. Format: "css:.selector" or "js:() => condition" WHEN TO USE: • Dynamic content that loads after page (AJAX, lazy load) • Elements that appear after animations/transitions • Content loaded by JavaScript frameworks WHEN NOT TO USE: • Elements already in initial HTML (forms, static content) • Standard page elements (just use wait_until: "load") • Can cause timeouts/errors if element already exists! SELECTOR TIPS: Use get_html first to check if element exists Examples: "css:.ajax-content", "js:() => document.querySelector('.lazy-loaded')"
wait_for_timeoutNoMaximum milliseconds to wait for condition
delay_before_scrollNoMilliseconds to wait before scrolling. Allows initial content to render
scroll_delayNoMilliseconds between scroll steps for lazy-loaded content
process_iframesNoExtract content from embedded iframes including videos and forms
exclude_external_linksNoRemove links pointing to different domains for cleaner content
screenshotNoCapture full-page screenshot as base64 PNG
screenshot_directoryNoDirectory path to save screenshot (e.g., ~/Desktop, /tmp). Do NOT include filename - it will be auto-generated. Large screenshots (>800KB) won't be returned inline when saved.
pdfNoGenerate PDF as base64 preserving exact layout
cache_modeNoCache strategy. ENABLED: Use cache if available. BYPASS: Fetch fresh (recommended). DISABLED: No cacheBYPASS
timeoutNoOverall request timeout in milliseconds
verboseNoEnable server-side debug logging (not shown in output). Only for troubleshooting. Does not affect extraction results
wait_untilNoWhen to consider page loaded (use INSTEAD of wait_for for initial load): • "domcontentloaded" (default): Fast, DOM ready, use for forms/static content • "load": All resources loaded, use if you need images • "networkidle": Wait for network quiet, use for heavy JS apps WARNING: Don't use wait_for for elements in initial HTML!domcontentloaded
page_timeoutNoPage navigation timeout in milliseconds
wait_for_imagesNoWait for all images to load before extraction
ignore_body_visibilityNoSkip checking if body element is visible
scan_full_pageNoAuto-scroll entire page to trigger lazy loading. WARNING: Can be slow on long pages. Avoid combining with wait_until:"networkidle" or CSS extraction on dynamic sites. Better to use virtual_scroll_config for infinite feeds
remove_formsNoRemove all form elements from extracted content
keep_data_attributesNoPreserve data-* attributes in cleaned HTML
excluded_selectorNoCSS selector for elements to remove. Comma-separate multiple selectors. SELECTOR STRATEGY: Use get_html first to inspect page structure. Look for: • id attributes (e.g., #cookie-banner) • CSS classes (e.g., .advertisement, .popup) • data-* attributes (e.g., [data-type="ad"]) • Element type + attributes (e.g., div[role="banner"]) Examples: "#cookie-banner, .advertisement, .social-share"
only_textNoExtract only text content, no HTML structure
image_description_min_word_thresholdNoMinimum words for image alt text to be considered valid
image_score_thresholdNoMinimum relevance score for images (filters low-quality images)
exclude_external_imagesNoExclude images from external domains
screenshot_wait_forNoExtra wait time in seconds before taking screenshot
exclude_social_media_linksNoRemove links to social media platforms
exclude_domainsNoList of domains to exclude from links (e.g., ["ads.com", "tracker.io"])
simulate_userNoMimic human behavior with random mouse movements and delays. Helps bypass bot detection on protected sites. Slows crawling but improves success rate
override_navigatorNoOverride navigator properties for stealth
magicNoEXPERIMENTAL: Auto-handles popups, cookies, overlays. Use as LAST RESORT - can conflict with wait_for & CSS extraction Try first: remove_overlay_elements, excluded_selector Avoid with: CSS extraction, precise timing needs
virtual_scroll_configNoFor infinite scroll sites that REPLACE content (Twitter/Instagram feeds). USE when: Content disappears as you scroll (virtual scrolling) DON'T USE when: Content appends (use scan_full_page instead) Example: {container_selector: "#timeline", scroll_count: 10, wait_after_scroll: 1}
log_consoleNoCapture browser console logs for debugging

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure and excels at this. It explains critical behavioral traits: session persistence mechanics, server serialization errors with js_code, the need for screenshot:true with js_code, multi-step workflow requirements, and warnings about js_only usage. It provides rich context about what gets maintained (cookies, localStorage, page state) and operational constraints.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

While the description is well-structured with clear sections, it's quite lengthy with multiple bullet points and workflow examples. Some information could be more condensed, though most content earns its place by providing essential guidance. The front-loaded session persistence claim is effective, but overall length reduces conciseness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's high complexity (48 parameters, no annotations, no output schema), the description does an excellent job covering critical usage patterns, behavioral constraints, and workflow guidance. It compensates well for the lack of structured metadata, though some edge cases or error handling details could be more explicit for such a complex tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Despite 100% schema description coverage, the description adds significant value by explaining the semantic relationship between session_id and persistence, providing critical warnings about js_code usage patterns, and offering practical guidance on parameter combinations. It doesn't just repeat schema information but provides context about how parameters interact in real workflows.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose as the only tool with browser persistence, distinguishing it from all sibling tools. It specifies that it crawls URLs with optional JavaScript execution and session management, making the verb+resource+scope explicit and differentiating it from stateless alternatives like get_html.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides extensive explicit guidance on when to use sessions vs stateless approaches, when to use crawl vs other tools, and includes detailed workflow patterns. It explicitly names alternatives (stateless tools, get_html) and provides clear decision criteria for different scenarios like form filling, screenshot after JS, and data extraction.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_recursiveA

[STATELESS] Deep crawl a website following internal links. Use when: mapping entire sites, finding all pages, building comprehensive indexes. Control with max_depth (default 3) and max_pages (default 50). Note: May need JS execution for dynamic sites. Each page gets a fresh browser. For persistent operations use create_session + crawl.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesStarting URL to crawl from
max_depthNoMaximum depth to follow links
max_pagesNoMaximum number of pages to crawl
include_patternNoRegex to match URLs to crawl. Example: ".*\/blog\/.*" for blog posts only, ".*\.html$" for HTML pages
exclude_patternNoRegex to skip URLs. Example: ".*\/(login|admin).*" to avoid auth pages, ".*\.pdf$" to skip PDFs

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key traits: stateless operation ('[STATELESS]'), need for JS execution for dynamic sites, fresh browser per page, and limitations (defaults for max_depth and max_pages). It doesn't cover rate limits or error handling, but provides substantial context beyond basic functionality.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is efficiently structured with zero waste: it opens with the stateless hint, states the core purpose, provides usage guidelines, mentions key parameters with defaults, and notes behavioral considerations. Every sentence earns its place, and information is front-loaded appropriately.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (recursive crawling with multiple parameters) and no annotations or output schema, the description does well to cover purpose, usage, key behaviors, and parameter defaults. It could improve by briefly mentioning the return format or error scenarios, but it's largely complete for guiding agent selection and invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 5 parameters thoroughly. The description adds minimal value by mentioning defaults for max_depth and max_pages, but doesn't provide additional semantics beyond what's in the schema. This meets the baseline of 3 when schema coverage is high.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose with specific verbs ('deep crawl', 'following internal links') and resources ('a website'), distinguishing it from siblings like 'crawl' (likely simpler) and 'smart_crawl' (likely more intelligent). It explicitly mentions use cases like mapping entire sites and building comprehensive indexes.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit guidance on when to use ('Use when: mapping entire sites, finding all pages, building comprehensive indexes') and when not to use ('For persistent operations use create_session + crawl'), with clear alternatives named ('create_session + crawl'). It also distinguishes from other tools by mentioning JS execution needs and fresh browser contexts.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

execute_jsA

[STATELESS] Execute JavaScript and get return values + page content. Creates new browser each time. Use for: extracting data, triggering dynamic content, checking page state. Scripts with "return" statements return actual values (strings, numbers, objects, arrays). Note: null returns as {"success": true}. Returns values but page state is lost. For persistent JS execution, use crawl with session_id.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe URL to load
scriptsYesJavaScript to execute. Use "return" to get values back! Each string runs separately. Returns appear in results array. Examples: "return document.title", "return document.querySelectorAll('a').length", "return {url: location.href, links: [...document.links].map(a => a.href)}". Use proper JS syntax: real quotes, no HTML entities.

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key behavioral traits: the stateless nature ('[STATELESS]', 'Creates new browser each time', 'page state is lost'), return value handling ('Scripts with "return" statements return actual values', 'null returns as {"success": true}'), and execution characteristics ('Each string runs separately. Returns appear in results array'). The only minor gap is lack of explicit error handling or timeout information.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is efficiently structured with front-loaded key information ('[STATELESS] Execute JavaScript and get return values + page content'), followed by usage guidelines, behavioral notes, and sibling differentiation. Every sentence adds value without redundancy, and the information density is high while remaining readable.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (JavaScript execution in browser context), no annotations, and no output schema, the description does an excellent job covering execution behavior, return value handling, and sibling differentiation. The only gap is the lack of explicit information about what the output structure looks like (though this is somewhat implied by the return value explanations). For a tool with this complexity level, it's nearly complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 100% schema description coverage, the baseline is 3. The description adds meaningful context beyond the schema by explaining how return values work ('Scripts with "return" statements return actual values'), providing concrete examples of script syntax, and clarifying that 'null returns as {"success": true}'. This significantly enhances understanding of how to use the scripts parameter effectively.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose with specific verbs ('Execute JavaScript and get return values + page content') and distinguishes it from siblings by mentioning it 'Creates new browser each time' and contrasting with 'For persistent JS execution, use crawl with session_id.' This provides a clear differentiation from tools like crawl, manage_session, and smart_crawl.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit guidance on when to use this tool ('Use for: extracting data, triggering dynamic content, checking page state') and when not to use it ('For persistent JS execution, use crawl with session_id'). It names a specific alternative (crawl with session_id) and explains the trade-off between stateless execution and persistence.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_with_llmA

[STATELESS] Ask questions about webpage content using AI. Returns natural language answers. Crawls fresh each time. For dynamic content or sessions, use crawl with session_id first.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe URL to extract data from
queryYesYour question about the webpage content. Examples: "What is the main topic?", "List all product prices", "Summarize the key points", "What contact information is available?"

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden and discloses key behavioral traits: stateless operation (via [STATELESS] tag), fresh crawling each time, natural language return format, and limitations with dynamic/session content. It doesn't mention rate limits, authentication needs, or error handling, but covers core operational behavior well.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Perfectly front-loaded with the core purpose first, followed by key behavioral notes and usage guidance. Every sentence earns its place with zero wasted words, making it highly scannable and efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 2-parameter tool with no annotations and no output schema, the description provides good coverage of purpose, behavior, and usage context. It could be more complete by describing the return format in more detail (beyond 'natural language answers') or error scenarios, but it adequately supports agent decision-making given the tool's complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents both parameters thoroughly. The description adds marginal value by implying the relationship between URL and query (asking questions 'about webpage content') but doesn't provide additional syntax or format details beyond what the schema provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose with specific verbs ('Ask questions about webpage content using AI') and resources ('webpage content'), distinguishing it from siblings by focusing on AI-powered Q&A rather than raw crawling, screenshot capture, or link extraction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly provides when-to-use guidance ('Crawls fresh each time') and when-not-to-use alternatives ('For dynamic content or sessions, use crawl with session_id first'), naming a specific sibling tool (crawl) for comparison.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

generate_pdfA

[STATELESS] Convert webpage to PDF. Returns base64-encoded PDF data. Creates new browser each time. Cannot capture form fills or JS changes. For persistent PDFs use create_session + crawl(session_id, pdf:true).

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe URL to convert to PDF

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key traits: it's stateless ('[STATELESS]', 'Creates new browser each time'), has limitations ('Cannot capture form fills or JS changes'), and specifies the return format ('Returns base64-encoded PDF data'). This covers essential operational context beyond basic functionality.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is highly concise and well-structured: it uses a tag ('[STATELESS]') for immediate context, states the core function in the first sentence, adds behavioral details in subsequent sentences, and ends with alternative guidance. Every sentence adds value without redundancy, making it efficient and front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with one parameter, no annotations, and no output schema, the description is complete: it explains what the tool does, its stateless nature, limitations, return format, and when to use alternatives. This provides sufficient context for an agent to understand and invoke the tool correctly, compensating for the lack of structured metadata.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is 100%, with the single parameter 'url' clearly documented in the schema. The description does not add any additional meaning or context about the parameter beyond what the schema provides (e.g., URL format requirements). Given the high schema coverage, the baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the specific action ('Convert webpage to PDF') and resource ('webpage'), distinguishing it from sibling tools like capture_screenshot (which captures images) or get_html (which retrieves HTML). It explicitly mentions the output format ('base64-encoded PDF data'), making the purpose unambiguous and distinct.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit guidance on when to use this tool vs. alternatives: it states 'Cannot capture form fills or JS changes' and directs users to 'For persistent PDFs use create_session + crawl(session_id, pdf:true)'. This clearly defines limitations and names an alternative approach, helping the agent choose appropriately.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_htmlA

[STATELESS] Get sanitized/processed HTML for inspection and automation planning. Use when: finding form fields/selectors, analyzing page structure before automation, building schemas. Returns cleaned HTML showing element names, IDs, and classes - perfect for identifying selectors for subsequent crawl operations. Commonly used before crawl to find selectors for automation. Creates new browser each time.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe URL to extract HTML from

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key behaviors: it's stateless ('[STATELESS]'), returns cleaned HTML with specific details ('element names, IDs, and classes'), and has a side effect ('Creates new browser each time'). However, it doesn't mention potential limitations like rate limits, error handling, or authentication needs, which would be useful for a tool that creates browser instances.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is appropriately sized and front-loaded, starting with the core purpose. Most sentences earn their place by providing usage guidelines and behavioral context. However, the repetition of 'for automation' in 'automation planning' and 'for subsequent crawl operations' slightly reduces efficiency, preventing a perfect score.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity (single parameter, no output schema, no annotations), the description is largely complete. It covers purpose, usage, behavior, and output characteristics. The main gap is the lack of output schema, so the description doesn't detail the exact structure of the returned HTML (e.g., format, size limits), but it does describe the content ('cleaned HTML showing element names, IDs, and classes').

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents the single 'url' parameter. The description doesn't add any additional semantic information about parameters beyond what's in the schema (e.g., URL format requirements, handling of invalid URLs). This meets the baseline of 3 when the schema provides complete parameter documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Get sanitized/processed HTML for inspection and automation planning.' It specifies the verb ('Get'), resource ('sanitized/processed HTML'), and distinguishes from siblings by focusing on HTML extraction rather than crawling, screenshot capture, or other operations listed in sibling tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly states when to use this tool: 'Use when: finding form fields/selectors, analyzing page structure before automation, building schemas.' It also provides context on alternatives by noting it's 'commonly used before crawl to find selectors for automation,' distinguishing it from actual crawl operations like 'crawl' or 'smart_crawl'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_markdownA

[STATELESS] Extract content as markdown with filtering options. Supports: raw (full content), fit (optimized, default), bm25 (keyword search), llm (AI-powered extraction). Use bm25/llm with query for specific content. Creates new browser each time. For persistence use create_session + crawl.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe URL to extract markdown from
filterNoFilter type: raw (full), fit (optimized), bm25 (search), llm (AI extraction)fit
queryNoQuery string for bm25/llm filters. Required when using bm25 or llm filter.
cacheNoCache-bust parameter (use different values to force fresh extraction)0

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden and discloses key behavioral traits: statelessness ('[STATELESS]'), browser creation ('Creates new browser each time'), and persistence alternatives. It does not cover rate limits or auth needs, but adds significant context beyond basic function.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is appropriately sized and front-loaded, with each sentence adding value: stating purpose, listing filters, explaining usage, and noting behavioral aspects. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no annotations and no output schema, the description is fairly complete for a tool with 4 parameters and stateless behavior. It covers purpose, usage, and key traits, but lacks details on output format or error handling, which would enhance completeness.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all parameters. The description adds some meaning by explaining filter purposes (e.g., 'raw (full content), fit (optimized, default)'), but does not provide additional syntax or format details beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose with specific verbs ('extract content as markdown') and distinguishes it from siblings by mentioning filtering options and browser creation, unlike tools like 'get_html' or 'extract_links'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It provides clear context on when to use specific filters (e.g., 'Use bm25/llm with query for specific content') and mentions alternatives for persistence ('For persistence use create_session + crawl'), but does not explicitly state when not to use this tool versus all siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

manage_sessionA

[SESSION MANAGEMENT] Unified tool for managing browser sessions. Supports three actions:

• CREATE: Start a persistent browser session that maintains state across calls • CLEAR: Remove a session from local tracking • LIST: Show all active sessions with age and usage info

USAGE EXAMPLES:

  1. Create session: {action: "create", session_id: "my-session", initial_url: "https://example.com"}

  2. Clear session: {action: "clear", session_id: "my-session"}

  3. List sessions: {action: "list"}

Browser sessions maintain ALL state (cookies, localStorage, page) across multiple crawl calls. Essential for: forms, login flows, multi-step processes, maintaining state across operations.

ParametersJSON Schema
NameRequiredDescriptionDefault
actionYesAction to perform: create, clear, or list
session_idNoSession identifier. Required for action="clear". Optional for create (auto-generated if omitted).
initial_urlNoURL to load when creating session (action="create").
browser_typeNoBrowser engine for the session (action="create").chromium

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden and does well by disclosing key behavioral traits: sessions are persistent, maintain ALL state (cookies, localStorage, page) across multiple calls, and support three distinct actions. It doesn't mention rate limits, authentication needs, or error handling, but covers the core functionality thoroughly.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with clear sections (SESSION MANAGEMENT header, bulleted actions, usage examples, and essential use cases). It's appropriately sized but could be slightly more concise by integrating the usage examples more tightly with the action descriptions rather than as a separate section.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 4 parameters, 100% schema coverage, and no output schema, the description provides good contextual completeness. It explains what the tool does, when to use it, and key behavioral characteristics. The main gap is lack of information about return values or output format, which would be helpful given no output schema exists.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description adds minimal parameter semantics beyond the schema - it mentions 'initial_url' in the CREATE example and that session_id is 'auto-generated if omitted' for create actions, but doesn't significantly enhance understanding of the four parameters beyond what the schema already provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose as 'Unified tool for managing browser sessions' and specifies the three supported actions (CREATE, CLEAR, LIST). It distinguishes this from sibling tools by focusing on session state management rather than crawling or content extraction operations.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit guidance on when to use this tool ('Essential for: forms, login flows, multi-step processes, maintaining state across operations'). It differentiates from sibling tools by explaining this is for maintaining persistent browser state across multiple operations, unlike one-off crawl tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

parse_sitemapA

[STATELESS] Extract URLs from XML sitemaps. Use when: discovering all site pages, planning crawl strategies, or checking sitemap validity. Supports regex filtering. Try sitemap.xml or robots.txt first. Creates new browser each time.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesURL of the sitemap (e.g., https://example.com/sitemap.xml)
filter_patternNoOptional regex pattern to filter URLs

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden and does well by disclosing key behavioral traits: the stateless nature ('[STATELESS]'), that it 'Creates new browser each time' (implying isolated execution), and supports 'regex filtering' for customization. It doesn't mention error handling or rate limits, keeping it from a perfect score.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with key information, uses bullet-like phrasing efficiently, and every sentence earns its place by adding distinct value (purpose, usage guidelines, behavioral notes). No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity (2 parameters, no output schema, no annotations), the description is largely complete: it covers purpose, usage, key behavior, and hints at parameters. It lacks details on output format or error cases, but for a stateless extraction tool, this is sufficient though not exhaustive.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents both parameters fully. The description adds minimal value beyond the schema by mentioning 'Supports regex filtering' which loosely relates to 'filter_pattern', but doesn't provide additional syntax or format details. Baseline 3 is appropriate when the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the specific action ('Extract URLs from XML sitemaps') and resource ('XML sitemaps'), distinguishing it from siblings like 'extract_links' or 'crawl' by focusing specifically on sitemap parsing rather than general link extraction or crawling.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly provides when-to-use guidance ('Use when: discovering all site pages, planning crawl strategies, or checking sitemap validity'), when-not-to-use alternatives ('Try sitemap.xml or robots.txt first'), and practical context for application.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

smart_crawlA

[STATELESS] Auto-detect and handle different content types (HTML, sitemap, RSS, text). Use when: URL type is unknown, crawling feeds/sitemaps, or want automatic format handling. Adapts strategy based on content. Creates new browser each time. For persistent operations use create_session + crawl.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe URL to crawl intelligently
max_depthNoMaximum crawl depth for sitemaps
follow_linksNoFor sitemaps/RSS: crawl found URLs (max 10). For HTML: no effect
bypass_cacheNoForce fresh crawl

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key traits: 'Adapts strategy based on content' (dynamic behavior), 'Creates new browser each time' (stateless operation, implying no session persistence), and '[STATELESS]' tag (explicit statelessness). However, it lacks details on rate limits, error handling, or output format, which are important for a crawl tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is highly concise and well-structured: it starts with a [STATELESS] tag for quick insight, states the purpose, provides usage guidelines, and notes behavioral traits in three clear sentences. Every sentence adds value without redundancy, making it front-loaded and efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (crawling with auto-detection) and no annotations or output schema, the description does a good job covering purpose, usage, and key behaviors. However, it lacks details on output (what is returned, e.g., extracted data or links) and error scenarios, which are critical for a tool with no output schema. This gap prevents a perfect score.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all parameters well. The description does not add any parameter-specific semantics beyond what's in the schema (e.g., it doesn't explain url formats or max_depth implications further). Baseline score of 3 is appropriate as the schema handles parameter documentation adequately.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Auto-detect and handle different content types (HTML, sitemap, RSS, text)' with specific verbs ('detect', 'handle') and resources ('content types'). It distinguishes from siblings by mentioning 'automatic format handling' versus more specific tools like parse_sitemap or get_html.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit usage guidelines: 'Use when: URL type is unknown, crawling feeds/sitemaps, or want automatic format handling.' It also specifies alternatives: 'For persistent operations use create_session + crawl,' though create_session is not listed as a sibling, implying an external or implied tool. This gives clear when-to-use and when-not-to-use scenarios.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 13 tool updates
    • First observedbatch_crawl
    • First observedcapture_screenshot
    • First observedcrawl
    • First observedcrawl_recursive
    • First observedexecute_js
    • First observedextract_links
    • First observedextract_with_llm
    • First observedgenerate_pdf
    • First observedget_html
    • First observedget_markdown
    • First observedmanage_session
    • First observedparse_sitemap
    • First observedsmart_crawl

TDQS

A4.3/5.0

Scored across 13 tools

Disambiguation3/5

There is significant overlap between tools, particularly between crawl, batch_crawl, crawl_recursive, and smart_crawl, which all handle URL crawling with different scopes or features. However, descriptions help clarify distinctions, such as crawl supporting sessions while others are stateless, and tools like extract_links or parse_sitemap have more unique purposes.

Naming Consistency4/5

Most tools follow a consistent snake_case pattern with descriptive names (e.g., batch_crawl, capture_screenshot, extract_links), but manage_session uses a different structure with action parameters, and there are minor deviations like extract_with_llm using 'llm' instead of a full word. Overall, naming is mostly predictable and readable.

Tool Count5/5

With 13 tools, the count is well-scoped for a web crawling and automation server, covering various aspects like data extraction, session management, and format handling. Each tool appears to serve a distinct function within the domain, avoiding bloat while providing comprehensive coverage.

Completeness5/5

The tool set provides complete coverage for web crawling and automation, including session management (create_session via manage_session), data extraction (get_html, get_markdown, extract_with_llm), crawling variants (batch, recursive, smart), and additional utilities like screenshot and PDF generation. No obvious gaps are present for the intended domain.

Maintenance

ActivityInactive
ResponsivenessUnresponsive

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    D
    maintenance
    A TypeScript server that provides web search and content fetching capabilities through Claude CLI integration, allowing Claude Desktop and other MCP clients to access real-time web information.
    39 npm
    1
    MIT
  • A
    license
    A
    quality
    C
    maintenance
    An MCP server that enables AI assistants to fetch web content in multiple formats (HTML, JSON, text, Markdown) with intelligent content extraction, chunk management, and browser automation support.
    5
    44 npm
    15
    MIT
  • A
    license
    A
    quality
    C
    maintenance
    MCP server that provides read_page, screenshot, and pdf tools using a real browser, enabling agents to fetch clean markdown, screenshots, and PDFs from any URL.
    5
    19 npm
    MIT