Skip to main content
Glama
Sign In

Web Crawler MCP by InsightfulPipe

by com.insightfulpipe
OverviewDiscussionsAdmin

Server Details

Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.

Ownership verified
Status
Healthy
Uptime
100.0% over the last 24 hours
OAuth
Works in Glama
Last Tested
Transport
Streamable HTTP · MCP 2025-11-25
URL
Repository
Insightful-Pipe/web-crawler-mcp-server
GitHub Stars
0
Server Listing
Web Crawler MCP Server
ChatGPT
–
Claude
–

TDQS

B3.4/5.0

Scored across 7 tools

Disambiguation2/5

The two central tools, execute_action and query_data, have nearly identical descriptions and are both labeled as the final workflow step, making it hard for an agent to know which to choose. crawler_run_checker also overlaps with execute_action since it is a convenience wrapper for checker actions. Other tools like crawler_generate_sitemap, crawler_generate_robots_txt, and prompts_menu are more distinct, but the core overlap is significant.

Naming Consistency3/5

Three tools use a clear crawler_ prefix with verb_noun naming, while execute_action, query_contexts, and query_data use verb_noun without the prefix, and prompts_menu uses a noun_noun pattern. The verb-oriented core is readable, but the mixed prefixing and one outlier make the set only partly predictable.

Tool Count5/5

Seven tools is well within the typical 3-15 range for a focused crawler integration. Each broad area—planning, execution, checking, generation, and prompt access—has at least one tool, so the count is appropriately scoped.

Completeness4/5

The surface covers workspace discovery, action discovery, action details, execution, data querying, checker runs, robots.txt and sitemap generation, and prompt access. Minor gaps remain around crawl job lifecycle management, such as status checks or cancellation, but core crawler workflows are represented.

Available Tools

7 tools
crawler_generate_robots_txtRobots.txt GeneratorCInspect

Generate a robots.txt using the InsightfulPipe crawler generator. Requires workspace_id, sitemap_url, and whether to block AI bots. Returns plaintext when successful so assistants can embed it directly.

Inputs: sitemap_url (required), workspace_id (required), block_ai_bots, platform_preset, +2 more. Returns: object

crawler_generate_sitemapSitemap GeneratorBInspect

Trigger the sitemap-generator action. Provide workspace_id and url. Supports payload_overrides (crawl depth, include_images, etc.). Returns XML or JSON wrapper depending on backend mode.

Inputs: url (required), workspace_id (required), timeout_seconds, payload_overrides. Returns: object

crawler_run_checkerCrawler Checker RunnerCInspect

Convenience wrapper for the URL-based crawler checkers (link-extractor, seo-audit-checker, meta-tags-checker, etc.). Provide workspace_id, action, and the target url. Optional: payload_overrides for checker-specific flags.

Inputs: url (required), action (required), workspace_id (required), extra_fields, +2 more. Returns: object

execute_actionCrawler Execute ActionA
Destructive
Inspect

Execute crawler action (Step 4 (FINAL) of workflow).

REQUIRED: Complete these steps first using query_contexts:

  1. request='accounts' - Get workspace_id and connected domains

  2. request='available_actions' - Get list of all available crawler actions

  3. request='actions_details' with actions=['xxxx'] - Get detailed body structure for specific action(s)

IMPORTANT: Always call step 3 to see the exact body structure before executing.

Rely on info from steps 1-3 to construct the action body. Follow the schema exactly. Automatically attaches platform='crawler' and handles JSON/text/XML responses.

Inputs: body (required), timeout_seconds. Returns: object

prompts_menuCrawler PromptsA
Read-only
Inspect

Access crawler prompt catalog with 4 operations:

  1. List Public Prompts: • request='menu' - No additional params needed • Returns: All public/system prompts available for crawler

  2. List Custom Prompts: • request='menu', prompt_type='custom', workspace_id= • Returns: All custom prompts for the specified workspace

  3. Get Public Prompt Details: • request='prompt_details', prompt_id= • Returns: Full text/details of a public prompt

  4. Get Custom Prompt Details: • request='prompt_details', prompt_id=, is_custom=True, workspace_id= • Returns: Full text/details of a custom prompt

Inputs: request (required), is_custom, prompt_id, prompt_type, +1 more. Returns: object

query_contextsCrawler Setup HelperA
Read-only
Inspect

Helper for planning InsightfulPipe crawler queries.

Step 1: request='accounts' – Retrieve workspace_id, brand_id, connected domains, and other source metadata.

Step 2: request='available_actions' – List all available crawler actions/checkers you can query.

Step 3: request='actions_details' with actions=['table_a', 'table_b'] – REQUIRED before querying. Returns the exact request body, required identifiers, fields, and notes for each action.

Optional: request='brands' with brand_id + workspace_id to pull brand guidelines.

Inputs: request (required), actions, brand_id, platform, +1 more. Returns: object

query_dataCrawler Data QueryA
Read-only
Inspect

Query crawler data (Step 4 (FINAL) of workflow).

REQUIRED: Complete these steps first using query_contexts:

  1. request='accounts' - Get workspace_id and connected domains

  2. request='available_actions' - Get list of all available crawler actions

  3. request='actions_details' with actions=['xxxx'] - Get detailed body structure for specific action(s)

IMPORTANT: Always call step 3 to see the exact body structure before querying.

Rely on info from steps 1-3 to construct the query body. Follow the schema exactly. Automatically attaches platform='crawler' and handles JSON/text/XML responses.

Inputs: body (required), timeout_seconds. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    Try in Browser
    One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

    Glama MCP Gateway

    Add one secure layer between your agents and this server.

    Resources