Skip to main content
Glama
ClawyPro

Clawy MCP Server

by ClawyPro

firecrawl_crawl

Crawl websites starting from a URL to discover and scrape linked pages, with configurable limits and path filtering for targeted data extraction.

Instructions

Crawl a website starting from a URL. Discovers and scrapes linked pages. Cost is per page crawled.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYesStarting URL to crawl
limitNoMaximum pages to crawl (default 10)
maxDepthNoMaximum link depth (default 2)
includePathsNoGlob patterns to include (e.g., ['/blog/*'])
excludePathsNoGlob patterns to exclude

Implementation Reference

  • The 'firecrawl_crawl' tool is defined here as part of the 'firecrawlTools' array. It specifies the input schema and the API endpoint.
    {
      name: "firecrawl_crawl",
      description: "Crawl a website starting from a URL. Discovers and scrapes linked pages. Cost is per page crawled.",
      inputSchema: z.object({
        url: z.string().describe("Starting URL to crawl"),
        limit: z.number().optional().describe("Maximum pages to crawl (default 10)"),
        maxDepth: z.number().optional().describe("Maximum link depth (default 2)"),
        includePaths: z.array(z.string()).optional().describe("Glob patterns to include (e.g., ['/blog/*'])"),
        excludePaths: z.array(z.string()).optional().describe("Glob patterns to exclude"),
      }),
      endpoint: "/v1/firecrawl/crawl",
    },
  • The 'firecrawl_crawl' tool (and all other tools in 'allTools') is registered in the MCP server here. The handler executes the request using 'gatewayRequest', dynamically hitting the endpoint specified in the tool definition.
    for (const tool of allTools) {
      server.tool(
        tool.name,
        tool.description,
        tool.inputSchema.shape,
        async (params) => {
          const method = tool.method || "POST";
          const result = await gatewayRequest(method, tool.endpoint, params as Record<string, unknown>);
    
          if (result.error) {
            return {
              content: [{ type: "text" as const, text: `Error (${result.status}): ${result.error}` }],
              isError: true,
            };
          }
    
          const text = typeof result.data === "string"
            ? result.data
            : JSON.stringify(result.data, null, 2);
    
          return {
            content: [{ type: "text" as const, text }],
          };
        },
      );

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/ClawyPro/clawy-mcp-server'

If you have feedback or need assistance with the MCP directory API, please join our Discord server