firecrawl_crawl

firecrawl_crawl

Crawl multiple web pages starting from a specified URL. Control depth, filter paths, and receive webhook notifications for structured web scraping.

Instructions

Start an asynchronous crawl of multiple pages from a starting URL. Supports depth control, path filtering, and webhook notifications.

Input Schema

Name	Required	Description
`allowBackwardLinks`	No	Allow crawling links that point to parent directories
`allowExternalLinks`	No	Allow crawling links to external domains
`deduplicateSimilarURLs`	No	Remove similar URLs during crawl
`excludePaths`	No	URL paths to exclude from crawling
`ignoreQueryParameters`	No	Ignore query parameters when comparing URLs
`ignoreSitemap`	No	Skip sitemap.xml discovery
`includePaths`	No	Only crawl these URL paths
`limit`	No	Maximum number of pages to crawl
`maxDepth`	No	Maximum link depth to crawl
`scrapeOptions`	No	Options for scraping each page
`url`	Yes	Starting URL for the crawl
`webhook`	No

Input Schema (JSON Schema)

{
  "properties": {
    "allowBackwardLinks": {
      "description": "Allow crawling links that point to parent directories",
      "type": "boolean"
    },
    "allowExternalLinks": {
      "description": "Allow crawling links to external domains",
      "type": "boolean"
    },
    "deduplicateSimilarURLs": {
      "description": "Remove similar URLs during crawl",
      "type": "boolean"
    },
    "excludePaths": {
      "description": "URL paths to exclude from crawling",
      "items": {
        "type": "string"
      },
      "type": "array"
    },
    "ignoreQueryParameters": {
      "description": "Ignore query parameters when comparing URLs",
      "type": "boolean"
    },
    "ignoreSitemap": {
      "description": "Skip sitemap.xml discovery",
      "type": "boolean"
    },
    "includePaths": {
      "description": "Only crawl these URL paths",
      "items": {
        "type": "string"
      },
      "type": "array"
    },
    "limit": {
      "description": "Maximum number of pages to crawl",
      "type": "number"
    },
    "maxDepth": {
      "description": "Maximum link depth to crawl",
      "type": "number"
    },
    "scrapeOptions": {
      "description": "Options for scraping each page",
      "properties": {
        "excludeTags": {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "formats": {
          "items": {
            "enum": [
              "markdown",
              "html",
              "rawHtml",
              "screenshot",
              "links",
              "screenshot@fullPage",
              "extract"
            ],
            "type": "string"
          },
          "type": "array"
        },
        "includeTags": {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "onlyMainContent": {
          "type": "boolean"
        },
        "waitFor": {
          "type": "number"
        }
      },
      "type": "object"
    },
    "url": {
      "description": "Starting URL for the crawl",
      "type": "string"
    },
    "webhook": {
      "oneOf": [
        {
          "description": "Webhook URL to notify when crawl is complete",
          "type": "string"
        },
        {
          "properties": {
            "headers": {
              "description": "Custom headers for webhook requests",
              "type": "object"
            },
            "url": {
              "description": "Webhook URL",
              "type": "string"
            }
          },
          "required": [
            "url"
          ],
          "type": "object"
        }
      ]
    }
  },
  "required": [
    "url"
  ],
  "type": "object"
}

mcp-server-firecrawl

Instructions

Input Schema

Input Schema (JSON Schema)

Other Tools from mcp-server-firecrawl

Related Tools