crawl

crawl

Crawls a website starting from a base URL

Input Schema

Name	Required	Description
`allowBackwardLinks`	No	Allow crawling links that point to parent directories
`allowExternalLinks`	No	Allow crawling links to external domains
`excludePaths`	No	URL patterns to exclude
`ignoreQueryParameters`	No	Ignore URL query parameters when comparing URLs
`ignoreSitemap`	No	Ignore sitemap.xml during crawling
`includePaths`	No	URL patterns to include
`limit`	No	Maximum pages to crawl
`maxDepth`	No	Maximum crawl depth
`scrapeOptions`	No	Options for scraping crawled pages
`url`	Yes	Base URL to start crawling from
`webhook`	No	Webhook URL for progress notifications

Input Schema (JSON Schema)

{
  "properties": {
    "allowBackwardLinks": {
      "description": "Allow crawling links that point to parent directories",
      "type": "boolean"
    },
    "allowExternalLinks": {
      "description": "Allow crawling links to external domains",
      "type": "boolean"
    },
    "excludePaths": {
      "description": "URL patterns to exclude",
      "items": {
        "type": "string"
      },
      "type": "array"
    },
    "ignoreQueryParameters": {
      "description": "Ignore URL query parameters when comparing URLs",
      "type": "boolean"
    },
    "ignoreSitemap": {
      "description": "Ignore sitemap.xml during crawling",
      "type": "boolean"
    },
    "includePaths": {
      "description": "URL patterns to include",
      "items": {
        "type": "string"
      },
      "type": "array"
    },
    "limit": {
      "default": 10000,
      "description": "Maximum pages to crawl",
      "type": "number"
    },
    "maxDepth": {
      "default": 2,
      "description": "Maximum crawl depth",
      "type": "number"
    },
    "scrapeOptions": {
      "description": "Options for scraping crawled pages",
      "type": "object"
    },
    "url": {
      "description": "Base URL to start crawling from",
      "type": "string"
    },
    "webhook": {
      "description": "Webhook URL for progress notifications",
      "type": "string"
    }
  },
  "required": [
    "url"
  ],
  "type": "object"
}

Input Schema

Input Schema (JSON Schema)

Other Tools