map_url
Discover URLs on a website to find the right page before scraping. Crawls from a starting URL and returns a list of links, helping you locate documentation or specific content.
Instructions
Discover URLs on a website without fetching full page content.
Returns a list of URLs found by crawling from the starting URL. Use this to find the right page before scraping it.
Best for: Finding documentation pages, locating specific content on a site, understanding site structure before scraping.
Typical workflow: map_url to find URLs -> scrape_url on the right page.
Args: url: The starting URL to discover links from. limit: Maximum number of URLs to discover (default: 50, max: 100000). depth: How many links deep to follow (default: 2). include_subdomains: If true, follows links to subdomains. include_external_links: If true, follows links to external domains. include_patterns: Only visit URLs matching these wildcard patterns (e.g. "https://example.com/docs/**"). exclude_patterns: Skip URLs matching these wildcard patterns.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| depth | No | ||
| limit | No | ||
| exclude_patterns | No | ||
| include_patterns | No | ||
| include_subdomains | No | ||
| include_external_links | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |