web_map
Traverse a website's links to generate a structured site map, with controls for depth, breadth, and content filtering.
Instructions
Maps a website's structure by traversing it like a graph, discovering URLs and generating a comprehensive site map.
**Key Features:**
- **Graph Traversal:** Explores website structure starting from root URL.
- **Depth & Breadth Control:** Configure traversal limits to balance coverage and performance.
- **Instruction Filtering:** Use natural language to focus crawler on specific content types.
**Edge Cases & Best Practices:**
- Start with low max_depth (1-2) for initial exploration, increase if needed.
- Use instructions to filter for specific content (e.g., "only documentation pages").
- Large sites may hit timeout limits; adjust timeout and limit parameters accordingly.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Root URL to begin the mapping (e.g., 'https://docs.example.com'). | |
| limit | No | Total number of links to process before stopping. | |
| timeout | No | Maximum time in seconds for the operation. | |
| max_depth | No | Maximum depth of mapping from the base URL. | |
| max_breadth | No | Maximum number of links to follow per page. | |
| instructions | No | Natural language instructions for the crawler to filter or focus on specific content. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |