add_web_source
Fetch documentation from a web URL by crawling HTML pages, with configurable depth, pattern, and directory restrictions to add a new source.
Instructions
Fetch documentation from a web URL (or re-download existing source).
If url is not provided, re-downloads an existing web source by slug.
By default, only fetches URLs in the same directory or below the initial URL. For example, fetching https://example.com/docs/api/index.html will only fetch files under https://example.com/docs/api/ and not https://example.com/docs/other/.
Only HTML files (.html, .htm, or no extension) are fetched and followed. Non-HTML files (CSS, JS, images, fonts, etc.) are automatically skipped.
Args: slug: Unique identifier for this web source url: Base URL to fetch from (optional for re-download) name: Display name (defaults to slug) max_depth: Recursion depth for crawling (default: 2) pattern: Regex pattern for URLs to fetch (default: .*.html?$) url_prefix: Optional URL prefix to restrict crawling (default: directory of initial URL)
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | ||
| name | No | ||
| slug | Yes | ||
| pattern | No | .*\.html?$ | |
| max_depth | No | ||
| url_prefix | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |