Fetch & Index URL(s)
ctx_fetch_and_indexFetch URL content, convert to markdown, index into searchable knowledge base, and return a preview. Parallel batch fetching enables multi-URL research.
Instructions
Fetches URL content, converts HTML to markdown, indexes into searchable knowledge base, and returns a ~3KB preview. Full content stays in sandbox — use ctx_search() for deeper lookups.
Better than WebFetch: preview is immediate, full content is searchable, raw HTML never enters context.
Content-type aware: HTML is converted to markdown, JSON is chunked by key paths, plain text is indexed directly.
PARALLELIZE I/O: For multi-URL research (library evaluation, migration scans, doc comparisons), pass requests: [{url, source}, ...] with concurrency: 4-8 — speeds up by 3-5x on real workloads.
✅ Use concurrency: 4-8 for: library docs sweep, multi-changelog scan, competitive pricing pages, multi-region docs, GitHub raw file pulls.
❌ Single URL → use the legacy {url, source} shape (concurrency irrelevant).
Example: requests: [{url: 'https://react.dev/...', source: 'react'}, {url: 'https://vuejs.org/...', source: 'vue'}], concurrency: 5.
Fetches parallelize up to your concurrency setting; FTS5 indexing serializes the writes after (SQLite single-writer rule).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | Single URL to fetch and index (legacy single-shape) | |
| source | No | Label for the indexed content when using single `url` (e.g., 'React useEffect docs', 'Supabase Auth API'). For batch, put source in each requests entry. | |
| requests | No | Batch shape: array of {url, source?} entries. Use with concurrency>1 for parallel fetch. Each request indexed under its own source label. Output preserves input order. | |
| concurrency | No | Max URLs to fetch in parallel (1-8, default: 1). Use 4-8 for I/O-bound multi-URL batches (library docs, changelogs, pricing pages). Capped by os.cpus().length on small machines (response notes when capped). Indexing is always serial regardless — only fetches race. | |
| force | No | Skip cache and re-fetch even if content was recently indexed |