Fetch & Index URL(s)
ctx_fetch_and_indexFetch web pages, convert them to markdown, and store them in a searchable knowledge base so raw bytes stay out of context—only small previews are returned for later retrieval.
Instructions
Fetches URL content, converts HTML to markdown (JSON chunked by key paths, plain text indexed directly), persists it in a searchable knowledge base, returns a small preview window per source. Raw page bytes never enter your conversation — they live in storage, retrievable via ctx_search. Every fetch is cached on disk for 24h by default; override per-call with ttl.
WHEN:
You need web content (docs, changelogs, API references, spec pages) and the raw bytes should NOT enter your conversation
Multi-URL research (library evaluation, migration scans, doc comparisons): pass the
requestsarray andconcurrency2-8 for parallel I/OYou want repeat lookups against the same URL to be cheap (cache hits return only a hint, no re-fetch)
You want a guaranteed-fresh fetch (
ttl: 0orforce: true)
WHEN NOT:
You already have the content locally — store it via the inline index tool
The page is SPA-rendered (JavaScript-required) — this is a plain HTTP fetch, no headless browser
RETURNS:
Per-source preview windows around indexable headings plus indexing metadata (chunk counts, source labels, cache). Raw content is NOT echoed back. concurrency parallelizes the fetch phase only — FTS5 is single-writer, so index writes always serialize. Use 4-8 for I/O-bound batches; lower it under per-IP rate limits.
EXAMPLE: ctx_fetch_and_index( requests: [{url: "https://react.dev/...", source: "react"}, {url: "https://vuejs.org/...", source: "vue"}], concurrency: 5 )
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| ttl | No | Override the cache freshness window for this call, in milliseconds. `ttl: 0` bypasses the cache like `force: true`; omit to use the default 24h TTL. | |
| url | No | Single URL to fetch and index (legacy single-shape) | |
| force | No | Skip cache and re-fetch even if content was recently indexed | |
| source | No | Label for the indexed content when using single `url` (e.g., 'React useEffect docs', 'Supabase Auth API'). For batch, put source in each requests entry. | |
| requests | No | Batch shape: array of {url, source?} entries. Use with concurrency>1 for parallel fetch. Each request indexed under its own source label. Output preserves input order. | |
| concurrency | No | Max URLs to fetch in parallel (1-8, default: 1). Use 4-8 for I/O-bound multi-URL batches (library docs, changelogs, pricing pages). Capped by os.cpus().length on small machines (response notes when capped). Indexing is always serial regardless — only fetches race. |