Fetch & Index URL(s)
ctx_fetch_and_indexFetch web content, convert to markdown, and index into a searchable knowledge base. Retrieve sections on-demand without raw content entering your conversation.
Instructions
Fetches URL content, converts HTML to markdown (JSON is chunked by key paths, plain text indexed directly), persists it in a searchable knowledge base, and returns a small preview window per source. The raw page bytes never enter your conversation — they live in storage and you retrieve any section on-demand via ctx_search.
Caching: every fetch is cached on disk and reused for repeat calls within the TTL window. The default TTL is 24 hours; override per-call with the ttl parameter (milliseconds, ttl: 0 bypasses cache like force: true). Stored content older than 14 days is cleaned up on startup.
WHEN:
You need web content (docs, changelogs, API references, spec pages) and the raw page bytes should NOT enter your conversation
Multi-URL research (library evaluation, migration scans, doc comparisons): pass the
requestsarray and aconcurrencyvalue 2-8 for parallel I/OYou want repeat lookups against the same URL to be cheap (TTL cache hits return only a hint, no re-fetch)
You want a long-lived cache window (override
ttlupward for stable specs) or a guaranteed-fresh fetch (ttl: 0orforce: true)
WHEN NOT:
You already have the content locally — store it via the inline index tool
The page is SPA-rendered (JavaScript-required to materialize content) — this is a plain HTTP fetch, no headless browser
RETURNS: Per-source preview windows extracted around indexable headings plus indexing metadata (chunk counts, source labels, cache state). Raw content is NOT echoed back — retrieve any section on-demand via ctx_search(source: ""). Concurrency parallelizes the fetch phase up to your chosen value (capped by the host's logical CPU count); the FTS5 write phase always runs serially because SQLite is a single-writer store. Net latency = max(fetch latency across the pool) + sum(per-source index write time). Cache hits skip both phases and return a small freshness hint instead of re-fetching. Use 4-8 for stable I/O-bound batches; lower the value when the target host enforces a per-IP rate limit you cannot raise.
EXAMPLE: ctx_fetch_and_index( requests: [{url: "https://react.dev/...", source: "react"}, {url: "https://vuejs.org/...", source: "vue"}], concurrency: 5 )
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | Single URL to fetch and index (legacy single-shape) | |
| source | No | Label for the indexed content when using single `url` (e.g., 'React useEffect docs', 'Supabase Auth API'). For batch, put source in each requests entry. | |
| requests | No | Batch shape: array of {url, source?} entries. Use with concurrency>1 for parallel fetch. Each request indexed under its own source label. Output preserves input order. | |
| concurrency | No | Max URLs to fetch in parallel (1-8, default: 1). Use 4-8 for I/O-bound multi-URL batches (library docs, changelogs, pricing pages). Capped by os.cpus().length on small machines (response notes when capped). Indexing is always serial regardless — only fetches race. | |
| force | No | Skip cache and re-fetch even if content was recently indexed | |
| ttl | No | Override the cache freshness window for this call, in milliseconds. `ttl: 0` bypasses the cache like `force: true`; omit to use the default 24h TTL. |