Fetch & Index URL(s)
ctx_fetch_and_indexFetches web content, converts it to markdown, and indexes it into a searchable knowledge base so you can retrieve sections on demand without raw bytes entering the conversation.
Instructions
Fetches URL content, converts HTML to markdown (JSON is chunked by key paths, plain text indexed directly), persists it in a searchable knowledge base, and returns a small preview window per source. The raw page bytes never enter your conversation — they live in storage and you retrieve any section on-demand via ctx_search.
Caching: every fetch is cached on disk and reused for repeat calls within the TTL window. The default TTL is 24 hours; override per-call with the ttl parameter (milliseconds, ttl: 0 bypasses cache like force: true). Stored content older than 14 days is cleaned up on startup.
WHEN:
You need web content (docs, changelogs, API references, spec pages) and the raw page bytes should NOT enter your conversation
Multi-URL research (library evaluation, migration scans, doc comparisons): pass the
requestsarray and aconcurrencyvalue 2-8 for parallel I/OYou want repeat lookups against the same URL to be cheap (TTL cache hits return only a hint, no re-fetch)
You want a long-lived cache window (override
ttlupward for stable specs) or a guaranteed-fresh fetch (ttl: 0orforce: true)
WHEN NOT:
You already have the content locally — store it via the inline index tool
The page is SPA-rendered (JavaScript-required to materialize content) — this is a plain HTTP fetch, no headless browser
RETURNS: Per-source preview windows extracted around indexable headings plus indexing metadata (chunk counts, source labels, cache state). Raw content is NOT echoed back — retrieve any section on-demand via ctx_search(source: ""). Concurrency parallelizes the fetch phase up to your chosen value (capped by the host's logical CPU count); the FTS5 write phase always runs serially because SQLite is a single-writer store. Net latency = max(fetch latency across the pool) + sum(per-source index write time). Cache hits skip both phases and return a small freshness hint instead of re-fetching. Use 4-8 for stable I/O-bound batches; lower the value when the target host enforces a per-IP rate limit you cannot raise.
EXAMPLE: ctx_fetch_and_index( requests: [{url: "https://react.dev/...", source: "react"}, {url: "https://vuejs.org/...", source: "vue"}], concurrency: 5 )
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| ttl | No | Override the cache freshness window for this call, in milliseconds. `ttl: 0` bypasses the cache like `force: true`; omit to use the default 24h TTL. | |
| url | No | Single URL to fetch and index (legacy single-shape) | |
| force | No | Skip cache and re-fetch even if content was recently indexed | |
| source | No | Label for the indexed content when using single `url` (e.g., 'React useEffect docs', 'Supabase Auth API'). For batch, put source in each requests entry. | |
| requests | No | Batch shape: array of {url, source?} entries. Use with concurrency>1 for parallel fetch. Each request indexed under its own source label. Output preserves input order. | |
| concurrency | No | Max URLs to fetch in parallel (1-8, default: 1). Use 4-8 for I/O-bound multi-URL batches (library docs, changelogs, pricing pages). Capped by os.cpus().length on small machines (response notes when capped). Indexing is always serial regardless — only fetches race. |