Search Toolkit

Web search, image search, and page reading for AI agents, routed across many providers by measured quality, with persistent multi-key rotation. Runs as an MCP server (STDIO or Streamable HTTP), a CLI, and an Agent Skill.

简体中文

What it gives an agent

Three tools cover almost every lookup. Each picks a provider, falls back when one fails, and says which one answered.

search_auto modes, in balanced quality:

quality: max is for broad or multi-hop questions: English general starts with Exa Auto with highlights, Exa Lite becomes Exa Auto elsewhere, and Parallel, You.com, Brave, and Tavily switch to their deeper settings. Every mode accepts includeDomains, excludeDomains, freshness, and maxCharsPerResult, translated into each provider's own parameters. crossCheck: true asks two independent indexes at once and marks which found each result.

Where the orders come from

npm run eval:search runs the queries in evals/search-queries.json (English and Chinese, across web, news, scholar, and code) on every provider variant and scores, without an LLM judge, whether an authoritative source is in the top three, the share of authoritative sources, language match, dates, freshness, duplicates, and latency. It spends real quota. The October 2026 run, the end-to-end check of the shipped routes, and a comparison with the Artificial Analysis Search Index are in docs/search-eval-2026-10.md.

Related MCP server: qsearch

Providers

Remote MCP servers (Exa, Tavily, LinkUp, AnySearch) and the STDIO Firecrawl server are proxied with their official tool names and schemas. Everything else is a thin adapter over the provider's official REST API.

Install

git clone https://github.com/TheWiseWolfHolo/search-toolkit
cd search-toolkit
npm install
npm run build

Node.js 22 or newer is required; Node.js 24 is recommended. Key health, the result cache, and provider health live in a SQLite file through the built-in node:sqlite.

Configure

Copy config.example.json to a private path outside the repository and replace the placeholder keys. The default path is %USERPROFILE%/.config/search-toolkit/providers.json on Windows (outside AppData/Local, so MSIX-packaged apps and ordinary processes read the same file) and ~/.config/search-toolkit/providers.json elsewhere. SEARCH_TOOLKIT_CONFIG or --config points somewhere else.

A provider entry lists what it may be chosen for without being named:

"brave": {
  "enabled": true,
  "auto": ["search", "images"],
  "keys": ["key-1", "key-2"],
  "integration": { "kind": "rest", "adapter": "brave" }
}

auto holds any of search, images, fetch. An empty list makes a provider manual-only: its tools can still be called directly, but search_auto, search_images, and fetch_auto never pick it, not even as a fallback. Doubao ships that way.

Optional top-level settings:

• profile: full (default), curated, or lean; see Context cost. --profile and SEARCH_TOOLKIT_PROFILE override it.

• cache: enabled (default true), searchTtlSeconds (900), newsTtlSeconds (300, for mode: current), fetchTtlSeconds (1800); see Cache.

• shaping: maxDescriptionChars (default 700) and maxParamDescriptionChars (default 220); 0 keeps upstream text untouched.

• Per provider, toolPolicy.allow, toolPolicy.deny, and toolPolicy.descriptions (replacement descriptions keyed by upstream tool name). Firecrawl defaults to seven retrieval tools; toolPolicy.allow: ["*"] exposes its whole catalog.

Version 1 configs (automatic / manualOnly) still load. To upgrade one:

node dist/src/cli.js migrate-config           # dry run: lists the changes
node dist/src/cli.js migrate-config --write   # backs the file up, then writes v2

CLI

node dist/src/cli.js search "MCP session management" --quality max --domains modelcontextprotocol.io,github.com --cross
node dist/src/cli.js search "台风 预警" --mode current
node dist/src/cli.js images "Shorncliffe pier at sunset"
node dist/src/cli.js images https://example.com/photo.jpg   # search by image
node dist/src/cli.js fetch https://example.com/article --section "Rate Limits" --max-chars 8000
node dist/src/cli.js fetch https://example.com/a https://example.com/b --max-chars 6000
node dist/src/cli.js call serper_scholar '{"query":"retrieval augmented generation survey","limit":5}'
node dist/src/cli.js tools --profile curated  # names; --json for full schemas
node dist/src/cli.js status --verbose         # masked key health, cool-downs, last errors
node dist/src/cli.js reset brave              # clear key cool-downs (optionally one slot)
node dist/src/cli.js cache-clear

search, images, fetch, and call print what a model would read. --json prints the complete structured result, --no-cache skips the cache.

MCP

Build first, then register the STDIO server with your client. Claude Code:

claude mcp add searchToolkit --scope user -- node C:/path/to/search-toolkit/dist/src/mcp-server.js --config C:/Users/you/.config/search-toolkit/providers.json

Codex (~/.codex/config.toml):

[mcp_servers.searchToolkit]
command = "C:/path/to/node.exe"
args = ["C:/path/to/search-toolkit/dist/src/mcp-server.js", "--config", "C:/Users/you/.config/search-toolkit/providers.json", "--profile", "lean"]
startup_timeout_sec = 30
tool_timeout_sec = 120
default_tools_approval_mode = "writes"

Besides the three unified tools, the server exposes search_pool_status (masked key diagnostics), search_rotation_probe (a live, quota-consuming rotation proof), and, depending on the profile, the provider tools directly or through the provider_tools / provider_call gateway. Tools that create, update, start jobs, or send feedback are not marked read-only, and delete tools are marked destructive.

What the model reads

Each result starts with one compact route block, then readable text:

{"searchToolkitRoute":{"provider":"parallel","tool":"parallel_search"},"searchToolkitAuto":{"mode":"general","quality":"balanced"}}

1. Page title (2026-10-15)
   https://example.com/page
   Excerpt with the sentences that mention the query…

attempts (such as ["parallel_search:429", "you_search:success"]) appears only when a fallback happened, and "cached":"4m" only when the answer was reused. Excerpts are bounded, lose page furniture (copy buttons, anchors, table rules, image and link syntax), and keep the sentences that mention the query when they are too long. Image results list the image, thumbnail, source page, and publisher.

Over MCP only content and _meta are sent. Claude Code and other clients show structuredContent to the model instead of content when both are present, which would replace the readable text with raw JSON, and upstream outputSchema declarations are dropped for the same reason. The full structured result is available from the CLI with --json.

Cache

search_auto, search_images, and fetch_auto keep successful answers for a short time in the shared state database, so a search or page repeated by another turn, agent, or client within the window costs no provider call. A cached page also serves later offset and section reads of the same document. Failures and thin fetch fallbacks are never cached. Turn it off with cache.enabled: false, SEARCH_TOOLKIT_CACHE=off, or --no-cache.

Result hygiene and provider health

Within one result list, duplicate URLs (including .md / .html documentation mirrors) and the same page in another release (same site, same title once x.y.z versions are dropped) keep only their best-ranked copy, and providers are asked for two extra results so the list stays full. A provider whose automatic calls failed twice in a row is tried after its healthy peers until one success or a quiet half hour; search_pool_status shows the score as recentFailure.

Context cost

Every tool schema a client loads costs context. Descriptions are trimmed at a sentence boundary and parameter descriptions are capped; types, enums, defaults, and required lists stay as the upstream defined them. With every provider in the example config enabled:

npm run measure -- --profile lean reports startup time and tool-list size.

Startup

Providers are discovered concurrently and each upstream's tool catalog is cached beside the state database, so later starts are ready in under half a second and connect to an upstream (including spawning Firecrawl) only when one of its tools is first called. A catalog older than 12 hours is refreshed in the background and announced with notifications/tools/list_changed when it changed.

Streamable HTTP

For mobile or remote clients, serve the same toolkit over stateful Streamable HTTP. The server stores only SHA-256 hashes of client tokens:

$env:SEARCH_TOOLKIT_HTTP_TOKENS = '[{"hash":"<owner-sha256-hex>"},{"hash":"<guest-sha256-hex>","tools":["search_auto","search_images"],"requestsPerMinute":30,"maxSessions":8}]'
$env:SEARCH_TOOLKIT_HTTP_ALLOWED_HOSTS = 'search-mcp.example.com'
node dist/src/http-server.js --config C:/Users/you/.config/search-toolkit/providers.json

Clients connect to /mcp with Authorization: Bearer <client-token>. A token without tools is an owner token. Shared tokens should carry a tool allowlist, a requestsPerMinute limit, and optionally maxSessions. allowedArgumentValues limits argument values for every tool, for example {"depth":["fast","deep"]} keeps a shared token off 16-agent Grok research; an omitted argument is allowed. Put public deployments behind HTTPS; provider keys stay on the server.

The HTTP endpoint uses MCP SDK 1.32.1, limits JSON requests to 100 KiB and tool arguments to 10,000 total array elements and object keys before schema validation. Each session has its own server and transport. These HTTP limits do not apply to STDIO.

Key rotation

Each request takes one healthy key from the provider's pool. The cursor persists across restarts and is shared by concurrent processes through SQLite. Health is recorded per key fingerprint, so reordering or trimming a pool does not misattribute history.

Benches are never permanent. Only error text clearly about the key (invalid key, quota, balance, rate limit) counts as a key fault; a site answering a reader with 403 does not. A pool whose keys are all benched fails at once and says when the first recovers; reset <provider> clears benches by hand. providers.json, raw keys, and state are ignored by Git, and output shows only masked key slots.

Skill

skills/search-toolkit/ is an Agent Skill: copy or link it into your agent's skill directory. SKILL.md holds the routing rules an agent needs on every search; references/provider-routing.md holds provider roles, mode details, and how to read results.

Development

npm test
npm run smoke:mcp
npm run measure
npm run eval:search -- --variants auto          # spends quota
npm run smoke:http -- https://search-mcp.example.com/mcp C:/private/client-token.txt
npm pack --dry-run

License

MIT