mcpy

Web tools for AI agents (fetch, grep, search), as:

• an SDK on npm: mcpy, usable from any TypeScript project

• a hosted server on Cloudflare Workers: apps/server, exposing the same tools over MCP (/mcp) and REST (/v1/*), built only on the SDK

Callers never see which backend served a request. Under the hood: a fallback chain of providers (direct fetch, Cloudflare Browser Run, Firecrawl, Bright Data), retries with backoff, per-attempt timeouts and an overall deadline, a circuit breaker, caching, context-size limits, and OpenTelemetry-shaped traces and structured logs tagged with the calling client's name.

mcpy used to be a local single-binary MCP server. That version is archived at ebursztein/mcpy-legacy.

Design and rules: docs/ARCHITECTURE.md.

Tools

Parameters

• user_agent: default (honest mcpy agent), chrome, safari, firefox, mobile, googlebot.

• type: web, news, images, videos, shopping, jobs, places.

• recency: hour, day, week, month, year. device: desktop, mobile.

• country / language: two-letter codes (us, fr, en). Hints: a source that cannot honor them still answers.

• sites / exclude_sites: hostnames (comma-separated in GET query strings).

• mode: fast (first source that answers) or thorough (every source that supports the request, merged; results carry sources, the number of sources that returned them).

• Long pages: pass the returned next_offset as offset to read on.

Operators also get GET /v1/admin/providers (admin clients only).

curl -s https://tools.example.com/v1/fetch \
  -H "Authorization: Bearer $KEY" -H "content-type: application/json" \
  -d '{"url":"https://elie.net","max_chars":5000}'

# URL-only agents can pass everything as query parameters:
curl -s "https://tools.example.com/v1/search?query=cloudflare&type=news&recency=week&max_results=5&key=$KEY"

Related MCP server: searxNcrawl

Run your own server

npm install
cp apps/server/.dev.vars.example apps/server/.dev.vars   # local secrets, never committed
npm run dev                                              # http://localhost:8787

npx wrangler secret put MCPY_CLIENTS -c apps/server/wrangler.jsonc   # name:key[:admin],...
npx wrangler secret put FIRECRAWL_API_KEY -c apps/server/wrangler.jsonc
npm run deploy

Wrangler creates the KV namespace (cache), Analytics Engine dataset (metrics) and rate limiter on first deploy. The lytl environment in apps/server/wrangler.jsonc is the maintainers' deployment at tools.ly.tl, a complete custom-domain example (npm run deploy -- --env lytl).

Clients and keys

MCPY_CLIENTS lists name:key[:admin] entries (keys 24+ characters, may contain ":"). A key is accepted as Authorization: Bearer, X-API-Key, or ?key= for web agents that can only be given a URL. Give those their own key: URLs can end up in browser history and proxy logs. (Cloudflare invocation logs, which record full URLs, are disabled in wrangler.jsonc.) Each client is rate-limited separately and named in every log line, span and metric; failed authentications are rate-limited per IP. With no clients configured, everything is rejected. Non-admin calls to admin routes get 403.

Configuration

Everything is validated with zod at startup; a bad value fails closed with a clear log line. A provider without credentials is skipped.

Paper indexes

OpenAlex (250M+ works with citations, venues and open-access links), arXiv (preprints, always free to read) and PubMed (37M+ biomedical citations, with free PubMed Central full text when available) need no key; Semantic Scholar is used when SEMANTIC_SCHOLAR_API_KEY is set. collection: preprints or collection: biomedical targets the indexes covering those collections directly; without it, every index is eligible. Dates are when a paper became available: an article published online in 2026 for a 2027 issue shows 2026. In thorough mode the same paper found in several indexes is merged by DOI, arXiv id or title, keeping the highest citation count. Sorting by citations or newest matches title and abstract only (OpenAlex full-text matching ranked off-topic classics first). arXiv asks for one request every 3 s and throttles otherwise, so mcpy spaces its requests and retries patiently; its answers can take up to ~30 s.

Bright Data setup

Create two zones in the Bright Data control panel (Proxies & Scraping):

• Web Unlocker API (e.g. mcpy_unlocker) → BRIGHTDATA_UNLOCKER_ZONE

• SERP API with data format Full JSON (e.g. mcpy_serp) → BRIGHTDATA_SERP_ZONE. Light JSON only carries top organic results, so news, images, videos, shopping and places would come back empty.

Measured on 2026-10-22: unlocker requests took 2-38 s and SERP requests 4-50 s, so mcpy gives them 45-50 s attempt budgets and keeps them after the faster providers in the default order. Searches without a country default to US English (Bright Data otherwise geolocates at random). Google Jobs through the SERP API failed upstream, so type=jobs is not offered for Bright Data.

• Logs: one structured JSON line per operation in Workers Logs, with client.name, mcpy.request.id, trace ids, cache status, cost.

• Traces: set OTEL_EXPORTER_OTLP_ENDPOINT to send spans and logs to any OTLP collector (Cloudflare Traces, Grafana, Honeycomb, ...). Incoming traceparent headers are honored.

• Metrics: Analytics Engine, one row per operation and provider attempt. Column layout in packages/mcpy/src/telemetry/sinks/analytics.ts.

SELECT blob7 AS client, blob3 AS provider, blob4 AS outcome, count() AS n,
       avg(double1) AS avg_ms, sum(double3) AS usd
FROM mcpy WHERE blob1 = 'mcpy.attempt' AND timestamp > NOW() - INTERVAL '7' DAY
GROUP BY client, provider, outcome ORDER BY n DESC

Develop

npm test            # unit, contract, regression, structure, perf
npm run test:e2e    # real Worker in workerd, MCP + REST against elie.net
npm run test:live   # real provider APIs (keys from apps/server/.dev.vars)
npm run typecheck

License

Apache-2.0