# mcpy

Web and paper tools for AI agents (fetch, grep, search, paper search), as:

- **an SDK** on npm: [`mcpy`](packages/mcpy), usable from any TypeScript project
- **a hosted server** on Cloudflare Workers: [`apps/server`](apps/server), exposing the
  same tools over MCP (`/mcp`) and REST (`/v1/*`), built only on the SDK

Callers never see which backend served a request. Under the hood: a fallback
chain of providers (direct fetch, Cloudflare Browser Run, Firecrawl, Bright
Data), retries with backoff, per-attempt timeouts and an overall deadline, a
circuit breaker, caching, context-size limits, and OpenTelemetry-shaped
traces and structured logs tagged with the calling client's name.

> mcpy used to be a local single-binary MCP server. That version is archived
> at [ebursztein/mcpy-legacy](https://github.com/ebursztein/mcpy-legacy).

Design and rules: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).

**Giving mcpy to an agent?** Hand it [`skills/mcpy/SKILL.md`](skills/mcpy/SKILL.md)
([raw](https://raw.githubusercontent.com/ebursztein/mcpy/main/skills/mcpy/SKILL.md))
with a base URL and its own API key. It covers connecting over MCP or REST,
which tool to use when, errors, and safety rules.

## Tools

| MCP tool | REST | Does |
| --- | --- | --- |
| `web_fetch` | `GET\|POST /v1/fetch` | Page as markdown (or HTML), in parts |
| `web_grep` | `GET\|POST /v1/grep` | Find lines in a page (regex or plain text), with context |
| `web_search` | `GET\|POST /v1/search` | Search the web, news, images, videos, shopping, jobs or places |
| `paper_search` | `GET\|POST /v1/papers` | Search scholarly papers: authors, year, venue, citations, abstract, free PDF |

### Parameters

| Tool | Parameters (all but the first are optional; papers needs a query, an author, or both) |
| --- | --- |
| fetch | `url`, `format` (`markdown`\|`html`), `offset`, `max_chars`, `user_agent`, `fresh` |
| grep | `url`, `pattern`, `literal`, `context`, `max_matches`, `format`, `user_agent`, `fresh` |
| search | `query`, `type`, `max_results`, `page`, `country`, `language`, `recency`, `device`, `sites`, `exclude_sites`, `mode`, `fresh` |
| papers | `query` and/or `author`, `max_results`, `page`, `year_from`, `year_to`, `open_access`, `sort` (`relevance`\|`citations`\|`newest`), `collection` (`preprints`\|`biomedical`), `mode`, `fresh` |

- `user_agent`: `default` (honest mcpy agent), `chrome`, `safari`, `firefox`, `mobile`, `googlebot`.
- `type`: `web`, `news`, `images`, `videos`, `shopping`, `jobs`, `places`.
- `recency`: `hour`, `day`, `week`, `month`, `year`. `device`: `desktop`, `mobile`.
- `country` / `language`: two-letter codes (`us`, `fr`, `en`). Hints: a source that cannot honor them still answers.
- `sites` / `exclude_sites`: hostnames (comma-separated in GET query strings).
- `mode`: `fast` (first source that answers) or `thorough` (every source that supports the request, merged; results carry `sources`, the number of sources that returned them).
- Long pages: pass the returned `next_offset` as `offset` to read on.

Operators also get `GET /v1/admin/providers` and `GET /v1/admin/spending`
(today's estimated spend and cap per client), admin clients only.

```bash
curl -s https://tools.example.com/v1/fetch \
  -H "Authorization: Bearer $KEY" -H "content-type: application/json" \
  -d '{"url":"https://elie.net","max_chars":5000}'

# URL-only agents can pass everything as query parameters:
curl -s "https://tools.example.com/v1/search?query=cloudflare&type=news&recency=week&max_results=5&key=$KEY"
```

## Run your own server

```bash
npm install
cp apps/server/.dev.vars.example apps/server/.dev.vars   # local secrets, never committed
npm run dev                                              # http://localhost:8787

npx wrangler secret put MCPY_CLIENTS -c apps/server/wrangler.jsonc   # name:key[:admin],...
npx wrangler secret put FIRECRAWL_API_KEY -c apps/server/wrangler.jsonc
npm run deploy
```

Wrangler creates the KV namespace (cache), Analytics Engine dataset
(metrics), rate limiter and spending Durable Object on first deploy. The `lytl` environment in
`apps/server/wrangler.jsonc` is the maintainers' deployment at
`tools.ly.tl`, a complete custom-domain example
(`npm run deploy -- --env lytl`).

### Clients and keys

`MCPY_CLIENTS` lists `name:key[:admin]` entries (keys 24+ characters, may
contain ":"). A key is accepted as `Authorization: Bearer`, `X-API-Key`, or
`?key=` for web agents that can only be given a URL. Give those their own
key: URLs can end up in browser history and proxy logs. (Cloudflare
invocation logs, which record full URLs, are disabled in `wrangler.jsonc`.)
Each client is rate-limited separately and named in every log line, span and
metric; failed authentications are rate-limited per IP. With no clients
configured, everything is rejected. Non-admin calls to admin routes get 403.

### Spending caps

`DAILY_BUDGET_USD` caps what each client may spend on paid sources per UTC
day, using mcpy's per-call cost estimates (Firecrawl, Bright Data, Cloudflare
Browser Run, OpenAlex). `CLIENT_DAILY_BUDGETS_USD` overrides it per client:
`cardy:0.5,ops:unlimited`. Over its cap a client still gets cached results
and free sources (direct fetch, arXiv, PubMed); when those cannot answer it
gets `budget_exceeded` (429) with `Retry-After` until midnight UTC. Totals
live in the `BUDGET` Durable Object, one per client; requests already in
flight when the cap is crossed still finish.

### Configuration

| Var | Default | Purpose |
| --- | --- | --- |
| `MCPY_CLIENTS` (secret) | - | API clients |
| `PAGE_PROVIDERS` | `direct,cloudflare_browser,firecrawl,brightdata` | Page fallback order |
| `SEARCH_PROVIDERS` | `firecrawl,brightdata` | Search fallback order |
| `PAPER_PROVIDERS` | `openalex,arxiv,pubmed,semantic_scholar` | Paper index order |
| `ATTEMPT_TIMEOUT_MS` / `DEADLINE_MS` | `20000` / `60000` | Per attempt / per call |
| `RETRIES` | `2` | Retries per upstream request |
| `DAILY_BUDGET_USD` | - (`5` in `wrangler.jsonc`) | Estimated USD each client may spend on paid sources per UTC day |
| `CLIENT_DAILY_BUDGETS_USD` | - | Per-client caps: `name:usd` or `name:unlimited`, comma-separated |
| `MAX_RESPONSE_BYTES` | `5000000` | Largest upstream body |
| `PAGE_CACHE_TTL_SECONDS` / `SEARCH_CACHE_TTL_SECONDS` | `3600` / `900` | Cache TTLs |
| `DEFAULT_USER_AGENT` | `default` | `default`, `chrome`, `safari`, `firefox`, `mobile`, `googlebot` |
| `USER_AGENT_STRING` | - | Replace the `default` user-agent string |
| `ALLOWED_HOSTNAMES` | - | Accepted Host headers (set on custom domains) |
| `ALLOWED_ORIGIN_HOSTNAMES` | - | Extra browser origins allowed to call `/mcp` |
| `LOG_LEVEL` | `info` | `debug`, `info`, `warn`, `error` |
| `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_HEADERS` | - | Export traces + logs (OTLP/HTTP JSON) |
| `OTEL_SERVICE_NAME` / `DEPLOYMENT_ENV` | `mcpy` / `production` | Resource attributes |
| `CF_ACCOUNT_ID`, `CF_BROWSER_API_TOKEN` | - | Cloudflare Browser Run |
| `FIRECRAWL_API_KEY`, `FIRECRAWL_API_URL` | - | Firecrawl (URL for self-hosted) |
| `BRIGHTDATA_API_KEY`, `BRIGHTDATA_UNLOCKER_ZONE`, `BRIGHTDATA_SERP_ZONE` | - | Bright Data |
| `OPENALEX_API_KEY` | - | Free at openalex.org/settings/api. Needed on Cloudflare: without it OpenAlex shares one small daily budget per IP, and Workers share IPs, so it answers 429 |
| `SEMANTIC_SCHOLAR_API_KEY` | - | Enables Semantic Scholar (anonymous calls are rate-limited) |
| `NCBI_API_KEY`, `NCBI_EMAIL` | - | Optional for PubMed: a free key raises the limit from 3 to 10 requests/s |
| `PAPER_CACHE_TTL_SECONDS` | `86400` | Paper search cache TTL |
| `THOROUGH_GRACE_MS` | `8000` | Thorough mode: how long to wait for other sources after the first answers |

Everything is validated with zod at startup; a bad value fails closed with a
clear log line. A provider without credentials is skipped.

### Paper indexes

OpenAlex (250M+ works with citations, venues and open-access links), arXiv
(preprints, always free to read) and PubMed (37M+ biomedical citations, with
free PubMed Central full text when available) work without a key, but on
Cloudflare OpenAlex needs one in practice (see `OPENALEX_API_KEY`): it is the
only keyless index that sorts by citations; Semantic Scholar
is used when `SEMANTIC_SCHOLAR_API_KEY` is set. `collection: preprints` or
`collection: biomedical` targets the indexes covering those collections
directly; without it, every index is eligible. Dates are when a paper became
available: an article published online in 2026 for a 2027 issue shows 2026. In `thorough` mode the same
paper found in several indexes is merged by DOI, arXiv id or title, keeping
the highest citation count. `author` resolves the name to an OpenAlex author profile, and maps to arXiv
`au:` terms and PubMed `[au]` (Semantic Scholar cannot filter by author); with
no query, an author's papers rank by citations. Sorting by citations or newest matches title and
abstract only (OpenAlex full-text matching ranked off-topic classics first).
arXiv asks for one request every 3 s and throttles otherwise, so mcpy spaces
its requests and retries patiently; its answers can take up to ~30 s.

### Bright Data setup

Create two zones in the Bright Data control panel (Proxies & Scraping):

- **Web Unlocker API** (e.g. `mcpy_unlocker`) → `BRIGHTDATA_UNLOCKER_ZONE`
- **SERP API** with data format **Full JSON** (e.g. `mcpy_serp`) → `BRIGHTDATA_SERP_ZONE`.
  Light JSON only carries top organic results, so news, images, videos,
  shopping and places would come back empty.

Measured on 2026-10-22: unlocker requests took 2-38 s and SERP requests
4-50 s, so mcpy gives them 45-50 s attempt budgets and keeps them after the
faster providers in the default order. Searches without a country default to
US English (Bright Data otherwise geolocates at random). Google Jobs through
the SERP API failed upstream, so `type=jobs` is not offered for Bright Data.

## Observability

- **Logs**: one structured JSON line per operation in Workers Logs, with
  `client.name`, `mcpy.request.id`, trace ids, cache status, cost.
- **Traces**: set `OTEL_EXPORTER_OTLP_ENDPOINT` to send spans and logs to
  any OTLP collector (Cloudflare Traces, Grafana, Honeycomb, ...). Incoming
  `traceparent` headers are honored.
- **Metrics**: Analytics Engine, one row per operation and provider attempt.
  Column layout in `packages/mcpy/src/telemetry/sinks/analytics.ts`.

```sql
SELECT blob7 AS client, blob3 AS provider, blob4 AS outcome, count() AS n,
       avg(double1) AS avg_ms, sum(double3) AS usd
FROM mcpy WHERE blob1 = 'mcpy.attempt' AND timestamp > NOW() - INTERVAL '7' DAY
GROUP BY client, provider, outcome ORDER BY n DESC
```

## Develop

```bash
npm test            # unit, contract, regression, structure, perf
npm run test:e2e    # real Worker in workerd, MCP + REST against elie.net
npm run test:live   # real provider APIs (keys from apps/server/.dev.vars)
npm run typecheck
```

## License

Apache-2.0