# Scrapyard

**Your own web search and page extraction, running on your own machine.**

Give it a query, get back real search results. Give it any URL, get back clean markdown your agent can actually read — articles, JavaScript-heavy pages, PDFs, all of it.

No API key. No account. No credits to run out mid-task. No monthly bill. It's yours.

---

## Why this exists

Every AI agent needs to read the web. The usual answer is a paid scraping API — until the credits run out in the middle of a research task, or the bill arrives, or something breaks and there's nothing you can do but file a ticket and wait.

Scrapyard runs on your hardware. It doesn't ask permission, it doesn't meter you, and when something goes wrong you can actually look inside and fix it.

For the vast majority of what agents need — documentation, articles, GitHub repos, references, news, PDFs — it does the job, and it does it for free.

---

## See it work

```console
$ curl -H "X-WebTools-Token: $TOKEN" \
    'http://127.0.0.1:8377/search?q=fastapi+background+tasks&limit=2'
{
  "query": "fastapi background tasks",
  "engine_used": "bing",
  "adapter_used": "first-party-bing-html",
  "cached": false,
  "results": [
    {
      "url": "https://fastapi.tiangolo.com/",
      "title": "FastAPI - FastAPI",
      "description": "FastAPI is a modern, fast (high-performance), web framework for building APIs with Python based on standard Python type hints."
    },
    {
      "url": "https://fastapi.tiangolo.com/tutorial/",
      "title": "Tutorial - User Guide - FastAPI",
      "description": "This tutorial shows you how to use FastAPI with most of its features, step by step."
    }
  ]
}
```

Any page, straight to markdown:

```console
$ curl -H "X-WebTools-Token: $TOKEN" \
    'http://127.0.0.1:8377/extract?url=https://www.iana.org/help/example-domains'
{
  "url": "https://www.iana.org/help/example-domains",
  "title": "Example Domains",
  "content": "# Example Domains\n\nAs described in [RFC 2606](...) and [RFC 6761](...), a number of domains such as example.com ...",
  "method": "trafilatura",
  "error": null,
  "pipeline_steps": ["httpx", "trafilatura"],
  "upstream_status": 200,
  "final_url": "https://www.iana.org/help/example-domains"
}
```

That `pipeline_steps` field is there on purpose. Scrapyard tells you exactly how it got the content and what the upstream server actually said. If a page returned a 500, you'll know — it never gets dressed up as a success.

---

## Get started

Python 3.12 or newer. Install the published package and its Chromium runtime:

```sh
python3.12 -m pip install scrapyard
python3.12 -m playwright install chromium
scrapyard
```

The first run creates a mode-`0600` token at
`~/.config/scrapyard/token` and stores the search cache beneath
`~/.local/share/scrapyard/`. Chromium is a required post-install step for
JavaScript-rendered extraction; `pip` cannot install the browser binary.
Scrapyard runs on `127.0.0.1:8377` by default.

For a source checkout instead:

```sh
git clone https://github.com/artboarding-hash/scrapyard.git && cd scrapyard
scripts/install.sh
scripts/run.sh
```

The source installer creates `.venv`, installs the exact dependency pins and
Chromium, and writes a checkout-local token.

---

## What you get

**`/search`** — Real web search with automatic engine fallback. If one engine goes down, the next one picks up. Results are cached, and `type=news` gives you date-sorted news.

**`/extract`** — Any URL to clean markdown. Static pages, JavaScript-rendered pages, and PDFs all work. It also pulls out the author, publication date and site name when the page provides them.

**`/map`** — Discover the URLs on a site via `robots.txt` and `sitemap.xml`. Fast, polite, and it reads only what sites publish for exactly this purpose.

**`/research`** — Search and extract the top results in a single call, so your agent gets a usable answer in one round trip instead of five.

**`/status`** — Per-engine health, cache stats, and what's working right now.

---

## It watches itself

Search engines change their HTML. Every scraper eventually breaks — the difference is whether you find out from your monitoring or from a failed task.

Scrapyard ships a daily canary that checks every engine and every extraction path independently. It stays quiet when everything's fine, and when something breaks it tells you *which* piece broke.

And it comes with the tools to fix it:

```sh
scripts/check_upstream.py     # often it's already fixed upstream
scripts/diagnose.py           # a ready-to-paste repair brief with the real failure data
scripts/repair.sh             # takes a backup, hands you the brief
scripts/verify_repair.py      # the gate: security suite + canary + a live check
```

Hand the brief to Claude Code, Codex, Cursor, whatever you use. The verification step is what makes the fix trustworthy — nothing is accepted until the security tests pass, the canary is green, and the engine returns real results again.

All on your machine, with your agent, on your budget. Nothing is reported anywhere.

---

## Built to be safe

Scrapyard fetches URLs you hand it, so it's careful about where those point:

- HTTP and HTTPS only, ports 80 and 443
- Private, loopback, link-local and IPv6 internal addresses are refused
- Every redirect is re-checked, so a redirect can't sneak into your network
- Download, PDF and output size caps
- `/map` filters unsafe URLs out of its own results
- Constant-time token comparison on every route

Backed by a 24-case test suite that runs against the live service:

```sh
python tests/test_security.py        # 17 passed
python tests/test_security_extra.py  # 7 passed
```

`SECURITY.md` has the full threat model. Keep it on loopback or a private network — it's built for you, not for the open internet.

---

## Zero telemetry

Scrapyard makes no outbound request except the ones you ask for. No analytics, no phone-home, no usage tracking. The canary writes to a local file and nowhere else.

---

## Good to know

Scrapyard requests come from your own IP, so sites behind heavy bot protection may block it where a commercial API with a proxy pool gets through. Google is intentionally not included — it serves a JavaScript challenge that no HTML client can pass. And `/map` finds URLs but doesn't crawl entire sites.

For everything else — which is almost everything — it just works.

---

## Staying current

For a PyPI installation:

```sh
python3.12 -m pip install --upgrade scrapyard
python3.12 -m playwright install chromium
```

For a git source checkout, use the guarded updater:

```sh
scripts/update.sh
```

It refuses a dirty tree, backs up the repository and local SQLite databases,
pulls fast-forward-only, reinstalls the pinned dependencies, then runs the
17-case security suite and the canary. It prints an exact restore command and
never restarts a service; after a successful run, restart Scrapyard using the
command owned by your deployment. `scripts/update.sh --dry-run` makes the
backup and runs both verification gates without pulling or installing.

At startup Scrapyard checks the public GitHub `releases/latest` endpoint in a
background thread and caches a successful result on disk for at least 24 hours.
`/status` reports `version`, `latest_version`, `update_available`, and
`update_check`. This is a read-only GET: it sends no body, query parameters,
cookies, custom User-Agent, install ID, usage counts, or user data; the HTTP
client emits only the protocol-required `Host` header. It fetches only the
latest public release JSON and reads `tag_name`. It never applies an update.

Disable the check completely with one flag:

```sh
SCRAPYARD_UPDATE_CHECK=0 scrapyard
```

Set `SCRAPYARD_UPDATE_REPOSITORY=owner/repo` if using a fork. The source default
is the publish-time placeholder `artboarding-hash/scrapyard`; maintainers should
replace it with the final public owner before release. Network or API failures
are reported as `update_check: "failed"` and never prevent service startup.

---

## License

Apache-2.0 — use it, fork it, ship it. Copyright 2026 scrapyard.dev

---

## Connect your agent with MCP

Scrapyard ships a stdio MCP server for Claude Code, Claude Desktop, Cursor,
Hermes, and other MCP-capable clients. From this checkout:

```sh
scripts/install.sh
uv pip install --python .venv/bin/python "mcp==2.0.0"
scripts/run.sh
hermes mcp add scrapyard --command "$PWD/.venv/bin/python" --args "$PWD/scrapyard_mcp/server.py"
hermes mcp test scrapyard
```

No token needs to be pasted into the agent config: the server reads `.token` by
default. See [docs/MCP.md](docs/MCP.md) for copy-pasteable Claude Code, Claude
Desktop, Cursor, remote-service, and troubleshooting configurations.