Scrapyard

Your own web search and page extraction, running on your own machine.

Give it a query, get back real search results. Give it any URL, get back clean markdown your agent can actually read — articles, JavaScript-heavy pages, PDFs, all of it.

No API key. No account. No credits to run out mid-task. No monthly bill. It's yours.

Why this exists

Every AI agent needs to read the web. The usual answer is a paid scraping API — until the credits run out in the middle of a research task, or the bill arrives, or something breaks and there's nothing you can do but file a ticket and wait.

Scrapyard runs on your hardware. It doesn't ask permission, it doesn't meter you, and when something goes wrong you can actually look inside and fix it.

For the vast majority of what agents need — documentation, articles, GitHub repos, references, news, PDFs — it does the job, and it does it for free.

Related MCP server: web-basics-mcp

See it work

$ curl -H "X-WebTools-Token: $TOKEN" \
    'http://127.0.0.1:8377/search?q=fastapi+background+tasks&limit=2'
{
  "query": "fastapi background tasks",
  "engine_used": "bing",
  "adapter_used": "first-party-bing-html",
  "cached": false,
  "results": [
    {
      "url": "https://fastapi.tiangolo.com/",
      "title": "FastAPI - FastAPI",
      "description": "FastAPI is a modern, fast (high-performance), web framework for building APIs with Python based on standard Python type hints."
    },
    {
      "url": "https://fastapi.tiangolo.com/tutorial/",
      "title": "Tutorial - User Guide - FastAPI",
      "description": "This tutorial shows you how to use FastAPI with most of its features, step by step."
    }
  ]
}

Any page, straight to markdown:

$ curl -H "X-WebTools-Token: $TOKEN" \
    'http://127.0.0.1:8377/extract?url=https://www.iana.org/help/example-domains'
{
  "url": "https://www.iana.org/help/example-domains",
  "title": "Example Domains",
  "content": "# Example Domains\n\nAs described in [RFC 2606](...) and [RFC 6761](...), a number of domains such as example.com ...",
  "method": "trafilatura",
  "error": null,
  "pipeline_steps": ["httpx", "trafilatura"],
  "upstream_status": 200,
  "final_url": "https://www.iana.org/help/example-domains"
}

That pipeline_steps field is there on purpose. Scrapyard tells you exactly how it got the content and what the upstream server actually said. If a page returned a 500, you'll know — it never gets dressed up as a success.

Get started

Python 3.12 or newer. Install the published package and its Chromium runtime:

python3.12 -m pip install scrapyard
python3.12 -m playwright install chromium
scrapyard

The first run creates a mode-0600 token at ~/.config/scrapyard/token and stores the search cache beneath ~/.local/share/scrapyard/. Chromium is a required post-install step for JavaScript-rendered extraction; pip cannot install the browser binary. Scrapyard runs on 127.0.0.1:8377 by default.

For a source checkout instead:

git clone https://github.com/artboarding-hash/scrapyard.git && cd scrapyard
scripts/install.sh
scripts/run.sh

The source installer creates .venv, installs the exact dependency pins and Chromium, and writes a checkout-local token.

What you get

/search — Real web search with automatic engine fallback. If one engine goes down, the next one picks up. Results are cached, and type=news gives you date-sorted news.

/extract — Any URL to clean markdown. Static pages, JavaScript-rendered pages, and PDFs all work. It also pulls out the author, publication date and site name when the page provides them.

/map — Discover the URLs on a site via robots.txt and sitemap.xml. Fast, polite, and it reads only what sites publish for exactly this purpose.

/research — Search and extract the top results in a single call, so your agent gets a usable answer in one round trip instead of five.

/status — Per-engine health, cache stats, and what's working right now.

It watches itself

Search engines change their HTML. Every scraper eventually breaks — the difference is whether you find out from your monitoring or from a failed task.

Scrapyard ships a daily canary that checks every engine and every extraction path independently. It stays quiet when everything's fine, and when something breaks it tells you which piece broke.

And it comes with the tools to fix it:

scripts/check_upstream.py     # often it's already fixed upstream
scripts/diagnose.py           # a ready-to-paste repair brief with the real failure data
scripts/repair.sh             # takes a backup, hands you the brief
scripts/verify_repair.py      # the gate: security suite + canary + a live check

Hand the brief to Claude Code, Codex, Cursor, whatever you use. The verification step is what makes the fix trustworthy — nothing is accepted until the security tests pass, the canary is green, and the engine returns real results again.

All on your machine, with your agent, on your budget. Nothing is reported anywhere.

Built to be safe

Scrapyard fetches URLs you hand it, so it's careful about where those point:

• HTTP and HTTPS only, ports 80 and 443

• Private, loopback, link-local and IPv6 internal addresses are refused

• Every redirect is re-checked, so a redirect can't sneak into your network

• Download, PDF and output size caps

• /map filters unsafe URLs out of its own results

• Constant-time token comparison on every route

Backed by a 24-case test suite that runs against the live service:

python tests/test_security.py        # 17 passed
python tests/test_security_extra.py  # 7 passed

SECURITY.md has the full threat model. Keep it on loopback or a private network — it's built for you, not for the open internet.

Zero telemetry

Scrapyard makes no outbound request except the ones you ask for. No analytics, no phone-home, no usage tracking. The canary writes to a local file and nowhere else.

Good to know

Scrapyard requests come from your own IP, so sites behind heavy bot protection may block it where a commercial API with a proxy pool gets through. Google is intentionally not included — it serves a JavaScript challenge that no HTML client can pass. And /map finds URLs but doesn't crawl entire sites.

For everything else — which is almost everything — it just works.

Staying current

For a PyPI installation:

python3.12 -m pip install --upgrade scrapyard
python3.12 -m playwright install chromium

For a git source checkout, use the guarded updater:

scripts/update.sh

It refuses a dirty tree, backs up the repository and local SQLite databases, pulls fast-forward-only, reinstalls the pinned dependencies, then runs the 17-case security suite and the canary. It prints an exact restore command and never restarts a service; after a successful run, restart Scrapyard using the command owned by your deployment. scripts/update.sh --dry-run makes the backup and runs both verification gates without pulling or installing.

At startup Scrapyard checks the public GitHub releases/latest endpoint in a background thread and caches a successful result on disk for at least 24 hours. /status reports version, latest_version, update_available, and update_check. This is a read-only GET: it sends no body, query parameters, cookies, custom User-Agent, install ID, usage counts, or user data; the HTTP client emits only the protocol-required Host header. It fetches only the latest public release JSON and reads tag_name. It never applies an update.

Disable the check completely with one flag:

SCRAPYARD_UPDATE_CHECK=0 scrapyard

Set SCRAPYARD_UPDATE_REPOSITORY=owner/repo if using a fork. The source default is the publish-time placeholder artboarding-hash/scrapyard; maintainers should replace it with the final public owner before release. Network or API failures are reported as update_check: "failed" and never prevent service startup.

License

Apache-2.0 — use it, fork it, ship it. Copyright 2026 scrapyard.dev

Connect your agent with MCP

Scrapyard ships a stdio MCP server for Claude Code, Claude Desktop, Cursor, Hermes, and other MCP-capable clients. From this checkout:

scripts/install.sh
uv pip install --python .venv/bin/python "mcp==2.0.0"
scripts/run.sh
hermes mcp add scrapyard --command "$PWD/.venv/bin/python" --args "$PWD/scrapyard_mcp/server.py"
hermes mcp test scrapyard

No token needs to be pasted into the agent config: the server reads .token by default. See docs/MCP.md for copy-pasteable Claude Code, Claude Desktop, Cursor, remote-service, and troubleshooting configurations.