Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@scrapyard search for Python 3.12 release notes"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Your own web search and page extraction, running on your own machine.
Give it a query, get back real search results. Give it any URL, get back clean markdown your agent can actually read — articles, JavaScript-heavy pages, PDFs, all of it.
No API key. No account. No credits to run out mid-task. No monthly bill. It's yours.
Every AI agent needs to read the web. The usual answer is a paid scraping API — until the credits run out in the middle of a research task, or the bill arrives, or something breaks and there's nothing you can do but file a ticket and wait.
Scrapyard runs on your hardware. It doesn't ask permission, it doesn't meter you, and when something goes wrong you can actually look inside and fix it.
For the vast majority of what agents need — documentation, articles, GitHub repos, references, news, PDFs — it does the job, and it does it for free.
Related MCP server: free-search-mcp
$ curl -H "X-WebTools-Token: $TOKEN" \
'http://127.0.0.1:8377/search?q=fastapi+background+tasks&limit=2'
{
"query": "fastapi background tasks",
"engine_used": "bing",
"adapter_used": "first-party-bing-html",
"cached": false,
"results": [
{
"url": "https://fastapi.tiangolo.com/",
"title": "FastAPI - FastAPI",
"description": "FastAPI is a modern, fast (high-performance), web framework for building APIs with Python based on standard Python type hints."
},
{
"url": "https://fastapi.tiangolo.com/tutorial/",
"title": "Tutorial - User Guide - FastAPI",
"description": "This tutorial shows you how to use FastAPI with most of its features, step by step."
}
]
}Any page, straight to markdown:
$ curl -H "X-WebTools-Token: $TOKEN" \
'http://127.0.0.1:8377/extract?url=https://www.iana.org/help/example-domains'
{
"url": "https://www.iana.org/help/example-domains",
"title": "Example Domains",
"content": "# Example Domains\n\nAs described in [RFC 2606](...) and [RFC 6761](...), a number of domains such as example.com ...",
"method": "trafilatura",
"error": null,
"pipeline_steps": ["httpx", "trafilatura"],
"upstream_status": 200,
"final_url": "https://www.iana.org/help/example-domains"
}That pipeline_steps field is there on purpose. Scrapyard tells you exactly how it got the content and what the upstream server actually said. If a page returned a 500, you'll know — it never gets dressed up as a success.
Python 3.12 or newer. Install the published package and its Chromium runtime:
python3.12 -m pip install scrapyard
python3.12 -m playwright install chromium
scrapyardThe first run creates a mode-0600 token at
~/.config/scrapyard/token and stores the search cache beneath
~/.local/share/scrapyard/. Chromium is a required post-install step for
JavaScript-rendered extraction; pip cannot install the browser binary.
Scrapyard runs on 127.0.0.1:8377 by default.
For a source checkout instead:
git clone https://github.com/artboarding-hash/scrapyard.git && cd scrapyard
scripts/install.sh
scripts/run.shThe source installer creates .venv, installs the exact dependency pins and
Chromium, and writes a checkout-local token.
/search — Real web search with automatic engine fallback. If one engine goes down, the next one picks up. Results are cached, and type=news gives you date-sorted news.
/extract — Any URL to clean markdown. Static pages, JavaScript-rendered pages, and PDFs all work. It also pulls out the author, publication date and site name when the page provides them.
/map — Discover the URLs on a site via robots.txt and sitemap.xml. Fast, polite, and it reads only what sites publish for exactly this purpose.
/research — Search and extract the top results in a single call, so your agent gets a usable answer in one round trip instead of five.
/status — Per-engine health, cache stats, and what's working right now.
Search engines change their HTML. Every scraper eventually breaks — the difference is whether you find out from your monitoring or from a failed task.
Scrapyard ships a daily canary that checks every engine and every extraction path independently. It stays quiet when everything's fine, and when something breaks it tells you which piece broke.
And it comes with the tools to fix it:
scripts/check_upstream.py # often it's already fixed upstream
scripts/diagnose.py # a ready-to-paste repair brief with the real failure data
scripts/repair.sh # takes a backup, hands you the brief
scripts/verify_repair.py # the gate: security suite + canary + a live checkHand the brief to Claude Code, Codex, Cursor, whatever you use. The verification step is what makes the fix trustworthy — nothing is accepted until the security tests pass, the canary is green, and the engine returns real results again.
All on your machine, with your agent, on your budget. Nothing is reported anywhere.
Scrapyard fetches URLs you hand it, so it's careful about where those point:
HTTP and HTTPS only, ports 80 and 443
Private, loopback, link-local and IPv6 internal addresses are refused
Every redirect is re-checked, so a redirect can't sneak into your network
Download, PDF and output size caps
/map filters unsafe URLs out of its own results
Constant-time token comparison on every route
Backed by a 24-case test suite that runs against the live service:
python tests/test_security.py # 17 passed
python tests/test_security_extra.py # 7 passedSECURITY.md has the full threat model. Keep it on loopback or a private network — it's built for you, not for the open internet.
Scrapyard makes no outbound request except the ones you ask for. No analytics, no phone-home, no usage tracking. The canary writes to a local file and nowhere else.
Scrapyard requests come from your own IP, so sites behind heavy bot protection may block it where a commercial API with a proxy pool gets through. Google is intentionally not included — it serves a JavaScript challenge that no HTML client can pass. And /map finds URLs but doesn't crawl entire sites.
For everything else — which is almost everything — it just works.
For a PyPI installation:
python3.12 -m pip install --upgrade scrapyard
python3.12 -m playwright install chromiumFor a git source checkout, use the guarded updater:
scripts/update.shIt refuses a dirty tree, backs up the repository and local SQLite databases,
pulls fast-forward-only, reinstalls the pinned dependencies, then runs the
17-case security suite and the canary. It prints an exact restore command and
never restarts a service; after a successful run, restart Scrapyard using the
command owned by your deployment. scripts/update.sh --dry-run makes the
backup and runs both verification gates without pulling or installing.
At startup Scrapyard checks the public GitHub releases/latest endpoint in a
background thread and caches a successful result on disk for at least 24 hours.
/status reports version, latest_version, update_available, and
update_check. This is a read-only GET: it sends no body, query parameters,
cookies, custom User-Agent, install ID, usage counts, or user data; the HTTP
client emits only the protocol-required Host header. It fetches only the
latest public release JSON and reads tag_name. It never applies an update.
Disable the check completely with one flag:
SCRAPYARD_UPDATE_CHECK=0 scrapyardSet SCRAPYARD_UPDATE_REPOSITORY=owner/repo if using a fork. The source default
is the publish-time placeholder artboarding-hash/scrapyard; maintainers should
replace it with the final public owner before release. Network or API failures
are reported as update_check: "failed" and never prevent service startup.
Apache-2.0 — use it, fork it, ship it. Copyright 2026 scrapyard.dev
Scrapyard ships a stdio MCP server for Claude Code, Claude Desktop, Cursor, Hermes, and other MCP-capable clients. From this checkout:
scripts/install.sh
uv pip install --python .venv/bin/python "mcp==2.0.0"
scripts/run.sh
hermes mcp add scrapyard --command "$PWD/.venv/bin/python" --args "$PWD/scrapyard_mcp/server.py"
hermes mcp test scrapyardNo token needs to be pasted into the agent config: the server reads .token by
default. See docs/MCP.md for copy-pasteable Claude Code, Claude
Desktop, Cursor, remote-service, and troubleshooting configurations.
Fetch one public URL and return clean readable markdown plus extraction provenance.
Use for a known article, documentation page, JavaScript-rendered page, or PDF. Check error
before relying on content. Preserve and cite method, pipeline_steps, upstream_status,
and final_url: they distinguish static parsing, browser rendering, PDF extraction, redirects,
and truthful upstream failures. Private-network and unsafe URLs are intentionally rejected.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Absolute public HTTP(S) URL of one HTML page or PDF to convert to clean markdown. | |
| char_limit | No | Optional maximum characters of returned content; null uses the service default. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
While annotations declare readOnlyHint=true and openWorldHint=true, the description enriches beyond that by explaining what the output contains (provenance, error handling) and how to interpret fields like method, pipeline_steps, upstream_status, and final_url. It also discloses the rejection of private-network URLs. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is four sentences, each with a distinct purpose: primary action, usage context, caution about error handling, and a note on provenance. No fluff; front-loaded with the core function.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that an output schema exists and annotations cover read-only/open-world aspects, the description covers the essential operational details: single-URL fetching, return format (markdown + provenance), error handling, and URL restrictions. Nothing critical is missing for an agent to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with both url and char_limit having descriptions. The description adds a small additional nuance by mentioning JavaScript-rendered pages, which is not in the schema's url description, but mostly reiterates the schema. Since the schema is strong, this earns a 4 rather than 3.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a clear verb-resource statement: 'Fetch one public URL and return clean readable markdown plus extraction provenance.' It further specifies the intended inputs (known article, documentation page, JavaScript-rendered page, or PDF) and implicitly differentiates from sibling tools (search, map, research, status) by focusing on a single URL extraction.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use: 'Use for a known article, documentation page, JavaScript-rendered page, or PDF.' Also provides actionable guidance like 'Check error before relying on content' and warns that private-network and unsafe URLs are rejected, which helps the agent decide and handle failures correctly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Discover a site's published URLs from robots.txt and sitemap XML without crawling pages.
Use this to inventory documentation, blog, or product URLs before selecting pages to extract.
The response reports the discovery source, total count, filtered_count, and URL/lastmod
entries. It intentionally does not follow ordinary page links or perform a full crawl.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Absolute public HTTP(S) site URL whose robots.txt and sitemap XML should be inspected. | |
| limit | No | Maximum number of safe sitemap URLs to return (1-5000). |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations include readOnlyHint=true and openWorldHint=true, so safety profile is already declared. Description adds that it intentionally does not follow links or crawl, and reports source, count, filtered_count, and entries. This adds behavioral context beyond annotations without contradicting them.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two compact paragraphs. First sentence states purpose and scope immediately. Second paragraph gives usage guidance and output details. No fluff, all sentences earn their place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present, the description need not detail return values, and it only briefly mentions the source/count fields. It effectively communicates the tool's non-crawling nature and inventory use case. Slight gap: no mention of rate limits or error behavior, but minor given annotations and schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with descriptions for both parameters. The description mentions the discovery source and counts, and while it implies the output shape, it doesn't add parameter-level syntax beyond schema. Baseline 3 is appropriate when schema covers all parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clear verb+resource: 'Discover a site's published URLs from robots.txt and sitemap XML'. Distinguishes from crawling. Sibling tools include scrapyard_extract and scrapyard_search, and description explicitly states it does not follow page links or crawl, so differentiation is clear.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
States when to use: 'inventory documentation, blog, or product URLs before selecting pages to extract'. Implies alternative scrapyard_extract but doesn't explicitly name alternatives. Provides context but no exclusions or direct comparison to siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Search and extract several domain-deduplicated sources in one bounded request.
Use when the task needs source text rather than only result snippets. The response keeps
engine_used; each source keeps method, content_source, and extraction metadata so you can
distinguish parsed content from snippet fallback and avoid overstating provenance. For exhaustive
work, search first and extract selected URLs individually instead.
| Name | Required | Description | Default |
|---|---|---|---|
| n | No | Number of domain-deduplicated sources to extract (1-5). | |
| query | Yes | Focused research question or search query used to find and extract sources. | |
| render | No | Request browser rendering for source pages when true; slower and usually unnecessary. | |
| char_limit | No | Maximum extracted characters per source (1-15000). |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint and openWorldHint, but the description adds valuable behavioral context: it explains the response structure (engine_used, per-source method, content_source, extraction metadata) and warns about overstating provenance by distinguishing parsed content from snippet fallback. This goes beyond the annotations and helps the agent use results correctly. Minor absence of rate-limit/auth notes, but for a read-only tool this is sufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact—two sentences—yet packs the core action, usage trigger, and behavioral caveat. Every sentence earns its place, and the most critical information (when to use) is front-loaded. No filler or repetition of schema details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given there is a schema for parameters and an output schema (indicated), the description need not restate return types. It still covers when to use, how it differs from siblings, and the key behavioral nuance about provenance. For a read-only, bounded tool, nothing essential is missing—an agent can confidently decide and invoke it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so each parameter (query, n, render, char_limit) already has a clear description. The tool description does not add additional semantics beyond what the schema provides, but it also doesn't conflict. Baseline 3 is appropriate because the schema carries the weight; the description offers only a high-level hint about 'bounded' which is already reflected in min/max constraints.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action ('Search and extract'), a resource ('several domain-deduplicated sources'), and a scope ('in one bounded request'). It also names sibling tools by implication, distinguishing this combined operation from separate search and extract tools. The purpose is unambiguous and differentiating.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use this tool: 'when the task needs source text rather than only result snippets.' It also provides an exclusion: 'For exhaustive work, search first and extract selected URLs individually instead,' routing the agent to an alternative approach. This is clear, direct guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Search the live web or news index and return ranked URLs, titles, and snippets.
Prefer this before extraction when you do not already know the target URL. The response
preserves engine_used and adapter_used so you can report where results came from;
cached says whether Scrapyard reused a prior response. This tool does not extract page bodies.
| Name | Required | Description | Default |
|---|---|---|---|
| type | No | Use 'web' for general results or 'news' for recent news results with dates. | web |
| limit | No | Maximum number of search results to return (1-20). | |
| query | Yes | Natural-language web search query. Include distinctive names, dates, or quoted phrases when precision matters. | |
| engine | No | Optional exact search adapter to probe. Leave null for Scrapyard's fallback chain; set only when diagnosing or requiring a specific configured engine. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations (readOnlyHint, openWorldHint) cover the read-only and open-world nature, but the description adds valuable behavioral context beyond these: it discloses that the response retains `engine_used`, `adapter_used`, and `cached` flags, enabling the agent to report provenance and caching behavior. It also explicitly states a limitation (does not extract page bodies). No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, each earning its place: purpose, usage guidance with response metadata, and a limitation. The core purpose is front-loaded in the first sentence. No filler or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's moderate complexity, the presence of an output schema (which obviates the need to detail return values), and the thorough schema descriptions, the description covers all necessary context: when to use, how it behaves, what metadata it returns, and what it does not do. An agent has everything needed to invoke correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so every parameter is already fully explained in the schema (e.g., 'type' enum, 'limit' range, 'query' natural-language advice, 'engine' fallback chain). The tool description adds no parameter-specific detail beyond what the schema provides, so the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Search'), a resource ('the live web or news index'), and the output ('ranked URLs, titles, and snippets'). It clearly distinguishes from siblings like scrapyard_extract and scrapyard_map by focusing on web/news search rather than extraction or mapping. No ambiguity about what the tool accomplishes.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use this tool: 'Prefer this before extraction when you do not already know the target URL.' It also names the alternative (extraction) and provides exclusion guidance ('This tool does not extract page bodies.'). This leaves no inference needed for an agent to decide whether to call search or reactive siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return Scrapyard version, search-engine breaker state, adapter registry, and cache counters.
Use this when search/research fails, results unexpectedly come from a fallback adapter, or you need to verify which engines are configured and healthy. This is operational status, not a web search and not merely a liveness probe.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations already declare readOnlyHint=true (safe read) and openWorldHint=false (does not depend on external world state). The description adds concrete behavioral context by listing what the tool returns (version, breaker state, registry, counters) and clarifies it is operational status, not a liveness probe. It does not dig into error conditions or rate limits, but given the read-only nature is already covered, a 4 is appropriate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is just two sentences. The first sentence front-loads the return payload; the second gives usage context. Every sentence earns its place with no fluff. The structure is excellent.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple status tool with no parameters and an output schema already provided, the description covers all necessary context: what is returned, when to use it, and how it differs from siblings. Nothing an agent needs to correctly invoke it is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so there is nothing for the description to explain about input. The description instead focuses on output, which is appropriate. The baseline for 0 params is 4, and the description does not need to add anything about parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific verb ('Return') and the exact resources: version, search-engine breaker state, adapter registry, and cache counters. It also explicitly differentiates itself from sibling tools by stating 'not a web search and not merely a liveness probe', which leaves no ambiguity about its role.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit when-to-use scenarios ('when search/research fails, results unexpectedly come from a fallback adapter, or you need to verify which engines are configured and healthy') and also what it is not for ('not a web search and not merely a liveness probe'). This gives an agent clear routing guidance relative to the sibling search/research tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Each tool has a clearly distinct purpose: search returns snippets, extract fetches full content, map discovers URLs via sitemaps, research combines search+extract, and status reports operational health. No overlap or ambiguity among them.
All tools follow a consistent 'scrapyard_<verb>' pattern using snake_case, making it easy to predict naming for future tools. The verbs are descriptive and match the tool's function.
Five tools is well-scoped for a web intelligence server, covering search, extraction, discovery, combined research, and status. Each tool earns its place without redundancy or bloat.
The toolset covers the full lifecycle of web research: discover URLs (map), search (search/research), extract content (extract/research), and verify system health (status). No obvious gaps like missing crawl functionality, and the descriptions explicitly note limitations where appropriate.
Docs: https://docs.keenable.ai/mcp-server Keenable is a free, remote MCP server that gives agents access to the web index. Search the web with ranked results and date/site filters, then fetch any indexed page as clean markdown. Works out of the box with no account or API key.
Hosted MCP server: convert PDFs to clean, LLM-ready Markdown with tables, formulas and OCR.
MCP server (stdio): fetch web pages as clean readable markdown via the AgentForge API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/artboarding-hash/scrapyard'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
