AI-native web scraper. Single binary with a bundled Claude Code skill.
Built by HighCraft.io.
macOS / Linux (Homebrew):
brew install alex-on-ai/webreaper/webreaperAny POSIX shell (install.sh):
curl -fsSL https://raw.githubusercontent.com/alex-on-ai/WebReaper/master/scripts/install.sh | sh.NET library:
dotnet add package WebReaperWindows binaries are on the GitHub Releases page; winget and Scoop are on the v10.1 roadmap.
Updating: brew upgrade webreaper (Homebrew), or re-run the install.sh line with --upgrade appended (| sh -s -- --upgrade), or dotnet add package WebReaper (library). When a newer release exists, scrape / crawl / map print a one-line upgrade hint on stderr (interactive terminals only, never in a pipe or CI); disable that check with WEBREAPER_NO_UPDATE_CHECK=1.
Related MCP server: Robot Resources Scraper
$ webreaper scrape https://news.ycombinator.com
# Hacker News
- [Show HN: ...](https://news.ycombinator.com/item?id=...)
- [Ask HN: ...](https://news.ycombinator.com/item?id=...)
...
$ webreaper init
Wrote WebReaper Agent Skill to .claude/skills/webreaper/SKILL.md
Try it out:
webreaper scrape https://example.com
webreaper map https://example.comAfter webreaper init, the next Claude Code session picks up the skill and routes scraping intents ("give me the markdown of X", "what blog posts are on Y", "scrape the top 5 articles") to webreaper automatically.
# One page as Markdown
webreaper scrape https://example.com
# Save Markdown to a file
webreaper scrape https://example.com --output page.md
# Discover URLs on a site
webreaper map https://example.com --search /blog/ --max-urls 50
# Crawl a whole site recursively (every on-domain page) to JSON Lines
webreaper crawl https://example.com > pages.jsonl
# Structured fields with a JSON schema (output: JSON; multi-page: JSON Lines)
webreaper scrape https://example.com --schema schema.json
# Schema-free extraction with an LLM (bring your own OpenAI-compatible endpoint)
webreaper scrape https://example.com --prompt "title and author" \
--model gpt-4o-mini --llm-url https://api.openai.com/v1
# Whole site, fields, cheaply: infer a schema once, then extract the rest
webreaper crawl https://example.com --infer "product name and price" \
--model gpt-4o-mini --llm-url https://api.openai.com/v1 --output-dir ./out
# JS-rendered single-page app
webreaper scrape https://example.com --browser
# Bot-protected site: a plain scrape already auto-climbs HTTP -> browser on a
# block; --stealth starts at a stealth backend (--auto-stealth = no prompt, for CI)
webreaper scrape https://example.com --stealth
# Install the Claude Code skill
webreaper initThe CLI is built Native-AOT (ADR-0043), ships as a single binary on every tagged GitHub release across six RIDs (linux-x64, linux-arm64, osx-x64, osx-arm64, win-x64, win-arm64), and is block-aware with automatic browser/stealth escalation (ADR-0083). The macOS binaries are Apple codesigned and notarized (ADR-0071); Homebrew installs run without Gatekeeper warnings on a clean machine.
using WebReaper.Builders;
var engine = await ScraperEngineBuilder
.Crawl("https://news.ycombinator.com")
.AsMarkdown()
.WriteToConsole()
.BuildAsync();
await engine.RunAsync();That is HTTP-only, no extra packages, no schema. For structured fields, swap AsMarkdown() for Extract(schema); for JS-rendered pages, Crawl for CrawlWithBrowser plus a transport satellite (WebReaper.Playwright or WebReaper.Cdp). The full surface is in the API overview.
Most scrapers make you opt into a browser up front and guess when a site is blocking you. WebReaper detects the block and escalates on its own, one page at a time:
HTTP -> browser (Chromium) -> stealth (CloakBrowser)
(climbs only when a page actually looks blocked)Automatic. A plain scrape or crawl starts on a fast HTTP fetch and climbs to a real browser only when a page looks blocked (a challenge status, response header, or body marker). No flag needed for the browser fallback.
Per page, host-sticky. The first confirmed block on a host lifts that host's floor, so the rest of a whole-site crawl starts at the working tier instead of re-paying the failed one. You pay for the climb once, not per page.
No garbage in your data. A page still blocked at the top tier is dropped, never written to your output, and the run exits non-zero so an unattended job knows. Clean data or a clear signal, never a challenge page masquerading as content.
You decide on stealth. --stealth starts at the stealth backend; --auto-stealth (or WEBREAPER_AUTO_STEALTH=1) enables it unattended; --no-auto-stealth caps the climb at a vanilla browser. The ~220 MB stealth backend downloads only if you opt in.
Self-hosted, single binary, no cloud round-trip (ADR-0083).
Onboarding an agent from zero? Hand it docs/AI-ONBOARDING.md — install → verify → choose-your-path on one page, no credentials anywhere.
webreaper initWrites a polished SKILL.md to .claude/skills/webreaper/. Claude Code loads it on the next session and routes scraping intents to the CLI: "scrape the top 5 stories on HN and summarize each", "give me the markdown of this article", "this Cloudflare-protected site is blocking me". The skill describes when to prefer webreaper over the built-in WebFetch (artifacts and structured data go through webreaper; conversational answers stay on WebFetch).
Two MCP servers expose the same tools (scrape, map, extract, extract_with_prompt, extract_inferred, crawl) for clients that speak the Model Context Protocol:
WebReaper.Mcp (ADR-0049) speaks stdio, for local clients that spawn a process: Cursor, Claude Desktop, Copilot Studio.
WebReaper.Mcp.AspNetCore (ADR-0086) speaks Streamable HTTP, for clients that connect to a URL. The headline case is n8n, whose MCP Client node is URL-only and cannot reach a stdio server. Run the Chromium-baked image, point n8n's MCP Client node at the URL with a bearer token, and call the tools from a workflow:
docker run -p 8080:8080 -e WEBREAPER_MCP_TOKEN=your-secret \
ghcr.io/alex-on-ai/webreaper-mcp-http:latestSee the n8n quickstart.
Both are thin facades over the library API; the primary agent surface remains the CLI.
The single binary works inside any shell-spawning agent: LangChain ShellTool, OpenAI Assistants code-interpreter, GitHub Actions, internal scripts. Zero runtime to install; one syscall to invoke.
await ScraperEngineBuilder
.Crawl("https://example.com")
.AsMarkdown() // ADR-0040 + ADR-0063
.WriteToConsole()
.BuildAsync()
.Result.RunAsync();The Markdown extractor (HtmlToMarkdown primitive, ADR-0063; MarkdownContentExtractor adapter, ADR-0040) emits {title, markdown} per page. Pass the output straight into a follow-up LLM prompt.
using WebReaper.Extraction.Attributes;
[ScrapeSchema]
public partial class Article
{
[ScrapeField("h1")] public string? Title { get; set; }
[ScrapeField(".views", Type = SchemaFieldType.Integer)] public int Views { get; set; }
[ScrapeField(".tag", IsList = true)] public List<string> Tags { get; set; } = new();
}
// Emitted at compile time, reflection-free, AOT-clean:
// public static Schema Schema { get; }
// public static Article Materialize(JsonObject json)
var engine = await ScraperEngineBuilder
.Crawl("https://example.com/post")
.Extract(Article.Schema)
.Subscribe(p => HandleArticle(Article.Materialize(p.Data)))
.BuildAsync();The WebReaper.Extraction.Generators Roslyn analyzer (ADR-0045) emits the schema and a Materialize function. Schema typos are compile errors; the generated path uses no reflection so it's AOT-clean.
Three composable patterns where the LLM only fires when the deterministic path can't deliver. Same proposer-validator shape across all three.
using WebReaper.AI;
// (a) Fire LLM only when a field returns empty (ADR-0046)
.WithLlmFallback(chatClient)
// (b) Repair a broken selector once per (Schema, field), cache forever (ADR-0047)
.WithLlmSelfHealing(chatClient)
// (c) No schema at all: infer it from the URL, re-infer on validator failure (ADR-0067 + ADR-0069)
.UseAi(chatClient, AiPolicyMode.Inferred)Stable pages cost zero LLM calls; broken pages cost one call per (page, field) and cache. Schema inference is one call per site (cached for the run). The WebReaper.AI satellite is built on Microsoft.Extensions.AI so any IChatClient works.
using WebReaper.AI;
var result = await LlmAgent.RunAsync(
"https://example.com",
goal: "Find the contact email and phone number for the support team.",
chatClient);AgentEngine (ADR-0051) runs a sequential decide → persist → execute loop over a closed-sum AgentDecision (Extract | Follow | Act | Stop). The brain picks each step from the bounded AgentState view; the engine validates (visited-link enforcement, MaxSteps cap), persists (IAgentRunStore), and dispatches. Durable resume across process restarts.
ScraperEngineBuilder
.CrawlWithBrowser(url, actions => actions
.Do(PageAction.SemanticAct("click 'sign in'")) // ADR-0050
.Do(PageAction.WaitForNetworkIdle())
.Build())
.WithLlmActionResolver(chatClient)
// ...A natural-language PageAction.SemanticAct(intent) is one of the ten closed-sum arms (ADR-0050 added it; ADR-0074 added Fill / Press / ScrollIntoView for form interactions). The transport resolves it once via the registered IActionResolver (LLM-backed by default), dispatches the concrete arm, and caches the resolution per crawl by intent string. First page pays the LLM, subsequent same-intent pages dispatch the cached arm with no LLM call.
Examples/WebReaper.AiNativeShowcase wires every feature in this section:
dotnet run --project Examples/WebReaper.AiNativeShowcase -- markdown
dotnet run --project Examples/WebReaper.AiNativeShowcase -- sourcegen
dotnet run --project Examples/WebReaper.AiNativeShowcase -- llm
dotnet run --project Examples/WebReaper.AiNativeShowcase -- router
dotnet run --project Examples/WebReaper.AiNativeShowcase -- changetrackBuild LLM context from blog or docs sites. webreaper map plus webreaper scrape per URL, piped into a prompt or a vector DB.
Monitor competitor pricing or status pages for changes. Schedule the CLI with cron or a worker, store records in MongoDB or SQLite, plug in .WithChangeTracking() (ADR-0048) so the sink fires only on diff. Hash-based dedup; cron-friendly.
Run an autonomous research agent. LlmAgent.RunAsync(url, goal, chatClient) decides which links to follow until the goal is met. Durable resume across restarts.
Scrape Cloudflare-protected catalogs. A plain scrape auto-climbs HTTP to a browser on a block; add --stealth (or --auto-stealth for unattended runs) to escalate to a stealth backend. Blocked pages are dropped, never emitted as challenge-page garbage.
Generate clean datasets from semi-structured pages. [ScrapeSchema] POCO plus the source generator; reflection-free, AOT-compiles into a native binary.
Embed a scraping primitive in your own app. dotnet add package WebReaper; the public registration seam lets you plug Redis, Cosmos DB, your own sink.
The release ships fifteen packages (one core, fourteen satellites), all versioned in lockstep. The core stays dependency-light and Native-AOT-publishable with zero warnings; satellites bring their own SDK dependencies and quarantine them off the core graph (ADR-0009).
Package | Add it for | Key builder calls |
WebReaper | Core. HTTP crawl and parse, in-memory and file scheduler / visited-link tracker / cookie and config storage, Console / CSV / JSON-Lines sinks, Markdown extractor, schema fold. Dependency-light, Native-AOT-ready, Newtonsoft-free. |
|
WebReaper.Cdp | Raw CDP |
|
WebReaper.Playwright | Microsoft.Playwright-backed transport (ADR-0053). Multi-browser (Chromium default; Firefox / WebKit opt-in). All ten |
|
WebReaper.Stealth.CloakBrowser | First stealth-backend satellite (ADR-0054). Auto-downloads CloakBrowser on first use; composes on |
|
WebReaper.AI | LLM extraction, LLM action resolver, LLM brain, LLM self-healing, LLM schema inferrer (ADR-0044 / 0050 / 0051 / 0067). Built on |
|
WebReaper.AI.Http | AOT-safe OpenAI-compatible |
|
WebReaper.Extraction.Attributes | The |
|
WebReaper.Extraction.Generators | Roslyn source generator that emits | compile-time only |
WebReaper.Mcp | MCP server | the package is the executable |
WebReaper.Mcp.AspNetCore | MCP server over Streamable HTTP (ADR-0086) for URL-based clients like n8n; same tools as | the package is the executable; or |
WebReaper.Mongo | MongoDB result sink and MongoDB-backed config / cookie storage. |
|
WebReaper.Redis | Redis scheduler, visited-link tracker, result sink, config / cookie storage. |
|
WebReaper.AzureServiceBus | Distributed scheduler over an Azure Service Bus queue. |
|
WebReaper.Cosmos | Azure Cosmos DB result sink. |
|
WebReaper.Sqlite | Local durable scheduler and visited-link tracker on an embedded SQLite store; resume is a query, no position file. Opt-in robust-local tier (no server, unlike Redis). |
|
WebReaper.Cli (the AOT single-binary; ADR-0043) is not a NuGet package; it ships as platform binaries on every GitHub release (Native-AOT plus dotnet tool install are mutually incompatible on one target). Install via Homebrew or install.sh, or build from source.
WebReaper | Firecrawl | Crawl4AI | WebFetch (Claude) | |
License | MIT | AGPL-3.0 (plus commercial) | Apache 2.0 | bundled with Claude |
Install | one binary, ~12 MB | Docker + Postgres + Redis (self-host) or hosted | Docker + Python + Playwright | nothing to install |
Cost | free | metered API plus free tier | free | included with Claude |
BYO LLM | any | no (their model) | yes (LiteLLM) | Claude only |
Autonomous agent |
|
| code it yourself | not available |
Whole-site crawl |
|
| deep-crawl strategies (code it yourself) | no (single fetch) |
Page actions | 10 declarative arms: | 9 actions: | JS hooks; no closed-sum vocabulary | none (single-fetch only) |
Bot-protected | automatic HTTP → browser → stealth climb, per page, host-sticky, self-hosted | cloud yes; self-host degraded (no Fire-engine) | BYO | no |
Claude Code skill |
| community | none official | not applicable |
Crawlee (Apify's Node/Python library) is also worth knowing; it covers similar ground to the WebReaper library API but doesn't ship a binary, a Claude Code skill, or a built-in LLM safety net. Use it if you're already in the Apify ecosystem.
The closest reference is Firecrawl: same AI-native positioning, opposite distribution shape. Firecrawl optimises for the hosted-API flow; WebReaper optimises for the local-binary flow. If you want a managed cloud with someone else's proxies and infra, Firecrawl is the buy. If you want a binary that runs locally with your own LLM key and no metering, WebReaper is the build.
The library is a fluent builder over a small set of seams. For the deep seam-by-seam reference (interfaces, main entities, custom sinks), see docs/architecture.md.
using WebReaper.Builders;
var engine = await ScraperEngineBuilder
.Crawl("https://www.alexpavlov.dev/blog")
.Extract(new()
{
new("title", ".text-3xl.font-bold"),
new("text", ".max-w-max.prose.prose-dark")
})
.Follow("a.text-gray-900.transition")
.WriteToJsonFile("output.json")
.PageCrawlLimit(10)
.WithParallelismDegree(30)
.LogToConsole()
.BuildAsync();
await engine.RunAsync();Each new("field", "css-selector") is a leaf; nest schemas for objects, set IsList = true for arrays, set Attr = "href" to read an HTML attribute instead of inner text.
Get the scraped records back in your own process, no file and no custom sink. Pass Subscribe (ADR-0038) the Add of a thread-safe collection, then read them after the run:
using System.Collections.Concurrent;
using WebReaper.Builders;
using WebReaper.Sinks.Models;
var records = new ConcurrentBag<ParsedData>();
var engine = await ScraperEngineBuilder
.Crawl("https://news.ycombinator.com")
.AsMarkdown()
.Subscribe(records.Add) // called concurrently; collect into a thread-safe type
.BuildAsync();
await engine.RunAsync();
foreach (var record in records)
Console.WriteLine($"{record.Url}: {record.Data["markdown"]?.GetValue<string>()?.Length ?? 0} chars");The Crawl driver fans out to sinks concurrently, so a ConcurrentBag is the right collector, not a bare List. For structured fields, read your schema keys out of record.Data instead of "markdown".
For JS-rendered pages, swap Crawl for CrawlWithBrowser and register a browser transport. Two satellites are available.
WebReaper.Playwright is the modern default (ADR-0053): multi-browser, all ten PageAction arms.
using WebReaper.Builders;
using WebReaper.Playwright;
await ScraperEngineBuilder
.CrawlWithBrowser("https://example.com")
.Extract(new() { new("title", "h1") })
.WithPlaywrightPageLoader()
.BuildAsync();WebReaper.Cdp is the AOT-clean alternative (ADR-0052): raw CDP over System.Net.WebSockets, AOT-publishable. Use for AOT consumers or as the base for stealth backends.
using WebReaper.Cdp;
.WithCdpPageLoader(new CdpLaunchOptions { Headless = true })
// or .WithCdpPageLoader(cdpUrl: "http://localhost:9222") for BYO browserFor bot-protected sites, layer WebReaper.Stealth.CloakBrowser (ADR-0054) on top of WebReaper.Cdp:
using WebReaper.Stealth.CloakBrowser;
.WithCloakBrowser() // auto-downloads CloakBrowser on first use, ~220 MBFor visible-browser debugging, add .HeadlessMode(false).
Drive the page as it loads. Pass an actions lambda.
using WebReaper.Builders;
using WebReaper.Playwright;
using WebReaper.Domain.PageActions;
await ScraperEngineBuilder
.CrawlWithBrowser("https://www.reddit.com/r/dotnet/", actions => actions
.ScrollToEnd()
.Build())
.Extract(new() { new("title", "h1") })
.WithPlaywrightPageLoader()
.BuildAsync();PageActionBuilder exposes Click, Wait, ScrollToEnd, ScrollIntoView, WaitForSelector, WaitForNetworkIdle, EvaluateExpression, Fill, Press, SemanticAct, Repeat / RepeatWithDelay, and Build(). Fill(selector, value) / Press(key) / ScrollIntoView(selector) (ADR-0074) carry an implicit 30 s auto-wait and use the React-friendly native-setter trick on the CDP transport, so controlled components in React / Vue / Svelte observe the change. SemanticAct (ADR-0050) accepts a natural-language intent and resolves it via the registered IActionResolver (see AI features §5).
Survive kill -9 and resume across restarts. Two adapters: file-backed (zero deps, in core) and SQLite-backed (durable, opt-in via WebReaper.Sqlite).
using WebReaper.Builders;
using WebReaper.Sqlite;
await ScraperEngineBuilder
.Crawl("https://example.com")
.Extract(new() { new("name", "h1") })
.Follow(".forumlink>a")
.Paginate("a.torTopic", ".pg")
.WriteToJsonFile("result.json")
.WithSqliteScheduler("crawl/state.db") // resume is a query, not a position file
.TrackVisitedLinksInSqlite("crawl/state.db") // the table is the set
.BuildAsync();Pass dataCleanupOnStart: true to any sink, tracker, or scheduler method to wipe its store at start (note: WriteToJsonFile defaults this to true; the others default to false).
If the site needs cookies, call SetCookies and fill the container. You perform the login yourself.
using System.Net;
using WebReaper.Builders;
await ScraperEngineBuilder
.Crawl("https://example.com/protected")
.Extract(new() { new("name", "h1") })
.SetCookies(cookies =>
{
cookies.Add(new Cookie("AuthToken", "123"));
})
.BuildAsync();Swap the scheduler, config storage, and link tracker to Redis or Azure Service Bus; multiple workers or serverless functions share one crawl. Examples/WebReaper.AzureFuncs shows the serverless shape (two functions: StartScraping seeds the work, WebReaperSpider is the distributed Crawl driver). Examples/WebReaper.DistributedScraperWorkerService shows the worker-service shape.
DistributedSpiderBuilder.BuildSpider() returns a bare ISpider without a Crawl seed (ADR-0009 / ADR-0025: "two seams, not one bug" split). The worker's config is persisted separately by the start endpoint.
Every backend is a swappable seam. In-memory is the default; file-backed lives in core; the rest come from satellites.
Seam | Core (in-memory default + file) | Satellite options |
Scheduler | in-memory, |
|
Visited-link tracker | in-memory, |
|
Config storage | in-memory, |
|
Cookie storage | in-memory, |
|
Agent run store | in-memory, file (ADR-0051) |
|
Result sink |
|
|
Page loader transport | HTTP (default) |
|
Custom sinks, the full interface index, and the main domain entities live in docs/architecture.md.
Project | Description |
| The core library (the |
| Raw CDP transport satellite (ADR-0052). |
| Microsoft.Playwright transport satellite (ADR-0053). |
| First stealth-backend satellite (ADR-0054). |
| LLM extraction, action resolver, agent brain, self-healing, schema inferrer. |
|
|
| Roslyn source generator (ADR-0045). |
| MCP server satellite (ADR-0049). |
| MongoDB sink plus config / cookie storage. |
| Redis scheduler, tracker, sink, config / cookie storage. |
| Azure Service Bus distributed scheduler. |
| Azure Cosmos DB sink. |
| Local durable scheduler and visited-link tracker over embedded SQLite. |
| AOT single-binary CLI (ADR-0043). |
| Using WebReaper in a console application. |
| Runnable demos for every AI feature in this README. |
| Demos for ADR-0067 / 0068 / 0069 schema inference. |
| Using WebReaper in a .NET Worker Service. |
| Distributed crawl across workers sharing crawl state. |
| Serverless crawl with Azure Functions plus Azure Service Bus. |
| A real-world e-commerce scraper example. |
| Example proxy-provider implementations. |
WebReaper is MIT-licensed (ADR-0017). All NuGet packages plus the WebReaper.Cli binary ship under the same terms. Use it commercially, embed it in proprietary software, fork it, modify it, redistribute it; the only ask is that you keep the copyright notice.
Prior to the 10.0.0 wave, WebReaper was GPL-3.0-or-later. The relicense is strictly more permissive: every existing user is unaffected; new users who couldn't embed under GPL now can. Historical contributors are credited in CONTRIBUTORS.md. See docs/adr/0017-relicense-gpl-mit.md for the analysis and contributor consent path.
Contributions are welcome under the same MIT terms; sign-off via DCO (CONTRIBUTING.md).
Built by HighCraft.io — a software and AI engineering team.
Crawl a whole site: recursively follow on-domain links from the start URL and return one Markdown record per page as JSON Lines. WARNING: this is a single long BLOCKING call, bounded by max_pages (default 50, hard cap 1000). It emits MCP progress notifications per page for clients that render them (e.g. Claude Desktop); blocking clients like n8n just wait for the result. For a large site prefer 'map' to list URLs, then 'scrape' each URL, so every call stays short and you keep per-URL control.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The site root URL to crawl. | |
| browser | No | Use the headless browser for each page (JS-rendered sites). Default false. | |
| maxPages | No | Maximum pages to sweep. Default 50, hard cap 1000. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses critical behavioral traits: single long blocking call, MCP progress notifications, bounded max pages (default 50, hard cap 1000). Since no annotations are provided, the description fully carries the burden of transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise yet packed with essential information. It is front-loaded with purpose, followed by warnings and alternatives. Every sentence earns its place without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (blocking crawl, progress notifications, resource-heavy), the description adequately covers the blocking nature, bounds, progress reporting, and alternative approach. The return format is stated as 'Markdown record per page as JSON Lines', which is sufficient. Minor gaps in error handling and rate limiting are acceptable for this context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% coverage with descriptions for all 3 parameters. The description reinforces the maxPages cap and default but does not add new semantic information beyond what the schema already provides. Baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb ('crawl'), resource ('whole site'), and outcome ('return one Markdown record per page as JSON Lines'). It explicitly distinguishes from siblings like 'map' and 'scrape' by suggesting an alternative approach for large sites.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit when-to-use guidance (crawl entire site) and when-not-to ('For a large site prefer map to list URLs, then scrape each URL'). It also warns about the blocking nature and gives an alternative strategy.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract structured fields from a URL using a JSON schema. The schema mirrors the WebReaper Schema shape: { field, children: [ { field, selector, type, is_list }, ... ] }. Returns the extracted record(s) as JSON Lines.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to extract from. | |
| browser | No | Use the headless browser. Auto-spawns a system Chrome / Chromium / Edge via WebReaper.Cdp; install a Chromium-family browser on the MCP host first. Default false. | |
| schemaJson | Yes | The extraction schema as JSON. See the WebReaper docs for the shape. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description reveals it uses WebReaper and returns JSON Lines, but lacks details on error handling, rate limits, or side effects. The `browser` parameter behavior is documented in parameters but not in the main description. Without annotations, the description carries the disclosure burden but is only partially transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the core purpose. Every sentence adds value—defining inputs, schema format, and outputs. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and moderate complexity, the description covers inputs, schema format, and output format. It does not mention potential failure modes or prerequisites (e.g., browser installation), but the parameter description fills in the browser requirement. Overall, it is adequately complete for basic use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, and the description adds clarity by explaining the schema shape and referencing WebReaper docs. This goes beyond the schema's property descriptions, which are mostly self-explanatory. The browser parameter's default behavior is noted in the description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Extract structured fields from a URL using a JSON schema.' It specifies the schema shape (WebReaper) and return format (JSON Lines), distinguishing it from sibling tools like `extract_inferred` and `extract_with_prompt`.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when you have a JSON schema, but does not explicitly state when to choose this over siblings or provide exclusions. No guidance on prerequisites (e.g., need for Chromium if using browser parameter), though that is in the parameter description.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract structured data from a URL WITHOUT writing a schema: an LLM infers the schema from the page (optionally steered by a goal), then WebReaper extracts deterministically. Cheaper and more consistent than extract_with_prompt across similarly shaped pages. Requires an OpenAI-compatible LLM endpoint on the host: WEBREAPER_LLM_MODEL + WEBREAPER_LLM_BASE_URL, key in WEBREAPER_LLM_API_KEY (or OPENAI_API_KEY). Returns the extracted record(s) as JSON Lines.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to extract from. | |
| goal | No | Optional goal to steer the inferred schema (e.g. "product name and price"). | |
| model | No | Optional model id, overriding WEBREAPER_LLM_MODEL for this call. | |
| browser | No | Use the headless browser (for JS-rendered pages). Default false. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden. It discloses the use of LLM for schema inference, deterministic extraction, dependencies on environment variables, and the cost/consistency benefits. It does not mention potential failure modes, but overall provides good behavioral insight beyond just the basic purpose.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise (3 sentences) and front-loaded with key information: the core behavior, comparison to alternatives, and configuration requirements. Every sentence adds value with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the tool's purpose, mechanism, comparison to a sibling, and configuration needs. It mentions the output format (JSON Lines). However, it does not address what happens on failure (e.g., if LLM call fails or page is inaccessible) or provide details on error handling. Minor gap for a tool with no output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%; the schema already describes each parameter. The description mentions the goal parameter ('optionally steered by a goal') but does not add significant meaning beyond what the schema provides. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool extracts structured data from a URL without writing a schema, using an LLM to infer the schema. It distinguishes itself from sibling tools, especially extract_with_prompt, by noting it's cheaper and more consistent for similarly shaped pages.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context on when to use this tool (when no schema is needed, for similarly shaped pages) and mentions requirements (OpenAI-compatible LLM endpoint). It explicitly compares to extract_with_prompt but does not address other siblings like crawl, scrape, or map, nor does it state when not to use this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract structured data from a URL with an LLM, using a natural-language instruction instead of a CSS schema (e.g. "each person's name, title, and email"). Returns the extracted record(s) as JSON Lines. Requires an OpenAI-compatible LLM endpoint configured on the MCP host: set WEBREAPER_LLM_MODEL and WEBREAPER_LLM_BASE_URL (e.g. https://api.openai.com/v1 or http://localhost:11434/v1), with the API key in WEBREAPER_LLM_API_KEY (or OPENAI_API_KEY). The optional model parameter overrides WEBREAPER_LLM_MODEL for this call. Costs one LLM call.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to extract from. | |
| model | No | Optional model id, overriding WEBREAPER_LLM_MODEL for this call (e.g. gpt-4o-mini). The API key is never a parameter; it stays in the environment. | |
| prompt | Yes | Natural-language description of the data to extract. | |
| browser | No | Use the headless browser (for JS-rendered pages). Auto-spawns a system Chrome / Chromium / Edge via WebReaper.Cdp. Default false. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully discloses behavior: uses LLM, costs one call, returns JSON Lines, requires specific environment variables, optional browser flag. Does not mention error handling, but sufficient for selection.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Concise and well-structured: first sentence states purpose, then configuration details, then optional parameters. No fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description covers return format (JSON Lines), cost, and all prerequisites. It is complete for an understanding of tool capabilities and requirements.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but description adds value beyond schemas: model parameter includes override explanation and API key clarification; browser parameter adds 'auto-spawns system Chrome'; prompt parameter includes example. Justifies score above baseline 3.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool extracts structured data using natural-language instructions, distinguishing it from sibling 'extract' which uses a CSS schema. The verb 'Extract' and resource 'URL with an LLM' are specific.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly describes when to use (natural-language extraction) and provides configuration prerequisites (LLM setup). Mentions optional model override. Lacks explicit when-not-to-use for siblings like extract_inferred, but still clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Discover URLs on a site via sitemap.xml + root-page link extraction. Returns a newline-separated list of URLs.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The site root URL. | |
| search | No | Optional case-insensitive substring filter on the returned URLs (e.g. "/blog/"). | |
| maxUrls | No | Cap on the number of URLs to return. Default 1000. | |
| allowOffsite | No | Keep off-site URLs in the result. Default false. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden for behavioral traits. It discloses the method (sitemap + link extraction) and output format (newline-separated list). However, it does not cover potential issues like rate limits, authentication, or behavior when sitemap is missing. The schema covers parameters well, but behavioral context is sparse.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise at one sentence, front-loading the core action and output. Every word serves a purpose with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 4 well-documented parameters, no output schema, and no annotations, the description provides enough information to use the tool effectively. It explains the output format, which is not in the schema. However, it lacks guidance on when to use this tool versus siblings, slightly reducing completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with descriptions for all 4 parameters. The description adds minimal additional meaning beyond the schema, only mentioning the newline-separated output format. It does not elaborate on parameter usage or constraints beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Discover URLs on a site via sitemap.xml + root-page link extraction.' It specifies the method and output format, distinguishing it from sibling tools like 'crawl' or 'scrape' which have different behaviors.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not provide explicit when-to-use or when-not-to-use guidance compared to siblings. While it mentions the method, it lacks context on when 'map' is preferred over 'crawl' or 'extract'. The sibling tool names are given but no differentiation is described.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fetch a URL and return its main content as LLM-ready Markdown. The lowest-cost call against any site. Useful for reading a page into context.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to scrape. | |
| browser | No | Use the headless browser (for JS-rendered pages). Auto-spawns a system Chrome / Chromium / Edge via WebReaper.Cdp; install a Chromium-family browser on the MCP host first. Default false. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Without annotations, the description carries the behavioral disclosure burden. It adds the claims of being the 'lowest-cost call' and returning 'main content as Markdown,' which provide some insight. However, it omits important behaviors like handling redirects, size limits, or error scenarios. The parameter descriptions complete the picture for the browser parameter, but the tool description itself lacks comprehensive behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise: two sentences with no wasted words. The first sentence directly states the action and output; the second adds a cost claim and use case. Every sentence is meaningful and earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with two parameters, no output schema, and no annotations, the description covers the core purpose and output format. It mentions the key extra feature of cost. However, it could be more complete by hinting at limitations or example use cases. Still, it is fairly complete for its complexity level.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The tool description does not add meaning beyond the parameter descriptions in the schema (url and browser). It repeats the general idea of fetching a URL but provides no new semantic detail.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly specifies the action ('Fetch a URL'), the resource ('return its main content'), and the output format ('LLM-ready Markdown'). It also distinguishes this tool as the 'lowest-cost call,' which differentiates it from sibling tools like crawl or extract.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for simple page reading ('useful for reading a page into context'), but provides no explicit guidance on when to use this tool versus alternatives (e.g., crawl for multiple pages, extract for structured data). No when-not-to-use information is given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
v0.1.0crawlextractextract_inferredextract_with_promptmapscrapeEach tool has a clearly distinct purpose: crawl for whole-site recursion, map for URL discovery, scrape for single-page Markdown, and three extraction tools that differ by whether a schema is provided, inferred via LLM, or prompted.
All tool names are lowercase verbs with underscores for compound terms (extract_inferred, extract_with_prompt), forming a predictable verb_descriptor pattern.
Six tools cover the core web scraping workflow (crawl, map, scrape, plus three extraction methods) without being excessive or insufficient.
The set covers crawling, scraping, URL discovery, and three extraction approaches, but lacks a bulk extraction or export tool, though it's not a critical gap.
Unclaimed servers have limited discoverability.
If you are the server author, to access and configure the admin panel.
Web scraping for AI agents. Converts URLs to clean, LLM-ready Markdown with anti-bot bypass.
Turn any URL into clean Markdown and structured data. Scrape, crawl, search and extract.
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Fetch any URL and get clean Markdown. Web scraping for AI agents.
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/alex-on-ai/WebReaper'
If you have feedback or need assistance with the MCP directory API, please join our Discord server