Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@reapfield scrape https://example.com with fields title, price"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
(No PyPI badge yet — nothing is published there. See Install.)
Give it a URL and a plain-language field spec, get structured JSON back — including on pages that only render under JavaScript.
$ git clone https://github.com/PedroHenriqueNS/reapfield.git && cd reapfield
$ uv sync
$ export ANTHROPIC_API_KEY=sk-...
$ uv run reapfield https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html \
--fields "title, price:float, in_stock:bool"
reapfield: mode=one records=1 llm_calls=1
{
"title": "A Light in the Attic",
"price": 51.77,
"in_stock": true
}That first run cost one LLM call — this page has no JSON-LD or other structured data, so
reapfield had to derive selectors. Run the identical command again and it's free:
$ uv run reapfield https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html \
--fields "title, price:float, in_stock:bool"
reapfield: mode=one records=1 llm_calls=0
{
"title": "A Light in the Attic",
"price": 51.77,
"in_stock": true
}Every one-off scraper gets written twice: once to find the selectors by hand, and again when the site changes.
reapfield pays an LLM once per field per domain to discover the CSS selectors, caches
them, and replays them deterministically forever after. It goes back to the LLM only when a
cached selector actually stops working. Steady-state cost is zero LLM calls — that
llm_calls=1 → llm_calls=0 transition above is the whole design in one line.
Before it ever considers an LLM it tries the free paths in order: JSON-LD, OpenGraph and
<meta>, __NEXT_DATA__, __NUXT__, inline JSON, then config-pinned selectors, then the
cache. Plenty of pages never reach the paid step at all.
Not a goal: defeating anti-bot systems. When a site refuses automated access,
reapfield names the system and stops.
Related MCP server: mcp-internet
From source — this is the path that actually works today:
git clone https://github.com/PedroHenriqueNS/reapfield.git
cd reapfield
uv sync
uv run playwright install chromium # only needed for JS-rendered pages
export ANTHROPIC_API_KEY=sk-... # only needed the first time a domain is scrapedEverything below assumes you're in that cloned directory, running commands with uv run.
Not yet on PyPI. pip install reapfield, uv tool install reapfield and
uvx reapfield are the intended install path once a release ships, but nothing is
published there yet — the commands above are correct on first release and dead until then.
reapfield <url> --fields "title, price:float, in_stock:bool"
[--format json|jsonl|csv] [--one | --many]
[--strict] [--refresh] [--no-llm] [--max-llm-calls N]
[--no-cache] [--cache-ttl SECONDS]
[--scroll N] [--paginate N] [--allow-private]Types are optional and inline: price:float, in_stock:bool, count:int. Unannotated
fields are strings. Field names are arbitrary — --fields "reactor_id, coolant_temp_c:float"
takes exactly the same code path as title, price.
Exit codes: 0 at least one field extracted · 1 nothing extracted, or --strict with any
miss, or blocked, or robots-disallowed, or an --one/--many conflict · 2 usage error.
A field that could not be found comes back as null, with the reason on stderr. That is a
result, not a crash.
There is no TTL on a selector. A selector that still works is still correct, and expiring it just buys LLM calls.
A cached selector is dropped and re-derived when it matches nothing, or when the value
it returns fails the declared type — price:float suddenly yielding "Add to basket"
means the page moved under us. That second case is the interesting one: nothing 404'd and
nothing expired, the selector simply started pointing at the wrong node.
Same core as the CLI, exposed over stdio. Run this from the cloned project directory so
uv run resolves to this checkout:
claude mcp add reapfield --scope local -- uv run reapfield-mcpTools: scrape, list_cached_selectors, refresh_selectors, prepare_issue_report. The
last one drafts a bug report about reapfield itself and returns a link — it never submits;
a human opens the link.
CLI URLs come from you; MCP URLs come from a model that may be acting on text it read off a
web page. That is why private, loopback and link-local addresses are blocked by default
on the CLI too, not only over MCP — including on redirects. A redirect target is chosen
by the server you were pointed at, not by whoever typed the original URL, so it earns no
more trust: redirects are followed by hand, capped at 5 hops, with this check, robots.txt,
rate limiting and the gated-platform guard re-applied at every hop. 169.254.169.254,
the cloud metadata endpoint, is the case that matters most. Checking the resolved IP
rather than the hostname string is what stops DNS rebinding.
Opt out with --allow-private on the CLI (for scraping localhost or a LAN host during
development) or REAPFIELD_MCP_ALLOW_PRIVATE=1 for MCP.
~/.config/reapfield/config.toml, then ./reapfield.toml overriding it per key.
concurrency = 4
user_agent = "reapfield/0.1 (+https://github.com/PedroHenriqueNS/reapfield)"
[domains."books.toscrape.com"]
fetcher = "http" # auto | http | browser
rate_limit = 2.0 # req/sec; robots.txt Crawl-delay is still a floor
wait_for = ".product_main" # browser mode only
pagination = "li.next > a"
[domains."books.toscrape.com".selectors]
price = ".price_color" # pinned: never derived, never touched by --refresh
_row = "article.product_pod" # the repeating container, for --many
[contribute]
reports = "ask" # ask | never -- see belowPinned selectors live in a separate store from the cache, so --refresh can never
overwrite a decision you made by hand. Append @attribute to read one: "h3 a@title".
The MCP server can invite a connected agent to draft a bug report about your session (see
prepare_issue_report above). [contribute] reports = "never", or the equivalent
REAPFIELD_ISSUE_REPORTS=never env var, turns that off entirely; the env var wins if both
are set. Default is "ask".
robots.txt is obeyed, with no override flag. Every domain is rate limited, and a
Crawl-delay in robots.txt always wins over the config. The user agent is honest and
identifiable, never randomized.
uv run pytest # offline: no network, no API keyThe suite stubs the derivation function, so nothing in it can reach Anthropic or the
network. See SPEC.md for the full design and docs/adapters.md for the adapter seam.
See CONTRIBUTING.md. Security issues go through SECURITY.md — please do not open public issues for them. Release history is in CHANGELOG.md.
Show the CSS selectors already learned for a domain. Costs nothing.
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses a key behavioral trait (costs nothing) and is clearly read-only via 'Show,' but it doesn't explain what happens when no selectors exist or if authentication is needed. Adequate but not rich.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is one short sentence with an added cost hint, both earning their place. It's front-loaded and efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple list tool with an output schema and a single parameter, the description covers the core purpose and cost behavior. It doesn't need to detail return values due to the output schema. Minor gap: no mention of empty results or preconditions, but acceptable for simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has zero description coverage for the 'domain' parameter. The description only repeats 'domain' without providing format, examples, or constraints, so it adds little meaning beyond the parameter name.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool shows CSS selectors already learned for a domain, using a specific verb ('Show') and resource. It distinguishes from siblings like scrape and refresh_selectors by focusing on cached selector lookup.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description notes that the operation 'costs nothing,' implying it's a low-cost read operation. While it doesn't explicitly name alternatives, this context suggests it can be used to check cache before more expensive operations. Could be more explicit, hence 4.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Prepare a bug report about reapfield itself. Does NOT submit it.
Call this only when you hit a real bug in reapfield during real work and
have verified a fix. It searches existing issues first; if `duplicate` is
true, stop. Otherwise show the user `body` and ask for authorization -- they
open `submit_url` themselves.
Never include credentials, private URLs, or personal data.
| Name | Required | Description | Default |
|---|---|---|---|
| cause | Yes | ||
| summary | Yes | ||
| suggested_fix | Yes | ||
| how_to_reproduce | Yes |
| Name | Required | Description |
|---|---|---|
| body | No | |
| title | No | |
| duplicate | No | |
| submit_url | No | Prefilled GitHub URL. A human must open it. |
| search_failed | No | |
| reports_enabled | No | |
| existing_issue_url | No | |
| existing_issue_title | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly states that the tool does not submit the report, checks for duplicates, stops if duplicate is true, and relies on the user to open submit_url. It also warns against including credentials, private URLs, or personal data, adding important context about what the tool does and does not do.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and front-loaded with the primary purpose and the critical non-submitting behavior. Each sentence earns its place by explaining the workflow, duplicate handling, authorization, or privacy constraints, with no wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description provides a complete overview of the tool's workflow, including duplicate detection, user authorization, and the fact that the user manually opens submit_url. Since an output schema exists, it needn't detail return values, but the mention of `duplicate`, `body`, and `submit_url` gives a good sense of expected output. The only gap is parameter semantics, which is covered in that dimension.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 0% description coverage, and the description never mentions the four required parameters (summary, how_to_reproduce, cause, suggested_fix). Although the parameter titles are self-explanatory, the description adds no parameter-level guidance or context, failing to compensate for the lack of schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool prepares a bug report about reapfield itself and explicitly highlights that it does NOT submit it. The verb 'prepare' plus the scope 'bug report about reapfield' distinguishes it from sibling tools that manage selectors, making its purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit when-to-use guidance: 'Call this only when you hit a real bug in reapfield during real work and have verified a fix.' It also outlines the workflow (search existing issues, stop if duplicate, show body and ask for authorization), which clarifies the intended usage and how it differs from a submission tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Forget the cached selectors for these fields so the next scrape re-derives them.
Returns the entries that were dropped. Re-derivation is lazy by design: it
needs a page to look at, and it happens on the next scrape of one.
Config-pinned selectors live in a separate store and are never touched.
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes | ||
| fields | Yes |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It reveals key traits: returns the dropped entries, lazy re-derivation requiring a page, and config-pinned selectors are never touched. This is meaningful beyond what a schema would provide, though it doesn't mention irreversibility or prerequisites.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is succinct and well-structured. The first sentence states the core purpose, followed by additional behavior in short, focused sentences. Every sentence contributes unique information, and the description is easy to scan.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the tool's purpose, return value, lazy re-derivation, and an important edge case (config-pinned selectors). However, given the missing parameter semantics and lack of explicit usage guidance, the description isn't fully complete for an agent to use the tool without further assumptions. The presence of an output schema reduces the need to explain return values, but the parameter gap remains.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description fails to compensate. It refers to 'these fields' but does not clarify the format or semantics of 'domain' or 'fields', leaving the agent without guidance on what values to provide. The parameters are only given as string types with no further explanation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action: 'Forget the cached selectors for these fields so the next scrape re-derives them.' The verb 'forget' (invalidate) and resource 'cached selectors' are specific, and this distinguishes it from siblings like list_cached_selectors (listing) and scrape (performing the scrape).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when this tool is relevant—when you want to force re-derivation of selectors on the next scrape. It notes that re-derivation is lazy and happens on next scrape, and that config-pinned selectors are untouched. However, it doesn't explicitly contrast with alternatives or state when not to use it, so it stays one point below the top.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract structured fields from a web page.
`fields` is comma-separated with optional inline types, e.g.
"title, price:float, in_stock:bool". Field names are arbitrary.
A field that could not be extracted comes back as null with an entry in
`misses` explaining why -- that is a normal result, not a failure.
Cost control: `no_llm=True` uses only structured data and already-cached
selectors, so the call is free; `max_llm_calls` caps what an uncached page
may spend. `scroll` and `paginate` cost extra fetches, not extra tokens.
There is no `strict`: it exists on the CLI only to pick an exit code, and
`misses` already tells you what was not found.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| mode | No | auto | |
| fields | Yes | ||
| no_llm | No | ||
| scroll | No | ||
| refresh | No | ||
| no_cache | No | ||
| paginate | No | ||
| cache_ttl | No | ||
| max_llm_calls | No |
| Name | Required | Description |
|---|---|---|
| url | Yes | |
| mode | Yes | |
| misses | No | field name -> why it is null. A populated misses is a result, not an error. |
| records | Yes | |
| llm_calls | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden of behavioral disclosure. It explains that missing fields yield null with a `misses` entry, that this is normal, and clarifies cost implications of `scroll`, `paginate`, and `no_llm`. It even preemptively notes that `strict` does not exist, which prevents confusion. This is thorough and transparent for a scraping tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise yet information-dense. It is structured into clear sections: purpose, field syntax, expected missing-field behavior, cost control, and a note about `strict`. Every sentence adds value with no fluff or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (10 params, no annotations, output schema present), the description covers core behavioral aspects such as output for missing fields, cost implications, and caching hints. It doesn't explain every parameter (mode, refresh, no_cache, cache_ttl) or discuss auth/rate limits, but the output schema and naming conventions cover some gaps. Overall, it is fairly comprehensive but not exhaustive.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% for the 10 parameters, so the description must compensate. It does provide detailed meaning for `fields` (comma-separated inline types), `no_llm`, `max_llm_calls`, `scroll`, and `paginate`. However, `mode`, `refresh`, `no_cache`, and `cache_ttl` are left unexplained, relying on their names for inference. The description adds significant value but doesn't fully cover all parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Extract structured fields from a web page.' This clearly states what the tool does and distinguishes it from sibling tools like list_cached_selectors or refresh_selectors, which manage selectors rather than perform extraction.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context on how to use the tool, including the `fields` syntax and cost-control options (e.g., no_llm, max_llm_calls, scroll/paginate costs). It does not explicitly mention when not to use it or name alternative tools, but the provided context is enough for an agent to decide when to invoke it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0list_cached_selectorsprepare_issue_reportrefresh_selectorsscrapeScored across 4 tools
Each tool has a clearly distinct purpose: scrape extracts data, list_cached_selectors shows cached selectors, refresh_selectors removes them, and prepare_issue_report handles bug reporting. No two tools overlap in function, making selection unambiguous.
Most tools follow a verb_noun pattern (list_cached_selectors, refresh_selectors, prepare_issue_report), but scrape is a bare verb. This is a minor deviation that does not impair readability or predictability.
With 4 tools, the server is well-scoped for its purpose. Each tool is necessary: scrape for core functionality, list and refresh for cache management, and prepare_issue_report for maintenance. No bloat or excessive thinness.
The core workflow of scraping with cached selectors is well covered: scrape handles extraction, list shows cache state, refresh allows re-derivation. Minor gaps exist, such as no direct way to manually set selectors, but the lazy re-derivation design mitigates this. The self-report tool is an optional extra rather than a missing operation.
Turn any website into structured JSON data matching your custom schema.
Turn any webpage into structured JSON with CSS-selector schemas - strings select text, attributes
Web scraping for AI agents. Converts URLs to clean, LLM-ready Markdown with anti-bot bypass.
Turn any URL into clean Markdown and structured data. Scrape, crawl, search and extract.