Scout MCP
Provides a precompiled recipe for crawling MikroTik product pages, enabling extraction of product specifications such as price, CPU, RAM, and max power consumption via the run tool with the mikrotik.com/specs recipe.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Scout MCPget the weight and dimensions from https://mikrotik.com/product/RB433AH"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Scout MCP
Scout is an MCP server for web crawling that improves with use. A capable model like Claude works a site out once. Scout compiles what it learned into a recipe that a 4B model can run with one instruction.
/scout https://mikrotik.com/product/RB433AH price|cpuClaude reaches the page, maps the site and finds the traps. It then hands back a one-line card that Qwen3.5-4B runs correctly from then on:
Call the tool run with recipe="mikrotik.com/specs" and input="hAP ax2". Reply with the RESULT line only.
RESULT: ok | mikrotik.com/specs | hAP ax2 | url=https://mikrotik.com/product/hap_ax2; title=MikroTik · hAP ax²; price=$99.00; cpu=IPQ-6010; ram=1 GB; max_power=27 WInstall
Claude Code (installs the MCP server and the /scout skill):
/plugin marketplace add sireggserroneous/scout-mcp
/plugin install scout@scout-mcpAny MCP client. Install uv, then add this to the client's MCP config:
{ "mcpServers": { "scout": { "command": "uvx",
"args": ["--from", "git+https://github.com/sireggserroneous/scout-mcp", "scout-mcp"] } } }Pages built by JavaScript need a headless browser. Add "--with", "playwright" to the uvx args, then run
uvx playwright install chromium once. If you already run Firecrawl or crawl4ai, set FIRECRAWL_URL or
CRAWL4AI_URL and Scout uses it as one more reader.
Related MCP server: mcp-web-search
From a maze to one line
Getting MikroTik specs looks like a one-step job. Here is what Claude actually ran into on the way:
Try | Result |
guess |
|
lowercase it, turn spaces into | Now |
also turn |
|
…but | |
look the url up in the sitemap, comparing both sides with | all four styles work |
Then the data: 4 fields, each a label on one line and its value on a later line, on a page that also has a 40-row throughput table, a documents list and a retailer map.
Written out as instructions, that is a 7-step program with a string-normalisation rule, a loop over 566 urls, a branch for "not found", and four lookups in a 7,000-character page. That is too much for a 4B model. So it is never handed to one. Claude compiles it instead:
{"name": "mikrotik.com/specs", "input_name": "model",
"examples": ["RB433AH", "hAP ac3", "CRS354-48G-4S+2Q+RM", "RB5009UG+S+IN", "CCR2004-1G-12S+2XS"],
"steps": [
{"map": "https://mikrotik.com", "filter": "/product/", "same": [["plus", ""], ["[^a-z0-9]", ""]]},
{"reach": "{url}", "want": "Specification"},
{"extract": {"price": "^\\s*-\\s*Suggested price\\s*\\n\\s*(\\S.*)$",
"cpu": "^\\s*-\\s*CPU\\s*\\n\\s*(\\S.*)$",
"ram": "^\\s*-\\s*Size of RAM\\s*\\n\\s*(\\S.*)$",
"max_power": "^\\s*-\\s*Max power consumption\\s*\\n\\s*(\\S.*)$"}}]}compile runs the recipe on every example and saves it only if all of them pass. The two wrong guesses above
were caught this way: each one passed on some examples and failed on others. Once saved, every branch lives in code.
The small model gets one card and one tool, and its whole job is a single call:
the small model gets | the small model must | |
without a recipe | 7 numbered steps, | normalise strings, scan 566 urls, branch, read a 7k-char page, format a line |
with a recipe | 1 line, the | make one call, copy one line |
Measured on Qwen3.5-4B (Q4_K_M, llama.cpp on a laptop iGPU), on 5 MikroTik models, as an A/B:
exact answer | tool calls | wall time (median) | |
A: the 7 steps as instructions, | 4 / 5 | 2–4 per model, 15 in all (5 of them wrong turns) | 389–671 s (414 s) |
B: the card, | 5 / 5 | 1 per model | 22–195 s (88 s) |
In arm A, Qwen filtered the sitemap with the raw model name, built a url with a space in it, and once dropped the $
from a price. It reached the right page in the end, but it took 3 to 11 minutes each time, mostly spent reading 566
urls and a 7,000-character page. In arm B the model never sees any of that. One arm-A run did catch a bug in the
recipe: the RAM regex had matched a sentence in the product description, not the spec list. The fix (anchoring each
field to its list item) went in, the recipe was recompiled with that model as a fifth example, and the arm-B run was
repeated on the fixed recipe. Both arms used the same model and the same Scout. Harness: bench/qwen_ab.py.
If a page changes, run fails loudly with a code: NO_MATCH at step 1, EXTRACT_MISS at step 3. The recipe's
score drops, and recipes() marks it last run failed … recompile. The next big-model session fixes the recipe. The
small model's card stays the same.
The CLI form fits a small model with a shell. It prints one line and exits 1 on failure:
$ scout-mcp run mikrotik.com/specs "CCR2004-16G-2S+"
RESULT: ok | mikrotik.com/specs | CCR2004-16G-2S+ | url=https://mikrotik.com/product/ccr2004_16g_2splus; ... price=$465.00; cpu=AL32400; ram=4 GB; max_power=48 W
$ scout-mcp run mikrotik.com/specs "nope 9"
RESULT: fail | mikrotik.com/specs | nope 9 | NO_MATCH at step 1Recipe steps
Step | Does |
| rewrites the input (slug rules) |
| the site's own urls. |
| reads a page with the full reader ladder (below) |
| the first link on the page whose url or text matches becomes |
| a link family from |
| reads the page's schema.org data by dotted path. A name ending in |
| group 1 of each regex. Every field must be found, except one whose name ends in |
How Scout learns: a maze, a Markov chain, and rewrite rules
Every way to reach a page is a move:
the readers:
direct,firecrawl,crawl4ai,browserurl_rewrite: a regex on the full urllocale_prefix×listing_root: where a site keeps its index of things, such as/en-us+/productsdetail_suffix: where the details live, such as/specifications
The idea comes from Markov chains and MarkovJunior-style rewrite rules. Each move takes Scout from one state to the next. The goal state is good information: real content (not a wall, a login page or an error page) that carries what you asked for.
flowchart LR
U[url + want] --> O[host's ordered moves<br/>winner first, recent failures last]
O --> J{good information?}
J -- yes --> W[return page · learn the winner]
J -- no --> N[next move: reader → url_rewrite → detail_suffix → on-page link]
N --> J
N -- maze exhausted --> E[coded error + next steps · logged]
W -. a big model compiles the path .-> R[recipe: one call for a small model]Each host keeps its readers in order. The reader that last worked goes first, and one that failed goes to the back for six hours. When excalidraw.com gives a plain fetch an empty JavaScript shell, Scout falls back to the browser. On the next visit it starts with the browser.
The register stores moves as data. Each move has a scope (
*, a domain or a host regex) and a Laplace score(wins + 1) / (tries + 2). A move nobody has tried scores 0.5. A move that loses its first five tries is retired. Agents propose moves withmoves(action='propose', …), and real traffic decides which ones stay.Parts compose. A listing url is a locale prefix (
"",/en-us,/us/en, …) plus a catalogue root (/products,/collections,/catalog, …). Scout ranks every pairing by the product of the two parts' scores. It puts the locale in the url you gave it, or the one the host taught it, first. Credit is by contrast: when/productswins after/en/productsfailed, only/enloses. When nothing wins, nothing is scored, because a site with no listing teaches nothing about the parts.Recipes are the compiled layer on top. Every run scores a recipe, the same way moves are scored.
The recipe book it ships with
scout_mcp/recipes.json was distilled from 417 sites that a larger crawler had learned. Every one was checked live
again on 2026-10-03 with Scout's own user agent, and only what held up was kept:
Shared structure became scored parts. The priors are measured.
/productswas the root of all 13 verified catalogue indexes. A bare root (no locale) won 8 of 8. The/enprefix the old crawler kept guessing was real on only 4 of 41 sites: the rest were 404s, homepage redirects, or soft 200s serving the homepage./specificationsis the detail page on 5 sites.Per-site facts, only where a site differs (147 sites): 56 need a rendering reader, 14 have a verified listing (7 with a locale), 28 have item url patterns proven by a real catalogue, 6 have a detail suffix, and 59 put up a wall (challenge, robots, refusal or login) that Scout reports before trying.
Dropped: unverified listings, including product pages, category pages, tag pages and a census.gov policy page that had all passed as "listings", plus item patterns a small model guessed but a catalogue never proved.
Point SCOUT_DB at a shared path and a whole team, people and agents alike, learns from each other's runs.
Link families: walking a site with no sitemap
These rules came from a sibling project: a recipe chain that rebuilt 16 US state codes from scratch, with 7 or 8 of every 8 sampled sections matching the held text word for word. Its hardest problems were about links, and the fixes were all general:
A link's family is its path with the numbers taken out.
Chapter_1.htmlandChapter_14A.htmlare one family, and so areNHTOC-I.htmandNHTOC-XIV.htm(Roman numerals). Links with no numbers group as siblings under their parent (/products/*). Only same-site families with 3 or more links count. Static assets,#anchors(each item is read once, not once per anchor) and a page's link to itself are ignored.Sample across the family, not its first links. A list opens with its odd items: a code's first chapter is often a one-line "Repealed". Requiring 2 of 3 good samples at every level compounded into no path at all, so it is 2 at the top and 1 below.
Vote on what the family leads to, not on whichever sample came back first. Section pages that also list their neighbours made one sample look like an index; the majority got it right.
Look deeper before trusting a weak match. A page that looks like one item, but still has families leading down, is walked further first.
Rank by yield through every level: links × share of good samples × what each yields below. Scoring one level at a time made a five-page index outrank 89 chapters of sections.
Tell content from menus by function words. The density of the, of, shall, any separates sentences (≈0.35–0.5) from navigation (≈0.15) where capitals and length could not. Every
reachresult carries thisprosescore.Skip families that lead to a different kind of document. Statute pages link to the bills that enacted them, and the chain followed them into scanned slip laws.
families(url, want?) runs this walk and returns the ranked families, their samples, and a recipe_hint draft for
compile. The links recipe step then lists a family's urls on sites with no sitemap.
Live, on two sites (2026-10-03):
Start | Best chain found by sampling | Estimate | Compiled hint |
Idaho statutes index |
| ≈2,516 chapter pages |
|
a MikroTik category page |
| 22 products |
|
MikroTik added two rules the law sites never needed:
A family the page above already shows is navigation. Every MikroTik page carries the same category menu, and without this rule the walk descended from the menu back into the menu.
Siblings group under their parent first. The number shape names the family only when one shape covers 80% of them (
Title<n>).RB<n>andcrs<n>_<n>_<n>_inare both just/product/*.
When a site blocks you
Also from the sibling project. A statute site blocked its crawler's address after a re-read at ten requests a second, and the lessons generalise:
A block page can come back as HTTP 200. That site answered
Blocked <your IP> … for assistance EMAIL: WebMaster@…as an ordinary page, so a check on status codes alone never saw it and the crawler kept going. Scout reads a page as a block only when it has both halves, a refusal ("blocked", "access denied", "too many requests") and something about the visitor (an IP address, "your IP", a CAPTCHA, "are you a robot"). That way a page that only mentions "blocked" isn't mistaken for one. It checks error bodies and 200 pages alike.A block stops the whole walk. Every reader leaves from the same address, so trying the next one only adds to the load. Scout stops, puts the host on hold for 30 minutes, and slows its pace there for good (5 s or slower). A 429 does the same, for its
Retry-Afteror 2 minutes.The probe is built in. When an automatic hold ends, the next reach is a single probe. If it is refused again, the hold doubles (up to a day) and the pace slows again. You don't need a separate watcher process.
Holds are shared. They live in Scout's memory, so every agent pointed at the same
SCOUT_DBstays off the site. A held host is not contacted at all:reach,site_mapandfamiliesreturnHELDand send no request.The way out is a person. The block page usually names a contact, and Scout keeps it with the host. The
BLOCKEDerror's first next step says who to email and what to say: what reads the site, that it went too fast, the new pace, and an offer to use a bulk download.hold(host, reason='unblock requested <date>')then keeps everyone off untilhold(host, lift=true). A hold set that way has no end date, so Scout never probes it.
Official routes and enrollment
When a site walls Scout, the best next step is usually the site's own front door: an open API, a bulk download, a feed, or an API a person signs up for. Scout keeps that per site as its access entry.
No signup needed (
open_api,bulk,feed,public_json,alternative): the wall's first next step is that route, as a readyreach(...).A person must enroll (
enroll_free,enroll_paid,enroll_oauth,commercial): Scout files an enrollment request the first time the wall is hit.enroll(action='list')is the checklist. Each entry says where to sign up, what it costs, the terms, and where the key goes.No route known yet: the error says so, and a big model looks for one (the API, developer program or bulk download) and records it with
enroll(action='request', …). Say you need LexisNexis but haven't set Scout up for it:/scout https://www.lexisnexis.com/...hits the login, Claude records LexisNexis's developer program, and it lands on the checklist.Updating Scout:
/scout-updateworks through the checklist with you, along with failing recipes, holds waiting on a site's answer, and recent failures. When you've put a key in place,enroll(action='done')tests it on the API's sample url before marking the site enrolled. From then on recipes call the API with no key in them; Scout adds it.
Keys never pass through a tool. They live in SCOUT_KEY_<NAME> environment variables or in
~/.config/scout-mcp/keys.env (NAME=value). Scout ignores that file unless its mode is 600. A key is added at fetch
time, only to its own API's host, and scrubbed by value from every page, url and error Scout returns. OAuth client
credentials (NAME_ID and NAME_SECRET) are exchanged for a token, which is cached until it expires.
The book ships with the official routes for 21 sites that challenged Scout, researched and tested on 2026-10-03:
Site | Official route | Signup | Tested from a datacenter address |
www.ecfr.gov, ecfr.federalregister.gov | eCFR API (+ GovInfo bulk XML) | none | 200 JSON, while the pages challenge |
DailyMed, openFDA and RxNav (the FDA's own label data) | none | 200 JSON | |
RSS feeds (full text needs the free Developer API key) | none | 200 RSS | |
the town's RSS feeds and GIS data hub | none | 200 RSS, 200 JSON | |
snuggymom.com | WordPress RSS and REST API | none | 200 JSON |
product literature on assets.dmp.com | none | 200 PDF | |
alaskacountyoffices.org | Alaska's state community database (ArcGIS) | none | 200 JSON |
legiscan.com | LegiScan Public API | free key | — |
Merriam-Webster Dictionary API | free key (non-commercial) | — | |
Britannica Syndication API | free key (non-commercial) | — | |
DigiKey Product Information v4 | free OAuth app | — | |
affiliate programs (product feeds unverified) | free application | — | |
us.rs-online.com, www.acronymfinder.com | licensed web services | contract | — |
webfiles.nycourts.gov, dictionary.cambridge.org, www.arcat.com, www.essentialbuildstore.com, aqcheckin.s3.amazonaws.com | none found: no API, a retired one, an unreachable one, or a private bucket | — | the reason is recorded, so no one searches again |
Several "challenged" sites serve their own APIs and feeds without any challenge. A wall in front of the pages is often not a wall in front of the data.
Lessons from a big catalogue: AMD
The sibling crawler's scout went after AMD's catalogue, and every mistake it made became a general rule:
A wall can be silent. From some networks amd.com answers no status at all: the plain fetch stalls until it times out, and the browser's HTTP/2 stream is reset. Scout used to call that
NETWORKand suggest checking the spelling. A stall or reset from a host that resolves is nowSILENT_REFUSAL: the site's edge refusing this client. It is treated as a wall (official route first, the host held so nothing keeps knocking), not as a typo.A map has to work when plain fetches don't.
site_mapreads sitemaps and walks links through the host's learned reader, not just a plain fetch.Map from the catalogue's root when nothing else looks like a catalogue. AMD's map ran out on blog posts. Walking
/en/productsfound 1,561 product pages. When the sitemap and the walk turn up no catalogue-shaped urls, Scout finds the root with its scored locale × root parts and walks from there. That way the step is a scored move, not a hard-wired rule.Editorial sections are never the items. The scout picked
/blogs/and/newsroom/as AMD's product pattern and "proved" 689 blog posts. Blog, news, press, events, careers, investors and similar patterns are flaggededitorialand ranked last insite_mapandfamilies.A title of
=====is not a title. It falls back to the page's first real heading."… Series" is a category. So are "… Family" and "… Lineup". Recipes pick their examples from single items.
A page with children under its own path is a hub, not an item.
familiescounts 3 or more child pages as an index. Partner, request-a-quote and contact pages are flagged with the editorial ones. Support pages are not flagged, because some makers keep their product pages under/support/.Listing a whole site again and again gets you blocked. Four dry runs in an hour, each re-listing amd.com, and its edge answered "Access Denied" to everything.
site_mapnow reuses a host's listing for 12 hours (fresh=trueto list again). The edge's denial page names no IP, only an incident reference (Reference #18.…). An incident reference (Akamai Reference #, Cloudflare Ray ID) now counts as the visitor half of a block, so the host is held, not retried.
Download once, read from the archive
Also from the sibling crawler: the way to map a whole site without loading it is to ask the library that already copied it. The Wayback Machine held 1,543 of AMD's product pages, the newest a day old, while amd.com refused every request.
Listing from the archive's index.
site_map(url, archive='only')lists a site from the Wayback Machine's public index without a single request to the site. Robots rules come from the site's archived robots.txt. With the defaultarchive='auto', the index is used when the site is on hold or listed nothing. amd.com's product section: 1,816 urls in 12 seconds.Reading from the archive. When a site walls Scout (blocked, a silent refusal, a challenge, a refusal, a login, or down), the next rung is the latest archived copy, as the site served it. A held site is read from the archive only, with no contact with the site at all. Every result says so:
archived: {captured: 2026-08-21, copy: …}, and recipe RESULT lines carryarchived=<date>. The url stays the site's own, so provenance is honest.The archive is never a host's winning reader. Live reading resumes as soon as the site allows it, and robots.txt still binds: a path the site disallows is not read from its archive either. The archive itself is paced at one request every 4.5 seconds, under its limit of about 15 copies a minute. Scout's first try at 2 seconds got it refused by archive.org, so a refusal from the archive now holds the archive too (10 minutes, doubling), and cache jobs wait out the hold, never knocking through it.
A local mirror. Every good page is kept on disk for a day (
SCOUT_MIRROR_DAYS). Re-runs, parser fixes and recipe compiles read the copy, not the site;reach(fresh=true)reads it again. Scout's pace was already a promise: it only ever slows down for a host, never speeds back up.
It compiles like any other recipe. amd.com/processor-specs ships in the book, built entirely from archived pages:
RESULT: ok | amd.com/processor-specs | EPYC 9575F | url=https://www.amd.com/en/products/processors/server/epyc/9005-series/amd-epyc-9575f.html; title=AMD EPYC™ 9575F; archived=2026-07-10; cores=64; threads=128; boost=Up to 5 GHz; tdp=400W; socket=SP5; launched=10/10/2024Cached mode: when you'll pull a lot from one site
Asking a site for the same pages again and again is how crawlers get blocked. If you're about to pull a lot from one site, download it once and work from the copy:
cache("https://www.amd.com/en/products/processors/", filter="(9000|9005)-series/.+\\.html$", source="archive")
cache(url, action="status") # listed, fetched, failed, eta
run("amd.com/processor-specs", inputs=["Ryzen 9 9950X3D", "EPYC 9575F", ...], cached=true)cachelists the site (sitemaps, link walk, catalogue root, or the Wayback index), then fetches every page once into Scout's copy, in the background, at the host's pace.source='archive'downloads from the Wayback Machine with no request to the site at all.'auto'reads live, with the archive where the site walls Scout. The copy is keptdays(default 7).cached=trueonreach,site_mapandrunreads only the copy and sends nothing anywhere. A page that isn't in it fails withNOT_CACHEDand says how to get it. A recipe over a cached site runs at memory speed, so a small model can work through a long input list without the site ever seeing it.run(recipe, inputs=[…])returns one RESULT line per input. On the command line:scout-mcp cache <url> [filter] [--archive]downloads in the foreground, andscout-mcp run <recipe> - --cached < models.txtprints one line per input.
What a scout checks first
These rules come from the sibling crawler's queue:
What already exists.
scout(url)returns the site's compiledrecipesfirst. A recipe that passes is the answer, so nobody re-scouts the site.Whose site it is.
own_hostslists the hosts the site links to that carry its name, such as regional sites or a store. A host that carries the name but isn't linked is a reseller lead, not the maker: meanwellsource.com had 1,827 "Mean Well" pages.The sibling the request names. When a page lacks what was asked, one of Scout's next steps climbs to the page that lists it and its siblings, and follows the sibling link that names two-thirds of the request's words. Depot held New York's Criminal Procedure Law and was asked for its Estates, Powers and Trusts Law; the CPL's parent page named it.
Counted is not read: lessons from a law re-read
The sibling crawler landed 17,126 law sections with zero errors. When someone finally read a sample, much of it was broken: 29,299 sections were a website's menu ("Skip navigation Home Documents…"), 23,530 were a heading with no body, and a double-escaped regex had turned every "t" of 9,308 Alabama sections into a space ("he books… o de ermine he accuracy"). Every check had counted; none had read. In Scout:
A content gate on recipe values. A value that is page chrome (menu text, raw markup, a stylesheet rule) or has lost common letters fails as
BAD_VALUE, at compile and on every run.A regex lint at compile. A double-escaped escape inside a character class (
[^\\t]means "not a backslash and not the letter t") is refused with that explanation.The skill says to read the test values yourself. A wrong but plausible value is something only a reader catches.
Text decoded the way it was written. Bytes that decode cleanly as UTF-8 are read as UTF-8. Otherwise Scout uses the declared charset, but never UTF-16/32 without a byte-order mark (one state's pages declare UTF-16 and are UTF-8), and falls back to windows-1252 (another state's pages lost every § when read as UTF-8).
A site that asks for slower requests gets them, for as long as it asks. That includes a
Retry-Aftergiven as an HTTP date.A download nobody is running says so. The re-reads died with the one-off script that started them, and nothing noticed. A cache job whose process is gone shows
interrupted, and starting it again resumes where it stopped.
The laws of the United States
Scout can read a state's whole code from the legislature's own site and write it out, one JSON line per section:
scout-mcp laws # the states it can read, with their citation form and source
scout-mcp laws us-tx --out texas.jsonl # every section of the Texas statutes
scout-mcp laws us-nv --limit 5 # a quick lookEach line is {"citation": "NRS 1.010", "heading": "...", "text": "...", "url": "<the page it came from>"}. A state is a
short script in scout_mcp/laws/<state>.json: a sections() generator that reads the legislature's table of contents,
API or bulk file and yields each section's own words. Every request goes through Scout's gate (robots.txt and its crawl
delay, holds), at the recipe's own pace when that is slower. Text that is page chrome or has lost its letters is skipped.
Big bulk files (Texas's code zips, North Dakota's single page) are downloaded once a day to Scout's data folder.
33 ready: AL, AS, CO, CT, GU, IA, IL, LA, MD, ME, MI, MN, MP, MT, ND, NE, NH, NJ, NV, NY, OR, PA, PR, RI, SD, TX, UT, VA, VT, WA, WI, WV, WY. These are the scripts a sibling crawler used in October 2026 to land each code whole: Texas 119,000 sections, Illinois 69,000, Nebraska 55,000, Nevada 50,000. A full state takes from minutes (Texas, 31 downloads) to hours (Illinois and New Hampshire ask for 10 and 11 seconds between requests).
New York reads the Senate's Open Legislation API, which needs a free key: put it in
SCOUT_KEY_NYSENATE.Louisiana takes its Revised Statutes list from a dated snapshot, because the site only opens a title by form post.
Not here yet: AK, AZ, DC, DE, FL, GA, HI, ID, KS, KY, MA, MO, NC and SC are read by the sibling crawler's page-pattern engine and still need porting to scripts. California needs its 1.28 GB bulk zip read entry by entry. Indiana serves its code zip only to browsers.
Need permission first: Arkansas, Mississippi, Tennessee and the Virgin Islands publish their official code only through LexisNexis, whose terms forbid automated reading. New Mexico's official site is on a platform whose terms require written consent. Ohio's and Oklahoma's robots.txt files disallow all crawlers. The routes section below says whom to ask.
The text is the law as each site publishes it. It is not an official or certified copy; cite the source url.
Whole catalogues
Seven makers so far: Intel, AMD, Samsung (the US site, ~2,500 SKUs), Honeywell Automation, Functional
Devices, QSC and Von Duprin. scout-mcp catalog lists the makers; scout-mcp catalog intel --out intel.jsonl reads one maker's whole catalogue, one JSON
line per product: {"model_number", "name", "family", "category", "url", "specs": {label: value}}. family is the product
line (AMD EPYC, AMD Ryzen, Intel Xeon, Intel Core) and model_number the model within it, so a shelf of 4,600 Intel
SKUs groups into its brands instead of one flat list.
Intel lists its SKUs from the Internet Archive's index: ARK draws its lists with JavaScript and the one full list is closed by robots.txt. Each SKU's specifications page is then read live. The Processor Number is the model and the Product Collection's brand the family. Prices are left out; a price is a link to a seller, not a number to copy.
AMD resets any client that isn't a browser, so its pages are read from the Internet Archive's latest copies (
"archive": "auto"in the maker's script). The site's own sitemap names ~400 product pages, and the spec table's Name row is the model as AMD writes it ("AMD Ryzen™ 9 9950X").Samsung pages list every SKU sold on them (sizes, colours) in the page's own data, and the specs come from the JSON the page itself loads, so one page lands a whole group. The page's feature copy is drawn later by JavaScript, so descriptions are left empty rather than filled with the SEO blurb.
Functional Devices runs a commerce store whose guest API lists every product with its spec fields; Scout reads that.
Panduit isn't here: its product data only answers requests that claim to come from its own pages.
A recipe field named like a description (description, summary, overview) must also read as one. The content gate
refuses a stripped breadcrumb ("HomeAll ProductsLow Voltage…"), a page's script data, and the site's lifecycle notice
("End of commercialization is the last date…"): 1,061 of one maker's descriptions in the sibling crawler were one of those.
Official routes to state codes
The sibling crawler rebuilt about 25 state codes on 2026-10-10. For most of them the best route was not the section pages
but something the legislature publishes for whole-code reading. Those routes are now in the book (enroll("list") shows
them, and a wall on these hosts leads with them):
Site | Official route | Why it beats the pages |
statutes.capitol.texas.gov | one zip of chapter HTML per code | all 30 codes, about 122,000 sections, in 31 requests |
olls.info | Colorado's title download (HTML zip) | the online C.R.S. is a commercial platform's |
le.utah.gov | XML per title | structured text, current and future versions marked |
law.lis.virginia.gov | CSV per title | 76 requests instead of 33,000 throttled pages |
ndlegis.gov | the whole Century Code on one page | one fetch |
nebraskalegislature.gov | range view, 400 sections a request | 16 minutes instead of 41 hours at its crawl delay |
sdlegislature.gov | Statutes API | a title per request (UTF-16 with no byte-order mark) |
mgaleg.maryland.gov | statute text API | an article's section list, then each section |
legislature.maine.gov, www.palegis.us | whole titles (.docx, HTML) | one file per title |
downloads.leginfo.legislature.ca.gov | pubinfo bulk zip (1.28 GB) | the website answers with a challenge |
nmonesource.com | the platform owner's written consent | its terms forbid automated reading |
codes.ohio.gov, oklegislature.gov, capitol.tn.gov | none: robots.txt disallows all, or the official copy is on a commercial platform | the request to make is recorded |
What else that rebuild taught:
A one-line robots.txt still means what it says. Ohio's reads
User-agent: * Disallow: /with the newline missing, so parsers find no rules. The intent is plain, so it is treated as disallow-all.Counted is still not read. Virginia's 32,000 sections, Ohio's 12,000 and Minnesota's 6,900 had landed as the legislature's own menus, version banners and footers. Each passed every count. Those phrases are in the content gate, and the missing-letters test now leaves short prose, all-capitals notices and lists alone.
More requests at once don't make one host faster. Its pace is fixed by its gap; 39 jobs behind one 2-second gate each moved a page a minute and looked dead.
Structured data, archived block pages, and other things worth knowing
schema.org data. Many product pages carry clean
Productdata as JSON-LD even when their html is a mess.reachreturns it asjsonld, and thejsonldrecipe step reads dotted paths (offers.price,sku). Title tails like "| CT4000T705SSD5 | crucial.com" are cut from names.crucial.com/ssdships in the book, built entirely from archived pages while crucial.com's edge rejects Scout:RESULT: ok | crucial.com/ssd | CT4000T705SSD5 | … archived=2026-07-20; name=Crucial T705 4TB PCIe Gen5 NVMe M.2 SSD with heatsink; sku=CT4000T705SSD5; price=565.99; currency=USDAn archived copy can itself be a block page. The newest Wayback copy of some AMD pages is Akamai's 403, and of some Crucial pages it's the edge's rejection page, captured with a 200. Scout tells a replayed error (it has a
Memento-Datetime) apart from the archive refusing Scout, so it no longer holds the archive by mistake. It then walks back through older, distinct captures until one is real content. The archive pace is 5 seconds, with a 15-minute hold when it refuses.Cloudflare's 2026 wall ("Performing security verification… protect against malicious bots") is read as a challenge, not content.
A map started inside a section stays inside it.
site_map("https://site/en/products/processors/")lists only that section. The site's root maps the whole site.Function words don't steer. "and", "the", "section", "law" no longer pick which link to follow.
A dropped connection to a renderer service gets one retry on a fresh connection before it counts as a failure.
Same-name companies and terms of use. Edwards Lifesciences is not Edwards fire safety, simplex.com is not Simplex fire, and sol.com is not Sol-Ark: the skill checks the site names what you mean. robots.txt is not the only rule either. LexisNexis's terms forbid automated access without written permission, so its entry in the book is a permission request, not a route.
Errors that say what to do next
When Scout fails, it returns an error code, a message, and next_steps written as concrete tool calls, plus the
tried trail it took through the maze. Any capable model can pick up from there.
Code | Next steps point to |
|
|
| loosening |
| adding a rendering reader, the site's JSON endpoints, a print view |
| the site's official route, its partner program, the same information published elsewhere |
| who to email to lift the block, |
| an official feed, waiting for |
| the cause, retrying later |
| what to recompile or propose |
Tools
Tool | For | Does |
| big model | reaches the page, maps url patterns, finds the listing, reports what was learned |
| big model | reads one page through the learned ladder |
| big model | the site's own urls and their patterns |
| big model | which link families lead to good pages, by sampling and vote |
| big model | tests a recipe and saves it, returns the small-model card |
| big model | lists, proposes or retires moves |
| big model | keeps every agent off a site (or lets them back), slows Scout's pace there |
| big model | records a site's official route, and lists, tests and closes enrollment requests |
| any model | the US state codes Scout reads whole from each legislature; a few sections of one (the whole code: |
| any model |
|
| big model | reader order, routes, moves and recent failures for a host |
| both | compiled recipes with scores, status and cards |
| small model | runs a recipe, returns one RESULT line (one per input) |
| big model | downloads a site once (live or from the Wayback Machine) for cached reads |
Give the small model only run, or the CLI. One tool means it has nothing to choose between.
Configuration
Variable | Default |
|
|
|
|
|
|
|
|
|
|
|
|
| unset |
Develop
uv venv && uv pip install -e . playwright
python -m scout_mcp.scout # self-checks, no networkThere are two modules. scout_mcp/scout.py is the engine, and scout_mcp/server.py holds the MCP tools and the CLI.
Seed moves and example recipes are in scout_mcp/recipes.json. MIT licensed.
Available Tools
13 toolscacheA
Cached mode, for pulling a lot from one site: list it once (filter narrows the urls) and download every page once
into Scout's copy, in the background, at the host's pace. source='archive' downloads from the Wayback Machine with no
request to the site; 'live' from the site; 'auto' live, with the archive where the site walls Scout. The copy is kept
days; reads and recipe runs with cached=true use only it. action=status follows the job; action=stop ends it.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| days | No | ||
| action | No | start | |
| filter | No | ||
| source | No | auto | |
| max_pages | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does well: it discloses background execution, pacing at the host's pace, retention for `days`, that reads/recipe runs with cached=true use only the copy, and that archive source makes no request to the site. It does not describe failure handling or what happens if a job is re-started.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three dense sentences, front-loaded with the scenario and the core behavior, with each clause carrying parameter or behavioral meaning. No filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 6-parameter, no-annotation, no-output-schema job-control tool, this covers purpose, mode selection, retention, and lifecycle actions. The omission of max_pages semantics and any job-failure/status-return detail are the only remaining gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate and largely does: it explains source values (archive/live/auto), the meaning of days (copy retention), filter (narrows the urls), and action (status/stop). Only max_pages and the url parameter are left without any semantic explanation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource ('download every page once into Scout's copy') and the scenario it serves ('pulling a lot from one site'). It is clear what the tool does, though it does not name or differentiate itself against any of the many sibling tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a clear when-to-use trigger ('for pulling a lot from one site') and describes the source selection trade-off (archive = no request to the site, auto = live with archive fallback). It never names an alternative sibling tool or an explicit when-not-to-use condition.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
compileA
Compile what you worked out about a site into a recipe a small model can run in one call. Scout runs it on every example and saves it only if all pass; the reply carries the one-line card for the small model. name: '/'. input_name: what the caller supplies ('model', 'sku'; empty for none). examples: 2+ inputs. steps, run in order, each a one-key object: {"input": [[regex, repl], ...], "lower": true} rewrite the input (slug rules) {"map": "https://site", "filter": "/product/{input}$"} the site's own urls; first match becomes {url} {"reach": "https://site/p/{input}" | "{url}", "want": regex} read a page {"find": regex} first link on the page matching (url or text) becomes {url} {"links": "/chapter//"} a link family from families(); last step = all its links, else the first is {url} {"jsonld": {"price": "offers.price", "sku": "sku"}, "type": "Product"} from the page's schema.org data {"extract": {"field": "regex with one group", "optional?": "..."}} every field must be found, except a name ending in ? Prefer map over a guessed url template: sites are inconsistent about case and slugs.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | ||
| about | No | ||
| steps | Yes | ||
| examples | No | ||
| input_name | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations, so the description carries the full burden, and it does disclose real behavior: the recipe is executed against every example and only saved if all pass, and the reply carries a one-line card. This is meaningful beyond a bare 'compile' claim, though side effects, failure messaging, and storage semantics are not covered.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads purpose, then presents the step types as a clean, indented reference where each line earns its place. It is long, but density is justified by the DSL it documents; nothing reads as filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex DSL-compiler tool with no annotations, no output schema, and 0% schema coverage, the description supplies the purpose, the saving gate, the reply contents, and a nearly complete step grammar. Remaining gaps ('about', error behavior on failed compilation) are small relative to the complexity handled.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate, and it documents name ('<host>/<slug>'), input_name (what the caller supplies, empty for none), examples (2+ inputs), and the entirety of the steps DSL. Only the 'about' parameter is left unexplained, which is a minor gap against otherwise strong compensation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: 'Compile what you worked out about a site into a recipe a small model can run in one call,' and adds the downstream workflow ('Scout runs it on every example and saves it only if all pass'). It is distinguishable from most siblings, though it leans on domain jargon and does not explicitly contrast itself with 'recipes', 'run', or 'scout'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied by the workflow narrative (compile then Scout validates) and one conditional hint is given: 'Prefer map over a guessed url template.' But there is no explicit statement of when to choose compile over siblings like recipes, run, or scout, nor any prerequisite conditions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
enrollA
Official routes and the enrollments they need: the checklist for whoever updates Scout.
action=list: open requests, each with a todo that says where to sign up, what it costs, and where the key goes.
action=request host info: record a site's official route. info = {route: open_api|bulk|feed|public_json|alternative
(no signup) or enroll_free|enroll_paid|enroll_oauth|commercial (a person must enroll), name, signup_url, docs_url,
base_url, auth: {kind: none|query|header|bearer|oauth2_client_credentials, name, token_url, headers}, cost, terms,
sample (a url that returns the data; leave the key out)}. Keyed routes become an enrollment request.
action=applied (signed up, waiting on approval) | done (Scout tests the key on the sample, then marks enrolled) | decline.
Keys are never passed here: they go in the keys file or SCOUT_KEY_ env vars.
| Name | Required | Description | Default |
|---|---|---|---|
| host | No | ||
| info | No | ||
| note | No | ||
| action | No | list | |
| include_all | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden and does so substantially: it discloses that recording a keyed route creates an enrollment request, that Scout tests the key on the sample before marking 'enrolled', and the key-handling constraint ('keys are never passed here: they go in the keys file or SCOUT_KEY_<NAME> env vars'). It still omits permissions required, reversibility, and what a call returns, which keeps it from a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The text is dense but nearly waste-free, with each action line earning its place and key-handling rules placed last as a constraint. Structure is poor, though: it is a run-on block with slash- and pipe-delimited fragments and no front-loaded summary, making it harder to scan than its length warrants.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 5-parameter, no-annotation, no-output-schema workflow tool, the description covers the action modes and the info payload adequately but leaves host/note/include_all undocumented and gives no return-value or state-transition details. It is usable for the common path but incomplete for an agent needing to call it precisely in every mode.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% across 5 parameters, so the description must compensate and it does so well for the two hardest inputs: the full action= vocabulary (which supplies the missing enum) and the nested info object's fields (route types, auth kinds, sample, cost). The host, note, and include_all parameters, however, are left completely unexplained in both schema and description, so compensation is only partial.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific resource ('official routes and the enrollments they need') and enumerates distinct operations via action= list/request/applied/done/decline, so an agent can tell it manages the enrollment workflow rather than generic site data. It does not name any sibling tool, however, and the referent 'Scout' is undefined, leaving some ambiguity about the boundary versus tools like site_map or scout.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The action= vocabulary implicitly conveys the workflow states (open request → applied → done/decline), which is useful triage context. But there is no explicit statement of when to choose enroll over its siblings, no preconditions, and no when-not-to-use condition; the agent must infer usage from the mode names.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
familiesA
Which of a page's link families lead to good pages, and how many. Each family (links whose paths differ only in
their numbers, like /chapter/1/ and /chapter/14A/) is sampled across its spread, the samples vote on what the family
leads to (an index of more links, or pages), indexes are walked one level deeper, and families are ranked by yield
through every level. best.recipe_hint is a draft recipe for compile. Use it on sites with no useful sitemap.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| want | No | ||
| depth | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations at all, this description carries the full disclosure burden and does unusually well: it explains the sampling/voting/walking/ranking mechanics and flags that `best.recipe_hint` is a draft recipe for `compile`. It omits operational traits an agent would want (network cost, rate limits, whether it is a safe read-only crawl), so not a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Purpose and the compile cross-reference are front-loaded, but the long middle sentence bundles four algorithm stages into one run-on clause, mixing helpful rationale with implementation detail. Adequately sized, not optimally structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a three-parameter, no-annotation, no-output-schema tool, the description explains the mechanism and one return field (`best.recipe_hint`) but leaves all inputs unexplained and says nothing about the shape of the ranking output. Partially complete rather than fully adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% for all three parameters. The description never explains `url`, `want`, or `depth`; only the passing phrase 'walked one level deeper' faintly gestures at the depth parameter. With low coverage the description was expected to compensate and largely does not.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific resource (a page's link "families") and what the tool produces (which families lead to good pages and how many), and it defines its own jargon with the /chapter/1/ vs /chapter/14A/ example. It is distinguishable from the cryptically named siblings, though the framing as a question about results rather than a crisp verb keeps it from a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives one explicit routing condition: use it on sites with no useful sitemap, which implicitly points away from the confusingly named `site_map` sibling. There is no explicit when-not guidance beyond that, but the key selecting condition is present.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
holdA
Keep every agent sharing Scout's memory off a host, or let them back. Use it when a site blocked you and a person has asked for the block to be lifted: reason says what was asked and when. min_interval (seconds) slows Scout's pace on that host for good. Scout holds a host by itself after a block page or a 429, and probes once when that ends.
| Name | Required | Description | Default |
|---|---|---|---|
| host | Yes | ||
| lift | No | ||
| reason | No | ||
| min_interval | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does disclose real behavior: automatic holds after a block page or 429, a single probe when the block ends, and that min_interval permanently slows Scout's pace on that host. It still omits scope and side effects — whether lifting affects in-flight requests, whether the hold persists across sessions, and what permissions are needed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The core purpose is front-loaded in the first clause, which is good, but the second sentence is a run-on whose referent ('reason says what was asked and when') is easy to misparse. The three sentences are dense and could be tightened without losing information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 4-parameter mutation tool with no annotations and no output schema, the description supplies an unusual amount of operational context (auto-hold triggers, probe behavior, permanent pacing). It is nearly complete, with the main remaining gap being persistence scope and any return/confirmation semantics.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate, and it largely does: 'lift' is defined by 'let them back', 'reason' by 'says what was asked and when', and 'min_interval (seconds)' is explained as a permanent pacing change. Only 'host' is left implicit, though it is inferable from 'off a host'.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb and resource: it keeps agents off a host, or lifts that block ('or let them back'). The hold/lift duality is unambiguous once read, but there is no differentiation from the sibling tools (scout, reach, cache, etc.), which are not referenced at all.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives an explicit trigger for the lift path ('when a site blocked you and a person has asked for the block to be lifted') and implies that manual holding is often unnecessary because 'Scout holds a host by itself after a block page or a 429'. That implicit when-not-to-use is valuable, but no alternative tool is ever named.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
lawsA
US state codes read whole from each legislature's own site. No state: the list of states Scout can read, with each
one's citation form and source. A state (e.g. 'us-tx'): its first limit sections (max 50) as citation, heading, text
and url. A whole code is tens of thousands of sections: run scout-mcp laws <state> --out file.jsonl for that.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| state | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden. It discloses the source (legislature sites), the max-50 limit, the scale warning for whole codes, and the returned fields (citation, heading, text, url). It does not cover permissions, rate limits, or error behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads the purpose, then walks through no-state, state, and whole-code behavior efficiently. It is dense but appropriately sized. Bullet formatting would improve scanability, but no sentence is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with no annotations and no output schema, the description covers both modes, return fields, and the scale limitation plus an alternative for full codes. It omits auth/permissions and pagination beyond limit, but is otherwise adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It explains the state format (e.g. 'us-tx'), the no-state mode, and the limit cap (max 50). The default limit of 10 is not restated, though the schema supplies it.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: reading US state codes from each legislature's own site. It also distinguishes two modes based on whether the state parameter is provided. It does not explicitly differentiate from sibling tools, but the scope is clear.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explains when to omit state (list of states with citation form/source) versus when to provide state (first limit sections of that state's code). It also gives an alternative CLI path for reading a whole code. No sibling-tool alternatives or exclusions are given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
memoryB
What Scout has learned about a host: winning reader, reader order, routes (listing, detail_suffix, patterns), the moves that apply, recent failures. No host: recent failures everywhere and the whole register.
| Name | Required | Description | Default |
|---|---|---|---|
| host | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It usefully discloses the returned content (winning reader, reader order, routes, moves, recent failures) and the no-host fallback behavior, but says nothing about read-only nature, side effects, or any limits, leaving gaps for a retrieval tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences, front-loading the with-host case before the no-host fallback, with no filler. Dense jargon slightly hurts readability but nothing is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a low-complexity single-parameter retrieval tool with no output schema and no annotations, the description adequately sketches the payload but leaves return shape and safety profile undefined, and the undefined terminology prevents fully confident invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% for the lone 'host' parameter, so the description must compensate — and it does, explaining both the per-host result set and what happens when host is omitted ('recent failures everywhere and the whole register'). That meaningfully exceeds the silent schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description conveys that this retrieves learned state about a host (readers, routes, moves, failures), which is a real purpose beyond the name. However it is saturated with unexplained domain jargon ('winning reader', 'the register', 'moves') and never distinguishes itself from siblings like scout, moves, recipes, or cache, so an agent cannot cleanly route to it over those.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It does specify the two operating modes via the host argument ('with host' vs 'No host'), which is genuine conditional guidance. But there is no explicit when-to-use/when-to-avoid statement relative to the many sibling tools, so the agent must infer the routing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
movesA
The moves register: rewrite rules Scout tries where its built-in moves fail, each scored (wins+1)/(tries+2) on real reaches and retired after 5 straight losses. action=list [kind, host] | propose kind scope spec [note] | retire id. kinds: url_rewrite {pattern, repl} (regex on the full url; repl is literal text plus \1 groups) · locale_prefix {prefix} and listing_root {path}, which compose into listing paths (/en-us + /products) and are ranked, never retired · listing_path {path} (a whole path) · detail_suffix {suffix}. scope: '*', a domain ('example.com' covers www.example.com) or a host regex. Propose only what you saw work.
| Name | Required | Description | Default |
|---|---|---|---|
| id | No | ||
| host | No | ||
| kind | No | ||
| note | No | ||
| spec | No | ||
| scope | No | * | |
| action | No | list |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does well: it discloses the scoring formula (wins+1)/(tries+2), the retirement rule (5 straight losses), and the exception that locale_prefix/listing_root are ranked and never retired. It omits auth/permission requirements and any notion of what a mutation returns or whether it is reversible.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Extremely dense but front-loaded: the registry purpose and scoring rule come first, then action syntax, then kind/scope vocabularies. The terse pseudo-grammar earns its space, though it reads as a compressed spec rather than prose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 7-parameter, no-annotation, no-output-schema tool, the description covers actions, parameter vocabularies, scoring, and lifecycle rules well. It does not describe the shape of list results or what propose returns (e.g. a new id), which an agent might need when chaining calls.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it largely does: it defines kind values with their spec shapes (url_rewrite {pattern, repl} with \1 group semantics, locale_prefix, listing_root, listing_path, detail_suffix) and explains scope ('*', domain with www coverage, or host regex). Minor gaps remain for host/note as list filters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names the resource (the moves register), its role (rewrite rules Scout tries when built-in moves fail), and enumerates the three concrete actions (list, propose, retire). An agent can distinguish this from siblings like scout, reach, or site_map without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives the operating context (rules that fill in for failed built-in moves) and a clear directive: 'Propose only what you saw work.' Action syntax shows how to invoke each mode, but no alternative tool is named and no explicit when-not-to-use exclusion is stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
reachA
Reach one page and judge whether it is good information. Tries the host's learned readers winner-first, then
scored url rewrites and detail sub-pages. Returns ok, markdown (excerpt unless full=true), the trail tried, and on
failure an error {code, message, next_steps}. A page read today comes from Scout's mirror (fresh=true reads it
again). When the site walls Scout, or is on hold, the latest Wayback Machine copy is read instead, and archived
says when it was captured. cached=true reads only Scout's copy of the site (see cache) and sends no request.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| full | No | ||
| want | No | ||
| fresh | No | ||
| cached | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does so richly: it names the retrieval cascade, the return fields (ok, markdown, tried, error with code/message/next_steps), the mirror-vs-Wayback fallback and the `archived` capture time, and that cached=true issues no network request. It does not address permissions, rate limits, or whether any state is mutated, which are the remaining gaps for a network-fetching tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The purpose is front-loaded in the first sentence and every following clause carries behavioral information (return shape, mirror, fallback, cache semantics) with little filler. It is dense and somewhat run-on, and jargon like 'winner-first' and 'detail sub-pages' costs a little clarity without being wasteful.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema and no annotations, so the description correctly takes on the job of describing return values and fallback semantics, which it does well. The main shortfall for a five-parameter tool is the unexplained `want` parameter and the absence of any auth or rate-limit context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It explains three of five parameters inline (full=true for full markdown, fresh=true to re-read, cached=true for no-request mirror reads), but `url` and especially `want` (default "") are left entirely undefined in both schema and description. Partial compensation for a real coverage gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a concrete action on a concrete resource: fetch one page and judge whether it is good information, then names the internal retrieval strategies it tries (learned readers, url rewrites, detail sub-pages). This is distinguishable from purpose-level siblings like site_map or cache, though it never explicitly routes away from its closest sibling, scout. The idiosyncratic verb 'reach' is only clarified by the body text rather than the opening clause.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied rather than stated: cached=true 'sends no request' and fresh=true re-reads, which tells the agent about modes but not when to pick this tool over scout or site_map. Automatic fallback behavior (wall/hold → Wayback) is described, but there is no when-not guidance or explicit alternative named for the general case. An agent can infer the single-page-fetch role, but must infer it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
recipesC
The compiled recipes (optionally for one host): what each takes, its score, whether its last run failed, and the one-line card a small model runs.
| Name | Required | Description | Default |
|---|---|---|---|
| host | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations at all, the description carries the full behavioral burden, and it partially discharges it by naming the returned payload: what each recipe takes, its score, last-run-failure status, and its card. It is silent on permissions, ordering, pagination, and whether 'failed' reflects the most recent run only, so it is adequate but not rich.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One dense sentence that front-loads the resource and then lists the salient returned fields. It earns its length, though the closing clause about 'the one-line card a small model runs' is jargon that costs a little clarity for no functional gain.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with one optional parameter and no output schema, the description does describe the return shape in lieu of an output schema, which is the right instinct. It still leaves the agent unsure of ordering, completeness, and how failure status is scoped, so it is only minimally sufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% (the single 'host' property has no description), so the description must compensate. The phrase 'optionally for one host' does add meaning: it signals the parameter is an optional filter and identifies the filtering dimension, which is more than the bare schema gives, but it omits format/matching semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description conveys that this returns the collection of compiled recipes, with the per-recipe fields enumerated, and that it can optionally be scoped to one host. However, it is phrased as a noun phrase ('The compiled recipes') rather than a verb+resource action, and it never distinguishes itself from siblings like compile, run, or cache, so the agent must infer the operation is a read/list.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The only usage information is the parenthetical 'optionally for one host', which explains filtering but not when to call this tool. There is no guidance on when to prefer recipes over compile or run, or any stated preconditions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
runB
Run a compiled recipe. Reply with the RESULT line. inputs runs it over many inputs (one RESULT line each);
cached=true reads only Scout's copy of the site, sending no request (use after cache).
| Name | Required | Description | Default |
|---|---|---|---|
| input | No | ||
| cached | No | ||
| inputs | No | ||
| recipe | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations, so the description carries the full burden. It usefully discloses that cached=true sends no request and reads Scout's copy, and that `inputs` yields one RESULT line per input — real behavioral detail. But it omits error behavior, whether the run mutates remote state, auth/permission needs, and rate limits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight sentences, front-loaded with the core purpose before parameter notes. Dense but every clause carries information; no padding.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema and no annotations, so the description should do more. It hints at the return format ('RESULT line') but doesn't define it, describe failure modes, or resolve the input/inputs relationship — leaving real gaps for a 4-param execution tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% across 4 params, so the description must compensate. It explains `inputs` (batch) and `cached` (local read, no request) meaningfully, but leaves `input` (singular) versus `inputs` and the required `recipe` value unexplained, so half the parameters remain ambiguous.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource: 'Run a compiled recipe.' This clearly separates it from the sibling 'compile' (which produces a recipe) and 'cache'. However, it doesn't explicitly name or contrast the sibling tools, so differentiation is inferred rather than stated.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a conditional usage hint for one parameter ('cached=true ... use after cache') and notes the batch mode for `inputs`, but never states when to choose `run` over `compile`/`cache`, nor any prerequisites for running a recipe. Usage is implied rather than spelled out.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scoutC
Scout a url: reach the page (as markdown), map the site's url patterns, find its listing page, and return what
Scout has learned about the host. want is an optional regex the page must carry (a model number, 'price|$',
a heading); without it, good = real content that is not a wall. On failure page.error says why and what to try.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| full | No | ||
| want | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations the description carries the full burden. It does disclose some behavior: the return is markdown, it maps url patterns, it finds a listing page, it reports failures via 'page.error', and 'good = real content that is not a wall' hints at a content-quality judgment. It says nothing about rate limits, auth, caching, or what the successful return shape looks like.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The action is front-loaded in the first sentence, which is good, but the remaining sentences are cryptic shorthand ('good = real content that is not a wall', 'what Scout has learned') that costs more parsing effort than it saves. Appropriately sized, poorly worded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a multi-step tool with three parameters, zero schema coverage, no annotations, and no output schema, the description is too thin. It hints at 'page.error' but never describes the successful return, and leaves 'full' unexplained, so an agent cannot call it confidently.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It explains 'want' (an optional regex the page must carry, with concrete examples), but 'url' is only implied and 'full' is never mentioned at all, leaving one of three parameters completely undocumented in both schema and prose.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description enumerates concrete steps (reach the page as markdown, map url patterns, find the listing page, return findings), so the agent gets a rough sense of the operation. But it leans on undefined internal jargon like 'what Scout has learned about the host,' and the verb 'scout' itself is vague, so the exact purpose stays fuzzy.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no when-to-use guidance and no differentiation from closely related siblings such as 'reach' and 'site_map', which appear to overlap heavily with this tool's fetch-and-map behavior. The agent must guess which of the several crawling/navigation tools to pick.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
site_mapA
A site's real urls (robots.txt sitemaps, a polite link walk, the catalogue's root) with their url patterns.
filter is a regex. A listing is reused for 12 hours; fresh=true lists the site again (only when it changed:
re-listing a whole site run after run is how crawlers get blocked). archive='only' lists the site from the Wayback
Machine's index without a single request to it; 'auto' does so when the site is on hold or listed nothing.
Use it instead of guessing paths.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| fresh | No | ||
| limit | No | ||
| filter | No | ||
| archive | No | auto |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden and does well: it discloses a 12-hour listing reuse window, explains that fresh=true re-lists only when the site changed, warns that repeated full listing can get crawlers blocked, and describes Wayback Machine behavior for archive modes. It does not cover return format, pagination, or error behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but front-loaded with the tool's purpose and mechanisms. Every sentence adds some operational detail, though the parameter explanations are interleaved rather than clearly structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema and no annotations, so the description must be complete. It covers important behavioral details and hints at returned URL patterns, but omits the limit parameter, return format specifics, and error or permission behavior.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It explains filter as a regex, fresh=true semantics, and archive='only'/'auto' behavior, but never explains the url parameter or the limit parameter, leaving two of five parameters undocumented.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the resource and method: it returns a site's real URLs gathered from robots.txt sitemaps, a polite link walk, and the catalogue root, along with their URL patterns. It distinguishes itself from generic guessing rather than from named sibling tools, so it is clear but lacks explicit sibling differentiation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives clear context for use with 'Use it instead of guessing paths,' and explains when to use fresh=true and archive='only' or 'auto.' It does not name alternative sibling tools or state explicit exclusions, but the usage context is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
13 tool updates
v0.4.0- First observed
cache - First observed
compile - First observed
enroll - First observed
families - First observed
hold - First observed
laws - First observed
memory - First observed
moves - First observed
reach - First observed
recipes - First observed
run - First observed
scout - First observed
site_map
TDQS
Scored across 13 tools
The umbrella `scout` tool overlaps with `reach`, `site_map`, and `memory`, since it reaches a page, maps site patterns, and returns learned host data. Descriptions help distinguish the targeted tools, but an agent could still misselect `scout` when a more specific tool would suffice.
All names are lowercase and mostly single words, with one snake_case compound (`site_map`). However, the set mixes verbs (`scout`, `reach`, `compile`, `run`) and nouns (`memory`, `moves`, `recipes`, `families`), so there is no predictable verb_noun pattern.
With 13 tools, the count sits comfortably in the ideal 3–15 range for a complex web-scouting server. Each tool appears to cover a distinct phase of discovery, reading, rule management, caching, or enrollment.
The surface covers discovery, page reading, site mapping, link families, memory, rewrite rules, recipe compilation/execution, caching, blocking, enrollment, and legal codes. Minor lifecycle gaps exist, such as no explicit recipe deletion or cache clearing, but agents can work around them.
Maintenance
Related MCP Connectors
Web scraping for agents. Point it at a URL and it returns the page as clean markdown, JavaScript-rendered pages included. Point it at a site and it maps the URLs or crawls the section you need in the background, a few pages at a time so results fit in the conversation. Search the web and read full pages, extract fields with a JSON schema you define (validated, never invented), read a store's catalogue or a blog's posts from the platform's own feed, and check whether a page has changed. Failed requests cost nothing. The free plan includes 1,500 credits a month.
Pay-per-call web jobs for agents: URL probe, page extract, watch, JSON shape, researched briefs.
Agent-readiness scanner (0-5 score), robots.txt + llms.txt generators, managed agent enablement.
Deterministic web intake and data utilities for autonomous agents.
Related MCP Servers
- AlicenseAqualityCmaintenanceEnables AI agents to read web pages reliably, returning clean markdown content, hyperlinks, and metadata without navigation or ad noise.36 npmMIT
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to search the web, read and extract content from webpages, fetch JSON from REST APIs, and collect links while bypassing anti-bot protections.26 npmISC
- AlicenseNot gradedqualityCmaintenanceEnables agents to extract article text from public URLs or HTML, verify results against acceptance contracts, and fall back across extractors while ranking future attempts using local observed outcomes.MIT
- AlicenseNot gradedqualityAmaintenanceEnables agents to save reusable extraction recipes for public web pages, inspect page structures, and fetch fresh JSON or TSV data over plain HTTP, with a headless browser fallback when needed.27 npmMIT