crawl-census
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@crawl-census preflight these domains for gptbot"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
crawl-census-client
Ask before you fetch. A drop-in client that stops your crawler spending requests on doors that are shut, and stops it routing around content someone is trying to sell.
Reading robots.txt answers one question and hides two others. Measured across 23,482 domains by Crawl Census:
2,874 domains permit AI agents in robots.txt and then refuse those same agents at the network edge. A parser sees permission; the fetch returns 403. You pay for the round trip and get nothing.
208 domains answer an AI user agent with
HTTP 402 Payment Required. That is a price, not a refusal. Treating it as a block walks away from content the operator wants to sell you. Retrying around it takes something they are charging for.
No dependencies. No key required.
MCP server
The same measurement is exposed as a remote MCP server, so an agent can ask before it fetches
rather than after it fails. Listed in the
official MCP registry as
io.github.taylorsmithgg/crawl-census.
{ "mcpServers": { "crawl-census": { "url": "https://crawlcensus.com/mcp" } } }Tool | Answers |
| will these domains serve my agent, refuse it, or charge it? |
| what does this census publish about my crawler, and how do I correct it? |
| the headline findings as dated records with denominators and citation lines |
| the stored audit for one domain |
| measure a domain now |
| corpus-level totals |
No authentication for read tools. Streamable HTTP.
Related MCP server: domain-intel-mcp
Install
npm i github:taylorsmithgg/crawl-census-client
pip install git+https://github.com/taylorsmithgg/crawl-census-clientUse
import { politeFetch } from "crawl-census-client";
const r = await politeFetch("https://example.com/", { agent: "gptbot" });
if (r.skipped) console.log(r.verdict, r.reason); // disallow | refuse | pay
else process(await r.response.text());from crawl_census import polite_fetch
r = polite_fetch("https://example.com/", agent="gptbot")
if r.skipped:
print(r.verdict, r.reason)
else:
process(r.body)Skipping is returned, not raised. It is the normal outcome for a large share of the web, and a crawl loop should be able to count skips without a try/except around every URL.
Split a queue before crawling it
One call per 1,000 domains instead of one per host:
const { crawl, skip, pay, unknown } = await partition(urls, { agent: "gptbot" });p = partition(urls, agent="gptbot")
p.crawl, p.skip, p.pay, p.unknownOr just take the file
For a fetcher that only needs a deny list in memory, skip the per-domain calls entirely:
curl https://crawlcensus.com/agents/gptbot/blocklist.txt # one domain per line, commented headerconst sync = await syncBlocklist("gptbot"); // full list once
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000); // then deltas only, a few hundred bytessync = BlocklistSync("gptbot")
if host in sync: skip()
sync.refresh() # {'added': 3, 'removed': 1, 'size': 3310, 'cursor': ...}The delta feed is https://crawlcensus.com/agents/<agent>/changes.json?since=<unix> and each
response carries next_since, so a long-running crawler stays current on a few hundred bytes
an hour instead of re-downloading the list.
That file covers robots.txt only. Edge refusal and HTTP 402 are per-request behaviours and
still need preflight or politeFetch.
What a crawl costs the census
Measured, not asserted. Twenty hosts fetched concurrently used to cost twenty preflight calls carrying one domain each; the same host requested three times at once cost three, because the cache only helps after the first lookup resolves. The anonymous allowance is 240 calls an hour, so a crawler hit its ceiling at 240 hosts when one call covers twenty-five.
politeFetch now shares work automatically: lookups issued in the same tick leave as one
batched call, and concurrent lookups for the same host await a single request.
pattern | before | now |
20 hosts, concurrent | 20 calls | 1 call of 20 |
1 host, 3 URLs, concurrent | 3 calls | 1 call |
60 hosts, concurrent | 60 calls | 3 calls (25 / 25 / 10) |
| 2 calls | 1 call |
batchSize defaults to 25, the per-call cap without a key. Raise it with a Pro or Data key.
batchWaitMs widens the coalescing window for concurrency that arrives in waves rather than
all at once; the default of zero flushes on the next tick.
Paying, when an origin quotes a price
A pay verdict carries the amount when the origin named one:
const r = await politeFetch(url, { agent: "claudebot" });
if (r.verdict === "pay") console.log(r.price); // "USD 0.5", or null if none was quotedTwo things worth knowing. Most origins answering HTTP 402 name no amount at all, so price
is usually null and the arrangement has to be made out of band. And pricing is per crawler:
across the measured corpus, 78 of 213 charging origins charge some agents and serve others
free, so ask with your own token rather than assuming a domain on the list will charge you.
Two kinds of unknown
partition splits a work queue into crawl, pay, skip, unknown and undecidable.
The last two look alike and are not. unknown means the census has not measured that domain
yet: submit it and the next pass gets a real verdict. undecidable means the site's robots.txt
disallows CrawlCensusBot, so this census will never measure it — retrying is guaranteed waste,
and a loop that resubmits its unknowns each pass would resubmit those forever. The server marks
the difference with a measurable boolean; read that, never the reason prose.
const p = await partition(urls, { agent: "gptbot" });
await Promise.all(p.crawl.map(politeFetchOne));
if (p.unmeasured.length) await submitUnmeasured(p, { agent: "gptbot" });
// p.undecidable: read their robots.txt yourself. Asking us again cannot help.Submission is a separate call on purpose. A library that quietly POSTs during what reads as a lookup is a bad citizen, and you should choose when your queue positions are spent.
Skipping everything that will not serve you
A deny list is the smaller half. Measured against the live census, a crawler that skips only robots disallows still spends around 2,900 requests a pass on domains that permit it in robots.txt and refuse it at the edge, or that answer HTTP 402 — for PerplexityBot that set is larger than its deny list. Those fetches return nothing and cost a round trip each.
const skip = await syncSkipList("gptbot");
if (skip.has(host)) continue; // disallowed, refused at the edge, or priced
skip.why(host); // "disallow" | "pay" | "refuse" | null
setInterval(() => skip.refresh(), 3600_000);agent | deny list | also skippable | total |
GPTBot | 3,542 | 2,944 | 6,486 |
ClaudeBot | 3,169 | 3,194 | 6,363 |
PerplexityBot | 1,128 | 3,658 | 4,786 |
The three sets are kept apart internally, so a change moves the one it belongs to. why()
follows the same precedence as preflight: a disallow outranks a price, because a price is not
permission.
Keeping a deny list current
syncBlocklist / BlocklistSync download the list once, then apply only what changed.
The list is served with the exact position in the change feed it was built at, in an
x-cursor header and a # cursor: comment. The clients read it and resume from there, so
there is no gap between the snapshot and the first poll, and no reliance on your clock being
in step with the server's. Polling by second cannot express a position inside a second, and a
crawl batch writes dozens of events into one, so a second-granularity resume can drop the
remainder of it: measured live, resuming after the first of three same-second changes
recovered both siblings by cursor and neither by second.
const sync = await syncBlocklist("gptbot"); // cursor comes from the list itself
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000); // a few hundred bytes per pollrefresh() applies only robots transitions to the list, because that is what the list is made
of. Edge refusals, new prices and llms.txt changes come back in other for you to act on
separately — an earlier version deleted those domains from the deny list, so a crawler resumed
fetching exactly what had just started refusing it.
Verdicts
The authoritative definition of each verdict — what it means, what it obliges a crawler to do,
and whether asking again could change it — is published as data at
/api/v1/verdicts. The list below is a summary; if
the two ever disagree, the endpoint is right and this file is stale.
politeFetch skips disallow, refuse and pay by default, which is the endpoint's derived
do_not_fetch set. The copy here is deliberate — a crawl loop should not need a network call to
decide — and a test compares the two so it cannot drift unnoticed.
Verdict | Meaning | Default behaviour |
| robots.txt permits this agent, and a live request carrying its user agent was served | fetch |
| robots.txt forbids this agent at the site root | skip |
| robots.txt permits it; the edge refused it anyway. The allowance is not real | skip |
| the origin answered HTTP 402. It will serve this agent on commercial terms | skip |
| not measured recently enough to answer | fetch |
onPay: "fetch" (on_pay="fetch") overrides the paywall default. It is an explicit opt-in and
is recorded on the result as paidRouteOverridden so it shows up in your logs.
It degrades, it does not fail
If the census is unreachable every verdict becomes unknown and your crawl proceeds as it
normally would. A third-party outage must never stop your pipeline. There is a live test for
exactly this.
What we publish about your agent
const p = await agentProfile("claudebot");
// robots disallow rate, edge refusal rate, operator page, correction channelIf a figure is wrong, the correction channel is in that response and on your operator page. Registry facts are corrected without argument; disputed measurements are published alongside the dispute with the underlying scan records, rather than quietly amended.
Limits
25 domains per preflight call anonymously, 200 with a Pro key, 1,000 with a Data key. Pass
apiKey. Details at https://crawlcensus.com/for-crawlers.
Tests
node test.mjs runs against the live census on purpose. The value of this client is whether
its verdicts match reality, and a mocked test would assert only that the mock agrees with itself.
MIT. Data is CC BY 4.0, attribute as "Source: Crawl Census (crawlcensus.com)".
Available Tools
7 toolsagent_profileAInspect
What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the correction channel and the public page a claim can be disputed against.
Inputs: agent (required).
census_factsAInspect
Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date, which is what makes the number wrong when it is repeated.
Inputs: no named parameters.
census_statsBInspect
Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.
Inputs: no named parameters.
crawl_preflightAInspect
Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it), refuse (robots permits it but the edge refused the agent anyway, so the allowance is not real), pay (the origin answered HTTP 402 Payment Required, meaning it will serve this agent on commercial terms), or unknown. The full definition of each, including what it obliges a crawler to do, is published at https://crawlcensus.com/api/v1/verdicts. Built for crawler operators rather than site owners: it prevents wasted fetches against doors that are shut, and flags content an operator is trying to sell rather than withhold.
Inputs: agent (required), domains (required).
scan_siteAInspect
Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.
Inputs: domain (required).
site_reportAInspect
Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.
Inputs: domain (required).
submit_domainsAInspect
Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a database write rather than a fetch, so the allowance is far higher than scan_site and submitted domains are measured ahead of the ranked backlog.
Inputs: domains (required).
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 7 tools
scan_site and site_report both return audits for a domain, but one is a live scan and the other is a stored result, so they are distinguishable. census_stats and census_facts also overlap somewhat as corpus-level outputs, but one gives direct statistics and the other gives citation-ready findings. The remaining tools each have clearly separate roles.
All names are lowercase snake_case and mostly combine a domain noun with a result or action, such as site_report, census_stats, agent_profile, and crawl_preflight. scan_site and submit_domains are verb-first while the others are noun-first, so the pattern is not perfectly uniform, but the style is still readable and predictable.
Seven tools is a well-scoped set for a crawl-census service: each tool covers a distinct consumer need, from live audits and stored reports to crawler preflight decisions, agent profiles, citation-ready facts, and domain submission. No tool is redundant, and the count feels appropriately sized for the domain.
The set covers the main workflows well: running and retrieving audits, querying corpus statistics and facts, profiling individual crawlers, deciding whether to fetch domains, and queueing unmeasured domains. Minor gaps include no obvious way to list all tracked agents or remove queued submissions, but these are not core to the documented purpose.
Maintenance
Related MCP Connectors
Policy-aware web access preflight: robots, licensing, payment and advertised routes for agents.
Flags phantom-squatted/typosquatted domains and known-malicious URLs before an agent follows them.
Check if a website will block your AI agent and find its llms.txt, MCP or OpenAPI door.
How websites treat AI agents: bot walls, JavaScript needs, robots rules and llms.txt. Look a site up before you fetch it, report what you saw after. Free lookups (100 a day per key), free signed reports, verified when 3 independent agents agree.
Related MCP Servers
- AlicenseAqualityCmaintenanceEnables AI agents to check whether a public website is crawlable, understandable, and ready for AI search workflows through local-only audits of robots.txt, sitemaps, metadata, and llms.txt.373 npm2MIT
- AlicenseAqualityBmaintenanceDomain and company intelligence for AI agents. Enables vetting companies, qualifying leads, and mapping targets from free public data without API keys.66 npm1MIT
- FlicenseNot gradedqualityCmaintenanceProvides pre-fetch routing intelligence for AI agents by recommending the cheapest reliable route (HTTP, browser, machine endpoint, or avoid) before visiting a URL.-
- AlicenseNot gradedqualityBmaintenanceProvides agents with tools to score a page's AI readiness, check which AI crawlers robots.txt blocks, and assess passage citability—all performed locally.58 npm2MIT