Skip to main content
Glama

crawl-census-client

Ask before you fetch. A drop-in client that stops your crawler spending requests on doors that are shut, and stops it routing around content someone is trying to sell.

Reading robots.txt answers one question and hides two others. Measured across 23,482 domains by Crawl Census:

  • 2,874 domains permit AI agents in robots.txt and then refuse those same agents at the network edge. A parser sees permission; the fetch returns 403. You pay for the round trip and get nothing.

  • 208 domains answer an AI user agent with HTTP 402 Payment Required. That is a price, not a refusal. Treating it as a block walks away from content the operator wants to sell you. Retrying around it takes something they are charging for.

No dependencies. No key required.

MCP server

The same measurement is exposed as a remote MCP server, so an agent can ask before it fetches rather than after it fails. Listed in the official MCP registry as io.github.taylorsmithgg/crawl-census.

{ "mcpServers": { "crawl-census": { "url": "https://crawlcensus.com/mcp" } } }

Tool

Answers

crawl_preflight

will these domains serve my agent, refuse it, or charge it?

agent_profile

what does this census publish about my crawler, and how do I correct it?

census_facts

the headline findings as dated records with denominators and citation lines

site_report

the stored audit for one domain

scan_site

measure a domain now

census_stats

corpus-level totals

No authentication for read tools. Streamable HTTP.

Related MCP server: domain-intel-mcp

Install

npm i github:taylorsmithgg/crawl-census-client
pip install git+https://github.com/taylorsmithgg/crawl-census-client

Use

import { politeFetch } from "crawl-census-client";

const r = await politeFetch("https://example.com/", { agent: "gptbot" });
if (r.skipped) console.log(r.verdict, r.reason);   // disallow | refuse | pay
else           process(await r.response.text());
from crawl_census import polite_fetch

r = polite_fetch("https://example.com/", agent="gptbot")
if r.skipped:
    print(r.verdict, r.reason)
else:
    process(r.body)

Skipping is returned, not raised. It is the normal outcome for a large share of the web, and a crawl loop should be able to count skips without a try/except around every URL.

Split a queue before crawling it

One call per 1,000 domains instead of one per host:

const { crawl, skip, pay, unknown } = await partition(urls, { agent: "gptbot" });
p = partition(urls, agent="gptbot")
p.crawl, p.skip, p.pay, p.unknown

Or just take the file

For a fetcher that only needs a deny list in memory, skip the per-domain calls entirely:

curl https://crawlcensus.com/agents/gptbot/blocklist.txt   # one domain per line, commented header
const sync = await syncBlocklist("gptbot");   // full list once
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000);  // then deltas only, a few hundred bytes
sync = BlocklistSync("gptbot")
if host in sync: skip()
sync.refresh()          # {'added': 3, 'removed': 1, 'size': 3310, 'cursor': ...}

The delta feed is https://crawlcensus.com/agents/<agent>/changes.json?since=<unix> and each response carries next_since, so a long-running crawler stays current on a few hundred bytes an hour instead of re-downloading the list.

That file covers robots.txt only. Edge refusal and HTTP 402 are per-request behaviours and still need preflight or politeFetch.

What a crawl costs the census

Measured, not asserted. Twenty hosts fetched concurrently used to cost twenty preflight calls carrying one domain each; the same host requested three times at once cost three, because the cache only helps after the first lookup resolves. The anonymous allowance is 240 calls an hour, so a crawler hit its ceiling at 240 hosts when one call covers twenty-five.

politeFetch now shares work automatically: lookups issued in the same tick leave as one batched call, and concurrent lookups for the same host await a single request.

pattern

before

now

20 hosts, concurrent

20 calls

1 call of 20

1 host, 3 URLs, concurrent

3 calls

1 call

60 hosts, concurrent

60 calls

3 calls (25 / 25 / 10)

partition then fetch

2 calls

1 call

batchSize defaults to 25, the per-call cap without a key. Raise it with a Pro or Data key. batchWaitMs widens the coalescing window for concurrency that arrives in waves rather than all at once; the default of zero flushes on the next tick.

Paying, when an origin quotes a price

A pay verdict carries the amount when the origin named one:

const r = await politeFetch(url, { agent: "claudebot" });
if (r.verdict === "pay") console.log(r.price);   // "USD 0.5", or null if none was quoted

Two things worth knowing. Most origins answering HTTP 402 name no amount at all, so price is usually null and the arrangement has to be made out of band. And pricing is per crawler: across the measured corpus, 78 of 213 charging origins charge some agents and serve others free, so ask with your own token rather than assuming a domain on the list will charge you.

Two kinds of unknown

partition splits a work queue into crawl, pay, skip, unknown and undecidable.

The last two look alike and are not. unknown means the census has not measured that domain yet: submit it and the next pass gets a real verdict. undecidable means the site's robots.txt disallows CrawlCensusBot, so this census will never measure it — retrying is guaranteed waste, and a loop that resubmits its unknowns each pass would resubmit those forever. The server marks the difference with a measurable boolean; read that, never the reason prose.

const p = await partition(urls, { agent: "gptbot" });
await Promise.all(p.crawl.map(politeFetchOne));
if (p.unmeasured.length) await submitUnmeasured(p, { agent: "gptbot" });
// p.undecidable: read their robots.txt yourself. Asking us again cannot help.

Submission is a separate call on purpose. A library that quietly POSTs during what reads as a lookup is a bad citizen, and you should choose when your queue positions are spent.

Skipping everything that will not serve you

A deny list is the smaller half. Measured against the live census, a crawler that skips only robots disallows still spends around 2,900 requests a pass on domains that permit it in robots.txt and refuse it at the edge, or that answer HTTP 402 — for PerplexityBot that set is larger than its deny list. Those fetches return nothing and cost a round trip each.

const skip = await syncSkipList("gptbot");
if (skip.has(host)) continue;        // disallowed, refused at the edge, or priced
skip.why(host);                      // "disallow" | "pay" | "refuse" | null
setInterval(() => skip.refresh(), 3600_000);

agent

deny list

also skippable

total

GPTBot

3,542

2,944

6,486

ClaudeBot

3,169

3,194

6,363

PerplexityBot

1,128

3,658

4,786

The three sets are kept apart internally, so a change moves the one it belongs to. why() follows the same precedence as preflight: a disallow outranks a price, because a price is not permission.

Keeping a deny list current

syncBlocklist / BlocklistSync download the list once, then apply only what changed.

The list is served with the exact position in the change feed it was built at, in an x-cursor header and a # cursor: comment. The clients read it and resume from there, so there is no gap between the snapshot and the first poll, and no reliance on your clock being in step with the server's. Polling by second cannot express a position inside a second, and a crawl batch writes dozens of events into one, so a second-granularity resume can drop the remainder of it: measured live, resuming after the first of three same-second changes recovered both siblings by cursor and neither by second.

const sync = await syncBlocklist("gptbot");   // cursor comes from the list itself
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000);  // a few hundred bytes per poll

refresh() applies only robots transitions to the list, because that is what the list is made of. Edge refusals, new prices and llms.txt changes come back in other for you to act on separately — an earlier version deleted those domains from the deny list, so a crawler resumed fetching exactly what had just started refusing it.

Verdicts

The authoritative definition of each verdict — what it means, what it obliges a crawler to do, and whether asking again could change it — is published as data at /api/v1/verdicts. The list below is a summary; if the two ever disagree, the endpoint is right and this file is stale.

politeFetch skips disallow, refuse and pay by default, which is the endpoint's derived do_not_fetch set. The copy here is deliberate — a crawl loop should not need a network call to decide — and a test compares the two so it cannot drift unnoticed.

Verdict

Meaning

Default behaviour

allow

robots.txt permits this agent, and a live request carrying its user agent was served

fetch

disallow

robots.txt forbids this agent at the site root

skip

refuse

robots.txt permits it; the edge refused it anyway. The allowance is not real

skip

pay

the origin answered HTTP 402. It will serve this agent on commercial terms

skip

unknown

not measured recently enough to answer

fetch

onPay: "fetch" (on_pay="fetch") overrides the paywall default. It is an explicit opt-in and is recorded on the result as paidRouteOverridden so it shows up in your logs.

It degrades, it does not fail

If the census is unreachable every verdict becomes unknown and your crawl proceeds as it normally would. A third-party outage must never stop your pipeline. There is a live test for exactly this.

What we publish about your agent

const p = await agentProfile("claudebot");
// robots disallow rate, edge refusal rate, operator page, correction channel

If a figure is wrong, the correction channel is in that response and on your operator page. Registry facts are corrected without argument; disputed measurements are published alongside the dispute with the underlying scan records, rather than quietly amended.

Limits

25 domains per preflight call anonymously, 200 with a Pro key, 1,000 with a Data key. Pass apiKey. Details at https://crawlcensus.com/for-crawlers.

Tests

node test.mjs runs against the live census on purpose. The value of this client is whether its verdicts match reality, and a mocked test would assert only that the mock agrees with itself.

MIT. Data is CC BY 4.0, attribute as "Source: Crawl Census (crawlcensus.com)".

Available Tools

7 tools
agent_profileAInspect

What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the correction channel and the public page a claim can be disputed against.

Inputs: agent (required).

census_factsAInspect

Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date, which is what makes the number wrong when it is repeated.

Inputs: no named parameters.

census_statsBInspect

Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.

Inputs: no named parameters.

crawl_preflightAInspect

Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it), refuse (robots permits it but the edge refused the agent anyway, so the allowance is not real), pay (the origin answered HTTP 402 Payment Required, meaning it will serve this agent on commercial terms), or unknown. The full definition of each, including what it obliges a crawler to do, is published at https://crawlcensus.com/api/v1/verdicts. Built for crawler operators rather than site owners: it prevents wasted fetches against doors that are shut, and flags content an operator is trying to sell rather than withhold.

Inputs: agent (required), domains (required).

scan_siteAInspect

Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.

Inputs: domain (required).

site_reportAInspect

Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.

Inputs: domain (required).

submit_domainsAInspect

Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a database write rather than a fetch, so the allowance is far higher than scan_site and submitted domains are measured ahead of the ranked backlog.

Inputs: domains (required).

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.9/5.0

    Scored across 7 tools

    Disambiguation4/5

    scan_site and site_report both return audits for a domain, but one is a live scan and the other is a stored result, so they are distinguishable. census_stats and census_facts also overlap somewhat as corpus-level outputs, but one gives direct statistics and the other gives citation-ready findings. The remaining tools each have clearly separate roles.

    Naming Consistency4/5

    All names are lowercase snake_case and mostly combine a domain noun with a result or action, such as site_report, census_stats, agent_profile, and crawl_preflight. scan_site and submit_domains are verb-first while the others are noun-first, so the pattern is not perfectly uniform, but the style is still readable and predictable.

    Tool Count5/5

    Seven tools is a well-scoped set for a crawl-census service: each tool covers a distinct consumer need, from live audits and stored reports to crawler preflight decisions, agent profiles, citation-ready facts, and domain submission. No tool is redundant, and the count feels appropriately sized for the domain.

    Completeness4/5

    The set covers the main workflows well: running and retrieving audits, querying corpus statistics and facts, profiling individual crawlers, deciding whether to fetch domains, and queueing unmeasured domains. Minor gaps include no obvious way to list all tracked agents or remove queued submissions, but these are not core to the documented purpose.

    Maintenance

    ActivityInactive
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      C
      maintenance
      Enables AI agents to check whether a public website is crawlable, understandable, and ready for AI search workflows through local-only audits of robots.txt, sitemaps, metadata, and llms.txt.
      3
      73 npm
      2
      MIT
    • A
      license
      A
      quality
      B
      maintenance
      Domain and company intelligence for AI agents. Enables vetting companies, qualifying leads, and mapping targets from free public data without API keys.
      6
      6 npm
      1
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Provides agents with tools to score a page's AI readiness, check which AI crawlers robots.txt blocks, and assess passage citability—all performed locally.
      58 npm
      2
      MIT