Crawl Census
Server Details
Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?
- Status
- Healthy
- Uptime
- 100.0% over the last 24 hours
- Last Tested
- Transport
- Streamable HTTP · MCP 2025-06-18
- URL
- Repository
- taylorsmithgg/crawl-census-client
- GitHub Stars
- 0
- Server Listing
- crawl-census
- ChatGPT
- –
- Claude
- –
TDQS
Scored across 7 tools
census_facts and census_stats are easy to conflate, and crawl_preflight, scan_site, and site_report overlap in that they each inspect a domain's crawler accessibility. The descriptions eventually separate them by output intent — verdicts vs audit vs stored report — but an agent must read carefully to avoid a wrong call.
All names use snake_case and are readable, but there is no consistent verb-noun rule: scan_site and submit_domains start with verbs, while census_facts, census_stats, agent_profle, crawl_preflight, and site_report are noun-like phrases. This mixed convention is understandable, though it misses the clarity of a uniform pattern.
Seven tools are well-scoped for a census-focused server: they cover agent lookups, corpus statistics, domain verdicts, audits, stored reports, and domain submission. None feels like filler, and the count stays small enough for an agent to navigate.
The core read, audit, report, and submission loop is covered: preflight verdicts, full scanned audits, cached reports, and bulk domain queueing. The only minor gap is a lack of tools to update or delete domain submissions or cached reports, but this does not seriously affect the intended census and crawler workflows.
Available Tools
7 toolsagent_profileAInspect
What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the correction channel and the public page a claim can be disputed against.
Inputs: agent (required).
census_factsAInspect
Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date, which is what makes the number wrong when it is repeated.
Inputs: no named parameters.
census_statsBInspect
Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.
Inputs: no named parameters.
crawl_preflightAInspect
Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it), refuse (robots permits it but the edge refused the agent anyway, so the allowance is not real), pay (the origin answered HTTP 402 Payment Required, meaning it will serve this agent on commercial terms), or unknown. The full definition of each, including what it obliges a crawler to do, is published at https://crawlcensus.com/api/v1/verdicts. Built for crawler operators rather than site owners: it prevents wasted fetches against doors that are shut, and flags content an operator is trying to sell rather than withhold.
Inputs: agent (required), domains (required).
scan_siteAInspect
Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.
Inputs: domain (required).
site_reportAInspect
Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.
Inputs: domain (required).
submit_domainsAInspect
Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a database write rather than a fetch, so the allowance is far higher than scan_site and submitted domains are measured ahead of the ranked backlog.
Inputs: domains (required).
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Related MCP Connectors
Ask before you scrape: is a site open, may I fetch this URL, and how do I extract a page. No key.
Policy-aware web access preflight: robots, licensing, payment and advertised routes for agents.
Whether an AI agent may fetch a URL, on what terms, and at what price. Free, no key, no signup.
Cited AI crawler registry (training, search or user fetch) plus robots.txt checks. No fetching.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables MCP clients to check whether a public domain is under observation in an EU evidentiary archive of the machine-readable web, and from which sealed day, suggest a domain for observation, and order qualified time-stamped records of up to twenty public addresses. It runs remotely over streamable HTTP without authentication and never returns file contents or charges payment itself.MIT
- AlicenseNot gradedqualityCmaintenanceEnables users to check whether GPTBot, ClaudeBot, PerplexityBot, Googlebot and other AI/search crawlers can reach a specified page, distinguishing robots.txt blocks from server or firewall refusals.MIT
- AlicenseNot gradedqualityDmaintenancePerforms comprehensive crawler compliance risk assessment for webpages, evaluating legal, ethical, and technical risks including robots.txt rules, anti-crawling mechanisms, sensitive data detection, and providing crawl permissibility ratings.3MIT

QueryRev AI Readinessofficial
AlicenseAqualityCmaintenanceChecks a site for AI crawlers: which of 11 AI crawlers its robots.txt lets in, whether its llms.txt follows llmstxt.org, and what its home page says without scripts. Findings with evidence; no scores, no model calls, no telemetry.549 npmMIT
Glama MCP Gateway
Add one secure layer between your agents and this server.