Skip to main content
Glama

crawl-census-client

가져오기 전에 물어보세요. 크롤러가 닫힌 문에 요청을 낭비하지 않고, 누군가 팔려는 콘텐츠를 우회하지 않도록 막아주는 드롭인 클라이언트입니다.

robots.txt를 읽는 것은 하나의 질문에 답하고 두 가지를 숨깁니다. Crawl Census가 측정한 23,482개 도메인 기준:

  • 2,874개 도메인은 robots.txt에서 AI 에이전트를 허용하면서도 네트워크 경계에서 동일한 에이전트를 거부합니다. 파서는 허가를 보고, 가져오기는 403을 반환합니다. 왕복 비용을 지불하고 아무것도 얻지 못합니다.

  • 208개 도메인은 AI 사용자 에이전트에 HTTP 402 Payment Required로 응답합니다. 그것은 거부가 아니라 가격입니다. 차단으로 취급하면 운영자가 판매하려는 콘텐츠에서 발을 빼는 것입니다. 우회해서 재시도하면 요금을 청구하는 것을 가져가는 것입니다.

의존성 없음. 키 불필요.

MCP 서버

동일한 측정이 원격 MCP 서버로 노출되어, 에이전트가 실패한 후가 아니라 가져오기 전에 물어볼 수 있습니다. 공식 MCP 레지스트리에 io.github.taylorsmithgg/crawl-census로 등재되어 있습니다.

{ "mcpServers": { "crawl-census": { "url": "https://crawlcensus.com/mcp" } } }

도구

답변

crawl_preflight

이 도메인들이 내 에이전트를 서빙할지, 거부할지, 청구할지?

agent_profile

이 인구조사가 내 크롤러에 대해 게시하는 내용과 수정 방법은?

census_facts

분모와 인용 줄이 있는 날짜 기록으로 된 주요 발견 사항

site_report

한 도메인에 대한 저장된 감사 기록

scan_site

지금 도메인 측정

census_stats

코퍼스 수준 합계

읽기 도구에 인증 불필요. Streamable HTTP.

Related MCP server: domain-intel-mcp

설치

npm i github:taylorsmithgg/crawl-census-client
pip install git+https://github.com/taylorsmithgg/crawl-census-client

사용

import { politeFetch } from "crawl-census-client";

const r = await politeFetch("https://example.com/", { agent: "gptbot" });
if (r.skipped) console.log(r.verdict, r.reason);   // disallow | refuse | pay
else           process(await r.response.text());
from crawl_census import polite_fetch

r = polite_fetch("https://example.com/", agent="gptbot")
if r.skipped:
    print(r.verdict, r.reason)
else:
    process(r.body)

건너뛰기는 발생시키지 않고 반환됩니다. 웹의 상당 부분에서 정상적인 결과이며, 크롤 루프는 모든 URL에 try/except 없이 건너뛰기를 셀 수 있어야 합니다.

크롤링 전에 큐 분할

호스트당 한 번이 아니라 1,000개 도메인당 한 번 호출:

const { crawl, skip, pay, unknown } = await partition(urls, { agent: "gptbot" });
p = partition(urls, agent="gptbot")
p.crawl, p.skip, p.pay, p.unknown

아니면 그냥 파일을 가져가세요

메모리에 거부 목록만 필요한 페처라면 도메인별 호출을 완전히 건너뛰세요:

curl https://crawlcensus.com/agents/gptbot/blocklist.txt   # one domain per line, commented header
const sync = await syncBlocklist("gptbot");   // full list once
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000);  // then deltas only, a few hundred bytes
sync = BlocklistSync("gptbot")
if host in sync: skip()
sync.refresh()          # {'added': 3, 'removed': 1, 'size': 3310, 'cursor': ...}

델타 피드는 https://crawlcensus.com/agents/<agent>/changes.json?since=<unix>이며 각 응답은 next_since를 전달하므로, 장기 실행 크롤러는 목록을 다시 다운로드하는 대신 시간당 수백 바이트로 최신 상태를 유지합니다.

그 파일은 robots.txt만 다룹니다. 경계 거부와 HTTP 402는 요청별 동작이므로 여전히 preflight 또는 politeFetch가 필요합니다.

크롤이 인구조사에 드는 비용

주장이 아닌 측정입니다. 동시에 가져온 20개 호스트는 각각 하나의 도메인을 담은 20번의 preflight 호출이 들었습니다. 같은 호스트를 세 번 동시에 요청하면 첫 조회가 해결된 후에만 캐시가 도움이 되므로 세 번의 비용이 들었습니다. 익명 허용량은 시간당 240회 호출이므로, 한 번 호출로 25개를 처리할 수 있을 때 크롤러는 240개 호스트에서 한계에 도달했습니다.

politeFetch는 이제 자동으로 작업을 공유합니다: 같은 틱에서 발행된 조회는 하나의 배치 호출로 나가고, 같은 호스트에 대한 동시 조회는 단일 요청을 기다립니다.

패턴

이전

지금

20개 호스트, 동시

20회 호출

20개 중 1회 호출

1개 호스트, 3개 URL, 동시

3회 호출

1회 호출

60개 호스트, 동시

60회 호출

3회 호출 (25 / 25 / 10)

partition 후 가져오기

2회 호출

1회 호출

batchSize는 키 없이 호출당 상한인 25가 기본값입니다. Pro 또는 Data 키로 올리세요. batchWaitMs는 한 번에 도착하는 것이 아니라 파도처럼 도착하는 동시성에 대한 병합 창을 넓힙니다. 기본값 0은 다음 틱에 플러시합니다.

출처가 가격을 제시할 때 지불

pay 판정은 출처가 금액을 명시했을 때 그 금액을 전달합니다:

const r = await politeFetch(url, { agent: "claudebot" });
if (r.verdict === "pay") console.log(r.price);   // "USD 0.5", or null if none was quoted

알아둘 두 가지가 있습니다. HTTP 402로 응답하는 대부분의 출처는 금액을 전혀 명시하지 않으므로 price는 보통 null이고 계약은 대역 외로 체결해야 합니다. 그리고 가격은 크롤러별로 다릅니다: 측정된 코퍼스에서 213개 청구 출처 중 78개는 일부 에이전트에게 청구하고 다른 에이전트에게는 무료로 서빙하므로, 목록에 있는 도메인이 당신에게 청구할 것이라고 가정하지 말고 자신의 토큰으로 물어보세요.

두 종류의 미지

partition은 작업 큐를 crawl, pay, skip, unknown, undecidable로 분할합니다.

마지막 두 개는 비슷해 보이지만 다릅니다. unknown은 인구조사가 아직 그 도메인을 측정하지 않았다는 뜻입니다: 제출하면 다음 패스에서 실제 판정을 받습니다. undecidable은 사이트의 robots.txt가 CrawlCensusBot을 허용하지 않아 이 인구조사가 결코 측정하지 못한다는 뜻입니다 — 재시도는 보장된 낭비이며, 매 패스 미지를 재제출하는 루프는 그것들을 영원히 재제출할 것입니다. 서버는 measurable 불리언으로 차이를 표시합니다. reason 문구가 아니라 그것을 읽으세요.

const p = await partition(urls, { agent: "gptbot" });
await Promise.all(p.crawl.map(politeFetchOne));
if (p.unmeasured.length) await submitUnmeasured(p, { agent: "gptbot" });
// p.undecidable: read their robots.txt yourself. Asking us again cannot help.

제출은 의도적으로 별도의 호출입니다. 조회처럼 보이는 동안 조용히 POST하는 라이브러리는 나쁜 시민이며, 큐 위치가 언제 소비될지 선택해야 합니다.

당신을 서빙하지 않을 모든 것 건너뛰기

거부 목록은 작은 절반입니다. 실시간 인구조사에 대해 측정했을 때, robots disallow만 건너뛰는 크롤러는 robots.txt에서 허용하면서 경계에서 거부하거나 HTTP 402로 응답하는 도메인에 패스당 약 2,900회의 요청을 여전히 소비합니다 — PerplexityBot의 경우 그 집합이 거부 목록보다 큽니다. 그 가져오기들은 아무것도 반환하지 않고 각각 왕복 비용이 듭니다.

const skip = await syncSkipList("gptbot");
if (skip.has(host)) continue;        // disallowed, refused at the edge, or priced
skip.why(host);                      // "disallow" | "pay" | "refuse" | null
setInterval(() => skip.refresh(), 3600_000);

에이전트

거부 목록

추가로 건너뛸 수 있음

합계

GPTBot

3,542

2,944

6,486

ClaudeBot

3,169

3,194

6,363

PerplexityBot

1,128

3,658

4,786

세 집합은 내부적으로 분리되어 있어 변경이 속한 집합만 이동합니다. why()는 preflight와 같은 우선순위를 따릅니다: disallow가 가격보다 우선합니다. 가격은 허가가 아니기 때문입니다.

거부 목록 최신 상태 유지

syncBlocklist / BlocklistSync는 목록을 한 번 다운로드한 다음 변경된 것만 적용합니다.

목록은 빌드된 변경 피드의 정확한 위치와 함께 x-cursor 헤더와 # cursor: 주석으로 제공됩니다. 클라이언트는 그것을 읽고 거기서 재개하므로 스냅샷과 첫 폴링 사이에 공백이 없고, 서버와 시계가 일치할 필요도 없습니다. 초 단위 폴링은 초 내부의 위치를 표현할 수 없고, 크롤 배치는 수십 개의 이벤트를 한 초에 쓰므로 초 단위 재개는 나머지를 놓칠 수 있습니다: 실시간 측정에서 같은 초의 세 가지 변경 중 첫 번째 후 재개했을 때 커서로는 형제 둘 다 복구했지만 초로는 둘 다 복구하지 못했습니다.

const sync = await syncBlocklist("gptbot");   // cursor comes from the list itself
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000);  // a few hundred bytes per poll

refresh()는 목록에 robots 전환만 적용합니다. 목록이 그것으로 구성되어 있기 때문입니다. 경계 거부, 새 가격, llms.txt 변경은 other로 돌아와 별도로 처리할 수 있습니다 — 이전 버전은 그 도메인들을 거부 목록에서 삭제하여 크롤러가 막 거부를 시작한 것을 정확히 다시 가져오기 시작했습니다.

판정

각 판정의 권위 있는 정의 — 의미, 크롤러가 해야 할 일, 다시 물어보면 바뀔 수 있는지 — 는 /api/v1/verdicts에 데이터로 게시되어 있습니다. 아래 목록은 요약입니다. 둘이 다르다면 엔드포인트가 맞고 이 파일이 낡은 것입니다.

politeFetch는 기본적으로 disallow, refuse, pay를 건너뛰며, 이는 엔드포인트의 파생 do_not_fetch 집합입니다. 여기의 복사본은 의도적입니다 — 크롤 루프는 결정을 위해 네트워크 호출이 필요하지 않아야 하며 — 테스트가 둘을 비교하여 눈에 띄지 않게 표류할 수 없게 합니다.

판정

의미

기본 동작

allow

robots.txt가 이 에이전트를 허용하고, 사용자 에이전트를 담은 실시간 요청이 서빙됨

가져오기

disallow

robots.txt가 사이트 루트에서 이 에이전트를 금지함

건너뛰기

refuse

robots.txt는 허용하지만 경계가 어쨌든 거부함. 허용은 실제가 아님

건너뛰기

pay

출처가 HTTP 402로 응답함. 상업적 조건으로 이 에이전트를 서빙할 것임

건너뛰기

unknown

답할 만큼 최근에 측정되지 않음

가져오기

onPay: "fetch" (on_pay="fetch")는 페이월 기본값을 재정의합니다. 명시적 옵트인이며 결과에 paidRouteOverridden으로 기록되어 로그에 표시됩니다.

저하되지만 실패하지 않음

인구조사에 연결할 수 없으면 모든 판정이 unknown이 되고 크롤은 평소처럼 진행됩니다. 제3자 장애가 파이프라인을 멈춰서는 안 됩니다. 정확히 이것을 위한 실시간 테스트가 있습니다.

우리가 당신의 에이전트에 대해 게시하는 것

const p = await agentProfile("claudebot");
// robots disallow rate, edge refusal rate, operator page, correction channel

수치가 틀렸다면 수정 채널이 그 응답과 운영자 페이지에 있습니다. 레지스트리 사실은 논쟁 없이 수정됩니다. 논쟁이 있는 측정은 조용히 수정하는 대신 기본 스캔 기록과 함께 논쟁과 함께 게시됩니다.

한도

익명으로 preflight 호출당 25개 도메인, Pro 키로 200개, Data 키로 1,000개. apiKey를 전달하세요. 자세한 내용은 https://crawlcensus.com/for-crawlers.

테스트

node test.mjs는 의도적으로 실시간 인구조사에 대해 실행됩니다. 이 클라이언트의 가치는 판정이 현실과 일치하는지이며, 모의 테스트는 모의가 자신과 일치한다는 것만 주장할 것입니다.

MIT. 데이터는 CC BY 4.0, 출처 표시는 "Source: Crawl Census (crawlcensus.com)".

Available Tools

7 tools
agent_profileAInspect

What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the correction channel and the public page a claim can be disputed against.

NameRequiredDescriptionDefault
agentYesCrawler token, e.g. gptbot, claudebot, ccbot, google-extended.

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the behavioral burden and does so well: it discloses what is measured, that live requests are checked at the network edge, that operator documentation is assessed, and that correction/dispute channels are included. It does not explicitly state read-only status or error behavior, but the 'measures and publishes' framing strongly implies a read operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose is front-loaded and the metric list is scannable. The second sentence slightly repeats the correction-channel idea already mentioned in the list, but it adds the audience and the dispute-page detail, so it earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter tool with no output schema, the description adequately conveys what data the profile returns and that correction channels are included. It could be more explicit about output format or unknown-agent handling, but the essential guidance for selecting and invoking the tool is present.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already documents `agent` with examples like gptbot and claudebot at 100% coverage. The description only adds that the tool is about 'one AI crawler,' contributing no new parameter syntax or constraints, so baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource: 'What this census measures and publishes about one AI crawler.' It then enumerates the exact metrics reported, clearly distinguishing this per-agent profile tool from sibling tools that focus on sites, stats, or submissions.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The intended use is clear: retrieve published census data about one AI crawler, aimed at both the crawler operator and external researchers. It does not explicitly name alternative sibling tools or exclusion conditions, but the usage context is unambiguous.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

census_factsAInspect

Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date, which is what makes the number wrong when it is repeated.

NameRequiredDescriptionDefault

No parameters

TDQS

A4.4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden, and it does so well. It discloses the structure of each returned record, the presence of caveats, and the important failure mode of losing denominators and dates when numbers are repeated. This gives an agent a clear picture of the tool's behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two dense, well-front-loaded sentences cover what the tool returns, how records are structured, and when to use it. Every clause earns its place, with no filler or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter tool with no output schema, the description is remarkably complete: it explains the record fields, caveats, and the specific use case. An agent can understand what it will get back and why it is the right tool without further clarification.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has zero parameters, so this is the baseline-4 case. The description focuses on output record semantics rather than parameters, which is appropriate since there is nothing to document.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states that this tool returns census headline findings as discrete, dated records with value, unit, denominator, date, page, and citation. It distinguishes itself from prose-style reporting, but it does not explicitly differentiate itself from the sibling census_stats, so some overlap remains.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives an explicit 'Use this when...' trigger tied to questions about how open the web is to AI crawlers, and explains why the dated/denominator-preserving format matters. It does not mention alternatives or when not to use it, so routing guidance is strong but not complete.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

census_statsBInspect

Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.

NameRequiredDescriptionDefault

No parameters

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of disclosing behavior. It states what data is included but does not reveal whether it is read-only, how recent the statistics are, whether any parameters or filters exist, or what the response structure looks like. The colon-led list of statistics is content, not behavioral transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence that leads with the core concept ('corpus-level statistics') and then lists specific examples. There is no filler or redundant language, making it easy to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a parameterless tool, the description gives enough to understand what kind of data comes back, but it omits an explicit return format and does not clarify the scope of 'measured domains' or any access prerequisites. The missing output schema means the description could reasonably be expected to say more about the result shape.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so the schema fully covers any input concerns. The description adds value by explaining what the resulting statistics measure, which complements the empty schema without needing to compensate for undocumented parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names the resource ('corpus-level statistics') and specifies three concrete metrics: per-crawler blocking counts, mean access score, and llms.txt adoption. This gives an agent a clear idea of what the tool computes, though it does not explicitly distinguish itself from the similarly named sibling census_facts.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'corpus-level statistics' implies this is for aggregate data, but there is no explicit statement of when to use this tool versus alternatives like census_facts or scan_site. No when-not conditions or alternative names are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_preflightAInspect

Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it), refuse (robots permits it but the edge refused the agent anyway, so the allowance is not real), pay (the origin answered HTTP 402 Payment Required, meaning it will serve this agent on commercial terms), or unknown. The full definition of each, including what it obliges a crawler to do, is published at https://crawlcensus.com/api/v1/verdicts. Built for crawler operators rather than site owners: it prevents wasted fetches against doors that are shut, and flags content an operator is trying to sell rather than withhold.

NameRequiredDescriptionDefault
agentYesCrawler token, e.g. gptbot, claudebot, perplexitybot, oai-searchbot, ccbot.
domainsYesDomains to check. Up to 25 per call anonymously; send an Authorization: Bearer key for more. An over-large batch is refused outright rather than partly answered.

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and goes beyond the basics: it reveals that a live request is sent to verify serving, that unrecognised agents are resolved from robots.txt rather than refused, and that 'refuse' means an allowance is not real. It also links to full verdict definitions. Minor omissions like rate-limit behavior are already covered in the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but every sentence earns its place: purpose, verdict list, unknown-agent behavior, authoritative link, and audience. It is front-loaded with the core decision purpose. Slightly longer than necessary, but no filler or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description is complete despite having no output schema: it enumerates all possible return verdicts with definitions, explains edge-case behavior for unknown agents, and provides a link for full semantics. Combined with a 100%-covered input schema, an agent has all needed information to invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3. The description adds meaningful context for the 'agent' parameter by explaining it works for any crawler token and how unrecognised agents are handled, which goes beyond the schema's list of examples. It also clarifies the 'domains' parameter's per-domain output semantics.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description opens with a specific verb+resource ('Decide whether a crawler may fetch a list of domains'), then enumerates the five possible verdicts with precise meanings. It also distinguishes itself from sibling census/site tools by stating it is 'Built for crawler operators rather than site owners' and by handling 'any crawler token, not only the ones this census tracks.'

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly tells when to use the tool ('before spending requests on them') and for whom ('crawler operators rather than site owners'), which excludes site-owner workflows that would be served by other siblings. It also explains the unrecognised-agent behavior, aiding correct selection for non-census crawlers.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scan_siteAInspect

Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.

NameRequiredDescriptionDefault
domainYesBare hostname, for example example.com

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of disclosure. It transparently explains that the tool performs live probes, checks robots.txt and llms.txt, and returns a score with per-check detail. It does not mention potential side effects or operational caveats, but the core behavior is well disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single information-dense sentence that front-loads the purpose and uses a compact list to convey scope, then states the return format. No filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with one parameter and no output schema, the description is largely complete: it explains what checks are performed and what the return value looks like (score out of 100 with per-check detail). Minor gaps are the lack of alternative routing and any timing/side-effect context, but nothing critical is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, and the single parameter 'domain' is already well documented with an example. The tool description adds no additional parameter semantics beyond what the schema provides, so the baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states a specific verb ('Run') and resource ('a live AI-accessibility audit of a domain'), and enumerates concrete checks. It does not explicitly distinguish itself from sibling tools like site_report or crawl_preflight, but the scope is specific enough that an agent can infer its function.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when to use it: when a live AI-accessibility audit of a domain is needed. However, it gives no explicit guidance on when to choose this tool over siblings such as site_report or crawl_preflight, and offers no exclusions or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

site_reportAInspect

Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.

NameRequiredDescriptionDefault
domainYesBare hostname

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the burden. It discloses the key behavioral traits: it is a read of stored data, it does not trigger a scan, and it imposes no load on the target. It stops short of describing behavior when no stored audit exists, but the main side-effect profile is clear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with no wasted words; the primary action and the no-scan guarantee are front-loaded, followed by a benefit. Every phrase earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-parameter read-only report tool with no output schema, the description covers the core contract: what is returned, the absence of a scan, and the speed/load benefit. It does not specify the return format or error behavior, but the tool's simplicity keeps these gaps minor.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already describes domain as 'Bare hostname' with 100% coverage, so the description adds no additional parameter meaning. Baseline 3 applies because the schema handles parameter documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('Return'), resource ('most recent stored audit for a domain'), and explicitly distinguishes itself from a scan by saying it does not trigger a new scan. This makes its purpose clear and differentiates it from the sibling scan_site.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'without triggering a new scan' plus 'Faster and free of load' gives clear context for when to use this tool: when a stored report is acceptable and no active scan is desired. It does not explicitly name scan_site as the alternative, but the intended use case is evident.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

submit_domainsAInspect

Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a database write rather than a fetch, so the allowance is far higher than scan_site and submitted domains are measured ahead of the ranked backlog.

NameRequiredDescriptionDefault
domainsYesHostnames to queue. Up to 50 per call anonymously; an over-large batch is refused outright rather than partly queued.

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description correctly discloses the core side effect: 'Queueing is a database write rather than a fetch.' It also reveals reply behavior, naming already-fresh or never-measured domains, though exact rate limits and idempotency semantics are not fully specified.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences pack the purpose, workflow, and operational distinction without filler. The first sentence front-loads the core action, while the later sentences add necessary context about the crawl_preflight loop and the allowance difference from scan_site.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter mutation tool with no annotations and no output schema, the description covers when to call, what to submit, the side effect, and the response's role in convergence. It leaves numeric limits and idempotency guarantees implicit, but enough information exists for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with a clear field description, so the parameter itself is well documented. The tool description adds value by defining eligibility: only domains the census has not measured yet, specifically those flagged by crawl_preflight as unknown with measurable true.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and object: 'Queue domains the census has not measured yet.' It also explicitly ties the tool to crawl_preflight's 'unknown with measurable true' results, distinguishing it from retrieval-focused siblings like scan_site and site_report.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives an explicit selection rule: anything crawl_preflight returns as unknown with measurable true is worth submitting, and it warns that fresh or never-measured domains appear in the reply so a caller can avoid resubmission. It also contrasts the database-write allowance with scan_site, helping an agent choose between them.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 7 tool updatesv1.0.0
    • First observedagent_profile
    • First observedcensus_facts
    • First observedcensus_stats
    • First observedcrawl_preflight
    • First observedscan_site
    • First observedsite_report
    • First observedsubmit_domains

TDQS

A3.9/5.0

Scored across 7 tools

Disambiguation4/5

scan_site and site_report both return audits for a domain, but one is a live scan and the other is a stored result, so they are distinguishable. census_stats and census_facts also overlap somewhat as corpus-level outputs, but one gives direct statistics and the other gives citation-ready findings. The remaining tools each have clearly separate roles.

Naming Consistency4/5

All names are lowercase snake_case and mostly combine a domain noun with a result or action, such as site_report, census_stats, agent_profile, and crawl_preflight. scan_site and submit_domains are verb-first while the others are noun-first, so the pattern is not perfectly uniform, but the style is still readable and predictable.

Tool Count5/5

Seven tools is a well-scoped set for a crawl-census service: each tool covers a distinct consumer need, from live audits and stored reports to crawler preflight decisions, agent profiles, citation-ready facts, and domain submission. No tool is redundant, and the count feels appropriately sized for the domain.

Completeness4/5

The set covers the main workflows well: running and retrieving audits, querying corpus statistics and facts, profiling individual crawlers, deciding whether to fetch domains, and queueing unmeasured domains. Minor gaps include no obvious way to list all tracked agents or remove queued submissions, but these are not core to the documented purpose.

Maintenance

ActivityStale
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    B
    maintenance
    Enables AI agents to check whether a public website is crawlable, understandable, and ready for AI search workflows through local-only audits of robots.txt, sitemaps, metadata, and llms.txt.
    3
    71 npm
    1
    MIT
  • A
    license
    A
    quality
    B
    maintenance
    Domain and company intelligence for AI agents. Enables vetting companies, qualifying leads, and mapping targets from free public data without API keys.
    6
    5 npm
    1
    MIT
  • A
    license
    Not graded
    quality
    A
    maintenance
    Provides agents with tools to score a page's AI readiness, check which AI crawlers robots.txt blocks, and assess passage citability—all performed locally.
    23 npm
    2
    MIT