Skip to main content
Glama

crawl-census-client

先问再抓取。 一个即插即用的客户端,让你的爬虫不再把请求浪费在关闭的门上,也不再绕过别人试图出售的内容。

阅读 robots.txt 回答了一个问题,却隐藏了另外两个。根据 Crawl Census 对 23,482 个域名 的测量:

  • 2,874 个域名在 robots.txt 中允许 AI 代理,却在网络边缘拒绝这些相同的代理。 解析器看到的是允许;抓取返回 403。你为往返付费,却一无所获。

  • 208 个域名以 HTTP 402 Payment Required 回应 AI 用户代理。 这是一个价格,而不是拒绝。将其视为阻止,就会放弃运营商想要卖给你的内容。绕过它重试,则拿走了他们收费的东西。

无依赖。无需密钥。

MCP 服务器

同样的测量结果以远程 MCP 服务器的形式提供,因此代理可以在抓取之前询问,而不是在失败之后。已在 官方 MCP 注册表 中列为 io.github.taylorsmithgg/crawl-census。

{ "mcpServers": { "crawl-census": { "url": "https://crawlcensus.com/mcp" } } }

工具

回答

crawl_preflight

这些域名会为我的代理提供服务、拒绝它,还是向它收费?

agent_profile

这份普查发布了关于我的爬虫的哪些信息,我该如何纠正?

census_facts

以带日期、分母和引用行的记录形式呈现的标题性发现

site_report

单个域名的存储审计

scan_site

立即测量一个域名

census_stats

语料库级别的总计

读取工具无需认证。支持流式 HTTP。

Related MCP server: domain-intel-mcp

安装

npm i github:taylorsmithgg/crawl-census-client
pip install git+https://github.com/taylorsmithgg/crawl-census-client

使用

import { politeFetch } from "crawl-census-client";

const r = await politeFetch("https://example.com/", { agent: "gptbot" });
if (r.skipped) console.log(r.verdict, r.reason);   // disallow | refuse | pay
else           process(await r.response.text());
from crawl_census import polite_fetch

r = polite_fetch("https://example.com/", agent="gptbot")
if r.skipped:
    print(r.verdict, r.reason)
else:
    process(r.body)

跳过是返回的,而不是抛出的。对于网络上的大部分内容来说,这是正常结果,爬虫循环应该能够统计跳过次数,而无需在每个 URL 周围使用 try/except。

在爬取之前拆分队列

每 1,000 个域名一次调用,而不是每个主机一次:

const { crawl, skip, pay, unknown } = await partition(urls, { agent: "gptbot" });
p = partition(urls, agent="gptbot")
p.crawl, p.skip, p.pay, p.unknown

或者直接获取文件

对于只需要在内存中维护拒绝列表的抓取器,完全跳过按域名的调用:

curl https://crawlcensus.com/agents/gptbot/blocklist.txt   # one domain per line, commented header
const sync = await syncBlocklist("gptbot");   // full list once
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000);  // then deltas only, a few hundred bytes
sync = BlocklistSync("gptbot")
if host in sync: skip()
sync.refresh()          # {'added': 3, 'removed': 1, 'size': 3310, 'cursor': ...}

增量源是 https://crawlcensus.com/agents/<agent>/changes.json?since=<unix>,每个响应都带有 next_since,因此长时间运行的爬虫每小时只需几百字节即可保持最新,而无需重新下载列表。

该文件仅涵盖 robots.txt。边缘拒绝和 HTTP 402 是每次请求的行为,仍然需要 preflight 或 politeFetch。

一次爬取对普查的成本

这是测量出来的,不是断言。并发抓取二十个主机过去需要二十次 preflight 调用,每次携带一个域名;同一个主机同时请求三次需要三次,因为缓存只在第一次查找解析后才有帮助。匿名配额是每小时 240 次调用,因此当一次调用覆盖二十五个主机时,爬虫在 240 个主机时达到上限。

politeFetch 现在自动共享工作:在同一 tick 中发出的查找会合并为一次批量调用,同一主机的并发查找会等待单个请求。

模式

之前

现在

20 个主机,并发

20 次调用

1 次调用,20 个

1 个主机,3 个 URL,并发

3 次调用

1 次调用

60 个主机,并发

60 次调用

3 次调用(25 / 25 / 10)

partition 然后抓取

2 次调用

1 次调用

batchSize 默认为 25,即没有密钥时的每次调用上限。使用 Pro 或 Data 密钥可以提高它。batchWaitMs 扩大了并发到达的合并窗口,而不是一次性全部到达;默认值为零,会在下一个 tick 刷新。

当源站报价时付费

当源站指定了金额时,pay 裁决会携带该金额:

const r = await politeFetch(url, { agent: "claudebot" });
if (r.verdict === "pay") console.log(r.price);   // "USD 0.5", or null if none was quoted

有两件事值得了解。大多数响应 HTTP 402 的源站根本没有指定金额,因此 price 通常为 null,安排必须在带外进行。而且定价是按爬虫计算的:在测量的语料库中,213 个收费源站中有 78 个对某些代理收费,对其他代理免费,因此请使用自己的令牌询问,而不是假设列表中的域名会向您收费。

两种未知

partition 将工作队列拆分为 crawl、pay、skip、unknown 和 undecidable。

最后两个看起来相似,但并非如此。unknown 表示普查尚未测量该域名:提交它,下一轮就会得到真正的裁决。undecidable 表示该网站的 robots.txt 不允许 CrawlCensusBot,因此这份普查永远不会测量它——重试是必然的浪费,每轮重新提交未知项的循环会永远重新提交这些项。服务器用 measurable 布尔值标记差异;读取该值,而不是 reason 文本。

const p = await partition(urls, { agent: "gptbot" });
await Promise.all(p.crawl.map(politeFetchOne));
if (p.unmeasured.length) await submitUnmeasured(p, { agent: "gptbot" });
// p.undecidable: read their robots.txt yourself. Asking us again cannot help.

提交是有意单独调用的。一个在看似查找的过程中悄悄 POST 的库是不良公民,您应该选择何时消耗队列位置。

跳过所有不会为您服务的内容

拒绝列表是较小的一半。根据实时普查测量,只跳过 robots 禁止的爬虫每轮仍会在 robots.txt 允许但在边缘拒绝的域名上花费约 2,900 个请求,或者响应 HTTP 402 的域名——对于 PerplexityBot 来说,这个集合比其拒绝列表还要大。这些抓取不会返回任何内容,每次都会花费一次往返。

const skip = await syncSkipList("gptbot");
if (skip.has(host)) continue;        // disallowed, refused at the edge, or priced
skip.why(host);                      // "disallow" | "pay" | "refuse" | null
setInterval(() => skip.refresh(), 3600_000);

代理

拒绝列表

也可跳过

总计

GPTBot

3,542

2,944

6,486

ClaudeBot

3,169

3,194

6,363

PerplexityBot

1,128

3,658

4,786

这三个集合在内部保持分离,因此更改会移动它所属的那个。why() 遵循与 preflight 相同的优先级:禁止高于价格,因为价格不是许可。

保持拒绝列表最新

syncBlocklist / BlocklistSync 下载一次列表,然后只应用更改的内容。

列表以构建时的变更源中的确切位置提供,位于 x-cursor 标头和 # cursor: 注释中。客户端读取它并从那里恢复,因此快照和第一次轮询之间没有间隙,也不依赖您的时钟与服务器同步。按秒轮询无法表达一秒内的位置,而爬取批次会将数十个事件写入一个事件中,因此按秒粒度恢复可能会丢失其余部分:实时测量,在三个同秒更改中的第一个之后恢复,通过游标恢复了两个兄弟,而按秒则一个也没有恢复。

const sync = await syncBlocklist("gptbot");   // cursor comes from the list itself
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000);  // a few hundred bytes per poll

refresh() 只将 robots 转换应用于列表,因为这就是列表的构成。边缘拒绝、新价格和 llms.txt 更改会返回到 other 中,供您单独处理——早期版本会从拒绝列表中删除这些域名,因此爬虫会重新开始抓取刚刚开始拒绝它的内容。

裁决

每个裁决的权威定义——它意味着什么,它要求爬虫做什么,以及再次询问是否会改变它——以数据形式发布在 /api/v1/verdicts。下面的列表是摘要;如果两者不一致,端点是正确的,此文件已过时。

politeFetch 默认跳过 disallow、refuse 和 pay,这是端点派生的 do_not_fetch 集合。这里的副本是故意的——爬虫循环不需要网络调用来决定——并且测试会比较两者,以免它悄悄漂移。

裁决

含义

默认行为

allow

robots.txt 允许此代理,并且携带其用户代理的实时请求已得到服务

抓取

disallow

robots.txt 在站点根目录禁止此代理

跳过

refuse

robots.txt 允许它;边缘仍然拒绝了它。允许不是真实的

跳过

pay

源站响应了 HTTP 402。它将按商业条款为这个代理提供服务

跳过

unknown

最近没有足够测量来回答

抓取

onPay: "fetch"(on_pay="fetch")覆盖付费墙默认值。这是一个明确的加入选项,并记录在结果中为 paidRouteOverridden,因此它会出现在您的日志中。

它会降级,不会失败

如果普查不可达,每个裁决都会变成 unknown,您的爬取会像往常一样继续。第三方中断绝不能停止您的管道。正好有一个实时测试来验证这一点。

我们发布关于您的代理的哪些信息

const p = await agentProfile("claudebot");
// robots disallow rate, edge refusal rate, operator page, correction channel

如果某个数字有误,更正渠道就在该响应和您的 操作员页面 中。注册表事实无需争论即可更正;有争议的测量结果会与争议和底层扫描记录一起发布,而不是悄悄修改。

限制

匿名每次 preflight 调用 25 个域名,使用 Pro 密钥 200 个,使用 Data 密钥 1,000 个。传递 apiKey。详情请见 https://crawlcensus.com/for-crawlers。

测试

node test.mjs 特意针对实时普查运行。这个客户端的价值在于其裁决是否与现实相符,而模拟测试只会断言模拟与自身一致。

MIT。数据采用 CC BY 4.0,署名方式为“来源:Crawl Census (crawlcensus.com)”。

Available Tools

7 tools
agent_profileAInspect

What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the correction channel and the public page a claim can be disputed against.

NameRequiredDescriptionDefault
agentYesCrawler token, e.g. gptbot, claudebot, ccbot, google-extended.

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the behavioral burden and does so well: it discloses what is measured, that live requests are checked at the network edge, that operator documentation is assessed, and that correction/dispute channels are included. It does not explicitly state read-only status or error behavior, but the 'measures and publishes' framing strongly implies a read operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose is front-loaded and the metric list is scannable. The second sentence slightly repeats the correction-channel idea already mentioned in the list, but it adds the audience and the dispute-page detail, so it earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter tool with no output schema, the description adequately conveys what data the profile returns and that correction channels are included. It could be more explicit about output format or unknown-agent handling, but the essential guidance for selecting and invoking the tool is present.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already documents `agent` with examples like gptbot and claudebot at 100% coverage. The description only adds that the tool is about 'one AI crawler,' contributing no new parameter syntax or constraints, so baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource: 'What this census measures and publishes about one AI crawler.' It then enumerates the exact metrics reported, clearly distinguishing this per-agent profile tool from sibling tools that focus on sites, stats, or submissions.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The intended use is clear: retrieve published census data about one AI crawler, aimed at both the crawler operator and external researchers. It does not explicitly name alternative sibling tools or exclusion conditions, but the usage context is unambiguous.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

census_factsAInspect

Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date, which is what makes the number wrong when it is repeated.

NameRequiredDescriptionDefault

No parameters

TDQS

A4.4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden, and it does so well. It discloses the structure of each returned record, the presence of caveats, and the important failure mode of losing denominators and dates when numbers are repeated. This gives an agent a clear picture of the tool's behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two dense, well-front-loaded sentences cover what the tool returns, how records are structured, and when to use it. Every clause earns its place, with no filler or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter tool with no output schema, the description is remarkably complete: it explains the record fields, caveats, and the specific use case. An agent can understand what it will get back and why it is the right tool without further clarification.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has zero parameters, so this is the baseline-4 case. The description focuses on output record semantics rather than parameters, which is appropriate since there is nothing to document.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states that this tool returns census headline findings as discrete, dated records with value, unit, denominator, date, page, and citation. It distinguishes itself from prose-style reporting, but it does not explicitly differentiate itself from the sibling census_stats, so some overlap remains.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives an explicit 'Use this when...' trigger tied to questions about how open the web is to AI crawlers, and explains why the dated/denominator-preserving format matters. It does not mention alternatives or when not to use it, so routing guidance is strong but not complete.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

census_statsBInspect

Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.

NameRequiredDescriptionDefault

No parameters

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of disclosing behavior. It states what data is included but does not reveal whether it is read-only, how recent the statistics are, whether any parameters or filters exist, or what the response structure looks like. The colon-led list of statistics is content, not behavioral transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence that leads with the core concept ('corpus-level statistics') and then lists specific examples. There is no filler or redundant language, making it easy to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a parameterless tool, the description gives enough to understand what kind of data comes back, but it omits an explicit return format and does not clarify the scope of 'measured domains' or any access prerequisites. The missing output schema means the description could reasonably be expected to say more about the result shape.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so the schema fully covers any input concerns. The description adds value by explaining what the resulting statistics measure, which complements the empty schema without needing to compensate for undocumented parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names the resource ('corpus-level statistics') and specifies three concrete metrics: per-crawler blocking counts, mean access score, and llms.txt adoption. This gives an agent a clear idea of what the tool computes, though it does not explicitly distinguish itself from the similarly named sibling census_facts.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'corpus-level statistics' implies this is for aggregate data, but there is no explicit statement of when to use this tool versus alternatives like census_facts or scan_site. No when-not conditions or alternative names are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_preflightAInspect

Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it), refuse (robots permits it but the edge refused the agent anyway, so the allowance is not real), pay (the origin answered HTTP 402 Payment Required, meaning it will serve this agent on commercial terms), or unknown. The full definition of each, including what it obliges a crawler to do, is published at https://crawlcensus.com/api/v1/verdicts. Built for crawler operators rather than site owners: it prevents wasted fetches against doors that are shut, and flags content an operator is trying to sell rather than withhold.

NameRequiredDescriptionDefault
agentYesCrawler token, e.g. gptbot, claudebot, perplexitybot, oai-searchbot, ccbot.
domainsYesDomains to check. Up to 25 per call anonymously; send an Authorization: Bearer key for more. An over-large batch is refused outright rather than partly answered.

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and goes beyond the basics: it reveals that a live request is sent to verify serving, that unrecognised agents are resolved from robots.txt rather than refused, and that 'refuse' means an allowance is not real. It also links to full verdict definitions. Minor omissions like rate-limit behavior are already covered in the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but every sentence earns its place: purpose, verdict list, unknown-agent behavior, authoritative link, and audience. It is front-loaded with the core decision purpose. Slightly longer than necessary, but no filler or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description is complete despite having no output schema: it enumerates all possible return verdicts with definitions, explains edge-case behavior for unknown agents, and provides a link for full semantics. Combined with a 100%-covered input schema, an agent has all needed information to invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3. The description adds meaningful context for the 'agent' parameter by explaining it works for any crawler token and how unrecognised agents are handled, which goes beyond the schema's list of examples. It also clarifies the 'domains' parameter's per-domain output semantics.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description opens with a specific verb+resource ('Decide whether a crawler may fetch a list of domains'), then enumerates the five possible verdicts with precise meanings. It also distinguishes itself from sibling census/site tools by stating it is 'Built for crawler operators rather than site owners' and by handling 'any crawler token, not only the ones this census tracks.'

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly tells when to use the tool ('before spending requests on them') and for whom ('crawler operators rather than site owners'), which excludes site-owner workflows that would be served by other siblings. It also explains the unrecognised-agent behavior, aiding correct selection for non-census crawlers.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scan_siteAInspect

Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.

NameRequiredDescriptionDefault
domainYesBare hostname, for example example.com

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of disclosure. It transparently explains that the tool performs live probes, checks robots.txt and llms.txt, and returns a score with per-check detail. It does not mention potential side effects or operational caveats, but the core behavior is well disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single information-dense sentence that front-loads the purpose and uses a compact list to convey scope, then states the return format. No filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with one parameter and no output schema, the description is largely complete: it explains what checks are performed and what the return value looks like (score out of 100 with per-check detail). Minor gaps are the lack of alternative routing and any timing/side-effect context, but nothing critical is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, and the single parameter 'domain' is already well documented with an example. The tool description adds no additional parameter semantics beyond what the schema provides, so the baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states a specific verb ('Run') and resource ('a live AI-accessibility audit of a domain'), and enumerates concrete checks. It does not explicitly distinguish itself from sibling tools like site_report or crawl_preflight, but the scope is specific enough that an agent can infer its function.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when to use it: when a live AI-accessibility audit of a domain is needed. However, it gives no explicit guidance on when to choose this tool over siblings such as site_report or crawl_preflight, and offers no exclusions or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

site_reportAInspect

Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.

NameRequiredDescriptionDefault
domainYesBare hostname

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the burden. It discloses the key behavioral traits: it is a read of stored data, it does not trigger a scan, and it imposes no load on the target. It stops short of describing behavior when no stored audit exists, but the main side-effect profile is clear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with no wasted words; the primary action and the no-scan guarantee are front-loaded, followed by a benefit. Every phrase earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-parameter read-only report tool with no output schema, the description covers the core contract: what is returned, the absence of a scan, and the speed/load benefit. It does not specify the return format or error behavior, but the tool's simplicity keeps these gaps minor.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already describes domain as 'Bare hostname' with 100% coverage, so the description adds no additional parameter meaning. Baseline 3 applies because the schema handles parameter documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('Return'), resource ('most recent stored audit for a domain'), and explicitly distinguishes itself from a scan by saying it does not trigger a new scan. This makes its purpose clear and differentiates it from the sibling scan_site.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'without triggering a new scan' plus 'Faster and free of load' gives clear context for when to use this tool: when a stored report is acceptable and no active scan is desired. It does not explicitly name scan_site as the alternative, but the intended use case is evident.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

submit_domainsAInspect

Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a database write rather than a fetch, so the allowance is far higher than scan_site and submitted domains are measured ahead of the ranked backlog.

NameRequiredDescriptionDefault
domainsYesHostnames to queue. Up to 50 per call anonymously; an over-large batch is refused outright rather than partly queued.

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description correctly discloses the core side effect: 'Queueing is a database write rather than a fetch.' It also reveals reply behavior, naming already-fresh or never-measured domains, though exact rate limits and idempotency semantics are not fully specified.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences pack the purpose, workflow, and operational distinction without filler. The first sentence front-loads the core action, while the later sentences add necessary context about the crawl_preflight loop and the allowance difference from scan_site.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter mutation tool with no annotations and no output schema, the description covers when to call, what to submit, the side effect, and the response's role in convergence. It leaves numeric limits and idempotency guarantees implicit, but enough information exists for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with a clear field description, so the parameter itself is well documented. The tool description adds value by defining eligibility: only domains the census has not measured yet, specifically those flagged by crawl_preflight as unknown with measurable true.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and object: 'Queue domains the census has not measured yet.' It also explicitly ties the tool to crawl_preflight's 'unknown with measurable true' results, distinguishing it from retrieval-focused siblings like scan_site and site_report.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives an explicit selection rule: anything crawl_preflight returns as unknown with measurable true is worth submitting, and it warns that fresh or never-measured domains appear in the reply so a caller can avoid resubmission. It also contrasts the database-write allowance with scan_site, helping an agent choose between them.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 7 tool updatesv1.0.0
    • First observedagent_profile
    • First observedcensus_facts
    • First observedcensus_stats
    • First observedcrawl_preflight
    • First observedscan_site
    • First observedsite_report
    • First observedsubmit_domains

TDQS

A3.9/5.0

Scored across 7 tools

Disambiguation4/5

scan_site and site_report both return audits for a domain, but one is a live scan and the other is a stored result, so they are distinguishable. census_stats and census_facts also overlap somewhat as corpus-level outputs, but one gives direct statistics and the other gives citation-ready findings. The remaining tools each have clearly separate roles.

Naming Consistency4/5

All names are lowercase snake_case and mostly combine a domain noun with a result or action, such as site_report, census_stats, agent_profile, and crawl_preflight. scan_site and submit_domains are verb-first while the others are noun-first, so the pattern is not perfectly uniform, but the style is still readable and predictable.

Tool Count5/5

Seven tools is a well-scoped set for a crawl-census service: each tool covers a distinct consumer need, from live audits and stored reports to crawler preflight decisions, agent profiles, citation-ready facts, and domain submission. No tool is redundant, and the count feels appropriately sized for the domain.

Completeness4/5

The set covers the main workflows well: running and retrieving audits, querying corpus statistics and facts, profiling individual crawlers, deciding whether to fetch domains, and queueing unmeasured domains. Minor gaps include no obvious way to list all tracked agents or remove queued submissions, but these are not core to the documented purpose.

Maintenance

ActivityStale
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    B
    maintenance
    Enables AI agents to check whether a public website is crawlable, understandable, and ready for AI search workflows through local-only audits of robots.txt, sitemaps, metadata, and llms.txt.
    3
    71 npm
    1
    MIT
  • A
    license
    A
    quality
    B
    maintenance
    Domain and company intelligence for AI agents. Enables vetting companies, qualifying leads, and mapping targets from free public data without API keys.
    6
    5 npm
    1
    MIT
  • A
    license
    Not graded
    quality
    A
    maintenance
    Provides agents with tools to score a page's AI readiness, check which AI crawlers robots.txt blocks, and assess passage citability—all performed locally.
    56 npm
    2
    MIT