crawl-census
crawl-census-client
取得する前に確認を。 クローラーが閉じたドアにリクエストを浪費するのを防ぎ、誰かが販売しようとしているコンテンツを迂回してしまうのを防ぐ、ドロップインクライアント。
robots.txt を読むことは、1つの質問に答え、2つの質問を隠すことです。Crawl Census による 23,482 ドメイン の測定結果:
2,874 ドメインが robots.txt で AI エージェントを許可し、その後ネットワークエッジで同じエージェントを拒否しています。 パーサーは許可を見ます。フェッチは 403 を返します。往復のコストを払って何も得られません。
208 ドメインが AI ユーザーエージェントに
HTTP 402 Payment Requiredで応答します。 それは価格であり、拒否ではありません。それをブロックとして扱うと、運営者が販売したいコンテンツから離れてしまいます。それを迂回して再試行すると、課金されているものを取得することになります。
依存関係なし。キー不要。
MCP サーバー
同じ測定結果がリモート MCP サーバーとして公開されており、エージェントは失敗した後にではなく、取得する前に確認できます。公式 MCP レジストリ に io.github.taylorsmithgg/crawl-census として登録されています。
{ "mcpServers": { "crawl-census": { "url": "https://crawlcensus.com/mcp" } } }ツール | 回答 |
| これらのドメインは私のエージェントにサービスを提供するか、拒否するか、課金するか? |
| このセンサスは私のクローラーについて何を公開しており、どう修正するか? |
| 分母と引用行付きの日付入りレコードとしての主要な調査結果 |
| 1つのドメインの保存された監査 |
| 今すぐドメインを測定する |
| コーパス全体の合計 |
読み取りツールに認証は不要。Streamable HTTP。
Related MCP server: domain-intel-mcp
インストール
npm i github:taylorsmithgg/crawl-census-client
pip install git+https://github.com/taylorsmithgg/crawl-census-client使用方法
import { politeFetch } from "crawl-census-client";
const r = await politeFetch("https://example.com/", { agent: "gptbot" });
if (r.skipped) console.log(r.verdict, r.reason); // disallow | refuse | pay
else process(await r.response.text());from crawl_census import polite_fetch
r = polite_fetch("https://example.com/", agent="gptbot")
if r.skipped:
print(r.verdict, r.reason)
else:
process(r.body)スキップは例外として送出されるのではなく、返されます。これはウェブの大部分にとって正常な結果であり、クロールループはすべての URL の周りに try/except を置かずにスキップを数えられるべきです。
クロール前にキューを分割する
ホストごとに1回ではなく、1,000 ドメインにつき1回の呼び出し:
const { crawl, skip, pay, unknown } = await partition(urls, { agent: "gptbot" });p = partition(urls, agent="gptbot")
p.crawl, p.skip, p.pay, p.unknownまたは単にファイルを取得する
メモリ内の拒否リストだけが必要なフェッチャーには、ドメインごとの呼び出しを完全にスキップします:
curl https://crawlcensus.com/agents/gptbot/blocklist.txt # one domain per line, commented headerconst sync = await syncBlocklist("gptbot"); // full list once
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000); // then deltas only, a few hundred bytessync = BlocklistSync("gptbot")
if host in sync: skip()
sync.refresh() # {'added': 3, 'removed': 1, 'size': 3310, 'cursor': ...}デルタフィードは https://crawlcensus.com/agents/<agent>/changes.json?since=<unix> で、各レスポンスは next_since を運びます。そのため、長時間実行されるクローラーは、リストを再ダウンロードする代わりに、1時間あたり数百バイトで最新の状態を維持できます。
そのファイルは robots.txt のみ を対象としています。エッジ拒否と HTTP 402 はリクエストごとの動作であり、依然として preflight または politeFetch が必要です。
クロールがセンサスに掛かるコスト
主張ではなく測定。20 ホストの並行フェッチは、かつては各1ドメインを運ぶ20回の preflight 呼び出しに相当しました。同じホストを3回同時に要求すると、最初のルックアップが解決した後にのみキャッシュが機能するため、3回のコストがかかりました。匿名の許容量は1時間あたり240回の呼び出しであり、1回の呼び出しで25ホストをカバーできる場合、クローラーは240ホストで上限に達しました。
politeFetch は現在、作業を自動的に共有します: 同じティックで発行されたルックアップは1つのバッチ呼び出しとして送信され、同じホストへの並行ルックアップは単一のリクエストを待機します。
パターン | 以前 | 現在 |
20 ホスト、並行 | 20 呼び出し | 20 の1呼び出し |
1 ホスト、3 URL、並行 | 3 呼び出し | 1 呼び出し |
60 ホスト、並行 | 60 呼び出し | 3 呼び出し (25 / 25 / 10) |
| 2 呼び出し | 1 呼び出し |
batchSize のデフォルトは 25 で、キーなしでの呼び出しあたりの上限です。Pro または Data キーで引き上げます。batchWaitMs は、一度にではなく波状に到着する並行処理のための合体ウィンドウを広げます。デフォルトのゼロは次のティックでフラッシュします。
オリジンが価格を示す場合の支払い
pay 判定は、オリジンが金額を指定した場合にその金額を運びます:
const r = await politeFetch(url, { agent: "claudebot" });
if (r.verdict === "pay") console.log(r.price); // "USD 0.5", or null if none was quoted知っておくべきことが2つあります。HTTP 402 で応答するほとんどのオリジンは金額をまったく指定しないため、price は通常 null であり、取り決めはバンド外で行う必要があります。また、価格設定はクローラーごとです: 測定されたコーパス全体で、213 の課金オリジンのうち 78 は一部のエージェントに課金し、他のエージェントには無料でサービスを提供します。そのため、リスト上のドメインがあなたに課金するだろうと想定するのではなく、自分のトークンで問い合わせてください。
2種類の不明
partition は作業キューを crawl、pay、skip、unknown、undecidable に分割します。
最後の2つは似ていますが、同じではありません。unknown は、センサスがそのドメインをまだ測定していないことを意味します: それを送信すると、次のパスで実際の判定が得られます。undecidable は、サイトの robots.txt が CrawlCensusBot を許可しないことを意味するため、このセンサスはそれを決して測定しません — 再試行は保証された無駄であり、毎パスで不明を再送信するループはそれらを永遠に再送信することになります。サーバーは measurable ブール値で違いをマークします。reason の散文ではなく、それを読んでください。
const p = await partition(urls, { agent: "gptbot" });
await Promise.all(p.crawl.map(politeFetchOne));
if (p.unmeasured.length) await submitUnmeasured(p, { agent: "gptbot" });
// p.undecidable: read their robots.txt yourself. Asking us again cannot help.送信は意図的に別の呼び出しです。ルックアップとして読めるものの間に静かに POST するライブラリは悪い市民であり、キュー位置がいつ使われるかを選択すべきです。
あなたにサービスを提供しないすべてをスキップする
拒否リストは小さい方の半分です。ライブセンサスに対して測定すると、robots の不許可のみをスキップするクローラーは、robots.txt で許可しながらエッジで拒否するドメイン、または HTTP 402 で応答するドメインに対して、1パスあたり約 2,900 リクエストを依然として費やします — PerplexityBot の場合、そのセットは拒否リストよりも大きいです。それらのフェッチは何も返さず、それぞれ往復のコストがかかります。
const skip = await syncSkipList("gptbot");
if (skip.has(host)) continue; // disallowed, refused at the edge, or priced
skip.why(host); // "disallow" | "pay" | "refuse" | null
setInterval(() => skip.refresh(), 3600_000);エージェント | 拒否リスト | 追加でスキップ可能 | 合計 |
GPTBot | 3,542 | 2,944 | 6,486 |
ClaudeBot | 3,169 | 3,194 | 6,363 |
PerplexityBot | 1,128 | 3,658 | 4,786 |
3つのセットは内部的に分離されているため、変更はそれが属するセットを移動します。why() は preflight と同じ優先順位に従います: 不許可は価格よりも優先されます。なぜなら、価格は許可ではないからです。
拒否リストを最新に保つ
syncBlocklist / BlocklistSync はリストを一度ダウンロードし、その後は変更されたものだけを適用します。
リストは、それが構築された変更フィード内の正確な位置とともに、x-cursor ヘッダーと # cursor: コメントで提供されます。クライアントはそれを読み、そこから再開するため、スナップショットと最初のポーリングの間にギャップはなく、サーバーと同期した時計に依存することもありません。秒単位のポーリングは秒内の位置を表現できず、クロールバッチは1秒に数十のイベントを書き込むため、秒単位の再開はその残りを落とす可能性があります: ライブで測定すると、同じ秒の3つの変更の最初の後に再開すると、カーソルでは両方の兄弟が回復しましたが、秒ではどちらも回復しませんでした。
const sync = await syncBlocklist("gptbot"); // cursor comes from the list itself
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000); // a few hundred bytes per pollrefresh() は robots の遷移のみをリストに適用します。なぜなら、リストはそれで構成されているからです。エッジ拒否、新しい価格、llms.txt の変更は、別途対応するために other で返されます — 以前のバージョンはそれらのドメインを拒否リストから削除したため、クローラーは拒否し始めたばかりのものを正確にフェッチを再開しました。
判定
各判定の権威ある定義 — それが何を意味するか、クローラーに何を義務付けるか、再度問い合わせることで変わる可能性があるか — は /api/v1/verdicts でデータとして公開されています。以下のリストは要約です。両者が矛盾する場合、エンドポイントが正しく、このファイルは古いものです。
politeFetch はデフォルトで disallow、refuse、pay をスキップします。これはエンドポイントの派生 do_not_fetch セットです。ここでのコピーは意図的です — クロールループは決定するためにネットワーク呼び出しを必要とすべきではありません — そしてテストが両者を比較するため、気付かれずにドリフトすることはありません。
判定 | 意味 | デフォルトの動作 |
| robots.txt はこのエージェントを許可し、そのユーザーエージェントを運ぶライブリクエストがサービスされた | フェッチ |
| robots.txt はサイトルートでこのエージェントを禁止する | スキップ |
| robots.txt は許可する。エッジはとにかく拒否した。許可は本物ではない | スキップ |
| オリジンは HTTP 402 で応答した。商業条件でこのエージェントにサービスを提供する | スキップ |
| 回答するほど最近測定されていない | フェッチ |
onPay: "fetch" (on_pay="fetch") はペイウォールのデフォルトを上書きします。これは明示的なオプトインであり、paidRouteOverridden として結果に記録されるため、ログに表示されます。
劣化しても失敗しない
センサスに到達できない場合、すべての判定は unknown になり、クロールは通常どおり進行します。サードパーティの障害がパイプラインを止めてはなりません。まさにこのためのライブテストがあります。
私たちがあなたのエージェントについて公開するもの
const p = await agentProfile("claudebot");
// robots disallow rate, edge refusal rate, operator page, correction channel数値が間違っている場合、修正チャネルはそのレスポンスとオペレーターページにあります。レジストリの事実は議論なしに修正されます。争われた測定値は、静かに修正されるのではなく、基礎となるスキャンレコードとともに論争と並べて公開されます。
制限
匿名での preflight 呼び出しあたり 25 ドメイン、Pro キーで 200、Data キーで 1,000。apiKey を渡します。詳細は https://crawlcensus.com/for-crawlers。
テスト
node test.mjs は意図的にライブセンサスに対して実行されます。このクライアントの価値は、その判定が現実と一致するかどうかであり、モックされたテストはモックが自分自身と一致することを主張するだけです。
MIT。データは CC BY 4.0、帰属は「Source: Crawl Census (crawlcensus.com)」。
Available Tools
7 toolsagent_profileAInspect
What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the correction channel and the public page a claim can be disputed against.
| Name | Required | Description | Default |
|---|---|---|---|
| agent | Yes | Crawler token, e.g. gptbot, claudebot, ccbot, google-extended. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the behavioral burden and does so well: it discloses what is measured, that live requests are checked at the network edge, that operator documentation is assessed, and that correction/dispute channels are included. It does not explicitly state read-only status or error behavior, but the 'measures and publishes' framing strongly implies a read operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The purpose is front-loaded and the metric list is scannable. The second sentence slightly repeats the correction-channel idea already mentioned in the list, but it adds the audience and the dispute-page detail, so it earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter tool with no output schema, the description adequately conveys what data the profile returns and that correction channels are included. It could be more explicit about output format or unknown-agent handling, but the essential guidance for selecting and invoking the tool is present.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already documents `agent` with examples like gptbot and claudebot at 100% coverage. The description only adds that the tool is about 'one AI crawler,' contributing no new parameter syntax or constraints, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb+resource: 'What this census measures and publishes about one AI crawler.' It then enumerates the exact metrics reported, clearly distinguishing this per-agent profile tool from sibling tools that focus on sites, stats, or submissions.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The intended use is clear: retrieve published census data about one AI crawler, aimed at both the crawler operator and external researchers. It does not explicitly name alternative sibling tools or exclusion conditions, but the usage context is unambiguous.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
census_factsAInspect
Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date, which is what makes the number wrong when it is repeated.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden, and it does so well. It discloses the structure of each returned record, the presence of caveats, and the important failure mode of losing denominators and dates when numbers are repeated. This gives an agent a clear picture of the tool's behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two dense, well-front-loaded sentences cover what the tool returns, how records are structured, and when to use it. Every clause earns its place, with no filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter tool with no output schema, the description is remarkably complete: it explains the record fields, caveats, and the specific use case. An agent can understand what it will get back and why it is the right tool without further clarification.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has zero parameters, so this is the baseline-4 case. The description focuses on output record semantics rather than parameters, which is appropriate since there is nothing to document.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that this tool returns census headline findings as discrete, dated records with value, unit, denominator, date, page, and citation. It distinguishes itself from prose-style reporting, but it does not explicitly differentiate itself from the sibling census_stats, so some overlap remains.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives an explicit 'Use this when...' trigger tied to questions about how open the web is to AI crawlers, and explains why the dated/denominator-preserving format matters. It does not mention alternatives or when not to use it, so routing guidance is strong but not complete.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
census_statsBInspect
Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden of disclosing behavior. It states what data is included but does not reveal whether it is read-only, how recent the statistics are, whether any parameters or filters exist, or what the response structure looks like. The colon-led list of statistics is content, not behavioral transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence that leads with the core concept ('corpus-level statistics') and then lists specific examples. There is no filler or redundant language, making it easy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a parameterless tool, the description gives enough to understand what kind of data comes back, but it omits an explicit return format and does not clarify the scope of 'measured domains' or any access prerequisites. The missing output schema means the description could reasonably be expected to say more about the result shape.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the schema fully covers any input concerns. The description adds value by explaining what the resulting statistics measure, which complements the empty schema without needing to compensate for undocumented parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names the resource ('corpus-level statistics') and specifies three concrete metrics: per-crawler blocking counts, mean access score, and llms.txt adoption. This gives an agent a clear idea of what the tool computes, though it does not explicitly distinguish itself from the similarly named sibling census_facts.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'corpus-level statistics' implies this is for aggregate data, but there is no explicit statement of when to use this tool versus alternatives like census_facts or scan_site. No when-not conditions or alternative names are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_preflightAInspect
Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it), refuse (robots permits it but the edge refused the agent anyway, so the allowance is not real), pay (the origin answered HTTP 402 Payment Required, meaning it will serve this agent on commercial terms), or unknown. The full definition of each, including what it obliges a crawler to do, is published at https://crawlcensus.com/api/v1/verdicts. Built for crawler operators rather than site owners: it prevents wasted fetches against doors that are shut, and flags content an operator is trying to sell rather than withhold.
| Name | Required | Description | Default |
|---|---|---|---|
| agent | Yes | Crawler token, e.g. gptbot, claudebot, perplexitybot, oai-searchbot, ccbot. | |
| domains | Yes | Domains to check. Up to 25 per call anonymously; send an Authorization: Bearer key for more. An over-large batch is refused outright rather than partly answered. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and goes beyond the basics: it reveals that a live request is sent to verify serving, that unrecognised agents are resolved from robots.txt rather than refused, and that 'refuse' means an allowance is not real. It also links to full verdict definitions. Minor omissions like rate-limit behavior are already covered in the schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but every sentence earns its place: purpose, verdict list, unknown-agent behavior, authoritative link, and audience. It is front-loaded with the core decision purpose. Slightly longer than necessary, but no filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is complete despite having no output schema: it enumerates all possible return verdicts with definitions, explains edge-case behavior for unknown agents, and provides a link for full semantics. Combined with a 100%-covered input schema, an agent has all needed information to invoke the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds meaningful context for the 'agent' parameter by explaining it works for any crawler token and how unrecognised agents are handled, which goes beyond the schema's list of examples. It also clarifies the 'domains' parameter's per-domain output semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description opens with a specific verb+resource ('Decide whether a crawler may fetch a list of domains'), then enumerates the five possible verdicts with precise meanings. It also distinguishes itself from sibling census/site tools by stating it is 'Built for crawler operators rather than site owners' and by handling 'any crawler token, not only the ones this census tracks.'
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly tells when to use the tool ('before spending requests on them') and for whom ('crawler operators rather than site owners'), which excludes site-owner workflows that would be served by other siblings. It also explains the unrecognised-agent behavior, aiding correct selection for non-census crawlers.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scan_siteAInspect
Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes | Bare hostname, for example example.com |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of disclosure. It transparently explains that the tool performs live probes, checks robots.txt and llms.txt, and returns a score with per-check detail. It does not mention potential side effects or operational caveats, but the core behavior is well disclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single information-dense sentence that front-loads the purpose and uses a compact list to convey scope, then states the return format. No filler or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with one parameter and no output schema, the description is largely complete: it explains what checks are performed and what the return value looks like (score out of 100 with per-check detail). Minor gaps are the lack of alternative routing and any timing/side-effect context, but nothing critical is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and the single parameter 'domain' is already well documented with an example. The tool description adds no additional parameter semantics beyond what the schema provides, so the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific verb ('Run') and resource ('a live AI-accessibility audit of a domain'), and enumerates concrete checks. It does not explicitly distinguish itself from sibling tools like site_report or crawl_preflight, but the scope is specific enough that an agent can infer its function.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use it: when a live AI-accessibility audit of a domain is needed. However, it gives no explicit guidance on when to choose this tool over siblings such as site_report or crawl_preflight, and offers no exclusions or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
site_reportAInspect
Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes | Bare hostname |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the burden. It discloses the key behavioral traits: it is a read of stored data, it does not trigger a scan, and it imposes no load on the target. It stops short of describing behavior when no stored audit exists, but the main side-effect profile is clear.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no wasted words; the primary action and the no-scan guarantee are front-loaded, followed by a benefit. Every phrase earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-parameter read-only report tool with no output schema, the description covers the core contract: what is returned, the absence of a scan, and the speed/load benefit. It does not specify the return format or error behavior, but the tool's simplicity keeps these gaps minor.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already describes domain as 'Bare hostname' with 100% coverage, so the description adds no additional parameter meaning. Baseline 3 applies because the schema handles parameter documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('Return'), resource ('most recent stored audit for a domain'), and explicitly distinguishes itself from a scan by saying it does not trigger a new scan. This makes its purpose clear and differentiates it from the sibling scan_site.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'without triggering a new scan' plus 'Faster and free of load' gives clear context for when to use this tool: when a stored report is acceptable and no active scan is desired. It does not explicitly name scan_site as the alternative, but the intended use case is evident.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
submit_domainsAInspect
Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a database write rather than a fetch, so the allowance is far higher than scan_site and submitted domains are measured ahead of the ranked backlog.
| Name | Required | Description | Default |
|---|---|---|---|
| domains | Yes | Hostnames to queue. Up to 50 per call anonymously; an over-large batch is refused outright rather than partly queued. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description correctly discloses the core side effect: 'Queueing is a database write rather than a fetch.' It also reveals reply behavior, naming already-fresh or never-measured domains, though exact rate limits and idempotency semantics are not fully specified.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences pack the purpose, workflow, and operational distinction without filler. The first sentence front-loads the core action, while the later sentences add necessary context about the crawl_preflight loop and the allowance difference from scan_site.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter mutation tool with no annotations and no output schema, the description covers when to call, what to submit, the side effect, and the response's role in convergence. It leaves numeric limits and idempotency guarantees implicit, but enough information exists for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with a clear field description, so the parameter itself is well documented. The tool description adds value by defining eligibility: only domains the census has not measured yet, specifically those flagged by crawl_preflight as unknown with measurable true.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and object: 'Queue domains the census has not measured yet.' It also explicitly ties the tool to crawl_preflight's 'unknown with measurable true' results, distinguishing it from retrieval-focused siblings like scan_site and site_report.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives an explicit selection rule: anything crawl_preflight returns as unknown with measurable true is worth submitting, and it warns that fresh or never-measured domains appear in the reply so a caller can avoid resubmission. It also contrasts the database-write allowance with scan_site, helping an agent choose between them.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
7 tool updates
v1.0.0- First observed
agent_profile - First observed
census_facts - First observed
census_stats - First observed
crawl_preflight - First observed
scan_site - First observed
site_report - First observed
submit_domains
TDQS
Scored across 7 tools
scan_site and site_report both return audits for a domain, but one is a live scan and the other is a stored result, so they are distinguishable. census_stats and census_facts also overlap somewhat as corpus-level outputs, but one gives direct statistics and the other gives citation-ready findings. The remaining tools each have clearly separate roles.
All names are lowercase snake_case and mostly combine a domain noun with a result or action, such as site_report, census_stats, agent_profile, and crawl_preflight. scan_site and submit_domains are verb-first while the others are noun-first, so the pattern is not perfectly uniform, but the style is still readable and predictable.
Seven tools is a well-scoped set for a crawl-census service: each tool covers a distinct consumer need, from live audits and stored reports to crawler preflight decisions, agent profiles, citation-ready facts, and domain submission. No tool is redundant, and the count feels appropriately sized for the domain.
The set covers the main workflows well: running and retrieving audits, querying corpus statistics and facts, profiling individual crawlers, deciding whether to fetch domains, and queueing unmeasured domains. Minor gaps include no obvious way to list all tracked agents or remove queued submissions, but these are not core to the documented purpose.
Maintenance
Related MCP Connectors
Policy-aware web access preflight: robots, licensing, payment and advertised routes for agents.
Flags phantom-squatted/typosquatted domains and known-malicious URLs before an agent follows them.
Check if a website will block your AI agent and find its llms.txt, MCP or OpenAPI door.
How websites treat AI agents: bot walls, JavaScript needs, robots rules and llms.txt. Look a site up before you fetch it, report what you saw after. Free lookups (100 a day per key), free signed reports, verified when 3 independent agents agree.
Related MCP Servers
- AlicenseAqualityBmaintenanceEnables AI agents to check whether a public website is crawlable, understandable, and ready for AI search workflows through local-only audits of robots.txt, sitemaps, metadata, and llms.txt.336 npm1MIT
- AlicenseAqualityBmaintenanceDomain and company intelligence for AI agents. Enables vetting companies, qualifying leads, and mapping targets from free public data without API keys.61 npm1MIT
- FlicenseNot gradedqualityBmaintenanceProvides pre-fetch routing intelligence for AI agents by recommending the cheapest reliable route (HTTP, browser, machine endpoint, or avoid) before visiting a URL.-
- AlicenseNot gradedqualityAmaintenanceProvides agents with tools to score a page's AI readiness, check which AI crawlers robots.txt blocks, and assess passage citability—all performed locally.23 npm2MIT