Skip to main content
Glama
Sign In

Crawlwise

by site.crawlwise
OverviewDiscussionsAdmin

Check AI crawler access

check_ai_crawlers
Read-onlyIdempotent

Fetch the URL as GPTBot, ClaudeBot and other AI crawlers and report allowed / blocked / challenged / not established. Does NOT measure whether a model would cite the page, brand mentions, or ranking in ChatGPT. A 200 is reachability, not visibility.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "check_ai_crawlers"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYesPublic http(s) URL.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish readOnly, idempotent, non-destructive and open-world behavior, so the bar is lower. The description adds real context beyond them: the enumeration of result states and the caveat that a 200 status reflects crawler reachability rather than visibility, which prevents misreading the result.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences, front-loaded with the action and result states, then the scope exclusion and the 200 caveat. Every sentence carries information; only the caveat could arguably be trimmed.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter, read-only probe with no output schema, the description supplies the missing return-value vocabulary (allowed/blocked/challenged/not established) plus the key interpretation caveat. Nothing essential is absent, though it does not mention rate limits or behavior on unreachable/invalid URLs.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There is only one parameter and schema description coverage is 100% ('Public http(s) URL'), so the schema carries this dimension. The description adds no extra meaning such as redirect handling, auth-protected URLs, or timeout behavior, making 3 the correct baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Names a specific verb and resource ('fetch the URL as GPTBot, ClaudeBot and other AI crawlers') and states the exact output states (allowed/blocked/challenged/not established). This distinguishes it from siblings like audit_page and crawl_site without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly bounds scope with 'Does NOT measure whether a model would cite the page, brand mentions, or ranking in ChatGPT' and warns that 'a 200 is reachability, not visibility', which tells an agent when this tool's answer is not the answer the user wants. It stops short of naming which sibling to use instead (e.g. audit_page) for those questions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All Crawlwise toolsInspect
Try in Browser
Claude Code, inside the Godot 4 editorFlockbay puts an AI agent in the real Godot 4 editor: it writes GDScript, runs your game and fixes what breaks. Free, or bring your own Claude Code or Codex.flockbay.comAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources