Rat Radar
Server Details
Ask before you scrape: is a site open, may I fetch this URL, and how do I extract a page. No key.
- Status
- Healthy
- Uptime
- 100.0% over the last 24 hours
- Last Tested
- Transport
- Streamable HTTP · MCP 2025-11-25
- URL
- ChatGPT
- –
- Claude
- –
TDQS
Scored across 8 tools
Each tool has a distinguishable role: dataset browsing (list_sites, search_sites, summary), site-level feasibility (check_site), URL-level robots.txt (can_fetch), and page-level extraction (xray_page, xray_diagram), plus workflow recipes (get_playbook). The main soft overlaps are can_fetch vs check_site (same underlying robots.txt topic at different granularity) and list_sites vs search_sites (both filter by category/verdict), but descriptions make the intended entry points clear. xray_page and xray_diagram operate on the same input yet return clearly different artifacts.
All names are snake_case and readable, but the convention is mixed: verb_noun (list_sites, search_sites, get_playbook), a modal verb (can_fetch), a near-synonym verb for a sibling tool (check_site), a bare noun (summary), and a product-prefix pair (xray_page, xray_diagram). The set is legible but does not follow a single predictable pattern.
Eight tools is well-scoped for a site-research and extraction-planning server. Each tool covers a distinct stage of the workflow (discover, assess, plan, build) with no filler.
The surface covers the full research lifecycle: discover sites, judge scrapeability, check a specific URL, plan extraction, and generate build recipes, with diagram output as a bonus. Gaps are minor: there is no tool to actually run a scrape or retrieve extracted records (xray_page returns proof rows only), and no way to submit or refresh a domain in the dataset beyond the live probe inside check_site.
Available Tools
8 toolscan_fetchAInspect
Does robots.txt allow fetching this exact URL? Fetches the live robots.txt and applies the longest-match rule for a user agent (default: * plus GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot). Returns allowed or not, the rule that decided it, and any crawl delay. A courtesy check, not a legal opinion.
Inputs: url (required), user_agent.
check_siteAInspect
Can I scrape this site, and how? Returns a verdict (open, guarded, hostile, blocked), a hostility score, a recommended approach with steps and risks, bot protections, robots.txt and AI-crawler policy, machine-readable surfaces (sitemap, llms.txt, JSON-LD), and eu_ai_training: the signals the EDPB draft Guidelines 03/2026 on web scraping for generative AI count against collecting personal data (robots.txt blocks on AI training crawlers, a Content-Signal of ai-train=no, ai.txt, CAPTCHAs, login walls), as none, some or strong. Signals, not legal advice. Uses a weekly-probed dataset of several hundred sites; for any other domain it probes live from a datacenter IP.
Inputs: domain (required).
get_playbookAInspect
What should I build with this site, and how? Give a domain and, optionally, a goal in plain words (for example "price alerts", "sales leads", "new jobs", "change log", "knowledge base"). Returns the best-matching workflow for that site: what it does, its cadence, a trigger, collect, shape, store, act pipeline, the fields to keep, and step-by-step build recipes for n8n, Make, Zapier and Apify (or just one with platform). Workflows marked found_on_this_site come from what the weekly probe saw on the site itself. Also returns the other workflows, the golden data, and the guardrails.
Inputs: domain (required), goal, platform.
list_sitesBInspect
List the sites in the dataset, optionally by category or verdict. Use it to find easy targets or to see what is covered.
Inputs: limit, verdict, category.
search_sitesAInspect
Find sites in the dataset by a word: a name, host, category (jobs, shop, news, gov), country or verdict. Use it to discover what is covered before calling check_site.
Inputs: query (required), limit.
summaryBInspect
Headline numbers across the dataset: verdict mix, how many sites block AI crawlers (and which), protection vendors, by category and country.
Inputs: no named parameters.
xray_diagramAInspect
Draw a page's anatomy as an Excalidraw diagram: its regions, the ranked ways to get its data, proof rows, the best route and what the page is made of. Returns a download link for the .excalidraw file (give it to the user; it opens at excalidraw.com or in Obsidian) and a file_name. With include_scene, also the scene itself (a few hundred KB).
Inputs: url (required), include_scene.
xray_pageAInspect
How do I extract the data from THIS page? Fetches one public page and returns a ranked extraction plan: embedded JSON, JSON-LD, repeated records with CSS selectors and field names, tables, feeds, hidden API calls. Each source has a reliability, proof rows pulled from the real page, and the plan notes pagination and blockers. Set include_code for a starter Python script. Works on the raw HTML only, so JavaScript-built pages report a blocker instead of data. Use check_site first to learn whether the site allows scraping at all.
Inputs: url (required), include_code.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Related MCP Connectors
Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?
Whether an AI agent may fetch a URL, on what terms, and at what price. Free, no key, no signup.
Check a page, AI-crawler access, llms.txt and backlinks of a site. Free, read-only.
Check which AI crawlers a site allows and see pages the way AI crawlers do. Free, read-only.
Related MCP Servers
- AlicenseNot gradedqualityBmaintenanceEnables AI agents and MCP clients to inspect a URL for reachability, then extract clean structured Markdown, headings, and links from public web pages, paid per successful request via x402 micropayments with no accounts or API keys.MIT
- FlicenseNot gradedqualityCmaintenanceEnables agents to read any web page or document as markdown, html, links, screenshots or extracted JSON, map and crawl entire sites, and drive a persistent browser session that stays open between calls so pages can be inspected and clicked without reloading. Callers pay per request with a wallet over x402 — no API key or signup — and every route answers its own free GET with the fields it expects.1-
- AlicenseNot gradedqualityAmaintenanceEnables MCP clients to extract structured data from web pages by supplying a URL and a prompt describing the desired fields.227 npmMIT
- AlicenseNot gradedqualityDmaintenanceRead-only observation of a single live URL: parses static HTML to report security posture, forms, links, accessibility signals, and leaks, with described fixes. SSRF-gated and safe, never executes JavaScript.85 npmMIT
Glama MCP Gateway
Add one secure layer between your agents and this server.