Agent Web Census
Server Details
Open data on MCP servers (with live health) and which AI crawlers top websites block.
- Status
- Healthy
- Uptime
- 100.0% over the last 24 hours
- Last Tested
- Transport
- Streamable HTTP · MCP 2025-11-25
- URL
- Repository
- jirizaloudek/agentwebcensus
- GitHub Stars
- 0
- ChatGPT
- –
- Claude
- –
TDQS
Scored across 4 tools
Each tool occupies a clearly distinct slot: ai_crawler_stats is the cross-site per-bot aggregate, site_ai_crawler_policy is the per-domain lookup, mcp_ecosystem_stats is registry-wide headline numbers, and find_mcp_servers is keyword search. The aggregate-vs-single-entity split mirrors the crawler pair on the MCP side, which makes the boundaries intuitive rather than confusing.
All names are snake_case and follow a domain_topic_noun shape (ai_crawler_stats, mcp_ecosystem_stats, site_ai_crawler_policy), so the grouping by subject area is readable. find_mcp_servers breaks the pattern with a verb-first form, a minor but noticeable deviation.
Four tools is light but defensible for a read-only statistics/lookup service with two query domains, each getting an aggregate plus a detail-style endpoint. Nothing redundant is included, though the surface sits near the thin end of the range.
The MCP side has search and ecosystem aggregates but no way to fetch a single server's full record by name, so a search hit can only be inspected via what the list view returns. The crawler side is better covered (per-bot aggregate and per-domain policy), but there is no way to enumerate available bot names or bulk-compare domains, leaving notable gaps for follow-up queries.
Available Tools
4 toolsai_crawler_statsARead-onlyInspect
Aggregate AI-crawler blocking statistics across the top 10,000 websites: percentage of sites with a reachable robots.txt that fully block, partly restrict or explicitly allow each AI bot. Give a bot name for one bot, or omit for the overall table.
Inputs: bot.
find_mcp_serversARead-onlyInspect
Search the official MCP Registry (~65,000 servers) by keyword. Matches name, title and description; ranks servers whose remote endpoint passed a live initialize check first, then by number of tools. Returns name, title, description, remote/package types, health (ok = works without auth, auth = alive but needs auth, fail = unreachable), tool count and a registry link.
Inputs: query (required), limit, category, transport, +1 more.
mcp_ecosystem_statsARead-onlyInspect
Headline numbers of the MCP ecosystem: servers in the official registry, new servers in the last 7/30 days, remote vs package share, top package registries and categories, plus the weekly health check of remote servers (share working, requiring auth, failing, median tools and latency).
Inputs: no named parameters.
site_ai_crawler_policyARead-onlyInspect
Which AI crawlers does a website block in robots.txt? Looks up a domain in our scan of the top 10,000 websites (Tranco list) and returns, per AI bot (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, ...), whether it is blocked, partly restricted or explicitly allowed, plus llms.txt presence.
Inputs: domain (required).
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Related MCP Connectors
One MCP server for all web data. Scrape, search, crawl, and AI research in one install.
A growing catalog of free data sources for AI agents, updated daily, plus MCP server audits.
Live index of AI agents, MCP servers and tools with observed liveness and capability search.
MCP registry: 138k servers crawled, handshake-validated, reliability-scored. 744 production-safe.
Related MCP Servers
- AlicenseNot gradedqualityAmaintenanceBehavioral trust scoring for MCP servers and AI agents. Live registry tracking 4,500+ servers with trust scores (0-100) based on interaction history, success rates, and latency2MIT
- AlicenseNot gradedqualityDmaintenanceOpen-source MCP server that audits websites for AI search readiness, providing deterministic scoring (0-100) and prioritized fix lists for metrics like JSON-LD, llms.txt, heading hierarchy, and AI crawler access.1MIT
- AlicenseNot gradedqualityDmaintenanceOpen-source MCP servers that give AI agents structured access to networking standards, device databases, and security data.MIT
- AlicenseAqualityBmaintenanceAI visibility tracker MCP server. Track brand citations across ChatGPT, Claude, Perplexity, Gemini & Google AI Overviews. Self-host on Cloudflare Workers. GEO/AEO.12238 npm40MIT
Glama MCP Gateway
Add one secure layer between your agents and this server.