Skip to main content
Glama
fairquery-web

fairquery-web

Official

Website Tech, SEO & llms.txt Check for Agents

Returns one row per website (tech stack, SEO score, robots.txt AI-crawler rules, sitemap and llms.txt) for $1.50 per 1,000 sites. You pay $0.0015 per site row, and error rows are free. Example: auditing 250 lead websites costs 250 × $0.0015 = $0.38, plus Apify's $0.00005 Actor-start fee.

Fairquery Apify Actor that returns one structured row per website: tech stack, title/meta/Open Graph/JSON-LD, an SEO issue list with a 0–100 score, robots.txt rules for AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, …), XML sitemap, llms.txt / llms-full.txt, security headers, and a 0–100 AI-readiness score. Plain HTTP (no browser), so it's fast and cheap.

When to use this Actor (LLM / tool selection)

Use it when an agent needs to:

  • Enrich a lead/company list with "what is this site built on?" (CMS, framework, analytics, chat, payments, CDN, hosting)

  • Run a quick on-page SEO check of homepages or landing pages

  • Answer "does this site have an llms.txt?" / "does it block GPTBot or ClaudeBot?"

  • Get a site's title, description, social links, feeds and structured-data types before summarizing it

Do not use it for full-site crawls, JS-rendered content (single-page apps may show THIN_CONTENT), Core Web Vitals, or backlink data.

Related MCP server: AI Tooling Detector MCP Server

How it works

Per site (max 3 in parallel): GET /robots.txt → GET <url> → sitemap (from robots Sitemap: or /sitemap.xml, /sitemap_index.xml) → /llms.txt and /llms-full.txt. Identifies as FairqueryBot/1.0; by default skips pages that robots.txt disallows for generic crawlers. Tech detection uses ~75 HTML/header/cookie/meta-generator signatures from a single response.

Input

Field

Type

Default

Description

urls

string[]

– (required; 3 sample sites are prefilled in the Console form)

Domains or URLs

checkRobots

bool

true

robots.txt + AI-bot access map

checkSitemap

bool

true

Find sitemap and count URLs

checkLlmsTxt

bool

true

Check /llms.txt and /llms-full.txt

respectRobotsTxt

bool

true

Skip URLs disallowed for generic crawlers

maxSites

int

500

Safety cap per run

{ "urls": ["stripe.com", "https://www.example.org/pricing"], "checkSitemap": false }

Output

Every row has schema_version (1.0.0), type (site | error), source_url (final fetched URL), checked_at.

{
  "type": "site", "domain": "vercel.com", "status_code": 200, "https": true, "response_ms": 310,
  "title": "Agentic Infrastructure - Vercel", "meta_description": "…", "canonical": "https://vercel.com/", "lang": "en",
  "h1": ["…"], "word_count": 256, "open_graph": { "title": "…", "image": "…" }, "json_ld_types": ["Organization", "WebSite"],
  "technologies": [{ "name": "Next.js", "category": "JS framework", "evidence": "html \"/_next/static\"" }, { "name": "Vercel", "category": "Hosting", "evidence": "header x-vercel-id" }],
  "security_headers": { "strict_transport_security": true, "content_security_policy": true },
  "robots_txt_found": true, "ai_bots": { "GPTBot": "partial", "ClaudeBot": "partial" },
  "sitemap_found": true, "sitemap_url_count": 8787, "llms_txt_found": true, "llms_txt_title": "Vercel",
  "seo_score": 100, "seo_issues": [], "ai_readiness_score": 85
}
  • ai_bots[bot]: allowed (no rules), partial (some paths disallowed), blocked (root disallowed).

  • seo_issues[]: {code, severity, message}; codes include NO_HTTPS, MISSING_TITLE, TITLE_TOO_LONG, MISSING_META_DESCRIPTION, MISSING_H1, MULTIPLE_H1, MISSING_CANONICAL, MISSING_LANG, MISSING_VIEWPORT, NOINDEX, MISSING_OPEN_GRAPH, MISSING_STRUCTURED_DATA, IMAGES_MISSING_ALT, THIN_CONTENT, MISSING_ROBOTS_TXT, MISSING_SITEMAP. Score = 100 − 15/8/3 per high/medium/low issue.

  • ai_readiness_score: llms.txt 25, llms-full.txt 5, AI bots allowed up to 20, JSON-LD 15, ≥150 words of server-rendered text 15, sitemap 10, meta description 10; −30 if noindex.

Error rows (machine-readable, never charged)

{ "type": "error", "code": "BLOCKED_OR_RATE_LIMITED", "message": "GET https://x.com/ returned HTTP 403.", "retryable": false, "status_code": 403, "input": "x.com" }

Codes: INPUT_EMPTY, INVALID_URL, MAX_SITES_EXCEEDED, ROBOTS_DISALLOWED, FETCH_FAILED, HTTP_ERROR, BLOCKED_OR_RATE_LIMITED.

Run summary

Key-value store record RUN_SUMMARY: sites checked, errors, HTTP request count, average SEO and AI-readiness scores, llms.txt count, echoed input.

Pricing

Event

Price

When

site-checked

$0.0015

Each website row returned

apify-actor-start

$0.00005

Once per run, per GB of memory (Apify default event)

Error rows (FETCH_FAILED, HTTP_ERROR, BLOCKED_OR_RATE_LIMITED, ROBOTS_DISALLOWED, …) are never charged.

Worked example: 1,000 domains from a lead list where 60 cannot be fetched: 940 × $0.0015 = $1.41, plus $0.00005.

To cap spend, set a maximum cost per run in the run options; the Actor stops cleanly at the limit. maxSites caps how many sites one run checks. Prices are from the Store pricing tab (October 2026).

Memory / runtime

Runs at a fixed 256 MB (default and maximum; the check is network-bound, so more memory does not help), no browser, no proxy. Typically 1–3 s per site, 3 sites in parallel. Slow or rate-limiting sites take longer.

MCP / Apify tools

Call from agents via Apify MCP (https://mcp.apify.com, authenticated with your Apify token), Actor fairquery/website-tech-seo-ai-readiness:

{ "actor": "fairquery/website-tech-seo-ai-readiness", "input": { "urls": ["stripe.com"] } }

Read dataset items (type == "site") and RUN_SUMMARY. A standalone stdio MCP server lives in the GitHub repo under mcp/.

Local stdio MCP server

This repo ships a stdio MCP server (TypeScript, @modelcontextprotocol/sdk) under mcp/ with single-purpose tools. Seven are free and run locally (no token); one optional bulk tool runs this Actor through the Apify API. initialize and tools/list work without a token.

Tool

What it does

Paging

Cost

web_quick_check

1-20 URLs: status, redirect chain, response time, content type, title

–

free

web_seo_audit

One URL: meta/OG/JSON-LD/headings/images, seo_issues, seo_score and ai_readiness_score (0-100); output is a reusable snapshot

–

free

web_ai_crawler_access

robots.txt rules per AI bot (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, ...), llms.txt / llms-full.txt

–

free

web_tech_stack

Frameworks, CMS, hosting/CDN, analytics with evidence; category filter

limit/offset

free

web_sitemap_info

Sitemap discovery, index/children, URL list with lastmod; contains, sort

limit/offset

free

web_security_headers

HSTS, CSP, X-Content-Type-Options, X-Frame-Options, Referrer-/Permissions-Policy, COOP; score + grade A-F

–

free

web_compare_snapshots

Diff two saved snapshots, a saved snapshot vs a fresh re-audit (follow-up), or two live URLs

–

free

web_batch_audit

Bulk audit of up to thousands of sites via the Apify Actor (formerly check_websites)

–

pay-per-event, needs APIFY_TOKEN

Errors are named codes: INVALID_URL, BLOCKED_PRIVATE_ADDRESS, DNS_FAILED, FETCH_FAILED, TIMEOUT, TOO_MANY_REDIRECTS, SITEMAP_NOT_FOUND, INVALID_ARGUMENTS, APIFY_TOKEN_MISSING, APIFY_ACTOR_RUN_FAILED.

Respectful fetching: User-Agent FairqueryBot ... Fairquery-MCP, 15 s timeout, 1.5 MB HTML / 10 MB sitemap cap, max 5 redirects, and an SSRF guard that refuses hosts (including redirect targets) resolving to private, loopback, link-local or reserved addresses. If you run behind a fake-IP DNS proxy that maps all hosts into 198.18.0.0/15, set FAIRQUERY_ALLOW_FAKE_IP_DNS=1.

Breaking change in v1.2.0: check_websites → web_batch_audit (same input, same billing); check_llms_txt was removed, use web_ai_crawler_access.

cd mcp && npm install && npm run build
npm run test:live                                    # live test of every free tool, no token
node scripts/smoke-test.mjs                          # initialize + tools/list, no token
APIFY_TOKEN=... node scripts/smoke-test.mjs --call   # real web_batch_audit call
{ "mcpServers": { "fairquery-web": { "command": "node", "args": ["/path/to/website-tech-seo-ai-readiness/mcp/dist/index.js"], "env": { "APIFY_TOKEN": "<optional, only for web_batch_audit>" } } } }

Docker: docker build -t fairquery-web-mcp . (the repo-root Dockerfile builds this MCP server and is what Glama uses; the Apify Actor image is built from .actor/Dockerfile). glama.json lists fairquery as maintainer.

Works well after

Troubleshooting

Symptom

Cause and fix

Run fails at once with INPUT_EMPTY

urls was empty. The three sample sites in the Console form are prefill only and are not sent with API calls.

BLOCKED_OR_RATE_LIMITED (HTTP 403/429)

The site blocks datacenter traffic or bots. These rows are free; the Actor does not use proxies or browsers to get around blocks.

ROBOTS_DISALLOWED

robots.txt disallows the page for generic crawlers. Set respectRobotsTxt to false only if you have the right to fetch the page.

Low SEO score or THIN_CONTENT on a modern site

The page renders its content with JavaScript. The Actor reads the server HTML only (no browser).

Technology missing from the list

Detection uses about 75 signatures from a single response, so some tools (for example tag-manager-only scripts) are not visible. The list is heuristic.

FAQ

Does it render JavaScript?

No. It makes plain HTTP requests, which keeps it fast and cheap. Content injected by JavaScript is not seen.

What does the AI-readiness score measure?

llms.txt (25), llms-full.txt (5), AI bots allowed (up to 20), JSON-LD (15), at least 150 words of server-rendered text (15), sitemap (10) and meta description (10); minus 30 for noindex. It is a heuristic, not a ranking factor.

How much does it cost?

$0.0015 per site row ($1.50 per 1,000), plus a $0.00005 start fee. Failed fetches are free.

How do I limit spend?

Set a maximum cost per run in the run options and/or lower maxSites.

Does it crawl the whole site?

No. Per site it fetches robots.txt, the given URL, the sitemap and /llms.txt + /llms-full.txt. Use a crawler for full-site audits.

Can an AI agent call it?

Yes, through Apify MCP with Actor fairquery/website-tech-seo-ai-readiness, or through the stdio MCP server in the GitHub repo, which also has free single-URL tools.

Fairquery family

Other Fairquery tools on the Apify Store. Each one returns structured rows with machine-readable error codes, and error rows are never charged:

All Fairquery Actors: https://apify.com/fairquery

Kits & templates

Optional guides from Fairquery for wiring this Actor into agents and automations (the Actor works fine without them):

Disclaimer

Detection is heuristic (single request, no JS execution). Only publicly served files are read; robots.txt is respected by default.

Development

Local development, tests and deployment notes are in DEVELOPMENT.md. The Apify Store README is .actor/README.md.

Available Tools

8 tools
web_ai_crawler_accessAI crawler access: robots.txt AI-bot rules + llms.txt (free)A
Read-onlyIdempotent
Inspect

Report whether AI crawlers may access a site, from its robots.txt: per bot "allowed" (no Disallow), "partial" (some paths disallowed) or "blocked" (Disallow: /). Default bots: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, Meta-ExternalAgent, Amazonbot. Also checks /llms.txt and /llms-full.txt (llmstxt.org): found, HTTP status, bytes, H1 title, link count (HTML soft-404s are not counted), and lists Sitemap: lines. WHEN TO USE: "can ChatGPT/Claude/Perplexity crawl this site?", GEO/AI-visibility checks, llms.txt adoption. Uses only the origin of the URL. Fetching: GETs from this server with User-Agent "FairqueryBot ... Fairquery-MCP", 15 s timeout per request, body capped (1.5 MB HTML), max 5 redirects, no JavaScript rendering; URLs resolving to private/loopback/link-local addresses are refused (SSRF guard). 3 requests per site. RETURNS: one "ai_crawler_access" object: origin, robots_txt_url, robots_txt_found, robots_txt_status, ai_bots{bot: status}, allowed_count, partial_count, blocked_count, note, sitemaps_declared[], llms_txt{url,found,status,bytes,title,links}, llms_full_txt{...}, checked_at. Errors (isError=true, JSON {error, message, retryable}): INVALID_URL, BLOCKED_PRIVATE_ADDRESS, DNS_FAILED, FETCH_FAILED, TIMEOUT, TOO_MANY_REDIRECTS. WHEN NOT TO USE: for full page SEO use web_seo_audit; for sitemap contents use web_sitemap_info; for many sites use web_batch_audit (paid). COST: free - runs locally in this MCP server (direct HTTP GETs), no APIFY_TOKEN, no Apify billing. PARAMETERS: url: only its origin (scheme + host + port) is used, so "nytimes.com/section/tech" and "nytimes.com" give the same result; www and non-www hosts are different origins with possibly different robots.txt files. bots replaces (does not extend) the default list - include the defaults you still want. Each token is matched like a crawler would: case-insensitively, against robots.txt User-agent groups whose name is a prefix of the token (so a "Googlebot" group also covers "Googlebot-Image"), falling back to the "*" group when no specific group exists; status is reported under the exact token you passed. If robots.txt is missing (or is an HTML soft-404) every bot is "allowed". bots does not affect the llms.txt / llms-full.txt checks. EXAMPLE: {"url":"nytimes.com","bots":["GPTBot","ClaudeBot"]} -> {ai_bots:{GPTBot:"allowed|partial|blocked", ClaudeBot:...}, blocked_count, llms_txt, ...}

NameRequiredDescriptionDefault
urlYesRequired. One public domain or http(s) URL (non-empty string), e.g. "vercel.com" or "https://www.python.org/about/". Bare domains are fetched over https; other schemes, credentials in the URL, dot-less hosts and private/loopback addresses are refused.
botsNoOptional, default = the 13 AI bots listed in the description. 1-50 robots.txt user-agent tokens (case-insensitive) to evaluate instead of the defaults. Example: ["GPTBot","ClaudeBot","Googlebot"]

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only cover the safety profile (readOnly, idempotent, non-destructive), so the description carries the rest and does so richly: User-Agent, 15s timeout, 1.5 MB body cap, max 5 redirects, no JS rendering, SSRF guard refusing private/loopback addresses, and 3 requests per site. Error codes and retryable flags are also disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Long but strictly sectioned (purpose, WHEN TO USE, Fetching, RETURNS, WHEN NOT TO USE, COST, PARAMETERS, EXAMPLE), with the core statement front-loaded. Each block is purposeful — the parameter details are needed given the matching subtleties, and the returns list compensates for the absent output schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers everything an agent needs: what it returns (full object shape), the error taxonomy, cost/model, fetching constraints, and origin-scoping behavior. With no output schema, the explicit RETURNS list and EXAMPLE fill the gap completely.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, yet the description still adds real meaning beyond the schema: only the URL origin is used, www vs non-www are distinct origins, and 'bots' replaces rather than extends the default list. It also explains the prefix-match/fallback matching semantics and that status is reported under the exact token passed.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a precise verb and resource — 'Report whether AI crawlers may access a site, from its robots.txt' — then names the secondary capability (llms.txt / llms-full.txt checks). It distinguishes itself from siblings by explicitly routing to web_seo_audit, web_sitemap_info, and web_batch_audit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Has explicit WHEN TO USE ('can ChatGPT/Claude/Perplexity crawl this site?', GEO/AI-visibility checks, llms.txt adoption) and WHEN NOT TO USE with named alternatives for each case. Nothing about tool selection is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_batch_auditBulk website tech, SEO & AI-readiness audit (Apify, paid)A
Read-onlyIdempotent
Inspect

Bulk audit up to thousands of websites in one call via the Fairquery Apify Actor fairquery/website-tech-seo-ai-readiness: technologies, on-page SEO, security headers, robots.txt AI-bot rules, XML sitemap, llms.txt, seo_score and ai_readiness_score per site. WHEN TO USE: lists of 20+ sites (lead enrichment, portfolio/market scans) where you want server-side parallelism and a dataset. For one or a few sites prefer the free tools: web_quick_check, web_seo_audit, web_ai_crawler_access, web_tech_stack, web_sitemap_info, web_security_headers. Rows can be stored and later diffed with web_compare_snapshots. BEHAVIOR: runs the Actor via the Apify API (run-sync), a few polite GET requests per site, respects robots.txt by default, no JS rendering. Requires APIFY_TOKEN (never logged). Typical 2-10 s per site, parallel. COST: paid (billed to your Apify account, pay-per-event): $0.00005 per run + $0.0015 per site row; error rows are free. 50 sites ~ $0.08. RETURNS: one "site" row per unique URL (a snapshot web_compare_snapshots accepts) or free "error" rows: code (INPUT_EMPTY, INVALID_URL, MAX_SITES_EXCEEDED, ROBOTS_DISALLOWED, FETCH_FAILED, HTTP_ERROR, BLOCKED_OR_RATE_LIMITED), message, retryable, input. Tool-level errors: APIFY_TOKEN_MISSING, APIFY_ACTOR_RUN_FAILED. WHEN NOT TO USE: for 1-20 sites or interactive questions use the free tools above (same checks). PARAMETERS: urls: bare domains use https, entries are deduped, and each unique URL is one billed site row; invalid, robots-disallowed or unreachable entries become free error rows. maxSites (default 500) caps unique URLs per run and is therefore your spend cap; extras get one free MAX_SITES_EXCEEDED row. The check* flags only remove work and fields, never change the price: checkRobots=false sets ai_bots / robots_txt_found to null, checkSitemap=false sitemap_*, checkLlmsTxt=false llms_txt_*; MISSING_ROBOTS_TXT / MISSING_SITEMAP issues are then not raised, so seo_score is not comparable across different flag settings. respectRobotsTxt is independent of checkRobots: robots.txt is still fetched when it is true, and a disallowed page yields a free ROBOTS_DISALLOWED row instead of an audit; set false only for sites you own or may audit. EXAMPLE: {"urls":["vercel.com","netlify.com"],"checkLlmsTxt":false} -> {count:2, items:[{type:"site", domain:"vercel.com", seo_score:..., ...}, ...]}

NameRequiredDescriptionDefault
urlsYesRequired. 1-5000 domains or http(s) URLs; bare domains use https; one billed row per unique URL. Example: ["vercel.com","https://www.python.org/about/"]
maxSitesNoOptional integer 1-10000, default 500. Safety/spend cap on unique URLs per run; extras produce one free MAX_SITES_EXCEEDED error row. Example: 100
checkRobotsNoOptional boolean, default true. Fetch robots.txt and report allowed/partial/blocked per AI bot; false = ai_bots null
checkLlmsTxtNoOptional boolean, default true. Check /llms.txt and /llms-full.txt; false = llms_txt fields null
checkSitemapNoOptional boolean, default true. Find the XML sitemap and count its URLs; false = sitemap fields null
respectRobotsTxtNoOptional boolean, default true (recommended). Skip pages robots.txt disallows for generic crawlers, emitting a free ROBOTS_DISALLOWED row

Output Schema

NameRequiredDescription
countYesNumber of rows returned
itemsYesResult rows; each has a "type" field (data rows or machine-readable "error" rows)

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/openWorld/non-destructive, but the description goes far beyond them: run-sync execution model, GET-only, no JS rendering, robots.txt compliance, APIFY_TOKEN requirement (with 'never logged'), 2-10s per site parallel timing, full pay-per-event cost breakdown with a worked 50-site example, and enumerated site-level and tool-level error codes. Nothing here contradicts the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Labeled sections (WHEN TO USE, BEHAVIOR, COST, RETURNS, WHEN NOT TO USE, PARAMETERS, EXAMPLE) keep a long description scannable and front-loaded with purpose. Slight redundancy remains — the WHEN NOT TO USE line repeats the 1-20 site / free-tools guidance already given — so it is dense but not maximal.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a paid, external-Actor tool with 6 parameters, the description covers cost, auth, timing, failure modes, output shape, and flag interactions; the output schema covers return structure and the description still sketches the row/error shapes an agent needs to reason about spend and retries. Nothing required for a correct call is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is already 100%, yet the description adds real semantics: bare domains default to https, dedupe/billing per unique URL, maxSites as the effective spend cap with free overflow rows, how each check* flag nulls specific field groups and breaks seo_score comparability across settings, and how respectRobotsTxt is independent of checkRobots. This is well beyond what the schema conveys.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource — bulk audit of websites via a named Apify Actor — and enumerates exactly what is checked (tech, SEO, security headers, robots.txt AI rules, sitemap, llms.txt, scores). It also names the sibling free tools it is distinct from, so an agent can route between bulk and single-site checks without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit WHEN TO USE (lists of 20+ sites, parallelism, dataset output) and WHEN NOT TO USE (1-20 sites or interactive questions use the free tools), with the alternative tools named by name. The threshold and the reason for choosing each path are both given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_compare_snapshotsCompare two SEO audits: before/after or site vs site (free)A
Read-onlyIdempotent
Inspect

Diff two SEO/AI-readiness snapshots field by field: score deltas, issues added/resolved, and changes to title, meta description, canonical, H1s, JSON-LD types, Open Graph, technologies, AI-bot access, llms.txt, sitemap, status and more. Three modes: (1) before + after = two snapshot objects you saved earlier from web_seo_audit (or site rows from web_batch_audit) - no fetching; (2) before + url = historical follow-up: re-audits url now and compares with the saved snapshot; (3) url + compareUrl = audits two live URLs (e.g. competitor vs own site, staging vs production). WHEN TO USE: track SEO regressions/improvements over time, verify a fix, compare competitors. Fetches like web_seo_audit (live modes only). RETURNS: {summary{before_url, after_url, before_checked_at, after_checked_at, seo_score_before, seo_score_after, seo_score_delta, ai_readiness_score_delta, issues_added, issues_resolved, changed_fields}, changes[{type:"change", field, before, after, delta?, added?, removed?} | {type:"issue_added"|"issue_resolved", code}], before, after (only fetched snapshots are echoed)}. Errors: INVALID_ARGUMENTS (wrong mode combination) plus INVALID_URL, BLOCKED_PRIVATE_ADDRESS, DNS_FAILED, FETCH_FAILED, TIMEOUT, TOO_MANY_REDIRECTS. WHEN NOT TO USE: for a single audit without comparison use web_seo_audit. COST: free (local fetches, no APIFY_TOKEN). PARAMETERS: exactly one of three combinations is accepted, any other combination (including all four or a single field) returns INVALID_ARGUMENTS without fetching: {before, after} | {before, url} | {url, compareUrl}. before/after must be whole snapshot objects as returned (type "seo_audit" from web_seo_audit, or a "site" row from web_batch_audit) - pass them unmodified; missing fields are compared as null. In {before, url} the live side is re-audited with checkRobots=true, so compare it only with a snapshot also taken with checkRobots=true. In {url, compareUrl} url is the "before" side and compareUrl the "after" side, so seo_score_delta = compareUrl score minus url score. url and compareUrl follow the same rules as web_seo_audit url (bare domain -> https, exact page). EXAMPLE: {"url":"mysite.com","compareUrl":"competitor.com"} -> {summary:{seo_score_delta, issues_added, ...}, changes:[...]}

NameRequiredDescriptionDefault
urlNoOptional; used with before or compareUrl. Public domain or http(s) URL audited live: the "after" side (with before) or the "before" side (with compareUrl). Example: "mysite.com"
afterNoOptional; used only with before. Later snapshot object of the same shape to compare against before
beforeNoOptional; used with after or url. Earlier snapshot object, unmodified output of web_seo_audit (type "seo_audit") or a web_batch_audit "site" row. Example: {"type":"seo_audit","final_url":"https://example.com/","seo_score":72,...}
compareUrlNoOptional; used only with url. Second public domain or http(s) URL audited live as the "after" side. Example: "competitor.com"

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations cover the safety profile (readOnly, idempotent, non-destructive, openWorld), and the description goes well beyond that: exactly one of three argument combinations is accepted or INVALID_ARGUMENTS is returned without fetching, live modes fetch like web_seo_audit, the live side uses checkRobots=true, snapshots must be passed unmodified, missing fields compare as null, and the tool is free with no APIFY_TOKEN.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core diff and the three modes, then cleanly sectioned into WHEN TO USE / RETURNS / WHEN NOT / COST / PARAMETERS / EXAMPLE. It is long, and the EXAMPLE plus the mode list duplicate some of the PARAMETERS bullets, but every section carries usable routing or invocation detail.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, yet the description spells out the full return shape (summary deltas, changes array with change/issue_added/issue_resolved entries, echoed snapshots) and enumerates the error codes. For a zero-required-parameter, multi-mode tool this is complete enough to invoke correctly without guessing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% so the baseline is 3, and the description earns extra credit by explaining combination validity, that before/after must be whole unmodified snapshots (with null for missing fields), and the delta direction in url+compareUrl mode (compareUrl minus url). The one gap is that the PARAMETERS prose partly restates the schema rather than adding new distinctions for each field.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (diff) over a specific resource (SEO/AI-readiness snapshots) and enumerates the exact fields compared. It also distinguishes itself from web_seo_audit by framing this as comparison rather than single audit work.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit WHEN TO USE (track regressions, verify a fix, compare competitors), three concrete modes with which arguments select each, and an explicit WHEN NOT TO USE naming web_seo_audit for single audits. Nothing is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_quick_checkQuick URL check: status, redirects, title, speed (free)A
Read-onlyIdempotent
Inspect

Lightweight reachability check for 1-20 URLs: final HTTP status, full redirect chain, response time (ms), content type, Server header and . WHEN TO USE: "is this site up?", "where does this domain redirect?", cheap pre-filtering of a lead list before deeper audits. For SEO details use web_seo_audit; for hundreds/thousands of sites use web_batch_audit. Fetching: GETs from this server with User-Agent "FairqueryBot ... Fairquery-MCP", 15 s timeout per request, body capped (1.5 MB HTML), max 5 redirects, no JavaScript rendering; URLs resolving to private/loopback/link-local addresses are refused (SSRF guard). Only the first 300 KB of each page is read. RETURNS: one "quick_check" or "error" row per input (error codes INVALID_URL, BLOCKED_PRIVATE_ADDRESS, DNS_FAILED, FETCH_FAILED, TIMEOUT, TOO_MANY_REDIRECTS); per-URL errors never fail the whole call. WHEN NOT TO USE: not for SEO/meta analysis (web_seo_audit), security headers (web_security_headers) or >20 URLs (web_batch_audit, paid). COST: free - runs locally in this MCP server (direct HTTP GETs), no APIFY_TOKEN, no Apify billing. PARAMETERS: urls: each entry is normalized independently and returns exactly one row, in input order: surrounding whitespace is trimmed, an entry without scheme gets "https://" (write "http://..." explicitly to test the plain-HTTP redirect), the #fragment is dropped, and the path/query are kept, so "vercel.com" and "vercel.com/pricing" are different checks. Entries that are not http(s), contain user:password@, have no dot in the host (e.g. "localhost") or resolve to private/loopback/link-local IPs become per-URL error rows (INVALID_URL / BLOCKED_PRIVATE_ADDRESS) while the other URLs still run; duplicates are fetched twice. All URLs are fetched in parallel, so total latency is roughly the slowest URL (max 15 s each). EXAMPLE: {"urls":["vercel.com"]} -> {count:1, items:[{type:"quick_check", final_url:"https://vercel.com/", status_code:200, ok:true, ...}]}

NameRequiredDescriptionDefault
urlsYesRequired. 1-20 non-empty strings, each a public domain or http(s) URL (bare domains use https; path kept, #fragment dropped). Example: ["vercel.com","http://github.com","https://www.python.org/about/"]

Output Schema

NameRequiredDescription
countYesNumber of rows returned
itemsYesResult rows; each has a "type" field (data rows or machine-readable "error" rows)

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes far beyond the readOnly/idempotent annotations: it discloses User-Agent, 15 s timeout, 1.5 MB body cap, 5-redirect limit, no JS rendering, SSRF refusal of private/loopback addresses, parallel fetching, and that per-URL errors do not fail the whole call. That is exactly the operational context an agent needs before invoking.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Well-labeled, front-loaded sections make a long description scannable and almost every sentence carries distinct information. There is minor redundancy (the free/local nature is restated as 'no APIFY_TOKEN, no Apify billing', and the SSRF guard is touched twice), which keeps it just short of a 5 for a one-parameter tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers purpose, routing, fetch constraints, per-URL error taxonomy, cost model, parameter normalization and an example call. With an output schema also present, there is no gap an agent would need filled before calling this tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, but the description adds real semantics on top: whitespace trimming, implicit https:// with explicit http:// to force plain-HTTP testing, fragment dropping, path/query retention, duplicate handling (fetched twice), and which entries become INVALID_URL / BLOCKED_PRIVATE_ADDRESS rows. These rules are not derivable from the schema alone.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('lightweight reachability check for 1-20 URLs') plus the exact outputs (status, redirect chain, response time, content type, Server header, title). It also distinguishes itself from web_seo_audit and web_batch_audit by name, so an agent can route without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit WHEN TO USE triggers ('is this site up?', 'where does this domain redirect?', pre-filtering leads), a WHEN NOT TO USE section naming the correct siblings (web_seo_audit, web_security_headers, web_batch_audit), and the threshold (>20 URLs) that switches tools. Nothing is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_security_headersHTTP security headers check with grade (free)A
Read-onlyIdempotent
Inspect

Check HTTP security response headers of one URL: Strict-Transport-Security (>=180 days), Content-Security-Policy (flags unsafe-inline/unsafe-eval), X-Content-Type-Options (nosniff), X-Frame-Options (or CSP frame-ancestors), Referrer-Policy, Permissions-Policy, Cross-Origin-Opener-Policy; plus information-disclosure headers (Server, X-Powered-By). Weighted score 0-100 and grade A-F (non-HTTPS capped at 40). WHEN TO USE: quick security posture / hygiene check of a site. Fetching: GETs from this server with User-Agent "FairqueryBot ... Fairquery-MCP", 15 s timeout per request, body capped (1.5 MB HTML), max 5 redirects, no JavaScript rendering; URLs resolving to private/loopback/link-local addresses are refused (SSRF guard). One request (after redirects); headers are taken from the final response. RETURNS: one "security_headers" object: input, final_url, status_code, https, score, grade, headers[{header, present, value, warning, weight}], missing[], info_disclosure_headers[{header, value}], checked_at. Errors (isError=true, JSON {error, message, retryable}): INVALID_URL, BLOCKED_PRIVATE_ADDRESS, DNS_FAILED, FETCH_FAILED, TIMEOUT, TOO_MANY_REDIRECTS. WHEN NOT TO USE: not a vulnerability/penetration scan; for dependency CVEs use an OSV tool; for SEO use web_seo_audit. COST: free - runs locally in this MCP server (direct HTTP GETs), no APIFY_TOKEN, no Apify billing. PARAMETERS: url: headers are read from the FINAL response after up to 5 redirects, so "github.com" grades https://github.com/ - the redirecting hop itself is not graded. Path matters when a site sets different headers per path (e.g. "/login" vs "/"); test the exact page you care about. A bare domain is requested over https; pass "http://..." explicitly to see whether plain HTTP redirects to HTTPS - a final response over plain HTTP caps the score at 40 (grade D at best). Only response headers are evaluated (one request, body capped at 100 KB), never page content. EXAMPLE: {"url":"github.com"} -> {type:"security_headers", score:0-100, grade:"A".."F", missing:[...], ...}

NameRequiredDescriptionDefault
urlYesRequired. One public domain or http(s) URL (non-empty string), e.g. "github.com" or "http://example.com/login". Bare domains use https; headers come from the final response after redirects; private/loopback hosts are refused.

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only declare readOnly/openWorld/idempotent/non-destructive; the description adds substantial context beyond that: User-Agent, 15 s timeout, body cap, max 5 redirects, no JS rendering, SSRF refusal of private/loopback hosts, headers read from the final response, and a full error taxonomy. The only blemish is an internal inconsistency between the '1.5 MB HTML' and '100 KB' body caps.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Long but front-loaded and cleanly sectioned (WHEN TO USE, RETURNS, WHEN NOT TO USE, COST, PARAMETERS, EXAMPLE), with each block earning its place. Minor redundancy and the conflicting body-cap figures keep it from a 5.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema, but RETURNS fully specifies the returned object shape and the error payloads, and the fetch/SSRF constraints are covered. An agent has everything needed to call and interpret this correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% so the baseline is 3, but the PARAMETERS section adds real meaning: redirecting hop is not graded, path-level differences matter, bare domains use https, and passing http:// explicitly reveals redirect behavior with a plain-HTTP score cap of 40.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('Check HTTP security response headers of one URL') and enumerates exactly which headers are evaluated. It explicitly differentiates from siblings in WHEN NOT TO USE, routing CVE work to an OSV tool and SEO to web_seo_audit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Has an explicit WHEN TO USE ('quick security posture / hygiene check') and a WHEN NOT TO USE naming the alternatives and the reason. The agent knows both when to pick this tool and which sibling to pick instead.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_seo_auditSingle-page SEO & AI-readiness audit with 0-100 score (free)A
Read-onlyIdempotent
Inspect

Full on-page audit of ONE URL: title, meta description, canonical, lang, meta robots, viewport, H1s, heading/word/link/image counts, images missing alt, Open Graph, Twitter card, JSON-LD @types, hreflang, feeds, social links, detected technologies, robots.txt/sitemap/llms.txt presence and AI-bot access, plus seo_score (0-100, deductions: high 15 / medium 8 / low 3) with seo_issues[{code, severity, message}] and ai_readiness_score (0-100). Issue codes: NO_HTTPS, MISSING_TITLE, TITLE_TOO_LONG, TITLE_TOO_SHORT, MISSING_META_DESCRIPTION, META_DESCRIPTION_TOO_LONG, MISSING_H1, MULTIPLE_H1, MISSING_CANONICAL, MISSING_LANG, MISSING_VIEWPORT, NOINDEX, MISSING_OPEN_GRAPH, MISSING_STRUCTURED_DATA, IMAGES_MISSING_ALT, THIN_CONTENT, MISSING_ROBOTS_TXT, MISSING_SITEMAP. WHEN TO USE: SEO health of a specific page, or to create a snapshot you can store and later pass to web_compare_snapshots for before/after tracking. For just one aspect use web_ai_crawler_access, web_tech_stack, web_sitemap_info or web_security_headers; for many sites use web_batch_audit. Fetching: GETs from this server with User-Agent "FairqueryBot ... Fairquery-MCP", 15 s timeout per request, body capped (1.5 MB HTML), max 5 redirects, no JavaScript rendering; URLs resolving to private/loopback/link-local addresses are refused (SSRF guard). Makes ~5 requests (page, robots.txt, sitemap, llms.txt, llms-full.txt) unless checkRobots=false. RETURNS: one "seo_audit" snapshot object (snapshot_schema "fairquery-web-snapshot/1") with the fields above plus input, final_url, status_code, https, response_ms, html_truncated, checked_at. Errors (isError=true, JSON {error, message, retryable}): INVALID_URL, BLOCKED_PRIVATE_ADDRESS, DNS_FAILED, FETCH_FAILED, TIMEOUT, TOO_MANY_REDIRECTS. WHEN NOT TO USE: only need reachability (web_quick_check), only AI-bot rules (web_ai_crawler_access), only stack (web_tech_stack), or many sites (web_batch_audit, paid). COST: free - runs locally in this MCP server (direct HTTP GETs), no APIFY_TOKEN, no Apify billing. PARAMETERS: url: the exact page is audited (path and query matter: "python.org" audits the homepage, "python.org/about/" that page), after following up to 5 redirects; on-page fields describe the final_url, while robots.txt, sitemap and llms.txt are always taken from the final URL's origin. Pass "http://..." explicitly to get the NO_HTTPS check on a site that does not redirect. checkRobots=false fetches only the page (1 request instead of ~5): robots_txt_found, sitemap_found, llms_txt_found and ai_bots become null, MISSING_ROBOTS_TXT / MISSING_SITEMAP are never raised and ai_readiness_score is computed from page signals only - so seo_score can be higher and the snapshot is not comparable with a checkRobots=true one in web_compare_snapshots. Keep the default true for snapshots you intend to compare later. EXAMPLE: {"url":"python.org/about/"} -> {type:"seo_audit", seo_score:0-100, ai_readiness_score:0-100, seo_issues:[{code, severity, message}], ...}

NameRequiredDescriptionDefault
urlYesRequired. One public domain or http(s) URL (non-empty string), e.g. "vercel.com" or "https://www.python.org/about/". Bare domains are fetched over https; other schemes, credentials in the URL, dot-less hosts and private/loopback addresses are refused.
checkRobotsNoOptional boolean, default true. true also fetches robots.txt, the XML sitemap, /llms.txt and /llms-full.txt from the page origin (~5 requests); false = page only (1 request, faster; robots/sitemap/llms fields null and their issues skipped). Example: false

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only declare the generic safety profile (readOnly, idempotent, openWorld, non-destructive); the description adds concrete operational behavior far beyond them: GET with a named User-Agent, 15s timeout, 1.5 MB body cap, max 5 redirects, no JS rendering, SSRF refusal of private/loopback addresses, ~5 outbound requests, and the free/local cost model. It also enumerates the error codes and the isError JSON shape.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Though long, it is tightly sectioned (purpose, issue codes, WHEN TO USE, fetching, RETURNS, WHEN NOT TO USE, COST, PARAMETERS, EXAMPLE) and front-loads the core purpose. The 18 issue codes are the only place those outputs are documented since no output schema exists, so every line earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the full burden and does so: it names the return object ('seo_audit' snapshot, snapshot_schema fairquery-web-snapshot/1), its fields, both scores and their deduction weights, and the error taxonomy. An agent has everything needed to call it and interpret the result.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% (baseline 3), but the description materially exceeds the schema: it explains that path/query select the exact page, that on-page fields describe the redirect-resolved final_url while robots/sitemap/llms come from that origin, and that checkRobots=false nulls those fields, suppresses MISSING_ROBOTS_TXT/MISSING_SITEMAP, and makes the snapshot incomparable in web_compare_snapshots.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource ('Full on-page audit of ONE URL') and enumerates the exact fields audited, then names each sibling it is not ('For just one aspect use web_ai_crawler_access, web_tech_stack...; for many sites use web_batch_audit'). An agent can select it correctly without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit WHEN TO USE and WHEN NOT TO USE sections name the alternatives (web_quick_check, web_ai_crawler_access, web_tech_stack, web_sitemap_info, web_security_headers, web_batch_audit) and the exact condition that selects each. It also explains the snapshot/web_compare_snapshots use case, leaving nothing to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_sitemap_infoXML sitemap discovery and URL listing (free, paged)A
Read-onlyIdempotent
Inspect

Find a site's XML sitemap (robots.txt Sitemap: lines, then /sitemap.xml, /sitemap_index.xml, or an explicit sitemapUrl), report whether it is a sitemap index, list child sitemaps, and page through the contained URLs with lastmod. WHEN TO USE: count/inspect indexed pages, find recently modified URLs (filter with contains / sort), feed URLs into web_seo_audit. Fetching: GETs from this server with User-Agent "FairqueryBot ... Fairquery-MCP", 15 s timeout per request, body capped (1.5 MB HTML), max 5 redirects, no JavaScript rendering; URLs resolving to private/loopback/link-local addresses are refused (SSRF guard). Sitemap files are capped at 10 MB each (meta.truncated=true when cut). RETURNS: paged "url" rows (lastmod is null when the sitemap gives none); meta says which sitemap was used and whether it is an index. Errors: SITEMAP_NOT_FOUND plus INVALID_URL, BLOCKED_PRIVATE_ADDRESS, DNS_FAILED, FETCH_FAILED, TIMEOUT, TOO_MANY_REDIRECTS. WHEN NOT TO USE: not a crawler and does not audit pages - pass selected URLs to web_seo_audit; for AI-bot robots.txt rules use web_ai_crawler_access. COST: free - runs locally in this MCP server (direct HTTP GETs), no APIFY_TOKEN, no Apify billing. PARAMETERS: url: only its origin is used for discovery (the path is ignored); discovery tries robots.txt Sitemap: lines, /sitemap.xml and /sitemap_index.xml (max 4 candidates) and, if nothing is found, retries once on the origin the homepage redirects to (example.com -> www.example.com). sitemapUrl skips discovery entirely (no robots.txt, no retry) - use it for sitemaps at non-standard paths; it must be an absolute http(s) URL. maxChildSitemaps only matters when the sitemap is an index: only the first N children (index order) are downloaded, so total counts URLs of those children only; 0 lists child_sitemaps without URLs. contains is a case-sensitive substring test on loc; sort "lastmod_desc" orders by lastmod (ISO strings) with undated URLs last. Both are applied before paging, so total reflects the filtered list; continue with offset = nextOffset. EXAMPLE: {"url":"python.org","contains":"/blog/","sort":"lastmod_desc","limit":10} -> paged url rows with loc and lastmod.

NameRequiredDescriptionDefault
urlYesRequired. One public domain or http(s) URL (non-empty string), e.g. "vercel.com" or "https://www.python.org/about/". Bare domains are fetched over https; other schemes, credentials in the URL, dot-less hosts and private/loopback addresses are refused.
sortNoOptional, one of "sitemap" | "lastmod_desc", default "sitemap" (file order); "lastmod_desc" = most recently modified first, undated last
limitNoOptional page size: number of items to return, integer 1-1000 (default 100). Example: 20
offsetNoOptional zero-based offset into the full result list, integer >= 0 (default 0); use nextOffset from the previous page, e.g. 50
containsNoOptional, default no filter. Case-sensitive substring that loc must contain. Example: "/blog/"
sitemapUrlNoOptional, default = auto-discovery. Absolute http(s) URL of a sitemap or sitemap index to use instead of discovery. Example: "https://www.python.org/sitemap.xml"
maxChildSitemapsNoOptional integer 0-20, default 3. For sitemap indexes only: number of child sitemaps (in index order) to fetch and list URLs from; 0 = list children only. Example: 10

Output Schema

NameRequiredDescription
metaNoTool-specific context (company, source_url, ...)
countYesNumber of items in this page
itemsYesResult rows for this page
totalYesTotal items matching the filters (before paging)
offsetYesOffset of the first item in this page
hasMoreYesTrue if more items exist after this page
nextOffsetYesOffset to pass for the next page, or null

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only/idempotent/open-world, but the description adds substantial context beyond them: User-Agent string, 15 s per-request timeout, 1.5 MB HTML cap, max 5 redirects, no JS rendering, an SSRF guard refusing private/loopback addresses, a 10 MB sitemap cap with meta.truncated signaling, the free/no-billing cost model, and a full error-code list.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Heavily but usefully sectioned (WHEN TO USE, Fetching, RETURNS, WHEN NOT TO USE, COST, PARAMETERS, EXAMPLE) and front-loaded with purpose. It is on the verbose side and some RETURN/error detail is arguably redundant given an output schema exists, but nearly every sentence carries operational information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 7-parameter, non-trivial discovery tool, the definition covers discovery behavior, fetch constraints, pagination, filtering, truncation, error taxonomy, cost, and a worked example. An output schema exists, yet no essential calling information is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Although schema coverage is 100%, the description adds semantics the schema does not carry: url uses only its origin for discovery, the candidate order (robots.txt Sitemap: lines, /sitemap.xml, /sitemap_index.xml, max 4) plus a redirect-origin retry, and that sitemapUrl skips discovery entirely. It also explains that maxChildSitemaps bounds the total count, that contains/sort apply before paging, and that pagination continues via nextOffset.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence states a specific verb and resource chain: 'Find a site's XML sitemap', report index status, list child sitemaps, and page through URLs with lastmod. It also explicitly distinguishes itself from web_seo_audit ('not a crawler and does not audit pages') and web_ai_crawler_access, so an agent can route without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit WHEN TO USE ('count/inspect indexed pages, find recently modified URLs, feed URLs into web_seo_audit') and WHEN NOT TO USE sections name the alternative tools for the excluded cases. The condition selecting sitemapUrl vs. auto-discovery is also spelled out, leaving nothing to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_tech_stackDetect website technology stack (free)A
Read-onlyIdempotent
Inspect

Detect technologies used by one page from its HTML and HTTP headers: frameworks (Next.js, Nuxt, React, ...), CMS (WordPress, Shopify, Webflow, ...), hosting/CDN (Vercel, Cloudflare, Netlify, ...), analytics/tag managers, and more, each with category and evidence. WHEN TO USE: lead enrichment ("what stack does this company use?"), competitor research. Signature-based on one response, so client-injected tools may be missed. For many sites use web_batch_audit. Fetching: GETs from this server with User-Agent "FairqueryBot ... Fairquery-MCP", 15 s timeout per request, body capped (1.5 MB HTML), max 5 redirects, no JavaScript rendering; URLs resolving to private/loopback/link-local addresses are refused (SSRF guard). RETURNS: paged "technology" rows, each with the evidence that matched; meta.final_url is the page actually analysed. Errors (isError=true, JSON {error, message, retryable}): INVALID_URL, BLOCKED_PRIVATE_ADDRESS, DNS_FAILED, FETCH_FAILED, TIMEOUT, TOO_MANY_REDIRECTS. WHEN NOT TO USE: for SEO metadata use web_seo_audit; for security headers use web_security_headers. COST: free - runs locally in this MCP server (direct HTTP GETs), no APIFY_TOKEN, no Apify billing. PARAMETERS: url: exactly that page is fetched (one request after redirects), so a marketing homepage and an app/shop subpage can show different stacks; meta.final_url shows what was analysed. category is a case-insensitive substring filter on the category name, applied before paging: existing categories are CMS, Headless CMS, Site builder, E-commerce, Payments, JS framework, JS library, Framework, CSS framework, Static site generator, Hosting, CDN, Web server, Security, Analytics, Tag manager, Advertising, Marketing automation, Consent, Live chat, Support, Docs, Fonts, Language - e.g. "cms" also matches "Headless CMS" and "framework" matches JS/CSS framework. An unmatched category returns an empty page (count 0), not an error. limit/offset page the filtered list; total = number of matching technologies. EXAMPLE: {"url":"vercel.com","category":"hosting"} -> {count:1, items:[{type:"technology", name:"Vercel", category:"hosting", evidence:"..."}], ...}

NameRequiredDescriptionDefault
urlYesRequired. One public domain or http(s) URL (non-empty string), e.g. "vercel.com" or "https://www.python.org/about/". Bare domains are fetched over https; other schemes, credentials in the URL, dot-less hosts and private/loopback addresses are refused.
limitNoOptional page size: number of items to return, integer 1-200 (default 50). Example: 20
offsetNoOptional zero-based offset into the full result list, integer >= 0 (default 0); use nextOffset from the previous page, e.g. 50
categoryNoOptional, default omitted = all categories. Case-insensitive substring of the category name (e.g. "analytics", "cms", "cdn", "hosting", "e-commerce"). Example: "analytics"

Output Schema

NameRequiredDescription
metaNoTool-specific context (company, source_url, ...)
countYesNumber of items in this page
itemsYesResult rows for this page
totalYesTotal items matching the filters (before paging)
offsetYesOffset of the first item in this page
hasMoreYesTrue if more items exist after this page
nextOffsetYesOffset to pass for the next page, or null

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/openWorld/idempotent/non-destructive, so the bar is lower, but the description still adds rich behavior: FairqueryBot User-Agent, 15s timeout, 1.5 MB HTML cap, max 5 redirects, no JS rendering, SSRF guard on private/loopback addresses, and a full error taxonomy (INVALID_URL, BLOCKED_PRIVATE_ADDRESS, TIMEOUT, etc.). Only the paging/return contract is left to the output schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with purpose and organized under scannable labels (WHEN TO USE, Fetching, RETURNS, WHEN NOT TO USE, COST, PARAMETERS, EXAMPLE), so nearly every line earns its place. Minor redundancy: 'free' appears in the title, COST section, and section header, and the Fetching block is dense.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, yet the definition still covers return shape (paged technology rows with evidence, meta.final_url) and error semantics, and it discloses a meaningful limitation (client-injected tools may be missed because detection is signature-based on one response). Complete for an agent to call it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is already 100%, yet the description adds real meaning: url fetches exactly that page after redirects so homepage and subpage may differ, category is a pre-paging case-insensitive substring filter with 'cms' also matching 'Headless CMS', and an unmatched category yields an empty page (count 0) rather than an error. This is substantive semantics beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('detect technologies used by one page from its HTML and HTTP headers') and enumerates concrete output categories (frameworks, CMS, hosting/CDN, analytics) with evidence. It explicitly distinguishes itself from siblings like web_seo_audit and web_security_headers.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit WHEN TO USE (lead enrichment, competitor research) and WHEN NOT TO USE (SEO metadata -> web_seo_audit; security headers -> web_security_headers), plus routes bulk work to web_batch_audit. Alternatives and selecting conditions are fully spelled out.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 8 tool updatesv1.2.4
    • Changedweb_ai_crawler_access2 fields changed
      • changedInput schema / properties / bots / description
        Previous value: -"User-agent tokens to evaluate instead of the default AI-bot list, e.g. [\"GPTBot\",\"Googlebot\"]"New value: +"Optional, default = the 13 AI bots listed in the description. 1-50 robots.txt user-agent tokens (case-insensitive) to evaluate instead of the defaults. Example: [\"GPTBot\",\"ClaudeBot\",\"Googlebot\"]"
      • changedInput schema / properties / url / description
        Previous value: -"Required. One public domain or http(s) URL, e.g. \"vercel.com\" or \"https://www.python.org/about/\". Bare domains are fetched over https; private/loopback hosts are refused."New value: +"Required. One public domain or http(s) URL (non-empty string), e.g. \"vercel.com\" or \"https://www.python.org/about/\". Bare domains are fetched over https; other schemes, credentials in the URL, dot-less hosts and private/loopback addresses are refused."
    • Changedweb_batch_audit6 fields changed
      • changedInput schema / properties / checkLlmsTxt / description
        Previous value: -"Check /llms.txt and /llms-full.txt (default true)"New value: +"Optional boolean, default true. Check /llms.txt and /llms-full.txt; false = llms_txt fields null"
      • changedInput schema / properties / checkRobots / description
        Previous value: -"Fetch robots.txt and report allow/partial/blocked per AI bot (default true)"New value: +"Optional boolean, default true. Fetch robots.txt and report allowed/partial/blocked per AI bot; false = ai_bots null"
      • changedInput schema / properties / checkSitemap / description
        Previous value: -"Find the XML sitemap and count URLs (default true)"New value: +"Optional boolean, default true. Find the XML sitemap and count its URLs; false = sitemap fields null"
      • changedInput schema / properties / maxSites / description
        Previous value: -"Safety cap on unique URLs per run (default 500); extras produce a MAX_SITES_EXCEEDED error row"New value: +"Optional integer 1-10000, default 500. Safety/spend cap on unique URLs per run; extras produce one free MAX_SITES_EXCEEDED error row. Example: 100"
      • changedInput schema / properties / respectRobotsTxt / description
        Previous value: -"Skip pages robots.txt disallows for generic crawlers, emitting ROBOTS_DISALLOWED (default true, recommended)"New value: +"Optional boolean, default true (recommended). Skip pages robots.txt disallows for generic crawlers, emitting a free ROBOTS_DISALLOWED row"
      • changedInput schema / properties / urls / description
        Previous value: -"Domains or URLs, e.g. [\"vercel.com\",\"https://www.python.org/about/\"]. Bare domains use https. One row per unique URL."New value: +"Required. 1-5000 domains or http(s) URLs; bare domains use https; one billed row per unique URL. Example: [\"vercel.com\",\"https://www.python.org/about/\"]"
    • Changedweb_compare_snapshots4 fields changed
      • changedInput schema / properties / after / description
        Previous value: -"Later snapshot object to compare against before"New value: +"Optional; used only with before. Later snapshot object of the same shape to compare against before"
      • changedInput schema / properties / before / description
        Previous value: -"Earlier snapshot object (output of web_seo_audit or a web_batch_audit site row)"New value: +"Optional; used with after or url. Earlier snapshot object, unmodified output of web_seo_audit (type \"seo_audit\") or a web_batch_audit \"site\" row. Example: {\"type\":\"seo_audit\",\"final_url\":\"https://example.com/\",\"seo_score\":72,...}"
      • changedInput schema / properties / compareUrl / description
        Previous value: -"Second live URL to audit and compare against url"New value: +"Optional; used only with url. Second public domain or http(s) URL audited live as the \"after\" side. Example: \"competitor.com\""
      • changedInput schema / properties / url / description
        Previous value: -"URL to audit live: the \"after\" side (with before) or the first side (with compareUrl)"New value: +"Optional; used with before or compareUrl. Public domain or http(s) URL audited live: the \"after\" side (with before) or the \"before\" side (with compareUrl). Example: \"mysite.com\""
    • Changedweb_quick_check1 field changed
      • changedInput schema / properties / urls / description
        Previous value: -"Required. 1-20 public domains or http(s) URLs (bare domains use https), e.g. [\"vercel.com\",\"http://github.com\"]"New value: +"Required. 1-20 non-empty strings, each a public domain or http(s) URL (bare domains use https; path kept, #fragment dropped). Example: [\"vercel.com\",\"http://github.com\",\"https://www.python.org/about/\"]"
    • Changedweb_security_headers1 field changed
      • changedInput schema / properties / url / description
        Previous value: -"Required. One public domain or http(s) URL, e.g. \"vercel.com\" or \"https://www.python.org/about/\". Bare domains are fetched over https; private/loopback hosts are refused."New value: +"Required. One public domain or http(s) URL (non-empty string), e.g. \"github.com\" or \"http://example.com/login\". Bare domains use https; headers come from the final response after redirects; private/loopback hosts are refused."
    • Changedweb_seo_audit2 fields changed
      • changedInput schema / properties / checkRobots / description
        Previous value: -"Also fetch robots.txt, sitemap and llms.txt for AI-bot/sitemap checks and scores (default true). false = page only, faster."New value: +"Optional boolean, default true. true also fetches robots.txt, the XML sitemap, /llms.txt and /llms-full.txt from the page origin (~5 requests); false = page only (1 request, faster; robots/sitemap/llms fields null and their issues skipped). Example: false"
      • changedInput schema / properties / url / description
        Previous value: -"Required. One public domain or http(s) URL, e.g. \"vercel.com\" or \"https://www.python.org/about/\". Bare domains are fetched over https; private/loopback hosts are refused."New value: +"Required. One public domain or http(s) URL (non-empty string), e.g. \"vercel.com\" or \"https://www.python.org/about/\". Bare domains are fetched over https; other schemes, credentials in the URL, dot-less hosts and private/loopback addresses are refused."
    • Changedweb_sitemap_info7 fields changed
      • changedInput schema / properties / contains / description
        Previous value: -"Only URLs containing this substring, e.g. \"/blog/\""New value: +"Optional, default no filter. Case-sensitive substring that loc must contain. Example: \"/blog/\""
      • changedInput schema / properties / limit / description
        Previous value: -"Optional page size: number of items to return, integer 1-1000 (default 100)"New value: +"Optional page size: number of items to return, integer 1-1000 (default 100). Example: 20"
      • changedInput schema / properties / maxChildSitemaps / description
        Previous value: -"For sitemap indexes: how many child sitemaps to fetch and list URLs from (default 3, max 20; 0 = list children only)"New value: +"Optional integer 0-20, default 3. For sitemap indexes only: number of child sitemaps (in index order) to fetch and list URLs from; 0 = list children only. Example: 10"
      • changedInput schema / properties / offset / description
        Previous value: -"Zero-based offset into the full result list (default 0); use nextOffset from the previous page"New value: +"Optional zero-based offset into the full result list, integer >= 0 (default 0); use nextOffset from the previous page, e.g. 50"
      • changedInput schema / properties / sitemapUrl / description
        Previous value: -"Explicit sitemap URL to use instead of auto-discovery"New value: +"Optional, default = auto-discovery. Absolute http(s) URL of a sitemap or sitemap index to use instead of discovery. Example: \"https://www.python.org/sitemap.xml\""
      • changedInput schema / properties / sort / description
        Previous value: -"\"sitemap\" = file order (default); \"lastmod_desc\" = most recently modified first"New value: +"Optional, one of \"sitemap\" | \"lastmod_desc\", default \"sitemap\" (file order); \"lastmod_desc\" = most recently modified first, undated last"
      • changedInput schema / properties / url / description
        Previous value: -"Required. One public domain or http(s) URL, e.g. \"vercel.com\" or \"https://www.python.org/about/\". Bare domains are fetched over https; private/loopback hosts are refused."New value: +"Required. One public domain or http(s) URL (non-empty string), e.g. \"vercel.com\" or \"https://www.python.org/about/\". Bare domains are fetched over https; other schemes, credentials in the URL, dot-less hosts and private/loopback addresses are refused."
    • Changedweb_tech_stack4 fields changed
      • changedInput schema / properties / category / description
        Previous value: -"Only return technologies whose category contains this text (case-insensitive), e.g. \"analytics\", \"cms\", \"cdn\""New value: +"Optional, default omitted = all categories. Case-insensitive substring of the category name (e.g. \"analytics\", \"cms\", \"cdn\", \"hosting\", \"e-commerce\"). Example: \"analytics\""
      • changedInput schema / properties / limit / description
        Previous value: -"Optional page size: number of items to return, integer 1-200 (default 50)"New value: +"Optional page size: number of items to return, integer 1-200 (default 50). Example: 20"
      • changedInput schema / properties / offset / description
        Previous value: -"Zero-based offset into the full result list (default 0); use nextOffset from the previous page"New value: +"Optional zero-based offset into the full result list, integer >= 0 (default 0); use nextOffset from the previous page, e.g. 50"
      • changedInput schema / properties / url / description
        Previous value: -"Required. One public domain or http(s) URL, e.g. \"vercel.com\" or \"https://www.python.org/about/\". Bare domains are fetched over https; private/loopback hosts are refused."New value: +"Required. One public domain or http(s) URL (non-empty string), e.g. \"vercel.com\" or \"https://www.python.org/about/\". Bare domains are fetched over https; other schemes, credentials in the URL, dot-less hosts and private/loopback addresses are refused."
  2. 9 tool updatesv1.2.1
    • Removedcheck_websites
    • Addedweb_ai_crawler_access
    • Addedweb_batch_audit
    • Addedweb_compare_snapshots
    • Addedweb_quick_check
    • Addedweb_security_headers
    • Addedweb_seo_audit
    • Addedweb_sitemap_info
    • Addedweb_tech_stack
  3. 1 tool updatev0.1.0
    • First observedcheck_websites

TDQS

A4.7/5.0

Scored across 8 tools

Disambiguation4/5

Each tool has a distinct primary purpose and the descriptions include explicit WHEN TO USE / WHEN NOT TO USE cross-references, which sharply reduces misselection. The main overlap is that web_seo_audit bundles robots.txt/llms.txt, AI-bot access, sitemap and technology detection that four single-purpose tools also cover, but the guidance redirects agents correctly.

Naming Consistency5/5

All eight tools use a consistent web_<noun_phrase> snake_case pattern (web_seo_audit, web_tech_stack, web_sitemap_info, etc.) with no style mixing. The prefix cleanly scopes the family and verbs/nouns are predictable.

Tool Count5/5

Eight tools is well-scoped for a web auditing server: one bulk tool, one diff tool, and six focused single-purpose checks. No tool feels redundant or out of place.

Completeness4/5

The surface covers reachability, on-page SEO, security headers, tech stack, sitemap, AI-crawler access, bulk audit and snapshot diffing, giving a coherent lifecycle with snapshot storage feeding comparison. Minor gaps remain (e.g. no explicit link/backlink or content-extraction tool), but agents can work around them.

Maintenance

ActivityNo data
ResponsivenessSyncing

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    C
    maintenance
    Enables AI agents to check whether a public website is crawlable, understandable, and ready for AI search workflows through local-only audits of robots.txt, sitemaps, metadata, and llms.txt.
    3
    80 npm
    2
    MIT
  • A
    license
    A
    quality
    B
    maintenance
    Enables AI clients to analyze any company domain and determine its AI maturity tier (commercialized, deployed, declared, or none) with supporting evidence, via a single tool backed by an Apify actor.
    1
    73 npm
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables AI agents to run a free AI-readiness audit of any URL, checking AI crawler rules, JavaScript-free page text, JSON-LD, llms.txt, sitemap, meta description, FAQ schema, and returning a score with findings and fixes. Also exposes the same capability via an A2A agent endpoint.
    MIT