ContrastAPI

Robots.txt

robots_txt

Read-onlyIdempotent

Fetch + parse the target domain's robots.txt — sitemaps, per-User-agent allow/disallow rules, crawl-delay, Host directive. Use BEFORE crawling/scraping a target site (seo_audit, brand_assets, redirect_chain) to honour the site's published rules. status_code=404 means no robots.txt exists = implicit allow-all per RFC 9309 §2.4. ContrastAPI fetches with User-agent: ContrastAPI/<version> (+https://contrastcyber.com/bot) so site operators can identify + opt out via robots.txt; we honour Disallow: / for our UA in seo_audit and brand_assets. Per-target eTLD+1 throttle (60 req/min) prevents weaponising this endpoint against a single site; subdomain rotation collapses to the same bucket. Free: 30/hr, Pro: 500/hr. Returns {domain, fetched_url, status_code, sitemaps, user_agents:{ua:{allow,disallow,crawl_delay}}, host, truncated, summary}. Returns 502 ErrorResponse if the target rejected the connection (DNS/TCP/TLS failure); the agent should NOT assume "no robots" in that case — it's an upstream-failure signal.

Input Schema

TableJSON Schema

Name	Required	Description	Default
`domain`	Yes	Registrable domain to fetch robots.txt for (e.g. 'example.com', 'github.com'). No scheme, no path, no port. Subdomains accepted; the bot fetches https://<domain>/robots.txt with HTTP fallback.

Output Schema

TableJSON Schema

Name	Required	Description	Default
`result`	Yes

Schema Changelog

Changes observed during successful MCP inspections.

Changed3 schema fields changedJun 2, 2026
- changedOutput schema / properties / result / properties / host / type
```
Previous value: -"string"New value: +[
+  "string",
+  "null"
+]
```
- changedOutput schema / properties / result / properties / next_calls / type
```
Previous value: -"array"New value: +[
+  "array",
+  "null"
+]
```
- changedOutput schema / properties / result / properties / verdict / type
```
Previous value: -"object"New value: +[
+  "object",
+  "null"
+]
```
Changed2 schema fields changedMay 25, 2026
- changedOutput schema / properties / result / properties / host / type
```
Previous value: -"object"New value: +"string"
```
- changedOutput schema / properties / result / properties / next_calls / type
```
Previous value: -"object"New value: +"array"
```

Changed1 schema field changedMay 22, 2026

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "properties": {
+    "result": {
+      "properties": {
+        "domain": {
+          "type": "string"
+        },
+        "fetched_url": {
+          "type": "string"
+        },
+        "host": {
+          "type": "object"
+        },
+        "next_calls": {
+          "type": "object"
+        },
+        "sitemaps": {
+          "type": "array"
+        },
+        "status_code": {
+          "type": "integer"
+        },
+        "summary": {
+          "type": "string"
+        },
+        "truncated": {
+          "type": "boolean"
+        },
+        "user_agents": {
+          "type": "object"
+        },
+        "verdict": {
+          "type": "object"
+        }
+      },
+      "required": [
+        "domain",
+        "fetched_url",
+        "status_code"
+      ],
+      "type": "object"
+    }
+  },
+  "required": [
+    "result"
+  ],
+  "type": "object"
+}

Changed1 schema field changedMay 22, 2026

changedOutput schema / (root)

Previous value: -{
-  "$defs": {
-    "ErrorDetail": {
-      "properties": {
-        "code": {
-          "enum": [
-            "invalid_argument",
-            "not_found",
-            "rate_limit_exceeded",
-            "auth_required",
-            "tier_limit",
-            "upstream_timeout",
-            "upstream_error",
-            "internal_error"
-          ],
-          "type": "string"
-        },
-        "docs_url": {
-          "anyOf": [
-            {
-              "type": "string"
-            },
-            {
-              "type": "null"
-            }
-          ]
-        },
-        "message": {
-          "maxLength": 500,
-          "type": "string"
-        },
-        "retry_after_seconds": {
-          "anyOf": [
-            {
-              "type": "integer"
-            },
-            {
-              "type": "null"
-            }
-          ]
-        },
-        "upgrade_url": {
-          "anyOf": [
-            {
-              "type": "string"
-            },
-            {
-              "type": "null"
-            }
-          ]
-        }
-      },
-      "required": [
-        "code",
-        "message"
-      ],
-      "type": "object"
-    },
-    "ErrorResponse": {
-      "properties": {
-        "error": {
-          "$ref": "#/$defs/ErrorDetail"
-        }
-      },
-      "required": [
-        "error"
-      ],
-      "type": "object"
-    },
-    "PivotHint": {
-      "additionalProperties": true,
-      "properties": {
-        "input": {
-          "type": "string"
-        },
-        "params": {
-          "anyOf": [
-            {
-              "additionalProperties": {
-                "type": "string"
-              },
-              "type": "object"
-            },
-            {
-              "type": "null"
-            }
-          ]
-        },
-        "reason": {
-          "type": "string"
-        },
-        "tool": {
-          "enum": [
-            "cve_lookup",
-            "cve_search",
-            "cve_leading",
-            "bulk_cve_lookup",
-            "calculate_risk_score",
-            "get_cvss_details",
-            "exploit_lookup",
-            "kev_detail",
-            "cwe_lookup",
-            "subdomain_enum",
-            "ssl_check",
-            "tech_fingerprint",
-            "asn_lookup",
-            "ip_lookup",
-            "ioc_lookup",
-            "bulk_ioc_lookup",
-            "hash_lookup",
-            "threat_intel",
-            "threat_report",
-            "audit_domain",
-            "domain_report",
-            "dns_lookup",
-            "whois_lookup",
-            "wayback_lookup",
-            "scan_headers",
-            "check_headers",
-            "check_secrets",
-            "check_injection",
-            "check_dependencies",
-            "email_mx",
-            "email_security_posture",
-            "email_disposable",
-            "email_verify",
-            "robots_txt",
-            "redirect_chain",
-            "brand_assets",
-            "seo_audit",
-            "phone_lookup",
-            "username_lookup",
-            "password_check",
-            "phishing_check",
-            "atlas_technique_lookup",
-            "atlas_technique_search",
-            "bulk_atlas_technique_lookup",
-            "atlas_case_study_lookup",
-            "atlas_case_study_search",
-            "d3fend_defense_lookup",
-            "d3fend_defense_search",
-            "d3fend_defense_for_attack",
-            "d3fend_attack_coverage",
-            "sigma_rule_lookup",
-            "bulk_sigma_rule_lookup",
-            "tech_stack_cve_audit"
-          ],
-          "type": "string"
-        }
-      },
-      "required": [
-        "tool",
-        "input",
-        "reason"
-      ],
-      "type": "object"
-    },
-    "RobotsRules": {
-      "additionalProperties": true,
-      "properties": {
-        "allow": {
-          "items": {
-            "type": "string"
-          },
-          "type": "array"
-        },
-        "crawl_delay": {
-          "anyOf": [
-            {
-              "type": "number"
-            },
-            {
-              "type": "null"
-            }
-          ]
-        },
-        "disallow": {
-          "items": {
-            "type": "string"
-          },
-          "type": "array"
-        }
-      },
-      "type": "object"
-    },
-    "RobotsTxtResponse": {
-      "properties": {
-        "domain": {
-          "type": "string"
-        },
-        "fetched_url": {
-          "type": "string"
-        },
-        "host": {
-          "anyOf": [
-            {
-              "type": "string"
-            },
-            {
-              "type": "null"
-            }
-          ]
-        },
-        "next_calls": {
-          "anyOf": [
-            {
-              "items": {
-                "$ref": "#/$defs/PivotHint"
-              },
-              "type": "array"
-            },
-            {
-              "type": "null"
-            }
-          ]
-        },
-        "sitemaps": {
-          "items": {
-            "type": "string"
-          },
-          "type": "array"
-        },
-        "status_code": {
-          "type": "integer"
-        },
-        "summary": {
-          "type": "string"
-        },
-        "truncated": {
-          "type": "boolean"
-        },
-        "user_agents": {
-          "additionalProperties": {
-            "$ref": "#/$defs/RobotsRules"
-          },
-          "type": "object"
-        },
-        "verdict": {
-          "anyOf": [
-            {
-              "$ref": "#/$defs/Verdict"
-            },
-            {
-              "type": "null"
-            }
-          ]
-        }
-      },
-      "required": [
-        "domain",
-        "fetched_url",
-        "status_code"
-      ],
-      "type": "object"
-    },
-    "Verdict": {
-      "properties": {
-        "completeness": {
-          "enum": [
-            "complete",
-            "partial",
-            "minimal"
-          ],
-          "type": "string"
-        },
-        "data_age_seconds": {
-          "anyOf": [
-            {
-              "type": "integer"
-            },
-            {
-              "type": "null"
-            }
-          ]
-        },
-        "deterministic": {
-          "type": "boolean"
-        },
-        "falsifiable_fields": {
-          "items": {
-            "type": "string"
-          },
-          "type": "array"
-        },
-        "sources_queried": {
-          "items": {
-            "type": "string"
-          },
-          "type": "array"
-        },
-        "sources_unavailable": {
-          "items": {
-            "type": "string"
-          },
-          "type": "array"
-        }
-      },
-      "required": [
-        "deterministic"
-      ],
-      "type": "object"
-    }
-  },
-  "properties": {
-    "result": {
-      "anyOf": [
-        {
-          "$ref": "#/$defs/RobotsTxtResponse"
-        },
-        {
-          "$ref": "#/$defs/ErrorResponse"
-        }
-      ]
-    }
-  },
-  "required": [
-    "result"
-  ],
-  "type": "object"
-}New value: +null

Changed80 schema fields changedMay 17, 2026

removedOutput schema / $defs / ErrorDetail / description

Removed value: -"Structured failure body. Codes mirror app/exceptions.AppException\nsubclasses; agent retry / upgrade decisions key off `code`, not `message`."

removedOutput schema / $defs / ErrorDetail / properties / code / description

Removed value: -"Stable machine-readable failure category. Agents key retry/upgrade decisions off this."

removedOutput schema / $defs / ErrorDetail / properties / code / title
```
Removed value: -"Code"
```
removedOutput schema / $defs / ErrorDetail / properties / docs_url / default
```
Removed value: -null
```

removedOutput schema / $defs / ErrorDetail / properties / docs_url / description

Removed value: -"Documentation pointer (e.g. tool input contract) when code='invalid_argument'."

removedOutput schema / $defs / ErrorDetail / properties / docs_url / title
```
Removed value: -"Docs Url"
```

removedOutput schema / $defs / ErrorDetail / properties / message / description

Removed value: -"Human-readable detail. Free text — never parse. Capped at 500 chars to prevent oversized upstream errors from bloating responses."

removedOutput schema / $defs / ErrorDetail / properties / message / title
```
Removed value: -"Message"
```
removedOutput schema / $defs / ErrorDetail / properties / retry_after_seconds / default
```
Removed value: -null
```

removedOutput schema / $defs / ErrorDetail / properties / retry_after_seconds / description

Removed value: -"When code='rate_limit_exceeded', the minimum seconds to wait before retrying."

removedOutput schema / $defs / ErrorDetail / properties / retry_after_seconds / title
```
Removed value: -"Retry After Seconds"
```
removedOutput schema / $defs / ErrorDetail / properties / upgrade_url / default
```
Removed value: -null
```

removedOutput schema / $defs / ErrorDetail / properties / upgrade_url / description

Removed value: -"Pricing/upgrade URL when code='tier_limit' or 'rate_limit_exceeded' on the Free tier."

removedOutput schema / $defs / ErrorDetail / properties / upgrade_url / title
```
Removed value: -"Upgrade Url"
```
removedOutput schema / $defs / ErrorDetail / title
```
Removed value: -"ErrorDetail"
```

removedOutput schema / $defs / ErrorResponse / description

Removed value: -"MCP error envelope. Tool return type is always\n`SpecificResponse | ErrorResponse` — Union flag tells the agent which arm\narrived without parsing the inner body."

removedOutput schema / $defs / ErrorResponse / title
```
Removed value: -"ErrorResponse"
```

removedOutput schema / $defs / PivotHint / description

Removed value: -"A suggested follow-up MCP tool call. Surfaced inside response.next_calls so\nLLM agents can chain related lookups without manual prompting. Each hint names\nthe tool, the input value to pass, and a short reason explaining why this\npivot adds value in the current context."

removedOutput schema / $defs / PivotHint / properties / input / description

Removed value: -"Suggested input value to pass to the tool — typically a CVE ID, CWE ID, domain, or IP. Pre-populated from the current response so the agent can call the next tool without re-deriving the argument."

removedOutput schema / $defs / PivotHint / properties / input / title
```
Removed value: -"Input"
```
removedOutput schema / $defs / PivotHint / properties / params / default
```
Removed value: -null
```

removedOutput schema / $defs / PivotHint / properties / params / description

Removed value: -"Optional extra kwargs to pass alongside `input`. Used by pivot generators when the next call benefits from a secondary parameter, e.g. {'exclude_id': 'AML.T0051'} to skip the originating technique from a sibling-tactic search. Omitted when no extra args are needed."

removedOutput schema / $defs / PivotHint / properties / params / title
```
Removed value: -"Params"
```

removedOutput schema / $defs / PivotHint / properties / reason / description

Removed value: -"Short rationale (one sentence) for why this follow-up call adds value, e.g. 'Federal patch deadline + ransomware association', 'Public exploits / PoC availability'."

removedOutput schema / $defs / PivotHint / properties / reason / title
```
Removed value: -"Reason"
```

removedOutput schema / $defs / PivotHint / properties / tool / description

Removed value: -"Canonical MCP tool name to call next. Constrained to known operation_ids in tools/list — adding a new tool here requires expanding the Literal."

removedOutput schema / $defs / PivotHint / properties / tool / title
```
Removed value: -"Tool"
```
removedOutput schema / $defs / PivotHint / title
```
Removed value: -"PivotHint"
```

removedOutput schema / $defs / RobotsRules / description

Removed value: -"Per-User-agent rule block parsed from a robots.txt file."

removedOutput schema / $defs / RobotsRules / properties / allow / description

Removed value: -"Paths the target site explicitly Allows for this UA. Each entry is verbatim from robots.txt (`_untrusted` — DO NOT execute or shell-out)."

removedOutput schema / $defs / RobotsRules / properties / allow / title
```
Removed value: -"Allow"
```
removedOutput schema / $defs / RobotsRules / properties / crawl_delay / default
```
Removed value: -null
```
removedOutput schema / $defs / RobotsRules / properties / crawl_delay / description
```
Removed value: -"Crawl-delay seconds for this UA, if specified."
```
removedOutput schema / $defs / RobotsRules / properties / crawl_delay / title
```
Removed value: -"Crawl Delay"
```

removedOutput schema / $defs / RobotsRules / properties / disallow / description

Removed value: -"Paths the target site Disallows for this UA. Empty Disallow per spec means allow-all."

removedOutput schema / $defs / RobotsRules / properties / disallow / title
```
Removed value: -"Disallow"
```
removedOutput schema / $defs / RobotsRules / title
```
Removed value: -"RobotsRules"
```
removedOutput schema / $defs / RobotsTxtResponse / description
```
Removed value: -"Parsed robots.txt for the target domain."
```
removedOutput schema / $defs / RobotsTxtResponse / properties / domain / description
```
Removed value: -"Queried domain (echoed)."
```
removedOutput schema / $defs / RobotsTxtResponse / properties / domain / title
```
Removed value: -"Domain"
```
removedOutput schema / $defs / RobotsTxtResponse / properties / fetched_url / description
```
Removed value: -"Final URL we fetched, e.g. https://example.com/robots.txt."
```
removedOutput schema / $defs / RobotsTxtResponse / properties / fetched_url / title
```
Removed value: -"Fetched Url"
```
removedOutput schema / $defs / RobotsTxtResponse / properties / host / default
```
Removed value: -null
```

removedOutput schema / $defs / RobotsTxtResponse / properties / host / description

Removed value: -"`Host:` directive (Yandex extension), if present. `_untrusted`."

removedOutput schema / $defs / RobotsTxtResponse / properties / host / title
```
Removed value: -"Host"
```
removedOutput schema / $defs / RobotsTxtResponse / properties / next_calls / default
```
Removed value: -null
```

removedOutput schema / $defs / RobotsTxtResponse / properties / next_calls / description

Removed value: -"Suggested follow-up MCP tool calls. Ordered by relevance; agents should chain these without re-prompting the user."

removedOutput schema / $defs / RobotsTxtResponse / properties / next_calls / title
```
Removed value: -"Next Calls"
```

removedOutput schema / $defs / RobotsTxtResponse / properties / sitemaps / description

Removed value: -"`Sitemap:` directives (URLs). Global, not per-UA. `_untrusted` — fetch only via SSRF-safe path."

removedOutput schema / $defs / RobotsTxtResponse / properties / sitemaps / title
```
Removed value: -"Sitemaps"
```

removedOutput schema / $defs / RobotsTxtResponse / properties / status_code / description

Removed value: -"HTTP status returned by the target. 404 = no robots.txt = implicit allow-all."

removedOutput schema / $defs / RobotsTxtResponse / properties / status_code / title
```
Removed value: -"Status Code"
```
removedOutput schema / $defs / RobotsTxtResponse / properties / summary / default
```
Removed value: -""
```
removedOutput schema / $defs / RobotsTxtResponse / properties / summary / description
```
Removed value: -"One-line human-readable summary."
```
removedOutput schema / $defs / RobotsTxtResponse / properties / summary / title
```
Removed value: -"Summary"
```
removedOutput schema / $defs / RobotsTxtResponse / properties / truncated / default
```
Removed value: -false
```

removedOutput schema / $defs / RobotsTxtResponse / properties / truncated / description

Removed value: -"True if the robots.txt body exceeded ROBOTS_MAX_BYTES and was truncated before parsing."

removedOutput schema / $defs / RobotsTxtResponse / properties / truncated / title
```
Removed value: -"Truncated"
```

removedOutput schema / $defs / RobotsTxtResponse / properties / user_agents / description

Removed value: -"Per-`User-agent:` rule blocks. Wildcard `*` is one of the keys when present."

removedOutput schema / $defs / RobotsTxtResponse / properties / user_agents / title
```
Removed value: -"User Agents"
```
removedOutput schema / $defs / RobotsTxtResponse / properties / verdict / default
```
Removed value: -null
```

removedOutput schema / $defs / RobotsTxtResponse / properties / verdict / description

Removed value: -"Falsifiability metadata: sources_queried, sources_unavailable, completeness, deterministic flag. Lets agents distinguish 'no data' from 'source failed' without re-running the call."

removedOutput schema / $defs / RobotsTxtResponse / title
```
Removed value: -"RobotsTxtResponse"
```
removedOutput schema / $defs / Verdict / properties / completeness / default
```
Removed value: -"complete"
```

removedOutput schema / $defs / Verdict / properties / completeness / description

Removed value: -"'complete' = every planned source returned data; 'partial' = at least one source in sources_unavailable failed or was skipped; 'minimal' = only the primary/required source returned, optional enrichment missing."

removedOutput schema / $defs / Verdict / properties / completeness / title
```
Removed value: -"Completeness"
```
removedOutput schema / $defs / Verdict / properties / data_age_seconds / default
```
Removed value: -null
```

removedOutput schema / $defs / Verdict / properties / data_age_seconds / description

Removed value: -"Seconds elapsed since the oldest cached source was fetched, or null when every source was queried live for this request. Use to judge freshness."

removedOutput schema / $defs / Verdict / properties / data_age_seconds / title
```
Removed value: -"Data Age Seconds"
```

removedOutput schema / $defs / Verdict / properties / deterministic / description

Removed value: -"True when the response is fully reproducible from the listed sources for the same input at the same moment (no randomness, no model inference). False for endpoints that include probabilistic scoring or LLM output."

removedOutput schema / $defs / Verdict / properties / deterministic / title
```
Removed value: -"Deterministic"
```

removedOutput schema / $defs / Verdict / properties / falsifiable_fields / description

Removed value: -"Top-level response fields whose values a caller can independently re-derive from the named upstream sources (e.g. 'dns', 'ssl', 'whois'). Fields not in this list are derived/computed and cannot be directly re-verified."

removedOutput schema / $defs / Verdict / properties / falsifiable_fields / title
```
Removed value: -"Falsifiable Fields"
```

removedOutput schema / $defs / Verdict / properties / sources_queried / description

Removed value: -"Canonical source identifiers successfully consulted for this response (e.g. 'ripe_stat', 'shodan_internetdb', 'firehol'). Agent-readable list, order not significant."

removedOutput schema / $defs / Verdict / properties / sources_queried / title
```
Removed value: -"Sources Queried"
```

removedOutput schema / $defs / Verdict / properties / sources_unavailable / description

Removed value: -"Sources that were expected but not returned — either intentionally skipped (lite mode, tier gating) or failed (quota, timeout, upstream down). Empty list means every planned source produced data."

removedOutput schema / $defs / Verdict / properties / sources_unavailable / title
```
Removed value: -"Sources Unavailable"
```
removedOutput schema / $defs / Verdict / title
```
Removed value: -"Verdict"
```
removedOutput schema / properties / result / title
```
Removed value: -"Result"
```
removedOutput schema / title
```
Removed value: -"robots_txtOutput"
```

Changed1 schema field changedMay 15, 2026

changedOutput schema / $defs / PivotHint / properties / tool / enum

Previous value: -[
-  "cve_lookup",
-  "cve_search",
-  "cve_leading",
-  "bulk_cve_lookup",
-  "calculate_risk_score",
-  "get_cvss_details",
-  "exploit_lookup",
-  "kev_detail",
-  "cwe_lookup",
-  "subdomain_enum",
-  "ssl_check",
-  "tech_fingerprint",
-  "asn_lookup",
-  "ip_lookup",
-  "ioc_lookup",
-  "bulk_ioc_lookup",
-  "hash_lookup",
-  "threat_intel",
-  "threat_report",
-  "audit_domain",
-  "domain_report",
-  "dns_lookup",
-  "whois_lookup",
-  "wayback_lookup",
-  "scan_headers",
-  "check_headers",
-  "check_secrets",
-  "check_injection",
-  "check_dependencies",
-  "email_mx",
-  "email_security_posture",
-  "email_disposable",
-  "email_verify",
-  "robots_txt",
-  "redirect_chain",
-  "brand_assets",
-  "seo_audit",
-  "phone_lookup",
-  "username_lookup",
-  "password_check",
-  "phishing_check",
-  "atlas_technique_lookup",
-  "atlas_technique_search",
-  "bulk_atlas_technique_lookup",
-  "atlas_case_study_lookup",
-  "atlas_case_study_search",
-  "d3fend_defense_lookup",
-  "d3fend_defense_search",
-  "d3fend_defense_for_attack",
-  "d3fend_attack_coverage",
-  "sigma_rule_lookup",
-  "bulk_sigma_rule_lookup"
-]New value: +[
+  "cve_lookup",
+  "cve_search",
+  "cve_leading",
+  "bulk_cve_lookup",
+  "calculate_risk_score",
+  "get_cvss_details",
+  "exploit_lookup",
+  "kev_detail",
+  "cwe_lookup",
+  "subdomain_enum",
+  "ssl_check",
+  "tech_fingerprint",
+  "asn_lookup",
+  "ip_lookup",
+  "ioc_lookup",
+  "bulk_ioc_lookup",
+  "hash_lookup",
+  "threat_intel",
+  "threat_report",
+  "audit_domain",
+  "domain_report",
+  "dns_lookup",
+  "whois_lookup",
+  "wayback_lookup",
+  "scan_headers",
+  "check_headers",
+  "check_secrets",
+  "check_injection",
+  "check_dependencies",
+  "email_mx",
+  "email_security_posture",
+  "email_disposable",
+  "email_verify",
+  "robots_txt",
+  "redirect_chain",
+  "brand_assets",
+  "seo_audit",
+  "phone_lookup",
+  "username_lookup",
+  "password_check",
+  "phishing_check",
+  "atlas_technique_lookup",
+  "atlas_technique_search",
+  "bulk_atlas_technique_lookup",
+  "atlas_case_study_lookup",
+  "atlas_case_study_search",
+  "d3fend_defense_lookup",
+  "d3fend_defense_search",
+  "d3fend_defense_for_attack",
+  "d3fend_attack_coverage",
+  "sigma_rule_lookup",
+  "bulk_sigma_rule_lookup",
+  "tech_stack_cve_audit"
+]

Changed1 schema field changedMay 12, 2026

changedOutput schema / $defs / PivotHint / properties / tool / enum

Previous value: -[
-  "cve_lookup",
-  "cve_search",
-  "cve_leading",
-  "bulk_cve_lookup",
-  "calculate_risk_score",
-  "get_cvss_details",
-  "exploit_lookup",
-  "kev_detail",
-  "cwe_lookup",
-  "subdomain_enum",
-  "ssl_check",
-  "tech_fingerprint",
-  "asn_lookup",
-  "ip_lookup",
-  "ioc_lookup",
-  "bulk_ioc_lookup",
-  "hash_lookup",
-  "threat_intel",
-  "threat_report",
-  "audit_domain",
-  "domain_report",
-  "dns_lookup",
-  "whois_lookup",
-  "wayback_lookup",
-  "scan_headers",
-  "check_headers",
-  "check_secrets",
-  "check_injection",
-  "check_dependencies",
-  "email_mx",
-  "email_security_posture",
-  "email_disposable",
-  "email_verify",
-  "robots_txt",
-  "redirect_chain",
-  "brand_assets",
-  "seo_audit",
-  "phone_lookup",
-  "username_lookup",
-  "password_check",
-  "phishing_check",
-  "atlas_technique_lookup",
-  "atlas_technique_search",
-  "bulk_atlas_technique_lookup",
-  "atlas_case_study_lookup",
-  "atlas_case_study_search",
-  "d3fend_defense_lookup",
-  "d3fend_defense_search",
-  "d3fend_defense_for_attack",
-  "d3fend_attack_coverage"
-]New value: +[
+  "cve_lookup",
+  "cve_search",
+  "cve_leading",
+  "bulk_cve_lookup",
+  "calculate_risk_score",
+  "get_cvss_details",
+  "exploit_lookup",
+  "kev_detail",
+  "cwe_lookup",
+  "subdomain_enum",
+  "ssl_check",
+  "tech_fingerprint",
+  "asn_lookup",
+  "ip_lookup",
+  "ioc_lookup",
+  "bulk_ioc_lookup",
+  "hash_lookup",
+  "threat_intel",
+  "threat_report",
+  "audit_domain",
+  "domain_report",
+  "dns_lookup",
+  "whois_lookup",
+  "wayback_lookup",
+  "scan_headers",
+  "check_headers",
+  "check_secrets",
+  "check_injection",
+  "check_dependencies",
+  "email_mx",
+  "email_security_posture",
+  "email_disposable",
+  "email_verify",
+  "robots_txt",
+  "redirect_chain",
+  "brand_assets",
+  "seo_audit",
+  "phone_lookup",
+  "username_lookup",
+  "password_check",
+  "phishing_check",
+  "atlas_technique_lookup",
+  "atlas_technique_search",
+  "bulk_atlas_technique_lookup",
+  "atlas_case_study_lookup",
+  "atlas_case_study_search",
+  "d3fend_defense_lookup",
+  "d3fend_defense_search",
+  "d3fend_defense_for_attack",
+  "d3fend_attack_coverage",
+  "sigma_rule_lookup",
+  "bulk_sigma_rule_lookup"
+]

Changed1 schema field changedMay 12, 2026

changedOutput schema / $defs / PivotHint / properties / tool / enum

Previous value: -[
-  "cve_lookup",
-  "cve_search",
-  "cve_leading",
-  "bulk_cve_lookup",
-  "calculate_risk_score",
-  "get_cvss_details",
-  "exploit_lookup",
-  "kev_detail",
-  "cwe_lookup",
-  "subdomain_enum",
-  "ssl_check",
-  "tech_fingerprint",
-  "asn_lookup",
-  "ip_lookup",
-  "ioc_lookup",
-  "bulk_ioc_lookup",
-  "hash_lookup",
-  "threat_intel",
-  "threat_report",
-  "audit_domain",
-  "domain_report",
-  "dns_lookup",
-  "whois_lookup",
-  "wayback_lookup",
-  "scan_headers",
-  "check_headers",
-  "check_secrets",
-  "check_injection",
-  "check_dependencies",
-  "email_mx",
-  "email_disposable",
-  "email_verify",
-  "robots_txt",
-  "redirect_chain",
-  "brand_assets",
-  "seo_audit",
-  "phone_lookup",
-  "username_lookup",
-  "password_check",
-  "phishing_check",
-  "atlas_technique_lookup",
-  "atlas_technique_search",
-  "bulk_atlas_technique_lookup",
-  "atlas_case_study_lookup",
-  "atlas_case_study_search",
-  "d3fend_defense_lookup",
-  "d3fend_defense_search",
-  "d3fend_defense_for_attack",
-  "d3fend_attack_coverage"
-]New value: +[
+  "cve_lookup",
+  "cve_search",
+  "cve_leading",
+  "bulk_cve_lookup",
+  "calculate_risk_score",
+  "get_cvss_details",
+  "exploit_lookup",
+  "kev_detail",
+  "cwe_lookup",
+  "subdomain_enum",
+  "ssl_check",
+  "tech_fingerprint",
+  "asn_lookup",
+  "ip_lookup",
+  "ioc_lookup",
+  "bulk_ioc_lookup",
+  "hash_lookup",
+  "threat_intel",
+  "threat_report",
+  "audit_domain",
+  "domain_report",
+  "dns_lookup",
+  "whois_lookup",
+  "wayback_lookup",
+  "scan_headers",
+  "check_headers",
+  "check_secrets",
+  "check_injection",
+  "check_dependencies",
+  "email_mx",
+  "email_security_posture",
+  "email_disposable",
+  "email_verify",
+  "robots_txt",
+  "redirect_chain",
+  "brand_assets",
+  "seo_audit",
+  "phone_lookup",
+  "username_lookup",
+  "password_check",
+  "phishing_check",
+  "atlas_technique_lookup",
+  "atlas_technique_search",
+  "bulk_atlas_technique_lookup",
+  "atlas_case_study_lookup",
+  "atlas_case_study_search",
+  "d3fend_defense_lookup",
+  "d3fend_defense_search",
+  "d3fend_defense_for_attack",
+  "d3fend_attack_coverage"
+]

Changed1 schema field changedMay 8, 2026

changedOutput schema / $defs / PivotHint / properties / tool / enum

Previous value: -[
-  "cve_lookup",
-  "cve_search",
-  "cve_leading",
-  "bulk_cve_lookup",
-  "exploit_lookup",
-  "kev_detail",
-  "cwe_lookup",
-  "subdomain_enum",
-  "ssl_check",
-  "tech_fingerprint",
-  "asn_lookup",
-  "ip_lookup",
-  "ioc_lookup",
-  "bulk_ioc_lookup",
-  "hash_lookup",
-  "threat_intel",
-  "threat_report",
-  "audit_domain",
-  "domain_report",
-  "dns_lookup",
-  "whois_lookup",
-  "wayback_lookup",
-  "scan_headers",
-  "check_headers",
-  "check_secrets",
-  "check_injection",
-  "check_dependencies",
-  "email_mx",
-  "email_disposable",
-  "email_verify",
-  "robots_txt",
-  "redirect_chain",
-  "brand_assets",
-  "seo_audit",
-  "phone_lookup",
-  "username_lookup",
-  "password_check",
-  "phishing_check",
-  "atlas_technique_lookup",
-  "atlas_technique_search",
-  "bulk_atlas_technique_lookup",
-  "atlas_case_study_lookup",
-  "atlas_case_study_search",
-  "d3fend_defense_lookup",
-  "d3fend_defense_search",
-  "d3fend_defense_for_attack",
-  "d3fend_attack_coverage"
-]New value: +[
+  "cve_lookup",
+  "cve_search",
+  "cve_leading",
+  "bulk_cve_lookup",
+  "calculate_risk_score",
+  "get_cvss_details",
+  "exploit_lookup",
+  "kev_detail",
+  "cwe_lookup",
+  "subdomain_enum",
+  "ssl_check",
+  "tech_fingerprint",
+  "asn_lookup",
+  "ip_lookup",
+  "ioc_lookup",
+  "bulk_ioc_lookup",
+  "hash_lookup",
+  "threat_intel",
+  "threat_report",
+  "audit_domain",
+  "domain_report",
+  "dns_lookup",
+  "whois_lookup",
+  "wayback_lookup",
+  "scan_headers",
+  "check_headers",
+  "check_secrets",
+  "check_injection",
+  "check_dependencies",
+  "email_mx",
+  "email_disposable",
+  "email_verify",
+  "robots_txt",
+  "redirect_chain",
+  "brand_assets",
+  "seo_audit",
+  "phone_lookup",
+  "username_lookup",
+  "password_check",
+  "phishing_check",
+  "atlas_technique_lookup",
+  "atlas_technique_search",
+  "bulk_atlas_technique_lookup",
+  "atlas_case_study_lookup",
+  "atlas_case_study_search",
+  "d3fend_defense_lookup",
+  "d3fend_defense_search",
+  "d3fend_defense_for_attack",
+  "d3fend_attack_coverage"
+]

AddedMay 2, 2026

TDQS

A4.7/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds substantial behavioral detail beyond the readOnly/idempotent annotations: the ContrastAPI user-agent string, honoring Disallow for internal tools, per-eTLD+1 throttling at 60 req/min, free/pro rate limits, and 502 error semantics. This level of transparency helps agents anticipate side effects and constraints.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Although the description is long, it is dense with non-redundant, operationally critical information: usage context, HTTP fallback, 404/502 semantics, UA identification, throttle, and rate limits. Every sentence earns its place and the main purpose is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Even with an output schema present, the description explains return fields, error responses, and failure modes. It covers edge cases (404 vs 502), rate throttling, and relation to sibling tools, making the tool fully self-sufficient for correct invocation and interpretation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already documents the single `domain` parameter with 100% coverage, including 'No scheme, no path, no port' and subdomain handling. The description adds no new parameter-level meaning; it stays at the baseline of relying on the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description starts with 'Fetch + parse the target domain's robots.txt' and enumerates exactly what it returns (sitemaps, allow/disallow rules, crawl-delay, Host). This clearly distinguishes it from siblings like dns_lookup or seo_audit by naming a unique resource and action.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states 'Use BEFORE crawling/scraping a target site (seo_audit, brand_assets, redirect_chain) to honour the site's published rules.' It also provides an exclusion: 502 means upstream failure, not 'no robots', and explains when no robots.txt exists (404 → allow-all). This is textbook usage guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All ContrastAPI tools Inspect

Try in Browser

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Robots.txt

Input Schema

Output Schema

Schema Changelog

TDQS

Glama MCP Gateway

Resources