Skip to main content
Glama

ContrastAPI

by com.contrastcyber

Robots.txt

robots_txt
Read-onlyIdempotent

Fetch + parse the target domain's robots.txt — sitemaps, per-User-agent allow/disallow rules, crawl-delay, Host directive. Use BEFORE crawling/scraping a target site (seo_audit, brand_assets, redirect_chain) to honour the site's published rules. status_code=404 means no robots.txt exists = implicit allow-all per RFC 9309 §2.4. ContrastAPI fetches with User-agent: ContrastAPI/<version> (+https://contrastcyber.com/bot) so site operators can identify + opt out via robots.txt; we honour Disallow: / for our UA in seo_audit and brand_assets. Per-target eTLD+1 throttle (60 req/min) prevents weaponising this endpoint against a single site; subdomain rotation collapses to the same bucket. Free: 30/hr, Pro: 500/hr. Returns {domain, fetched_url, status_code, sitemaps, user_agents:{ua:{allow,disallow,crawl_delay}}, host, truncated, summary}. Returns 502 ErrorResponse if the target rejected the connection (DNS/TCP/TLS failure); the agent should NOT assume "no robots" in that case — it's an upstream-failure signal.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
domainYesRegistrable domain to fetch robots.txt for (e.g. 'example.com', 'github.com'). No scheme, no path, no port. Subdomains accepted; the bot fetches https://<domain>/robots.txt with HTTP fallback.

Output Schema

TableJSON Schema
NameRequiredDescriptionDefault
resultYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds substantial behavioral detail beyond the readOnly/idempotent annotations: the ContrastAPI user-agent string, honoring Disallow for internal tools, per-eTLD+1 throttling at 60 req/min, free/pro rate limits, and 502 error semantics. This level of transparency helps agents anticipate side effects and constraints.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Although the description is long, it is dense with non-redundant, operationally critical information: usage context, HTTP fallback, 404/502 semantics, UA identification, throttle, and rate limits. Every sentence earns its place and the main purpose is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Even with an output schema present, the description explains return fields, error responses, and failure modes. It covers edge cases (404 vs 502), rate throttling, and relation to sibling tools, making the tool fully self-sufficient for correct invocation and interpretation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already documents the single `domain` parameter with 100% coverage, including 'No scheme, no path, no port' and subdomain handling. The description adds no new parameter-level meaning; it stays at the baseline of relying on the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description starts with 'Fetch + parse the target domain's robots.txt' and enumerates exactly what it returns (sitemaps, allow/disallow rules, crawl-delay, Host). This clearly distinguishes it from siblings like dns_lookup or seo_audit by naming a unique resource and action.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states 'Use BEFORE crawling/scraping a target site (seo_audit, brand_assets, redirect_chain) to honour the site's published rules.' It also provides an exclusion: 502 means upstream failure, not 'no robots', and explains when no robots.txt exists (404 → allow-all). This is textbook usage guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All ContrastAPI toolsInspect
Try in Browser
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

TDQS

A4.5/5.0
Disambiguation4/5

Most tools have clearly distinct purposes, with differences between lookup/search/scan/audit for each domain. However, some overlap exists (e.g., email_mx vs email_security_posture, scan_headers vs contrast_scan) which could cause occasional confusion. Overall, boundaries are well-defined.

Naming Consistency5/5

Tool names follow a consistent verb_noun pattern (e.g., cve_lookup, check_headers, bulk_cve_lookup) with all lowercase underscores. Variations like kev_detail or ssl_check are minor and still predictable. No chaotic mixing of conventions.

Tool Count4/5

54 tools is high but justified by the broad cybersecurity scope (CVE, ATLAS, D3FEND, Sigma, domain, email, IOC, scanning). Some redundancy exists (e.g., three email-related tools), but the count is not excessive given the API's comprehensive feature set.

Completeness5/5

The tool set thoroughly covers the threat intelligence and domain investigation lifecycle: CVE/KEV/exploit/CWE, ATLAS/D3FEND/Sigma, DNS/WHOIS/SSL/subdomains, email security, IOC enrichment, and active scanning. No significant gaps are apparent for the stated cybersecurity purpose.

Resources