"Web scraping and crawling tools" matching MCP tools:

• dossier_full

  drwho.me developer tools

  Aggregate dossier check: Run all 10 Domain Dossier checks — dns, mx, spf, dmarc, dkim, tls, redirects, headers, cors, web-surface — in parallel and return all results in a single response. Use when you need a comprehensive domain health snapshot in one call; counts as ONE paywall call regardless of how many checks run. For a single focused check, prefer the individual dossier_* tools to minimise latency. Fires all 10 checks concurrently via Cloudflare DoH or direct HTTPS, 5 s per-check timeout. Returns a JSON object keyed by check id (dns, mx, etc.), each value a CheckResult discriminated union ({status:"ok",...} or {status:"error", reason}).

  Connector

• robots_txt

  ContrastAPI

  Fetch + parse the target domain's robots.txt — sitemaps, per-User-agent allow/disallow rules, crawl-delay, Host directive. Use BEFORE crawling/scraping a target site (seo_audit, brand_assets, redirect_chain) to honour the site's published rules. status_code=404 means no robots.txt exists = implicit allow-all per RFC 9309 §2.4. ContrastAPI fetches with `User-agent: ContrastAPI/<version> (+https://contrastcyber.com/bot)` so site operators can identify + opt out via robots.txt; we honour `Disallow: /` for our UA in seo_audit and brand_assets. Per-target eTLD+1 throttle (60 req/min) prevents weaponising this endpoint against a single site; subdomain rotation collapses to the same bucket. Free: 30/hr, Pro: 500/hr. Returns {domain, fetched_url, status_code, sitemaps, user_agents:{ua:{allow,disallow,crawl_delay}}, host, truncated, summary}. Returns 502 ErrorResponse if the target rejected the connection (DNS/TCP/TLS failure); the agent should NOT assume "no robots" in that case — it's an upstream-failure signal.

  Connector

• meta_capabilities

  Colour Memory

  Return a live inventory of all active endpoints and MCP tools. Use this first to discover what the API can do before making calls. Returns tool count, endpoint list, MCP-exposed tools, and usage notes. Deterministic -- no LLM cost.

  Connector

• get_stratalize_overview

  Stratalize Oracle

  START HERE - Returns the complete Stratalize tool catalog: 216 governed MCP tools across 6 namespaces (crypto, finance, governance, healthcare, realestate, intelligence). 159 tools available via x402 (USDC micropayments on Base): $0.02 atomic · $0.10 benchmark · $0.50 synthesis · $1.00 premium; 159 priced tier tools + 0 free reference tools. 64 additional tools accessible via OAuth-authenticated MCP for organizations. Call this first to discover C-suite briefs (CEO, CFO, CRO, CMO, CTO, CHRO, CX, GC, COO), market benchmarks, governance compliance tools (EU AI Act, FS AI RMF, UK FCA), and org intelligence with role-based recommendations. No auth required.

  Connector

• meta_capabilities

  Colour Memory

  Return a live inventory of all active endpoints and MCP tools. Use this first to discover what the API can do before making calls. Returns tool count, endpoint list, MCP-exposed tools, and usage notes. Deterministic -- no LLM cost.

  Connector

• get_stratalize_overview

  Stratalize Governance

  START HERE - Returns the complete Stratalize tool catalog: 216 governed MCP tools across 6 namespaces (crypto, finance, governance, healthcare, realestate, intelligence). 159 tools available via x402 (USDC micropayments on Base): $0.02 atomic · $0.10 benchmark · $0.50 synthesis · $1.00 premium; 159 priced tier tools + 0 free reference tools. 64 additional tools accessible via OAuth-authenticated MCP for organizations. Call this first to discover C-suite briefs (CEO, CFO, CRO, CMO, CTO, CHRO, CX, GC, COO), market benchmarks, governance compliance tools (EU AI Act, FS AI RMF, UK FCA), and org intelligence with role-based recommendations. No auth required.

  Connector

• get_stratalize_overview

  Stratalize Finance

  START HERE - Returns the complete Stratalize tool catalog: 216 governed MCP tools across 6 namespaces (crypto, finance, governance, healthcare, realestate, intelligence). 159 tools available via x402 (USDC micropayments on Base): $0.02 atomic · $0.10 benchmark · $0.50 synthesis · $1.00 premium; 159 priced tier tools + 0 free reference tools. 64 additional tools accessible via OAuth-authenticated MCP for organizations. Call this first to discover C-suite briefs (CEO, CFO, CRO, CMO, CTO, CHRO, CX, GC, COO), market benchmarks, governance compliance tools (EU AI Act, FS AI RMF, UK FCA), and org intelligence with role-based recommendations. No auth required.

  Connector

• list_capabilities

  Axcess — Design Accessibility Evaluation

  Returns available evaluation tools, what they check, and their pricing. Call this first to understand what Axcess can evaluate and how much each evaluation costs. This tool is FREE. All evaluation tools require USDC payment on Base network. Returns: JSON with tool descriptions, pricing, and rubric categories.

  Connector

• list_tools

  foundation-discovery

  List available MCP tools and get detailed help. Use this tool to discover what tools are available and how to use them. Call without parameters to see all tools, or provide a tool name to get detailed help including parameters, examples, and related tools.

  Connector

• discover_tools

  Transit Land

  Find tools by describing the data or task. Use when you need to browse, search, look up, or discover what tools exist for: SEC filings, financials, revenue, profit, FDA drugs, adverse events, FRED economic data, Census demographics, BLS jobs/unemployment/inflation, ATTOM real estate, ClinicalTrials, USPTO patents, weather, news, crypto, stocks. Returns the top-N most relevant tools with names, descriptions, and full input schemas (with curated examples) — each result is ready to call directly, no second schema lookup needed. Call this FIRST when you have many tools available and want to see the option set (not just one answer).

  Connector

• vest_search_tools

  vest-mcp

  Search and browse AI tools available in Vest's cashback catalog. Returns names, slugs, categories, and live cashback rates. Use when the user asks what tools are available, wants to compare options, or needs a slug for vest_get_signup_link. Real triggers: 'what AI writing tools does Vest have?', 'show me coding tools with high cashback', 'find tools under $50/mo'. Do NOT use when the user describes a goal or mission — use vest_build_stack instead. Do NOT use to get a signup link — use vest_get_signup_link.

  Connector

• cyanheads_search

  cyanheads-mcp-server

  Search fleet tools and servers by natural-language description. Returns ranked matches with brief summaries and the server each tool belongs to. Use scope "servers" to find which server handles a workflow; use the default scope "tools" to find specific tools. Call cyanheads_describe on a result name to get install snippets and the connection URL.

  Connector

• get_item

  Sciencebase

  Get a single ScienceBase catalog item by id — full summary, categories, types, dates, contacts, web links, and attached files (download URLs). e.g. id "58f8be37e4b0b7ea5452260e". Keyless.

  Connector

• list_tools

  zero-core-tools

  List all available Zero Core Tools with pricing and input requirements. Use this for discovery.

  Connector

• get_stratalize_overview

  Stratalize Healthcare

  START HERE - Returns the complete Stratalize tool catalog: 216 governed MCP tools across 6 namespaces (crypto, finance, governance, healthcare, realestate, intelligence). 159 tools available via x402 (USDC micropayments on Base): $0.02 atomic · $0.10 benchmark · $0.50 synthesis · $1.00 premium; 159 priced tier tools + 0 free reference tools. 64 additional tools accessible via OAuth-authenticated MCP for organizations. Call this first to discover C-suite briefs (CEO, CFO, CRO, CMO, CTO, CHRO, CX, GC, COO), market benchmarks, governance compliance tools (EU AI Act, FS AI RMF, UK FCA), and org intelligence with role-based recommendations. No auth required.

  Connector

• mcp_capabilities

  FXMacroData

  Explain what the FXMacroData MCP server can do, which tools render MCP Apps, which tools return plain rows, what is public versus subscriber-only, and how to choose tools across ChatGPT, Claude, Cursor, Codex, and plain MCP clients. Use this when a user asks what is available, why visuals are not showing, or how to get the same result in a different interface.

  Connector

• onyx_dns_lookup

  onyx-mcp

  Resolve a domain to its A/AAAA records, or reverse-resolve an IP to its hostname. Useful for validating a domain exists before scraping, checking if two domains share infrastructure, mapping CDN origins, or doing safety lookups before agents call third-party APIs. Returns IPv4, IPv6, canonical hostname, and resolution time. Powered by stdlib so results are whatever the host's DNS resolver returns — typically 20-100ms. (price: $0.001 USDC, tier: metered)

  Connector

• eurlex_get_relations

  eur-lex-mcp-server

  Traverse the CELLAR CDM relationship graph for an EU work: what it amends, what amends it, its current consolidated version, its legal basis, and works that cite it. This is CELLAR's primary value over HTML scraping — the graph traversal that exposes the lifecycle and dependencies of an EU act. Returns one-hop direct relations only. For deeper traversal, use eurlex_query_sparql. The "consolidated_version" relation links to the current consolidated text (a separate CELEX-numbered work); fetch that work with eurlex_get_document. Requires a valid CELEX number or CELLAR work URI — use eurlex_lookup_celex to resolve identifiers first.

  Connector

• web_url_reader

  Inferventis — Financial Data, News & Web MCP

  Fetches any public web page and returns clean, readable plain text stripped of HTML, navigation, scripts, advertisements, and boilerplate. Returns the page title, meta description, word count, and main body text ready for analysis or summarisation. Use this tool when an agent needs to read the content of a specific web page or article URL — for example to summarise an article, extract facts from a page, verify a claim by reading the source, or convert a web page into plain text to pass to another tool. Pass article URLs returned by web_news_headlines to this tool to read full article content. Do not use this tool to discover current news headlines — use web_news_headlines instead. Does not execute JavaScript — best suited for standard HTML content pages. Will not work with paywalled, login-protected, or JavaScript-rendered single-page applications.

  Connector

• get_stratalize_overview

  Stratalize Real Estate

  START HERE - Returns the complete Stratalize tool catalog: 216 governed MCP tools across 6 namespaces (crypto, finance, governance, healthcare, realestate, intelligence). 159 tools available via x402 (USDC micropayments on Base): $0.02 atomic · $0.10 benchmark · $0.50 synthesis · $1.00 premium; 159 priced tier tools + 0 free reference tools. 64 additional tools accessible via OAuth-authenticated MCP for organizations. Call this first to discover C-suite briefs (CEO, CFO, CRO, CMO, CTO, CHRO, CX, GC, COO), market benchmarks, governance compliance tools (EU AI Act, FS AI RMF, UK FCA), and org intelligence with role-based recommendations. No auth required.

  Connector