"Information about crawer or crawler" matching MCP servers:

Enables autonomous web browsing and extraction by canonicalizing URLs, detecting anti-crawler traps such as circular paths and infinite pagination, pruning DOM content, mapping forms, converting tables to JSON, and evaluating agent trajectories.

Crawl entire websites, sitemaps, and documentation portals into clean, LLM-ready Markdown without ads, cookie popups, or noise for RAG pipelines and AI agents.

Enables users to audit personal reading streams by quantifying category diversity with Shannon entropy, detecting lexical redundancy and echo chambers via Jaccard similarity, and ranking content sources by decibel signal-to-noise ratio for pruning recommendations.

Enables agents to read any web page or document as markdown, html, links, screenshots or extracted JSON, map and crawl entire sites, and drive a persistent browser session that stays open between calls so pages can be inspected and clicked without reloading. Callers pay per request with a wallet over x402 — no API key or signup — and every route answers its own free GET with the fields it expects.

Provides access to Canadian federal parliamentary data (debates, bills, MPs, votes, Hansard transcripts) and legal information (case law and legislation through CanLII) for research and analysis.

Enables detection of AI-generated content in images, videos, audio, and text via the AI or Not API. Supports media analysis tools for deepfakes, synthetic voices, and AI-written text.

Free, open-source desktop SEO crawler with a built-in MCP server. 250 technical SEO, GEO and AI-search checks across 23 areas, run locally; MCP tools list crawls, inspect issues, run Lighthouse, check AI-crawler access, and apply confirmation-gated WordPress fixes with rollback.

MCP server: is this GitHub repo still maintained? A verdict.

Audit a site for AI readiness: AI-crawler access, schema, FAQ and content structure. Generates the fixes.

A read-only MCP server that gives LLM agents deterministic operational facts about Trello boards, including structural history, card movement, workflow flow, staleness, and due-date information.

Provides access to the SonoVault music metadata API, enabling search and retrieval of information about 90M+ tracks, artists, labels, and releases, including ISRC, ISWC, and cross-platform IDs.

Read-only access to your Citlyze AI search visibility workspace: visibility scores, tracked prompts, citations, competitor comparison, recommendations, and AI crawler analytics.

Provides access to Shodan API functionality, enabling AI assistants to query information about internet-connected devices for cybersecurity research and threat intelligence.

Agent-native Generative Engine Optimization (GEO) audits, crawler analysis (GPTBot, Perplexity, Claude), and Schema.org direct-answer remediation.

Provides tools to access information about religious institutions and community services in Kenya, including Catholic dioceses, Protestant denominations, Muslim communities, and welfare programs.

Open-source technical SEO crawler MCP server built on LibreCrawl. Runs full audits inside Claude, Cursor, or Codex — 50+ checks (hreflang, schema.org, security headers, WAF detection on 200-OK pages), chunked-progressive engine for large sites, ephemeral by design (server forgets every audit after download).

A Model Context Protocol server that enables AI assistants to search and retrieve information about security exploits and vulnerabilities from the Exploit Database, enhancing cybersecurity research capabilities.

Enables AI visibility audits by checking AI crawler access, identity files, and page signals locally, with hosted tools for identity scoring, fix plans, competitor comparison, history, and answer-engine analysis.

Enables users to check any public website's AI crawler access, GEO readiness, and latest public brand mention and citation reports in ChatGPT, Gemini, and Perplexity.

Enables searching and retrieving detailed information about Swedish companies, including financial data and annual reports from Bolagsverket (Swedish Companies Registration Office), with intelligent caching for fast responses.

This MCP server provides detailed GitHub repository information by integrating with mcp.deepwiki.com and GitHub API to fetch repository wiki content, README files, and enable intelligent question-answering about repositories.

Retrieves current weather information for any location using the OpenWeatherMap API with configurable temperature units (Celsius or Fahrenheit).

Run free live audience Q&A and polls from your agent: start an AskOne room, add and launch a poll, approve or hide questions and mark them answered, close the room, then read the questions as a FAQ draft and get the results. Free AI moderation. Uses an organization API token from askone.org

fixRAgent triages one photo of a building problem: what it is, how urgent (EMERGENCY, TODAY, THIS WEEK or WHENEVER), which trade to call, and what to tell the resident. Remote server at https://fixragent.com/mcp with three tools: try_sample, assess_property_photo, get_triage_profile. No key needed to start.

A locally runnable stdio MCP server for Fortrip hotel search, live room rates, hotel details, trip planning and booking payment links. Uses a Fortrip CLI token or saved CLI login; payments are completed by the user on Fortrip and Stripe.

Let Claude, Cursor or Cline fetch pages and run Google searches from a real Singapore mobile IP (Singtel/M1). Hosted; free 24-hour trial key.

Aziel Runtime is a node-meshed MCP software orchestration suite for digital forensics, auditing, provenance, research, intelligence-support, and secure agent workflows, with a single gated execution path, compartmentalized engines, verifiable receipts, temporal integrity, and explicit capability refusal. It is not merely an API aggregator or generic tool router.

Enables agents to discover, search, compare, and inspect tested MCP tools by problem or keyword, with certification verdicts, scorecard summaries, certified scope, and explicit not-tested details.

Enables AI clients to transcribe YouTube links and uploaded audio or video files, then retrieve the resulting transcripts as text, SRT, VTT, or JSON with timestamps and optional speaker labels. Also supports paging through long transcripts, checking upload progress, listing or deleting jobs, and viewing remaining account minutes.

CNN's Fear & Greed index for the US stock market: current score and rating, the seven component indicators, and about a year of daily history. Go standard library only, no API key required.