Skip to main content
Glama
125,016 servers. Updated
Browse by product
20 Best Web Scraping MCP Servers: compared and ranked, November 2026Ranked from 2,147 matching servers on stars, growth, downloads and maintenance. Updated .

Matching MCP tools:

Matching MCP Connectors:

"Information about crawer or crawler" matching MCP servers:

GET /v1/servers – MCP directory API reference
  • A
    license
    A
    quality
    D
    maintenance
    Web-crawler MCP server that fetches web pages with static parsing or headless browser fallback, enabling Claude to fetch, extract links, crawl sites, and select content via CSS selectors.
    4
    ISC
  • A
    license
    Not graded
    quality
    B
    maintenance
    Fetches web pages politely (it obeys robots.txt and sends one request at a time to each site), keeps them in a local store, and lets the client search them and verify quotes. Nine tools for Claude and other MCP clients.
    MIT
  • A
    license
    A
    quality
    D
    maintenance
    Enables site-wide crawling of internal links with BFS traversal, time/title/URL filtering, and page operations like summarization and link clicking, with results persisted to SQLite.
    6
    MIT
  • A
    license
    Not graded
    quality
    D
    maintenance
    Enables AI to autonomously search, filter, and download images from the web using natural language commands, with intelligent scoring and local storage.
    1
    MIT
  • A
    license
    Not graded
    quality
    D
    maintenance
    A specialized web crawler and scraper for the US Patent and Trademark Office (USPTO) website, built with Crawl4AI and MCP integration, enabling patent and trademark search, extraction, and status checking.
    4
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables web crawling and content extraction from web pages, supporting multiple output formats like text, markdown, XML, and JSON, with robots.txt compliance and rate limiting.
    14 npm
    1
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables AI assistants such as Claude, ChatGPT, Cursor, and Claude Code to run 25+ specialized SEO checks on any URL, generate XML sitemaps and robots.txt files, extract brand design elements, and audit technical SEO, content quality, mobile responsiveness, and security.
    MIT
  • F
    license
    Not graded
    quality
    C
    maintenance
    Provides lightweight web crawling and scraping capabilities, enabling users to fetch pages as clean Markdown, recursively crawl domains, extract metadata, and perform search-and-crawl operations.
    -
  • F
    license
    Not graded
    quality
    D
    maintenance
    An intelligent web crawling server that uses Cloudflare's headless browser to render dynamic pages and Workers AI to extract relevant links based on natural language queries. It enables AI assistants to search and filter website content while providing secure access through GitHub OAuth authentication.
    3
    -
  • A
    license
    A
    quality
    B
    maintenance
    Enables crawling sites from start URLs and retrieving internal and external links, file download links, HTML tables as rows, and lists, with controls for crawl depth, page limits, sitemaps, robots.txt, and proxy mode.
    1
    110 npm
    MIT
  • A
    license
    Not graded
    quality
    B
    maintenance
    Free, open-source desktop SEO crawler with a built-in MCP server. 250 technical SEO, GEO and AI-search checks across 23 areas, run locally; MCP tools list crawls, inspect issues, run Lighthouse, check AI-crawler access, and apply confirmation-gated WordPress fixes with rollback.
    22 PyPI
    24
    MIT
  • F
    license
    Not graded
    quality
    C
    maintenance
    Enables agents to read any web page or document as markdown, html, links, screenshots or extracted JSON, map and crawl entire sites, and drive a persistent browser session that stays open between calls so pages can be inspected and clicked without reloading. Callers pay per request with a wallet over x402 — no API key or signup — and every route answers its own free GET with the fields it expects.
    1
    -
  • F
    license
    A
    quality
    D
    maintenance
    A single-process MCP server that crawls web pages, handles anti-bot challenges, converts HTML to Markdown, and caches results in PostgreSQL, providing tools to fetch and read web content.
    6
    -
  • A
    license
    A
    quality
    B
    maintenance
    Lets an MCP client read allowlisted pages from the LiveSwapFace website, retrieve its official links, and estimate credits for a planned browser session using rates fetched at call time. It is read-only and never performs face swaps, signs in, uploads media, or touches a private API.
    3
    MIT
  • F
    license
    Not graded
    quality
    D
    maintenance
    MCP server for crawling and extracting data from web pages using Selenium with CSS/XPath selectors. Supports 17 tools including smart extraction, pagination, infinite scroll, and screenshots.
    3
    -
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables Claude and other MCP clients to read, extract, crawl, and export data from open browser tabs and specified URLs locally, including logged-in sessions, structured page data, contacts, images, sitemaps, and Shopify catalogues.
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Lets an AI assistant check any URL to see how much of a page's main content is present in the raw HTML versus only after JavaScript runs, showing which text blocks, meta fields, canonical tags, JSON-LD and links AI crawlers like GPTBot and ClaudeBot cannot read. It also reports what robots.txt allows for 22 AI bot tokens and how the site responds to each bot's user-agent.
    AGPL 3.0
  • A
    license
    Not graded
    quality
    B
    maintenance
    Enables users to check which page content AI crawlers like GPTBot and ClaudeBot cannot read because it only appears after JavaScript runs, by comparing raw HTML with rendered HTML in headless Chrome.
    AGPL 3.0
  • A
    license
    A
    quality
    B
    maintenance
    Companion to Prerender Buddy's AI visibility and website improvement platform. Three public tools inspect crawler-readable HTTP responses and discovery files without a key. Local installations can use scoped PB keys for workspace evidence and, with content-write permission, article drafts for review.
    3
    525 npm
    1
    Apache 2.0
  • A
    license
    A
    quality
    B
    maintenance
    Enables AI assistants to audit a website's AI visibility by checking crawler access, identity files, and page signals, and running a full scored audit across ChatGPT, Claude, Gemini, and Perplexity.
    10
    106 npm
    MIT
  • A
    license
    A
    quality
    B
    maintenance
    Autonomous LLM-powered web crawler with 17 browser tools and goal-driven agent. Single Rust binary with stealth browsing, provider-agnostic LLM support, and built-in MCP server.
    40
    18
    MIT
  • A
    license
    A
    quality
    C
    maintenance
    Enables AI agents to audit and improve a site's visibility in AI search by checking robots.txt against 17 major AI crawlers, detecting llms.txt, and returning a 0-100 visibility score with per-crawler verdicts; it also lists known AI crawlers and their purposes.
    2
    MIT
  • A
    license
    B
    quality
    D
    maintenance
    Enables web searching and webpage scraping using pure crawler technology without requiring official APIs. Supports Bing web and news search, batch webpage scraping, and content extraction through Puppeteer automation.
    4
    1
    MIT
  • A
    license
    B
    quality
    D
    maintenance
    An MCP server that allows users to check if a website is experiencing downtime by querying isitdownrightnow.com, providing status information and details about recent downtime events.
    1
    2
    MIT
  • A
    license
    B
    quality
    D
    maintenance
    Enables web search functionality through Perplexity AI's API with time-based filtering options. Allows users to search the web for recent information within specified time periods (day, week, month, or year).
    1
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables AI agents to run a free AI-readiness audit of any URL, checking AI crawler rules, JavaScript-free page text, JSON-LD, llms.txt, sitemap, meta description, FAQ schema, and returning a score with findings and fixes. Also exposes the same capability via an A2A agent endpoint.
    MIT
  • A
    license
    Not graded
    quality
    B
    maintenance
    Enables checking whether a page or domain can be found and cited by AI assistants and search engines, covering page-level fixes, AI-crawler access via robots.txt, llms.txt validity, and backlink strength. It is free, read-only, and requires no account.
    MIT
  • A
    license
    Not graded
    quality
    B
    maintenance
    Enables autonomous web agents to prune DOM trees, canonicalize URLs against anti-crawler traps, auto-map form input schemas, and convert HTML/Markdown tables into typed JSON records. It also scores agent execution trajectories against gold benchmarks using action-level precision, recall, and Levenshtein sequence edit distance.
    7
    MIT
  • A
    license
    Not graded
    quality
    A
    maintenance
    Drive Clearcote, a Chromium build with its fingerprint protection patched into the engine, from any MCP client: about 20 tools to navigate, read, extract, click, fill, screenshot and keep sessions, plus a CDP endpoint that hands the same browser to Playwright or Puppeteer. The server lives in the repo's mcp/ folder (npx clearcote-mcp or pip install clearcote-mcp).
    273
    BSD 3-Clause
  • A
    license
    Not graded
    quality
    D
    maintenance
    A web fetching MCP server with HTTP proxy support, allowing AI to retrieve web content in markdown, text, or HTML format while handling anti-crawler detection and redirects.
    2,530 npm
    1
    MIT
  • A
    license
    Not graded
    quality
    D
    maintenance
    Crawl4AI MCP Server is an intelligent information retrieval server offering robust search capabilities and LLM-optimized web content understanding, utilizing multi-engine search and intelligent content extraction to efficiently gather and comprehend internet information.
    155
    MIT
  • A
    license
    Not graded
    quality
    D
    maintenance
    Enables AI agents to perform instant SEO audits, check robots.txt, sitemaps, and AI crawler access for any URL without API keys.
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables a coding agent to audit any public page the way search engines and AI crawlers see it, including per-crawler robots.txt rules, CDN/firewall blocks, and no-JavaScript content. It returns a prioritized fix pack showing where each fix belongs in the codebase plus a command to verify it, so the agent can apply the changes, flag human-only settings, and re-run the audit to confirm.
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables AI assistants to list known AI crawlers such as GPTBot and ClaudeBot, generate and validate robots.txt rules, and check a live site's existing crawler directives. It runs as a remote Streamable HTTP MCP server exposing tools for crawler listing, rule generation, validation, and site checking.
    MIT
  • A
    license
    Not graded
    quality
    B
    maintenance
    Enables SEO auditing of websites from MCP-capable IDEs, providing crawler and rendered views, scoring, and ROI-ranked fix plans that agents can apply directly in the repository.
    MIT
  • A
    license
    Not graded
    quality
    B
    maintenance
    Enables checking a public site's AI-search visibility by auditing crawler access, homepage readability, and llms.txt presence, while also generating and validating llms.txt files and linting robots.txt for AI-crawler issues.
    MIT
  • A
    license
    Not graded
    quality
    B
    maintenance
    Enables AI agents to perform local-first web search, fetch, crawl, extract, cache, research, and autonomous information gathering through MCP, with no API keys or cloud dependencies.
    823 npm
    AGPL 3.0
  • F
    license
    Not graded
    quality
    B
    maintenance
    Enables AI agents to automate a user-owned local Chrome browser or personal Linux host by opening pages, reading controls, clicking, filling forms, extracting information, taking screenshots, and saving encrypted profiles, while pausing for brief human handoff during logins, CAPTCHAs, 2FA, or OAuth approval.
    1
    -
  • A
    license
    A
    quality
    B
    maintenance
    Enables finding information online, reading web pages, finding images, capturing screenshots, and performing multi-source research with per-answer confidence.
    5
    AGPL 3.0
  • A
    license
    A
    quality
    D
    maintenance
    Fetches a URL and returns the main content as clean Markdown, using plain HTTP when possible and headless Chromium for JavaScript-rendered or bot-protected pages.
    1
    21 PyPI
    MIT