Skip to main content
Glama
Sign In

Search for:

Using Crawl4AI to Crawl Websites

View all MCP Servers

  • Why this server?

    This server is explicitly named Crawl4AI and is described as an intelligent information retrieval server offering robust search capabilities and LLM-optimized web content understanding, which is a strong match for the user's query.

    Crawl4AI MCP Server

    weidwonder
    -
    security
    A
    license
    -
    quality
    Crawl4AI MCP Server is an intelligent information retrieval server offering robust search capabilities and LLM-optimized web content understanding, utilizing multi-engine search and intelligent content extraction to efficiently gather and comprehend internet information.
    Last updated -
    36
    Python
    MIT License
    • Apple
    • Linux

  • Why this server?

    This server supports web crawling and ingestion into a Graphlit project, enabling retrieval of relevant contents, aligning with the user's need to crawl websites.

    Graphlit MCP Serverofficial

    graphlit
    A
    security
    A
    license
    A
    quality
    The Model Context Protocol (MCP) Server enables integration between MCP clients and the Graphlit service. Ingest anything from Slack to Gmail to podcast feeds, in addition to web crawling, into a Graphlit project - and then retrieve relevant contents from the MCP client.
    Last updated -
    43
    1,676
    238
    TypeScript
    MIT License
    • Apple

  • Why this server?

    This server uses the Oxylabs Web Scraper API to fetch and process web content, providing flexible options for parsing and rendering pages, which is useful for web crawling.

    Oxylabs MCP Serverofficial

    oxylabs
    A
    security
    A
    license
    A
    quality
    A scraper tool that leverages the Oxylabs Web Scraper API to fetch and process web content with flexible options for parsing and rendering pages, enabling efficient content extraction from complex websites.
    Last updated -
    2
    21
    Python
    MIT License
    • Apple
    • Linux

  • Why this server?

    This server offers Google and individual website searching, along with content scraping, useful for obtaining web data.

    olostep-mcp

    olostep
    -
    security
    A
    license
    -
    quality
    Searching google, individual websites and scraping their content. Fast and cost-effective. ⚡️
    Last updated -
    16
    TypeScript
    MIT License

  • Why this server?

    This server is designed for web search, content extraction, web crawling, and scraping, fitting the user's need to crawl websites.

    WebSearch

    josemartinrodriguezmortaloni
    A
    security
    F
    license
    A
    quality
    Built as a Model Context Protocol (MCP) server that provides advanced web search, content extraction, web crawling, and scraping capabilities using the Firecrawl API.
    Last updated -
    4
    1
    Python
    • Apple
    • Linux

  • Why this server?

    This server fetches web content in various formats, enabling LLMs to process and understand web data, which is a component of crawling.

    MCP URL Fetcher

    nathanonn
    -
    security
    F
    license
    -
    quality
    A Model Context Protocol server that enables LLMs to fetch and process web content in multiple formats (HTML, JSON, Markdown, text) with automatic format detection.
    Last updated -
    TypeScript
    • Apple

  • Why this server?

    This server fetches web page content using Playwright with AI-powered capabilities for efficient information extraction, helpful for web crawling.

    Fetcher MCP

    everford
    A
    security
    A
    license
    A
    quality
    A server that allows fetching web page content using Playwright headless browser with AI-powered capabilities for efficient information extraction.
    Last updated -
    2
    509
    2
    TypeScript
    MIT License

  • Why this server?

    Provides web search functionality and can be part of a larger web crawling solution.

    DuckDuckGo MCP Server

    nickclyde
    A
    security
    A
    license
    A
    quality
    A Model Context Protocol (MCP) server that provides web search capabilities through DuckDuckGo, with additional features for content fetching and parsing.
    Last updated -
    2
    26
    Python
    MIT License
    • Apple

  • Why this server?

    Allows AI assistants to browse and read files from GitHub repositories, providing access to repository contents, which might be relevant for specific crawling tasks.

    MCP GitHub Repository Server

    loglmhq
    -
    security
    A
    license
    -
    quality
    A server that allows AI assistants to browse and read files from specified GitHub repositories, providing access to repository contents via the Model Context Protocol.
    Last updated -
    3
    JavaScript
    MIT License
    • Apple

  • Why this server?

    Enables web searches and provides real-time web information, which is crucial for crawling websites.

    Exa MCP Server

    geezerrrr
    -
    security
    A
    license
    -
    quality
    A server that enables AI assistants like Claude to perform web searches using the Exa AI Search API, providing real-time web information in a safe and controlled way.
    Last updated -
    1,832
    MIT License
    • Apple