Skip to main content
Glama
Sign In

Search for:

Understanding Web Crawlers

View all MCP Servers

  • Why this server?

    Provides web search capabilities through DuckDuckGo, which can be used to crawl and gather information from the web. It includes content fetching and parsing.

    DuckDuckGo MCP Server

    nickclyde
    A
    security
    A
    license
    A
    quality
    A Model Context Protocol (MCP) server that provides web search capabilities through DuckDuckGo, with additional features for content fetching and parsing.
    Last updated -
    2
    26
    Python
    MIT License
    • Apple

  • Why this server?

    Offers DuckDuckGo search functionality specifically tailored for Claude, providing web search with rate limiting.

    DuckDuckGo MCP Server

    zhsama
    A
    security
    A
    license
    A
    quality
    A Model Context Protocol server that provides DuckDuckGo search functionality for Claude, enabling web search capabilities through a clean tool interface with rate limiting support.
    Last updated -
    1
    60
    15
    TypeScript
    MIT License
    • Apple

  • Why this server?

    Enables integration with DuckDuckGo search capabilities for LLMs, supporting web search, regional filtering, and result types.

    MCP DuckDuckGo Search Server

    spences10
    -
    security
    A
    license
    -
    quality
    Enables integration with DuckDuckGo search capabilities for LLMs, supporting comprehensive web search, regional filtering, result types, and safe browsing with caching and customizable search parameters.
    Last updated -
    26
    2
    TypeScript
    MIT License

  • Why this server?

    Enables language models to perform web searches through SearXNG, a metasearch engine that aggregates results from various search engines.

  • Why this server?

    A headless browser MCP server that allows AI agents to fetch web content and perform Google searches without API keys, supporting various output formats which is essential for analyzing crawled data.

    Fetch Browser

    TheSethRose
    A
    security
    A
    license
    A
    quality
    A headless browser MCP server that allows AI agents to fetch web content and perform Google searches without API keys, supporting various output formats like Markdown, JSON, HTML, and text.
    Last updated -
    2
    4
    TypeScript
    MIT License

  • Why this server?

    Provides a tool to download entire websites using wget, preserving the website structure and converting links to work locally, useful for offline analysis of crawled websites.

    Website Downloader

    pskill9
    A
    security
    F
    license
    A
    quality
    Provides a tool to download entire websites using wget. It preserves the website structure and converts links to work locally.
    Last updated -
    1
    40
    JavaScript
    • Apple
    • Linux

  • Why this server?

    Enables users to download entire websites and their assets for offline access, supporting configurable depth and concurrency settings.

    MCP Windows Website Downloader Server

    angrysky56
    A
    security
    A
    license
    A
    quality
    This server enables users to download entire websites and their assets for offline access, supporting configurable depth and concurrency settings.
    Last updated -
    1
    4
    Python
    MIT License

  • Why this server?

    Provides human-like browser automation with Playwright, allowing control over browser actions such as navigation, element interaction, and scrolling - useful for web crawling.

    Playwright MCP Server

    showfive
    -
    security
    F
    license
    -
    quality
    Provides a server utilizing Model Context Protocol to enable human-like browser automation with Playwright, allowing control over browser actions such as navigation, element interaction, and scrolling.
    Last updated -
    2
    TypeScript

  • Why this server?

    Enables web content scanning and analysis by fetching, analyzing, and extracting information from web pages, including link extraction and site crawling.

    MCP Webscan Server

    bsmi021
    A
    security
    A
    license
    A
    quality
    Enables web content scanning and analysis by fetching, analyzing, and extracting information from web pages using tools like page fetching, link extraction, site crawling, and more.
    Last updated -
    6
    7
    TypeScript
    MIT License

  • Why this server?

    An MCP server implementation that integrates the SearxNG API, providing web search capabilities through a metasearch engine.

    SearXNG Server

    ihor-sokoliuk
    A
    security
    A
    license
    A
    quality
    An MCP server implementation that integrates the SearxNG API, providing web search capabilities.
    Last updated -
    2
    633
    63
    JavaScript
    MIT License
    • Linux
    • Apple