"Automatic detection of HTTP method (GET vs POST) for web crawling" matching MCP servers:

MCP server that gives AI agents access to current web information through web_search and fetch_url tools, using Tavily, Brave, and DuckDuckGo providers with automatic fallback.

MCP server for automating article posting to note.com from Markdown files using Playwright.

Enables automated publishing and draft saving to note.com platform using Playwright automation. Reads Markdown files with titles, body text, and tags to publish content directly to your note.com account.

Enables fetching HTTP resources through the MCP protocol.

Enables MCP clients to perform HTTP GET requests to arbitrary URLs and receive response details.

Enables agents to read one public X/Twitter post from a status URL without cookies, returning post text, quoted content, links, and media URLs as untrusted data.

An MCP server for safe web crawling and scraping of public and authenticated sites using Playwright and Crawlee. It provides tools for scraping single pages, crawling entire sites, validating sessions, and listing supported sites, with built-in security protections and adapter-based extensibility.

MCP server providing free, IP-direct tools for web search, crawling, anti-detect browsing, and structured data from Wikipedia, Wikidata, GitHub, and YouTube.

Enables a locally-run LLM to search the web, fetch pages as markdown, make arbitrary HTTP requests, and optionally render pages with headless Chromium.

MCP server for multi-engine web search and web page fetching, supporting parallel search, content extraction, and optional LLM-powered search summarization and deep search.

Enables an MCP client to read Discord servers, channels, threads, and messages, and to send messages, replies, reactions, and GIFs through an already-open Discord Web tab via a local Chrome extension and 127.0.0.1 bridge, without using a bot or the Discord HTTP API.

Provides eight tools for web search, content extraction, screenshots, PDFs, JavaScript execution, crawling, and Wayback Machine snapshots and archiving.

Enables compatible MCP clients to read JavaScript-rendered web pages as bounded Markdown or structured records, with URL policy and resource limits for safe, inspectable access.

Enables AI agents to search DuckDuckGo and fetch web content with human-assisted CAPTCHA handling, designed for local use where humans can intervene when bot detection occurs.

A self-hosted MCP server that reads web pages and converts them to Markdown, with built-in web search and anti-crawl fallback using Playwright.

Exposes web search, site mapping, scraping, crawling, schema-validated structured extraction, and stateful Playwright browser interaction as MCP tools, routing every call through the same authenticated REST gateway used by the platform's HTTP clients.

Converts web pages to Markdown for MCP clients like Claude, with support for single and batch reads, caching, fallback rendering, and SSRF protection.

Enables fetching, crawling, and browsing anti-bot protected web pages with real browser rendering, returning clean JSON or markdown, and solving captchas.

Enables cross-browser mobile-first responsive layout auditing and browser automation, including overflow detection, tap target checks, viewport/media query introspection, and screenshot matrices via MCP.

Enables AI-powered web searches using DuckDuckGo with automatic summarization via Google's Gemini AI. Supports both summary and detailed analysis modes for search results.