Web Scraper MCP
Optional web search backend for the search tool, configured via [BRAVE](/mcp/servers/integrations/brave)_API_KEY to use Brave Search.
Default web search backend for the search tool, providing privacy-focused search results without requiring additional configuration.
Alternative local LLM backend for the extract and deep_research tools, allowing use of self-hosted models (e.g., Qwen) via SCRAPER_[OLLAMA](/mcp/servers/integrations/ollama)_HOST and model environment variables.
Optional self-hosted web search backend for the search tool, configurable via SCRAPER_[SEARXNG](/mcp/servers/integrations/searxng)_URL to use a SearXNG instance.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Web Scraper MCP Scrape https://example.com and summarize it"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Web Scraper MCP
A self-hosted Model Context Protocol server for reading public web pages, bounded crawling, search, model-backed extraction and browser interaction. Supports Python 3.11–3.13 and stdio or authenticated HTTP.
Run it with your own models and infrastructure. Fetching is static by default; browser rendering is optional and uses sandboxed Chromium with guarded egress.
Tools
Tool | Behavior |
| Fetch a public URL and return its title, Markdown and optional capped links or HTML. |
| Start a background crawl with page/depth limits and an optional same-host policy. |
| Read progress/results or cancel a job while retaining completed pages. |
| Return canonical, deduplicated links; defaults to the source host. |
| Create or reuse a browser session and return an ARIA snapshot. |
| Click, fill, press, select or wait using a selector. Explicit actions may submit same-origin forms. |
| Close a session and release its page capacity. |
| Use Tavily, Brave, SearXNG or DuckDuckGo, in that priority order. |
| Ask Ollama or Anthropic for text or data matching a locally validated JSON Schema. |
| Search, read bounded sources and synthesize a report with checked citation indices and source failures. |
Related MCP server: Fetch MCP Server
Quick start
Install Python 3.11–3.13 and uv, then:
git clone https://github.com/Prog-up/web-scraper-mcp.git
cd web-scraper-mcp
uv sync --frozen
cp .env.example .envFor a local MCP client using stdio:
SCRAPER_TRANSPORT=stdio uv run --frozen web-scraper-mcpStdio does not require a token. Clients with an mcpServers configuration can use:
{
"mcpServers": {
"web-scraper": {
"command": "uv",
"args": ["--directory", "/absolute/path/to/web-scraper-mcp", "run", "--frozen", "web-scraper-mcp"],
"env": {"SCRAPER_TRANSPORT": "stdio"}
}
}
}For HTTP:
export SCRAPER_AUTH_TOKEN=$(openssl rand -hex 32)
./start.shConnect to http://127.0.0.1:8000/mcp with
Authorization: Bearer YOUR_TOKEN. Keep the token in your client's secret
configuration. HTTP startup rejects missing or blank tokens. The launcher stays
in the foreground; use a service manager for supervision.
Scraping and browser tools do not require an LLM. For extract and
deep_research, configure an installed Ollama model and reachable endpoint in
.env, or select Anthropic and supply its API key. Search defaults to DuckDuckGo;
other providers are optional. See configuration and
the environment template.
Docker and browser tools
For sandboxed rendering and browser sessions:
export SCRAPER_AUTH_TOKEN=$(openssl rand -hex 32)
docker compose up --buildCompose runs the app on an internal network with a separate guarded egress proxy, a read-only filesystem, resource limits and Chromium's sandbox enabled. Use Docker Engine 28 or newer and a host that supports unprivileged user namespaces. Sandbox startup failures stop rendering; there is no automatic fallback that disables the sandbox.
Compose does not start Ollama. Its model endpoint must be reachable from the isolated app network; an external endpoint needs a trusted route or relay. See deployment for static-only containers, networking, model connectivity and browser restrictions, and seccomp provenance.
Scope and limitations
Intended for public HTTP(S) pages. Private destinations are denied by default. Error status pages and unsupported content types are rejected.
Protected websites, CAPTCHAs and some JavaScript applications may fail. There is no managed proxy pool or guaranteed challenge bypass.
Browser sessions block downloads, WebSockets, service workers and background writes. Explicit actions allow writes only to the current page origin.
Crawl jobs and browser sessions live in memory. They expire and are lost on restart; this is not a durable distributed job system.
One HTTP bearer token grants all tools. There are no per-user quotas or tenant isolation; use TLS and trusted access boundaries beyond localhost.
JSON Schema validation checks output structure. Citation validation checks source indices, not whether a source supports every claim. Review important extracted facts against the original sources.
Self-hosting still requires maintenance and compute. Configured search/model providers receive their requests; provider fees and policies apply.
Development and support
See CONTRIBUTING.md for setup and checks, benchmarks for reproducible measurements, and SECURITY.md for vulnerability reporting and security boundaries.
CI checks Python 3.11–3.13, lint, formatting, typing, tests, secrets and locked dependencies. Container releases also test the exact sandboxed browser artifact, scan vulnerabilities, and sign the published digest with an SPDX SBOM. See release procedures, release history and the known security exceptions for 1.1.0 and 1.1.1.
Report bugs through GitHub issues. Support is community-based, with no service-level agreement.
License
See LICENSE for the existing GNU GPL version 3 terms.
Available Tools
4 toolsdeep_researchA
Search the web, read the top sources, and return a cited synthesis report.
Inputs: query (required), max_sources. Returns: object
extractB
Extract structured data (JSON matching json_schema) or a text answer from a page.
Inputs: url (required), prompt, render, json_schema. Returns: object
scrapeA
Scrape a single URL into clean markdown (boilerplate/ads stripped).
Tries a fast static fetch first and falls back to a stealth headless browser automatically when the page looks JS-gated.
Inputs: url (required), render, include_links, include_raw_html. Returns: object
searchA
Web search. Returns ranked results (title, url, content snippet).
Inputs: query (required), max_results. Returns: object
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 4 tools
scrape and extract both operate on a page but are clearly distinguished by output type: markdown document vs structured JSON/answer. search and deep_research are also distinct, though deep_research could be seen as a superset of search.
All tools use lowercase snake_case imperative verbs: scrape, extract, search, deep_research. The naming pattern is consistent and predictable.
Four tools is well-scoped for a web scraper/research server: one for raw page content, one for structured extraction, one for search, and one for synthesis. Each tool has a clear purpose with no redundant bloat.
The core web research workflow is covered: search, fetch, extract, and synthesize. A minor gap is the lack of batch crawling or link-following features, but most common agent tasks can be completed with these tools.
Maintenance
Related MCP Connectors
Scrape, map, crawl, search and monitor the web. Runs locally with no API key, cites every source.
Web scraping, search, crawling, structured extraction and research for AI agents.
Web search, scraping, crawling, research, a browser agent, and company and people data.
Enable language models to perform advanced AI-powered web scraping with enterprise-grade reliabili…
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables web scraping and crawling capabilities for LLM clients, supporting single-page scraping, multi-page website crawling, and web search with multiple engines (Playwright, Cheerio, Puppeteer) and flexible output formats including markdown, HTML, text, and screenshots.7MIT
- AlicenseNot gradedqualityCmaintenanceEnables LLMs to fetch and extract web content using browser automation, OCR, and multiple extraction methods, handling JavaScript rendering and anti-scraping techniques.17MIT
- AlicenseBqualityCmaintenanceEnables privacy-first web scraping and structured data extraction using a local headless browser and your own LLM key. Supports tools for scraping, batch scraping, data extraction with prompts or schemas, and screenshots.521 npm3MIT
- FlicenseNot gradedqualityAmaintenanceEnables language models to perform AI-powered web scraping, structured extraction, search, crawling, schema generation, and scheduled monitoring through an integrated web-scraping API.-