Website Scraper MCP Server
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Website Scraper MCP Server scrape and index https://example.com into AI Search"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Website Scraper MCP Server
A production-ready MCP (Model Context Protocol) server that allows any MCP-compatible AI agent to scrape websites, crawl internal pages, clean content, chunk it, and index everything into Azure AI Search — all through a clean, typed tool interface.
Table of Contents
Related MCP server: Crawl4AI RAG MCP Server
Architecture
website_scraper_mcp/
├── app.py ← Entry point (stdio / SSE transport)
├── server.py ← MCP server + tool dispatcher
├── config.py ← Pydantic Settings (env vars)
├── models.py ← Input/Output Pydantic models
└── tools/
├── scrape.py ← Tool 1 – static/dynamic detection + scraping
├── crawl.py ← Tool 2 – BFS crawler, robots.txt aware
├── clean.py ← Tool 3 – Trafilatura + BS4 content cleaning
├── chunk.py ← Tool 4 – sliding window chunking
└── azure_ai_search.py ← Tools 5 & 7 – index + searchTools
# | Tool | Description |
1 |
| Detect static/dynamic, scrape title/content/links |
2 |
| BFS crawl with depth limit + robots.txt |
3 |
| Strip noise HTML, return readable text |
4 |
| Sliding window chunks (~1 000 chars, 200 overlap) |
5 |
| Upload chunks to Azure AI Search |
6 |
| End-to-end pipeline: crawl → clean → chunk → index |
7 |
| Full-text search on the Azure AI Search index |
Installation
Prerequisites
Python 3.11+
Azure AI Search service (free tier works for testing)
Steps
# 1. Clone the repo
git clone https://github.com/your-org/website-scraper-mcp.git
cd website-scraper-mcp
# 2. Create and activate a virtual environment
python -m venv .venv
# Windows
.venv\Scripts\activate
# Linux/Mac
source .venv/bin/activate
# 3. Install dependencies
pip install -r requirements.txt
# 4. Install Playwright browsers (Chromium)
playwright install chromium
# 5. Copy and fill environment variables
cp .env.example .env
# Edit .env with your Azure credentialsRunning Locally
stdio mode (default — for MCP clients / AI agents)
python -m website_scraper_mcp.app
# or
python -m website_scraper_mcp.app --transport stdioSSE mode (HTTP endpoint for browser-based / HTTP clients)
python -m website_scraper_mcp.app --transport sse --port 8000
# Server available at http://localhost:8000/sseRunning with Docker
# Build and start in SSE mode
docker compose up --build
# Stop
docker compose downThe container exposes port 8000 for SSE transport.
Environment Variables
Variable | Default | Description |
| (required) | Azure AI Search service URL |
| (required) | Admin API key |
|
| Target index name |
|
| Playwright page load timeout (ms) |
|
| Run Chromium headless |
|
| Maximum crawl depth |
|
| Hard cap on pages per crawl |
|
| Polite delay between requests |
|
| Characters per chunk |
|
| Overlap between consecutive chunks |
|
| Python logging level |
Sample MCP Client
Run the included example after starting the server in SSE mode:
python examples/mcp_client_example.pyOr configure it in your MCP-compatible agent (e.g. Claude Desktop mcp_config.json):
{
"mcpServers": {
"website-scraper": {
"command": "python",
"args": ["-m", "website_scraper_mcp.app", "--transport", "stdio"],
"cwd": "/path/to/website-scraper-mcp",
"env": {
"AZURE_SEARCH_ENDPOINT": "https://your-service.search.windows.net",
"AZURE_SEARCH_KEY": "your-key",
"AZURE_SEARCH_INDEX_NAME": "website-content"
}
}
}
}Example API Requests
Via MCP client (Python SDK)
import asyncio
from mcp import ClientSession
from mcp.client.sse import sse_client
async def demo():
async with sse_client("http://localhost:8000/sse") as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
# Scrape a single page
result = await session.call_tool("scrape_website", {"url": "https://example.com"})
print(result)
# Full pipeline
result = await session.call_tool("index_website", {
"url": "https://example.com",
"max_depth": 2
})
print(result)
# Search
result = await session.call_tool("search_index", {
"query": "What services does the company provide?",
"top": 5
})
print(result)
asyncio.run(demo())Tool input/output examples
scrape_website
// Input
{"url": "https://example.com"}
// Output
{
"title": "Example Domain",
"url": "https://example.com",
"content": "This domain is for use in illustrative examples...",
"links": ["https://www.iana.org/domains/example"],
"is_dynamic": false,
"metadata": {"description": "..."}
}index_website
// Input
{"url": "https://example.com", "max_depth": 2}
// Output
{
"url": "https://example.com",
"pages_crawled": 4,
"total_chunks": 38,
"indexed_documents": 38,
"failed_documents": 0,
"status": "success",
"errors": []
}search_index
// Input
{"query": "What services does the company provide?", "top": 5}
// Output
{
"query": "What services does the company provide?",
"total_results": 3,
"hits": [
{
"id": "abc123",
"url": "https://example.com/services",
"title": "Our Services",
"content": "We provide cloud, AI, and data services...",
"chunk_number": 0,
"score": 9.8
}
]
}Error Handling
The server handles all errors gracefully and returns structured JSON error responses:
{
"error": "HTTP 404 when fetching https://example.com/missing",
"tool": "scrape_website"
}Handled errors include: invalid URLs, HTTP 4xx/5xx, timeouts, Playwright failures, Azure Search quota errors, network issues, and duplicate document IDs.
License
MIT
Available Tools
5 toolschunk_contentA
Split clean text into overlapping chunks (~1 000 characters each, 200-character overlap). Each chunk has a unique deterministic ID derived from the URL and position. Useful for preparing text for vector embedding or search indexing.
Inputs: text (required), url, title.
clean_contentA
Clean raw HTML by removing scripts, styles, navigation bars, footers, cookie banners, ads, and other noise. Returns readable plain text keeping only main article content, headings, paragraphs, tables, and lists.
Inputs: html (required), url.
crawl_websiteA
BFS-crawl an entire website starting from the given root URL. Only follows internal (same-domain) links. Respects robots.txt. Avoids duplicate URLs. Limits crawl depth and total page count. Returns every scraped page with title, content, and links.
Inputs: url (required), max_depth.
scrape_full_siteA
End-to-end pipeline: crawl every internal page of a website, clean the HTML of each page, and optionally split into chunks. Returns a structured result with every page's title, clean content, links, metadata, and (if requested) text chunks. Handles both static and dynamic pages automatically.
Inputs: url (required), chunk, clean, max_depth.
scrape_websiteA
Scrape a single web page. Automatically detects whether the page is static (uses httpx + BeautifulSoup) or dynamic/JS-rendered (uses Playwright headless Chromium). Returns the page title, clean content, all internal/external links, and page metadata.
Inputs: url (required).
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 5 tools
Each tool has a unique purpose: single-page scrape, full-site crawl, cleaning, chunking, and an end-to-end pipeline. No two tools overlap in functionality.
All tool names follow a consistent verb_noun pattern in snake_case (e.g., scrape_website, clean_content), making them predictable and easy to distinguish.
Five tools is appropriate for a website scraping server: it covers the essential operations without being overly sparse or bloated.
The tool surface covers the full workflow: scraping single pages, crawling entire sites, cleaning HTML, and chunking text. No essential operation is missing for the stated domain.
Maintenance
Related MCP Connectors
Web data for agents. Extract web pages, crawl websites, discover URLs, and search the web.
Web infrastructure for AI agents: scrape, crawl and search the web, run browsers and agents.
Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
Pages, crawls and URL batches as clean Markdown and JSON for AI agents, with an Evidence Record.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceProvides AI agents and coding assistants with advanced web crawling and RAG capabilities, allowing them to scrape websites and leverage that knowledge through various retrieval strategies.2MIT
- FlicenseNot gradedqualityDmaintenanceEnables AI assistants to crawl websites, extract and store web content with semantic search capabilities using vector embeddings, and retrieve information through natural language queries with tag-based filtering and intelligent content cleaning.-
- AlicenseNot gradedqualityDmaintenanceProvides AI agents and assistants with advanced web crawling and RAG capabilities, enabling them to scrape websites and perform semantic search over crawled content.1MIT
- FlicenseNot gradedqualityDmaintenanceEnables AI agents to perform web searches, extract webpage content, and conduct end-to-end search-and-extract operations using multiple search providers and content extraction methods.-