Website Scraper MCP Server
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| LOG_LEVEL | No | Python logging level | INFO |
| CHUNK_SIZE | No | Characters per chunk | 1000 |
| CHUNK_OVERLAP | No | Overlap between consecutive chunks | 200 |
| MAX_CRAWL_DEPTH | No | Maximum crawl depth | 2 |
| AZURE_SEARCH_KEY | Yes | Admin API key | |
| MAX_PAGES_PER_SITE | No | Hard cap on pages per crawl | 100 |
| CRAWL_DELAY_SECONDS | No | Polite delay between requests | 0.5 |
| PLAYWRIGHT_HEADLESS | No | Run Chromium headless | true |
| AZURE_SEARCH_ENDPOINT | Yes | Azure AI Search service URL | |
| PLAYWRIGHT_TIMEOUT_MS | No | Playwright page load timeout (ms) | 30000 |
| AZURE_SEARCH_INDEX_NAME | No | Target index name | website-content |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
This server publishes no instructions, or was last inspected before Glama recorded them.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": false
} |
| experimental | {} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| scrape_websiteA | Scrape a single web page. Automatically detects whether the page is static (uses httpx + BeautifulSoup) or dynamic/JS-rendered (uses Playwright headless Chromium). Returns the page title, clean content, all internal/external links, and page metadata. |
| crawl_websiteA | BFS-crawl an entire website starting from the given root URL. Only follows internal (same-domain) links. Respects robots.txt. Avoids duplicate URLs. Limits crawl depth and total page count. Returns every scraped page with title, content, and links. |
| clean_contentA | Clean raw HTML by removing scripts, styles, navigation bars, footers, cookie banners, ads, and other noise. Returns readable plain text keeping only main article content, headings, paragraphs, tables, and lists. |
| chunk_contentA | Split clean text into overlapping chunks (~1 000 characters each, 200-character overlap). Each chunk has a unique deterministic ID derived from the URL and position. Useful for preparing text for vector embedding or search indexing. |
| scrape_full_siteA | End-to-end pipeline: crawl every internal page of a website, clean the HTML of each page, and optionally split into chunks. Returns a structured result with every page's title, clean content, links, metadata, and (if requested) text chunks. Handles both static and dynamic pages automatically. |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 5 tools
Each tool has a unique purpose: single-page scrape, full-site crawl, cleaning, chunking, and an end-to-end pipeline. No two tools overlap in functionality.
All tool names follow a consistent verb_noun pattern in snake_case (e.g., scrape_website, clean_content), making them predictable and easy to distinguish.
Five tools is appropriate for a website scraping server: it covers the essential operations without being overly sparse or bloated.
The tool surface covers the full workflow: scraping single pages, crawling entire sites, cleaning HTML, and chunking text. No essential operation is missing for the stated domain.