webscout-mcp
Self-healing web access layer for AI agents: search, fetch, crawl, extract, analyze, monitor, and cache web content over MCP.
Search the web with multi-provider routing, health checks, and fallback.
Fetch URLs with main-content extraction and progressive windowed reading (
start_char), with browser escalation for challenge pages.Crawl sites concurrently with depth/page limits and robots.txt compliance.
Extract structured data using CSS selectors or a supplied JSON Schema, with JSON-LD/meta/semantic fallback.
Extract page metadata, parse RSS/Atom feeds, and assess content quality.
Check broken links and compare sites on SEO, performance, content, and links.
Build entity/relationship knowledge graphs and retrieve query-relevant RAG passages from a page.
Monitor URLs for text, price, keyword, or JSON-path changes and push alerts via webhook/DingTalk/Telegram/SMTP.
Inspect/clear cache, check search-backend health, and review fetch recovery policy.
Provides web search capabilities using DuckDuckGo's HTML interface as a fallback backend, returning structured results with titles, URLs, snippets, and backend metadata.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@webscout-mcp search for the latest news on artificial intelligence"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
webscout-mcp
Self-healing web access layer for AI Agents. Search, fetch, extract, and read long web content progressively โ with provider routing, deterministic recovery, browser fallback, caching, and observability over the Model Context Protocol (MCP). Everything stays on your machine.
๐ฏ Project Positioning: webscout-mcp is a self-healing web access layer for AI agents. The core MCP server exposes 11 tools:
web_search,web_fetch,web_crawl,web_extract,metadata_extract,rss_parse,content_quality,broken_links,cache_stats,cache_clear, andsearch_health. AI content understanding, vector search/RAG, headless browser automation, monitoring, SEO, and the optional Jev shadow evaluator are available as Python libraries / optional extras and are not required for core use. See Module Status for detailed stability and integration status.
ไธญๆ็ๆฌ็ฎไป | ๅฟซ้ไบ่งฃ้กน็ฎ๏ผ้ๅไธญๆ็จๆท้ ่ฏป
๐ฏ What's Included in MCP (Right Now)
The MCP server currently exposes these 11 tools:
Tool | Description | Stability |
| Self-healing multi-provider search with deterministic recovery and automatic fallback | โ Stable |
| Fetch a URL and return one window of extracted main content, with progressive continuation ( | โ Stable |
| Concurrent website crawling with depth limits | โ Stable |
| Structured content extraction with CSS selectors | โ Stable |
| Extract metadata (JSON-LD, OpenGraph, Twitter cards) from a page | ๐ถ Beta |
| Parse an RSS or Atom feed and return its entries | ๐ถ Beta |
| Analyze content quality of a fetched page | ๐งช Experimental |
| Check for broken links on a web page | ๐งช Experimental |
| View cache statistics and hit rates | โ Stable |
| Clear the search/fetch cache | โ Stable |
| Health report for all search backends | ๐ถ Beta |
Available as Python libraries / optional extras (not required for core use): AI content understanding, vector search & RAG, headless browser automation, web monitoring & alerting, SEO analysis, OCR, PDF processing, knowledge graphs, and the optional Jev shadow evaluator (pip install "webscout-mcp[jev]", disabled by default, BYOK). See Module Status for the full list.
Related MCP server: mcp-web-search
โจ Features
๐ Core Web Tools
Self-healing multi-provider search โ Bing, DuckDuckGo, SearXNG, Tavily with health-based ranking, deterministic recovery, and automatic fallback
Smart content extraction โ trafilatura + readability-lxml + html2text fallback, clean article content
Concurrent crawler โ BFS crawl with depth/page limits, robots.txt compliance, retry on failures
Structured data extraction โ CSS selectors, attributes, regex extraction
Metadata extraction โ JSON-LD, OpenGraph, Twitter Cards, article metadata, images, links
RSS/Atom support โ parse feeds and feed indexes
๐ค AI Content Understanding
Text summarization โ automatic article and page summarization
Question answering โ ask questions about fetched content
Key points extraction โ extract main ideas and takeaways
Content classification โ categorize content into custom categories
Tag generation โ auto-generate relevant tags
Sentiment analysis โ analyze text sentiment
Document comparison โ compare two documents side by side
Entity extraction โ extract people, places, organizations, dates
Multiple LLM backends โ Ollama (local/free), OpenAI, Doubao, custom OpenAI-compatible
๐ง Vector Search & RAG
Semantic search โ search by meaning, not just keywords
RAG (Retrieval-Augmented Generation) โ answer questions based on your crawled content
Local vector database โ ChromaDB persistent storage
Multiple embedding backends โ local sentence-transformers (free), OpenAI, custom
Document chunking โ automatic text splitting with overlap
Similarity threshold โ configurable relevance filtering
๐ Headless Browser Automation
JavaScript rendering โ fetch modern SPAs and dynamic content
User interaction simulation โ scroll, click, fill forms
Screenshot capture โ full-page screenshots
PDF export โ convert web pages to PDF
Login state management โ cookie persistence across sessions
Anti-detection stealth mode โ navigator.webdriver, plugins, languages spoofing
Resource blocking โ block images, media, CSS, fonts for faster loading
Proxy support โ HTTP/HTTPS proxy configuration
Multiple browsers โ Chromium, Firefox, WebKit
๐ก Web Monitoring & Alerting
Scheduled monitoring โ configurable check intervals
Content change detection โ text, HTML, specific element changes
Keyword monitoring โ appearance, disappearance, count changes
Price monitoring โ track price changes with threshold alerts
Change history โ persistent history with diff generation
Multi-channel alerts โ Webhook, Email (SMTP), DingTalk, WeCom
Configurable thresholds โ minimum change size, similarity thresholds
โก Performance & Security
Smart caching โ SQLite cache with TTL, size limits, automatic eviction
Rate limiting โ per-domain token-bucket rate limiting
SSRF protection โ blocks localhost, sensitive ports, invalid schemes
Browser fingerprint rotation โ random User-Agents + realistic headers
TLS fingerprint simulation โ realistic TLS ClientHello fingerprints
Connection pooling โ persistent HTTP connections
Cookie management โ automatic cookie handling and persistence
๐ Easy Setup & Deployment
One-click setup โ
webscout-mcp setupauto-installs all dependenciesSystem detection โ auto-detects OS, CPU, memory, GPU
Smart recommendations โ suggests optimal configuration based on hardware
Docker support โ pre-built images for amd64 and arm64
Docker Compose โ one-command deployment
systemd service โ Linux service file for production
Kubernetes โ deployment manifests for container orchestration
Configuration hot-reload โ reload config without restart
๐ Website Analysis & Optimization
SEO analyzer โ comprehensive SEO audit: meta tags, headings, images, links, URL structure, content length, Open Graph, Twitter Cards, Schema markup, with multi-dimensional scoring and actionable recommendations
Broken link checker โ detect broken links, redirect chains, invalid URLs, mixed content; classify internal/external/mailto/tel/javascript links; detailed reporting with statistics
Performance analyzer โ page performance audit: HTML size, DOM size, resource counts, render-blocking resources, inline CSS/JS, optimization techniques (lazy loading, preconnect, preload), compression/cache detection, performance scoring
Content quality assessor โ readability scores (Flesch-Kincaid, Gunning Fog, SMOG), keyword density, content structure analysis, duplicate content detection, quality scoring
๐ Export & Integration
Multiple export formats โ JSON, CSV, Excel, Parquet, SQLite, Markdown, HTML
Field selection & ordering โ export only specified fields with custom column order
Append mode โ incremental exports for CSV and SQLite
MCP server โ native Model Context Protocol support
CLI interface โ command-line tools for search, fetch, crawl
Python API โ full programmatic access to all features
Sitemap support โ parse sitemap.xml and sitemap indexes
Incremental crawling โ only re-fetch changed pages via ETag/Last-Modified
โ ๏ธ Search Backend Stability Notice
webscout-mcp uses direct HTML scraping for search backends (Bing, DuckDuckGo, Google, Brave) by default โ no API keys required. This makes it free to use, but please be aware of the stability trade-offs:
What can go wrong
DOM changes: Search engines frequently update their HTML structure, which can break scrapers
CAPTCHAs: Automated requests may trigger CAPTCHAs (especially Google and Brave)
Bot detection: Advanced bot detection may block or rate-limit requests
IP blocking: Sustained automated requests can lead to IP bans
Parameter changes: Search engines may change request parameters or headers
Mitigations built in
โ Multi-backend failover: If one backend fails, automatically try the next one
โ Realistic browser headers: Random User-Agents and realistic request headers
โ Rate limiting: Per-domain rate limiting to avoid overwhelming search engines
โ Caching: SQLite cache reduces repeated requests to the same queries
โ Retry with backoff: Exponential backoff on transient failures
For production use
For production workloads requiring higher reliability, consider:
Using official search APIs (Bing Search API, SerpAPI, etc.) โ planned for future releases
Deploying with rotating proxies
Increasing cache TTL to reduce request frequency
Monitoring search backend health and adjusting backends accordingly
Bottom line: webscout-mcp's default search is great for development, personal use, and low-volume workloads. For high-volume production use, plan for additional reliability measures.
๐ฆ Installation
Quick Install
pip install webscout-mcpRequires Python 3.10+.
One-Click Full Setup (Recommended)
# Install core package
pip install webscout-mcp
# Run setup to install all optional dependencies
webscout-mcp setup --playwright --ollama --vector-storeThe setup command will:
Detect your system configuration (OS, CPU, memory, GPU)
Install Playwright and Chromium browser
Install Ollama and download a local LLM (optional)
Install ChromaDB and sentence-transformers for vector search (optional)
Generate a configuration file
Run a health check to verify everything works
Optional Dependencies
# Browser automation (Playwright)
pip install webscout-mcp[browser]
playwright install chromium
# Vector search & RAG (ChromaDB + sentence-transformers)
pip install webscout-mcp[vector]
# AI content understanding (OpenAI client)
pip install webscout-mcp[ai]
# All features
pip install webscout-mcp[all]Docker
docker pull wxslang/webscout-mcp:latest
docker run -p 8000:8000 wxslang/webscout-mcp:latest๐ Quick Start
MCP Client Configuration
Add to your MCP client config:
Claude Desktop (~/Library/Application Support/Claude/claude_desktop_config.json on macOS, %APPDATA%\Claude\claude_desktop_config.json on Windows):
{
"mcpServers": {
"webscout": {
"command": "webscout-mcp",
"args": []
}
}
}Cursor (Settings โ MCP โ Add new MCP server):
{
"mcpServers": {
"webscout": {
"command": "webscout-mcp",
"args": []
}
}
}CLI Usage
# Search the web
webscout-mcp search "python web scraping" --max-results 10
# Fetch a page
webscout-mcp fetch https://example.com --format markdown
# Crawl a site
webscout-mcp crawl https://example.com --depth 2 --pages 50
# Run setup
webscout-mcp setup --playwright
# Start MCP server
webscout-mcp servePython API
from webscout_mcp import WebScout
# Initialize
scout = WebScout()
# Search
results = scout.search("AI agents", max_results=5)
for result in results:
print(result.title, result.url)
# Fetch and extract
page = scout.fetch("https://example.com/article")
print(page.title)
print(page.content) # Clean article text
# Crawl
pages = scout.crawl("https://example.com", depth=2, max_pages=20)๐ค AI Content Understanding
from webscout_mcp.ai_processor import AIProcessor, AIConfig
# Use local Ollama (free, no API key needed)
config = AIConfig(backend="ollama", model="qwen2.5:7b")
ai = AIProcessor(config=config)
# Summarize
summary = ai.summarize(page.content, max_length=500)
print(summary.content)
# Ask questions
answer = ai.answer_question(page.content, "What are the main points?")
print(answer.content)
# Extract key points
points = ai.extract_key_points(page.content, num_points=5)
print(points.content)
# Analyze sentiment
sentiment = ai.analyze_sentiment(page.content)
print(sentiment.content)Using OpenAI API:
config = AIConfig(
backend="openai",
model="gpt-4o",
api_key="your-api-key",
)Using Doubao (่ฑๅ ):
config = AIConfig(
backend="doubao",
model="ep-20240101",
api_key="your-api-key",
)๐ง Vector Search & RAG
from webscout_mcp.vector_store import VectorStore, RAGEngine, Document
# Initialize vector store (local, free)
store = VectorStore()
# Add documents
doc = Document.from_text(page.content, source=page.url)
store.add_document(doc)
# Semantic search
results = store.search("how to build AI agents", n_results=5)
for result in results:
print(f"[{result.score:.2f}] {result.document.content[:100]}")
# RAG: Ask questions based on your documents
rag = RAGEngine(vector_store=store)
answer = rag.query("What is the best approach for web scraping?")
print(answer["answer"])
print("Sources:", answer["sources"])๐ Headless Browser Automation
from webscout_mcp.browser_fetcher import BrowserFetcher, BrowserConfig
# Initialize
config = BrowserConfig(headless=True, block_media=True)
browser = BrowserFetcher(config=config)
# Fetch JS-rendered page
result = browser.fetch(
"https://example.com/spa",
wait_for_selector=".content",
scroll_to_bottom=True,
)
print(result.title)
print(result.content)
# Take screenshot
browser.take_screenshot("https://example.com", "screenshot.png", full_page=True)
# Export to PDF
browser.export_pdf("https://example.com", "page.pdf")
# Click element
result = browser.click_element("https://example.com", "button.load-more")
# Fill form
result = browser.fill_form(
"https://example.com/login",
{"#username": "user", "#password": "pass"},
submit_selector="button[type=submit]",
)
browser.close()๐ก Web Monitoring & Alerting
from webscout_mcp.monitor import WebMonitor, MonitorConfig, WebhookAlert, EmailAlert
# Initialize
config = MonitorConfig(check_interval=300, min_change_size=10)
monitor = WebMonitor(config=config)
# Add alert channels
monitor.add_alert_channel(WebhookAlert("https://hooks.example.com/alert"))
monitor.add_alert_channel(EmailAlert(
smtp_server="smtp.gmail.com",
smtp_port=587,
username="you@gmail.com",
password="app-password",
from_addr="you@gmail.com",
to_addrs=["recipient@example.com"],
))
# Check for changes
changes = monitor.check_url("https://example.com/pricing")
for change in changes:
print(f"{change.change_type}: {change.old_value} -> {change.new_value}")
# Get history
history = monitor.get_history("https://example.com/pricing")๐ Website Analysis & Optimization
SEO Analysis
from webscout_mcp.seo_analyzer import SEOAnalyzer
# Initialize
analyzer = SEOAnalyzer()
# Analyze a page
html = "<html>...</html>"
metrics = analyzer.analyze(html, url="https://example.com")
# Check scores
print(f"Overall SEO Score: {metrics.overall_score}/100")
print(f"Meta Score: {metrics.meta_score}")
print(f"Heading Score: {metrics.heading_score}")
print(f"Image Score: {metrics.image_score}")
# Check issues and recommendations
print("Issues:", metrics.issues)
print("Recommendations:", metrics.recommendations)Broken Link Checking
from webscout_mcp.broken_link_checker import BrokenLinkChecker
# Initialize
checker = BrokenLinkChecker(timeout=10.0, max_redirects=5)
# Check all links on a page
html = "<html>...</html>"
report = checker.check_page(html, base_url="https://example.com")
# Check statistics
print(f"Total links: {report.total_links}")
print(f"OK: {report.ok_links}")
print(f"Broken: {report.broken_links}")
print(f"Redirects: {report.redirect_links}")
print(f"Broken percentage: {report.broken_link_percentage}%")
# Get only broken links
broken = checker.get_broken_links(report)
for link in broken:
print(f"[{link.status}] {link.url} - {link.error_message}")
# Generate human-readable summary
print(checker.generate_summary(report))Performance Analysis
from webscout_mcp.performance_analyzer import PerformanceAnalyzer
# Initialize
analyzer = PerformanceAnalyzer()
# Analyze page performance
html = "<html>...</html>"
headers = {"Content-Encoding": "gzip", "Cache-Control": "max-age=3600"}
metrics = analyzer.analyze(html, url="https://example.com", response_headers=headers)
# Check scores
print(f"Overall Performance Score: {metrics.overall_score}/100")
print(f"HTML Size: {metrics.html_size_kb}KB (score: {metrics.html_size_score})")
print(f"DOM Nodes: {metrics.dom_node_count} (score: {metrics.dom_size_score})")
print(f"Requests: {metrics.request_count} (score: {metrics.request_count_score})")
# Check optimization techniques
print(f"Has gzip: {metrics.has_gzip}")
print(f"Has brotli: {metrics.has_brotli}")
print(f"Has lazy loading: {metrics.has_lazy_loading}")
print(f"Has preconnect: {metrics.has_preconnect}")
# Check issues and recommendations
print("Issues:", metrics.issues)
print("Warnings:", metrics.warnings)
print("Recommendations:", metrics.recommendations)Enhanced Data Export
from webscout_mcp.data_exporter import DataExporter, ExportConfig
# Sample data
data = [
{"title": "Result 1", "url": "https://example.com/1", "score": 0.95},
{"title": "Result 2", "url": "https://example.com/2", "score": 0.85},
]
# Export to JSON
config = ExportConfig(format="json", output_path="results.json", pretty_json=True)
exporter = DataExporter(config=config)
result = exporter.export(data)
print(f"Exported {result.record_count} records to {result.output_path}")
# Export to CSV with field selection
config = ExportConfig(
format="csv",
output_path="results.csv",
fields=["title", "url"], # Only export these fields
csv_delimiter=",",
)
exporter = DataExporter(config=config)
result = exporter.export(data)
# Export to Excel
config = ExportConfig(format="excel", output_path="results.xlsx", excel_sheet_name="Results")
exporter = DataExporter(config=config)
result = exporter.export(data)
# Export to SQLite
config = ExportConfig(format="sqlite", output_path="results.db", sqlite_table_name="search_results")
exporter = DataExporter(config=config)
result = exporter.export(data)
# Export to Parquet (columnar storage)
config = ExportConfig(format="parquet", output_path="results.parquet")
exporter = DataExporter(config=config)
result = exporter.export(data)
# Export to Markdown
config = ExportConfig(format="markdown", output_path="results.md")
exporter = DataExporter(config=config)
result = exporter.export(data)
# Export to HTML
config = ExportConfig(format="html", output_path="results.html")
exporter = DataExporter(config=config)
result = exporter.export(data)
# Using convenience function
from webscout_mcp.data_exporter import export_data
result = export_data(data, "results.json", export_format="json", fields=["title", "url"])โ๏ธ Configuration
Environment Variables
# Core
WEBSCOUT_CACHE_ENABLED=true
WEBSCOUT_CACHE_TTL=3600
WEBSCOUT_RATE_LIMIT_ENABLED=true
# Search
WEBSCOUT_SEARCH_DEFAULT_BACKEND=bing
WEBSCOUT_SEARCH_MAX_RESULTS=10
# AI
WEBSCOUT_AI_BACKEND=ollama
WEBSCOUT_AI_MODEL=qwen2.5:7b
WEBSCOUT_AI_API_KEY=your-key
# Vector Store
WEBSCOUT_VECTOR_DB=chroma
WEBSCOUT_EMBEDDING_BACKEND=local
WEBSCOUT_EMBEDDING_MODEL=BAAI/bge-small-zh-v1.5
# Browser
WEBSCOUT_BROWSER_TYPE=chromium
WEBSCOUT_BROWSER_HEADLESS=true
WEBSCOUT_BROWSER_BLOCK_MEDIA=true
# Monitor
WEBSCOUT_MONITOR_INTERVAL=300
WEBSCOUT_MONITOR_MIN_CHANGE=10
# Setup
WEBSCOUT_SETUP_PLAYWRIGHT=true
WEBSCOUT_SETUP_OLLAMA=false
WEBSCOUT_SETUP_CHROMADB=falseConfig File
Create ~/.config/webscout/config.toml:
[server]
host = "127.0.0.1"
port = 8000
[cache]
enabled = true
ttl = 3600
[search]
default_backend = "bing"
max_results = 10
[ai]
backend = "ollama"
model = "qwen2.5:7b"
[vector_store]
enabled = true
vector_db = "chroma"
embedding_backend = "local"
[browser]
enabled = true
headless = true
[monitor]
check_interval = 300๐ง Troubleshooting: pages that need a browser
Some sites answer a plain HTTP request with a page that says "checking your browser"
instead of the content you asked for. web_fetch detects this, and if a browser is
available it retries with one.
What it looks like when there is no browser
The payload does not look like a failure:
{
"status_code": 200,
"error": null,
"error_code": null,
"extracted": true,
"extraction_status": "ok",
"content": "็ซๅฑฑๅผๆ\n\nๆญฃๅจ่ฟ่กๅฎๅ
จๆฃๆต...\n\nไธบไฟ้ๆจ็่ฎฟ้ฎๅฎๅ
จ..."
}Status 200, no error, extraction "ok" - because a readable page was returned. It is just not the page you wanted. The signals that identify it are:
Signal | Value for a challenge page |
|
|
|
|
| very short (tens of characters) |
| often 200, not 403 |
If you only check status_code and error, you will treat a challenge page as
success. Check recovery_outcome when content looks thin.
Enabling browser escalation
pip install 'webscout-mcp[browser]'
playwright install chromiumThat is all. With the extra installed, web_fetch escalates automatically when a
page needs rendering - no sidecar, no Docker, no configuration.
Verify it took effect:
python -c "from webscout_mcp.local_browser_backend import LocalBrowserBackend; print(LocalBrowserBackend().is_available)"
# TrueWhen a browser is not enough
Measured against sixteen real sites, browser escalation fixes client-rendered pages but not all anti-bot walls. Sites that reject by IP reputation rather than by rendering keep returning 403/202 even through a real browser:
Site | Response with browser | What would actually help |
medium.com | 200, but navigation only | client-side rendering |
stackoverflow.com | 403 | residential IP |
zhihu.com | 403 | residential IP, login session |
amazon.com | 202 | JS + cookie handshake |
reuters.com | 200, full article | already works (TLS impersonation) |
A proxy can be configured for the browser:
from webscout_mcp.browser_fetcher import BrowserConfig, BrowserFetcher
fetcher = BrowserFetcher(BrowserConfig(proxy_server="http://host:port"))If a fetch returns unreadable text
content_quality grades each response and reports garbled when the bytes are not
readable prose - typically a decoding fault rather than a blocked page. The most
common historical cause was a brotli response decoded without the brotli package
(fixed in v1.13.0; upgrade if you see it).
To grade a payload yourself:
from webscout_mcp.content_grading import assess
print(assess(content).to_dict())
# {'grade': 'usable' | 'thin' | 'empty' | 'garbled', 'chars': ..., 'reason': ...}Diagnosing why an escalation did not happen
web_fetch returns a route_trace describing every decision:
{"stage": "recovery_classification", "reason": "CHALLENGE", "action": "BROWSER"}
{"capability": "browser", "provider": null, "result": "unavailable"}
{"stage": "recovery_execution", "outcome": "browser_unavailable:fallback_unavailable"}provider: null with result: unavailable means no browser backend was registered -
install the extra as above. A populated provider with result: failure means the
browser ran and the site rejected it, which points at the IP-reputation cases in the
table.
๐ Documentation
README โ This file
Module Status โ Module stability levels and MCP integration status
Project Introduction โ Detailed project overview and architecture
Deployment Guide โ Docker, systemd, Kubernetes deployment
Examples โ Usage examples and sample code
CHANGELOG โ Version history
CONTRIBUTING โ Contributing guidelines
CODE OF CONDUCT โ Community code of conduct
SECURITY โ Security policy and vulnerability reporting
๐งช Testing
# Install dev dependencies
pip install webscout-mcp[dev]
# Run all tests
pytest tests/
# Run with coverage
pytest tests/ --cov=webscout_mcp --cov-report=htmlTest coverage: 395+ tests covering all modules.
๐ค Contributing
Contributions are welcome! Please see CONTRIBUTING.md for guidelines.
Fork the repository
Create a feature branch
Make your changes
Add tests
Submit a pull request
๐ License
MIT License โ see LICENSE for details.
๐ Acknowledgments
trafilatura โ content extraction
readability-lxml โ readability fallback
Playwright โ browser automation
ChromaDB โ vector database
sentence-transformers โ text embeddings
Ollama โ local LLM runtime
httpx โ HTTP client
BeautifulSoup โ HTML parsing
Built with โค๏ธ for the AI agent community.
Available Tools
17 toolsbroken_linksA
Check for broken links on a web page.
Extracts all links from the page, checks each one's HTTP status, and returns a report with broken links, redirects, and link statistics.
Inputs: url (required), timeout. Returns: result
cache_clearA
Clear all cached entries. Returns the number of entries deleted.
Inputs: no named parameters. Returns: result
cache_statsA
Return cache statistics: entry count, total size, TTL, and limits.
Inputs: no named parameters. Returns: result
content_qualityA
Analyze content quality of a web page.
Evaluates readability scores (Flesch-Kincaid, Gunning Fog), keyword density, content structure, metadata quality, and provides actionable suggestions for improvement.
Inputs: url (required). Returns: result
metadata_extractA
Extract metadata from a web page.
Extracts JSON-LD, OpenGraph, Twitter Cards, article metadata, images, links, and other structured metadata from the page. Relative URLs are resolved against the final (post-redirect) URL. SSRF safety is enforced on the initial URL and every redirect hop.
Inputs: url (required). Returns: result
recovery_policyA
Get the fetch recovery policy table: thresholds, marker sets, reasonโaction rules, and classification stages, as data.
This is the single source the recovery classifier is bound to (the modules import their thresholds from this table, enforced by identity tests). Returns the rendered markdown table plus any drift warnings found while assembling it. Read-only, zero-arg โ use it to answer "why did this request escalate?" or to review the policy without reading the classifier source.
Inputs: no named parameters. Returns: result
rss_parseA
Parse an RSS or Atom feed and return its entries.
Fetches and parses RSS 2.0, RSS 1.0, and Atom feeds. Returns feed title, description, link, and a list of entries with title, link, description, publication date, and author.
Inputs: url (required), max_entries. Returns: result
search_healthA
Get health report for all search backends.
Returns overall health score, per-backend status (healthy/degraded/open/half-open), circuit breaker state, and request statistics. Use this to diagnose search failures.
Reports the active SearchService health first, with legacy SearchEngine health included as fallback reference during the migration period.
Inputs: no named parameters. Returns: result
web_compare_sitesA
Compare several websites side by side on SEO, performance, content and links.
Fetches each URL, computes metrics, and returns a ranked comparison with a per-metric matrix and prioritised recommendations. Use this to benchmark a page against competitors, or to check whether your own pages regress.
Scores are 0-100 per axis and a weighted overall. Comparisons are only as good as the markup available to a plain HTTP fetch: a JavaScript-rendered page may look thin even when it is not, and that will be visible as a low content score.
A URL that cannot be fetched is reported as an entry with an error and its
recovery reason, rather than failing the whole comparison.
Inputs: urls (required), names. Returns: result
web_crawlB
Crawl a website starting from a seed URL, respecting depth and page limits.
Fetches pages concurrently within each depth level. Respects robots.txt by default.
Inputs: seed_url (required), extract, max_depth, max_pages, +2 more. Returns: result
web_extractC
Extract structured data from a web page using CSS selectors.
Inputs: url (required), rules (required). Returns: result
web_extract_schemaA
Extract data matching a JSON Schema without writing selectors.
Pass the shape you want and get back an object; no CSS rules, no LLM key. Each field is resolved from the best available source, tried in this order:
json-ld- the page's own structured data, so nothing is guessedmeta- OpenGraph / Twitter cardssemantic-html- h1, , itemprop, and a few class-name hints
The response records sources per field, so a caller can tell a stated value from an
inferred one, and lists missing fields explicitly rather than leaving silent nulls.
Use web_extract instead when you already know the page's selectors: it is exact,
whereas this infers. A field whose structure this has never seen comes back null -
it will not invent a plausible value.
schema is JSON: {"type": "object", "properties": {"price": {"type": "number"}}}.
Inputs: url (required), schema (required). Returns: result
web_fetchA
Fetch a URL and return one window of its extracted main content.
Progressive delivery: each call returns at most max_chars chars
(default 8000). If the response contains continuation.has_more=true,
more content is available locally. To read the next window, call this
SAME tool again with the SAME url/extract/output_format/max_chars and set
start_char to the previous continuation.next_start_char. Follow-up
windows are served from a local snapshot (no new HTTP fetch, extraction,
or browser). Stop when has_more is false. You do not need to read the
whole page unless the task requires it.
Inputs: url (required), extract, max_chars, start_char, +2 more. Returns: result
web_knowledge_graphA
Build a knowledge graph of entities and relationships from a page or text.
Extracts named entities (people, organizations, locations, technologies) and the relationships between them, then ranks entities by centrality.
Provide either url (fetched first) or text (used directly). This is useful
for mapping who/what a document is about and how they connect, rather than
reading it linearly.
Relationship types are currently co_occurs_with only: relationships come
from proximity, not from parsing sentence semantics. Entity types are
rule-based, so some are reported as unknown rather than guessed.
Inputs: url, text, max_entities, include_centrality. Returns: result
web_monitorA
Check a URL for changes since the last check, and return what changed.
Compares the page against the last snapshot stored on disk and reports text changes, price moves, and keyword appearances or disappearances. Useful for tracking a page across sessions: docs, prices, availability, job postings.
This checks once per call. It does not schedule itself - an MCP tool is passive. To monitor continuously, call it again from your MCP client or an external scheduler. The first call for a URL only records a baseline and returns no changes.
Use ignore_text (regexes) for values that change on every request such as
timestamps or session ids, and ignore_selector (CSS) for regions that churn
independently such as ad slots. Without them most pages report a change every
time, which is why they exist.
Price watching is independent of min_change_size: a one-character move like
$100.00 to $900.00 is exactly what a text threshold would swallow, so it is
compared as a percentage when price_selector is set.
json_path watches one value inside a JSON response instead of the whole body:
json:$.product.price uses JSONPath, jq:.product.price uses jq (needs the
[json] extra; JSONPath needs it too). Reducing an API response to one field is
what makes it stable enough to monitor - markup and prose churn, a price field
does not. Without json_path the whole body is compared as before.
notify takes target URLs such as webhook://example.com/hook,
dingtalk://oapi.dingtalk.com/robot/send?access_token=X, telegram://TOKEN@CHAT
or smtp://user:pass@host:587?to=a@b.c. When a change is found it is pushed to
those channels, and the send result is reported in notifications. Credentials
stay in the URL and are never echoed back.
Inputs: url (required), notify, json_path, state_path, +7 more. Returns: result
web_rag_contextA
Retrieve the passages of a page most relevant to a query, compressed to fit a context window.
Splits the content into sentence-aligned passages, ranks them against query,
and returns the best ones together with a compressed context block. Use this
when a page is too long to hand to a model whole, or when you need to find the
specific passages that answer a question.
Provide either url (fetched first) or text (used directly). query is
required: this tool retrieves, it does not summarise.
Ranking is lexical (stop-word-filtered IDF over content terms), not embedding
based, so a query phrased with different vocabulary than the document will miss.
Use compressed_context as the text to pass to a model; chunks are for
inspection and citation.
Inputs: url, text, query, top_k, +1 more. Returns: result
web_searchB
Search the web and return structured results.
SearchService uses health-based provider ranking, deterministic recovery classification, and automatic fallback across registered search providers. Empty/invalid queries are rejected without network calls. No API key required for the default HTML providers.
Inputs: query (required), region, max_results, safe_search. Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelogโฆ
TDQS
Scored across 17 tools
Most tools target a clearly distinct operation: fetch vs crawl vs search vs monitor vs RSS vs cache management. The main overlap cluster is the page-analysis family (web_extract vs web_extract_schema vs metadata_extract), but descriptions explicitly disambiguate them (exact selectors vs schema inference vs metadata dump). recovery_policy and search_health both serve diagnostics, though their scopes are stated as policy table vs backend status.
There is a recognizable `web_` prefix convention for the core scraping tools, but eight tools (cache_stats, cache_clear, recovery_policy, broken_links, search_health, metadata_extract, rss_parse, content_quality) drop it. Verb ordering is also mixed between noun_verb (web_search, rss_parse) and noun_noun (cache_stats, content_quality, search_health), so the pattern is readable but not predictable.
At 17 tools this is on the heavier side, but the domain (web fetching, extraction, analysis, monitoring) genuinely spans that many operations and each tool has a defensible role. The analysis/audit cluster (broken_links, content_quality, web_compare_sites, metadata_extract) is where trimming could be argued.
The surface covers the full pipeline: search, fetch with continuation, crawl, selector-based and schema-based extraction, metadata, RSS, content analysis, monitoring, plus cache and health introspection. Minor gaps exist (no sitemap/robots introspection tool, no explicit pagination for search beyond what descriptions imply), but agents can work around these.
Maintenance
Related MCP Connectors
Web scraping, search, crawling, structured extraction and research for AI agents.
Web search, browser automation, scraping, crawling and CAPTCHA solving for AI agents.
Web data for agents. Extract web pages, crawl websites, discover URLs, and search the web.
Web search and any page as LLM-ready Markdown for AI agents. Free keyless tier, no signup.
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceEnables AI agents to perform web searches, extract webpage content, and conduct end-to-end search-and-extract operations using multiple search providers and content extraction methods.-
- AlicenseNot gradedqualityCmaintenanceEnables AI agents to search the web, read and extract content from webpages, fetch JSON from REST APIs, and collect links while bypassing anti-bot protections.35 npmISC
- FlicenseNot gradedqualityAmaintenanceEnables AI agents to search multiple web providers in parallel and crawl single pages or entire sites into clean, deduplicated Markdown or JSON, including authenticated crawls.-
- FlicenseNot gradedqualityAmaintenanceEnables AI agents to search the web through configured providers and crawl pages or entire sites into clean, deduplicated Markdown or JSON, including authenticated crawls.-