webscout-mcp
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| WEBSCOUT_AI_MODEL | No | AI model to use. Default: qwen2.5:7b | qwen2.5:7b |
| WEBSCOUT_CACHE_TTL | No | Cache TTL in seconds. Default: 3600 | 3600 |
| WEBSCOUT_VECTOR_DB | No | Vector database to use. Default: chroma | chroma |
| WEBSCOUT_AI_API_KEY | No | API key for AI backend (e.g., OpenAI, Doubao). Required for non-Ollama backends. | |
| WEBSCOUT_AI_BACKEND | No | AI backend to use for content understanding (ollama, openai, doubao, or custom). Default: ollama | ollama |
| WEBSCOUT_BROWSER_TYPE | No | Browser type for Playwright automation (chromium, firefox, webkit). Default: chromium | chromium |
| WEBSCOUT_SETUP_OLLAMA | No | Install and configure Ollama during setup. Default: false | false |
| WEBSCOUT_CACHE_ENABLED | No | Enable caching. Default: true | true |
| WEBSCOUT_SETUP_CHROMADB | No | Install ChromaDB during setup. Default: false | false |
| WEBSCOUT_EMBEDDING_MODEL | No | Embedding model name. Default: BAAI/bge-small-zh-v1.5 | BAAI/bge-small-zh-v1.5 |
| WEBSCOUT_BROWSER_HEADLESS | No | Run browser in headless mode. Default: true | true |
| WEBSCOUT_MONITOR_INTERVAL | No | Default monitoring check interval in seconds. Default: 300 | 300 |
| WEBSCOUT_SETUP_PLAYWRIGHT | No | Install Playwright and Chromium during setup. Default: true | true |
| WEBSCOUT_EMBEDDING_BACKEND | No | Embedding backend for vector search (local, openai, custom). Default: local | local |
| WEBSCOUT_MONITOR_MIN_CHANGE | No | Minimum change size to trigger monitoring alerts. Default: 10 | 10 |
| WEBSCOUT_RATE_LIMIT_ENABLED | No | Enable per-domain rate limiting. Default: true | true |
| WEBSCOUT_SEARCH_MAX_RESULTS | No | Maximum number of search results to return. Default: 10 | 10 |
| WEBSCOUT_BROWSER_BLOCK_MEDIA | No | Block media resources (images, videos) in browser automation. Default: true | true |
| WEBSCOUT_SEARCH_DEFAULT_BACKEND | No | Default search backend (e.g., bing, duckduckgo, google, brave). Default: bing | bing |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
Web search and fetch tools for AI agents. Use web_search to find information (Bing + DuckDuckGo failover), web_fetch to read a specific page's main content, web_crawl to explore a site concurrently, and web_extract to pull structured data via CSS selectors. All results are cached locally to avoid redundant requests. Crawler respects robots.txt by default.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": false
} |
| prompts | {
"listChanged": false
} |
| resources | {
"subscribe": false,
"listChanged": false
} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| web_searchB | Search the web and return structured results. SearchService uses health-based provider ranking, deterministic recovery classification, and automatic fallback across registered search providers. Empty/invalid queries are rejected without network calls. No API key required for the default HTML providers. |
| web_fetchA | Fetch a URL and return one window of its extracted main content. Progressive delivery: each call returns at most |
| web_crawlB | Crawl a website starting from a seed URL, respecting depth and page limits. Fetches pages concurrently within each depth level. Respects robots.txt by default. |
| web_extractC | Extract structured data from a web page using CSS selectors. |
| web_extract_schemaA | Extract data matching a JSON Schema without writing selectors. Pass the shape you want and get back an object; no CSS rules, no LLM key. Each field is resolved from the best available source, tried in this order:
The response records Use
|
| cache_statsA | Return cache statistics: entry count, total size, TTL, and limits. |
| cache_clearA | Clear all cached entries. Returns the number of entries deleted. |
| search_healthA | Get health report for all search backends. Returns overall health score, per-backend status (healthy/degraded/open/half-open), circuit breaker state, and request statistics. Use this to diagnose search failures. Reports the active SearchService health first, with legacy SearchEngine health included as fallback reference during the migration period. |
| metadata_extractA | Extract metadata from a web page. Extracts JSON-LD, OpenGraph, Twitter Cards, article metadata, images, links, and other structured metadata from the page. Relative URLs are resolved against the final (post-redirect) URL. SSRF safety is enforced on the initial URL and every redirect hop. |
| recovery_policyA | Get the fetch recovery policy table: thresholds, marker sets, reason→action rules, and classification stages, as data. This is the single source the recovery classifier is bound to (the modules import their thresholds from this table, enforced by identity tests). Returns the rendered markdown table plus any drift warnings found while assembling it. Read-only, zero-arg — use it to answer "why did this request escalate?" or to review the policy without reading the classifier source. |
| rss_parseA | Parse an RSS or Atom feed and return its entries. Fetches and parses RSS 2.0, RSS 1.0, and Atom feeds. Returns feed title, description, link, and a list of entries with title, link, description, publication date, and author. |
| content_qualityA | Analyze content quality of a web page. Evaluates readability scores (Flesch-Kincaid, Gunning Fog), keyword density, content structure, metadata quality, and provides actionable suggestions for improvement. |
| broken_linksA | Check for broken links on a web page. Extracts all links from the page, checks each one's HTTP status, and returns a report with broken links, redirects, and link statistics. |
| web_knowledge_graphA | Build a knowledge graph of entities and relationships from a page or text. Extracts named entities (people, organizations, locations, technologies) and the relationships between them, then ranks entities by centrality. Provide either Relationship types are currently |
| web_rag_contextA | Retrieve the passages of a page most relevant to a query, compressed to fit a context window. Splits the content into sentence-aligned passages, ranks them against Provide either Ranking is lexical (stop-word-filtered IDF over content terms), not embedding
based, so a query phrased with different vocabulary than the document will miss.
Use |
| web_compare_sitesA | Compare several websites side by side on SEO, performance, content and links. Fetches each URL, computes metrics, and returns a ranked comparison with a per-metric matrix and prioritised recommendations. Use this to benchmark a page against competitors, or to check whether your own pages regress. Scores are 0-100 per axis and a weighted overall. Comparisons are only as good as the markup available to a plain HTTP fetch: a JavaScript-rendered page may look thin even when it is not, and that will be visible as a low content score. A URL that cannot be fetched is reported as an entry with an |
| web_monitorA | Check a URL for changes since the last check, and return what changed. Compares the page against the last snapshot stored on disk and reports text changes, price moves, and keyword appearances or disappearances. Useful for tracking a page across sessions: docs, prices, availability, job postings. This checks once per call. It does not schedule itself - an MCP tool is passive. To monitor continuously, call it again from your MCP client or an external scheduler. The first call for a URL only records a baseline and returns no changes. Use Price watching is independent of
|
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 17 tools
Most tools target a clearly distinct operation: fetch vs crawl vs search vs monitor vs RSS vs cache management. The main overlap cluster is the page-analysis family (web_extract vs web_extract_schema vs metadata_extract), but descriptions explicitly disambiguate them (exact selectors vs schema inference vs metadata dump). recovery_policy and search_health both serve diagnostics, though their scopes are stated as policy table vs backend status.
There is a recognizable `web_` prefix convention for the core scraping tools, but eight tools (cache_stats, cache_clear, recovery_policy, broken_links, search_health, metadata_extract, rss_parse, content_quality) drop it. Verb ordering is also mixed between noun_verb (web_search, rss_parse) and noun_noun (cache_stats, content_quality, search_health), so the pattern is readable but not predictable.
At 17 tools this is on the heavier side, but the domain (web fetching, extraction, analysis, monitoring) genuinely spans that many operations and each tool has a defensible role. The analysis/audit cluster (broken_links, content_quality, web_compare_sites, metadata_extract) is where trimming could be argued.
The surface covers the full pipeline: search, fetch with continuation, crawl, selector-based and schema-based extraction, metadata, RSS, content analysis, monitoring, plus cache and health introspection. Minor gaps exist (no sitemap/robots introspection tool, no explicit pagination for search beyond what descriptions imply), but agents can work around these.