Skip to main content
Glama
webscout-mcp

webscout-mcp

by wxs-lang

Server Configuration

Describes the environment variables required to run the server.

NameRequiredDescriptionDefault
WEBSCOUT_AI_MODELNoAI model to use. Default: qwen2.5:7bqwen2.5:7b
WEBSCOUT_CACHE_TTLNoCache TTL in seconds. Default: 36003600
WEBSCOUT_VECTOR_DBNoVector database to use. Default: chromachroma
WEBSCOUT_AI_API_KEYNoAPI key for AI backend (e.g., OpenAI, Doubao). Required for non-Ollama backends.
WEBSCOUT_AI_BACKENDNoAI backend to use for content understanding (ollama, openai, doubao, or custom). Default: ollamaollama
WEBSCOUT_BROWSER_TYPENoBrowser type for Playwright automation (chromium, firefox, webkit). Default: chromiumchromium
WEBSCOUT_SETUP_OLLAMANoInstall and configure Ollama during setup. Default: falsefalse
WEBSCOUT_CACHE_ENABLEDNoEnable caching. Default: truetrue
WEBSCOUT_SETUP_CHROMADBNoInstall ChromaDB during setup. Default: falsefalse
WEBSCOUT_EMBEDDING_MODELNoEmbedding model name. Default: BAAI/bge-small-zh-v1.5BAAI/bge-small-zh-v1.5
WEBSCOUT_BROWSER_HEADLESSNoRun browser in headless mode. Default: truetrue
WEBSCOUT_MONITOR_INTERVALNoDefault monitoring check interval in seconds. Default: 300300
WEBSCOUT_SETUP_PLAYWRIGHTNoInstall Playwright and Chromium during setup. Default: truetrue
WEBSCOUT_EMBEDDING_BACKENDNoEmbedding backend for vector search (local, openai, custom). Default: locallocal
WEBSCOUT_MONITOR_MIN_CHANGENoMinimum change size to trigger monitoring alerts. Default: 1010
WEBSCOUT_RATE_LIMIT_ENABLEDNoEnable per-domain rate limiting. Default: truetrue
WEBSCOUT_SEARCH_MAX_RESULTSNoMaximum number of search results to return. Default: 1010
WEBSCOUT_BROWSER_BLOCK_MEDIANoBlock media resources (images, videos) in browser automation. Default: truetrue
WEBSCOUT_SEARCH_DEFAULT_BACKENDNoDefault search backend (e.g., bing, duckduckgo, google, brave). Default: bingbing

Instructions

Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.

Web search and fetch tools for AI agents. Use web_search to find information (Bing + DuckDuckGo failover), web_fetch to read a specific page's main content, web_crawl to explore a site concurrently, and web_extract to pull structured data via CSS selectors. All results are cached locally to avoid redundant requests. Crawler respects robots.txt by default.

Capabilities

Features and capabilities supported by this server

Protocol revision2025-11-25

CapabilityDetails
tools
{
  "listChanged": false
}
prompts
{
  "listChanged": false
}
resources
{
  "subscribe": false,
  "listChanged": false
}

Tools

Functions exposed to the LLM to take actions

NameDescription
web_searchB

Search the web and return structured results.

SearchService uses health-based provider ranking, deterministic recovery classification, and automatic fallback across registered search providers. Empty/invalid queries are rejected without network calls. No API key required for the default HTML providers.

web_fetchA

Fetch a URL and return one window of its extracted main content.

Progressive delivery: each call returns at most max_chars chars (default 8000). If the response contains continuation.has_more=true, more content is available locally. To read the next window, call this SAME tool again with the SAME url/extract/output_format/max_chars and set start_char to the previous continuation.next_start_char. Follow-up windows are served from a local snapshot (no new HTTP fetch, extraction, or browser). Stop when has_more is false. You do not need to read the whole page unless the task requires it.

web_crawlB

Crawl a website starting from a seed URL, respecting depth and page limits.

Fetches pages concurrently within each depth level. Respects robots.txt by default.

web_extractC

Extract structured data from a web page using CSS selectors.

web_extract_schemaA

Extract data matching a JSON Schema without writing selectors.

Pass the shape you want and get back an object; no CSS rules, no LLM key. Each field is resolved from the best available source, tried in this order:

  1. json-ld - the page's own structured data, so nothing is guessed

  2. meta - OpenGraph / Twitter cards

  3. semantic-html - h1, , itemprop, and a few class-name hints

The response records sources per field, so a caller can tell a stated value from an inferred one, and lists missing fields explicitly rather than leaving silent nulls.

Use web_extract instead when you already know the page's selectors: it is exact, whereas this infers. A field whose structure this has never seen comes back null - it will not invent a plausible value.

schema is JSON: {"type": "object", "properties": {"price": {"type": "number"}}}.

cache_statsA

Return cache statistics: entry count, total size, TTL, and limits.

cache_clearA

Clear all cached entries. Returns the number of entries deleted.

search_healthA

Get health report for all search backends.

Returns overall health score, per-backend status (healthy/degraded/open/half-open), circuit breaker state, and request statistics. Use this to diagnose search failures.

Reports the active SearchService health first, with legacy SearchEngine health included as fallback reference during the migration period.

metadata_extractA

Extract metadata from a web page.

Extracts JSON-LD, OpenGraph, Twitter Cards, article metadata, images, links, and other structured metadata from the page. Relative URLs are resolved against the final (post-redirect) URL. SSRF safety is enforced on the initial URL and every redirect hop.

recovery_policyA

Get the fetch recovery policy table: thresholds, marker sets, reason→action rules, and classification stages, as data.

This is the single source the recovery classifier is bound to (the modules import their thresholds from this table, enforced by identity tests). Returns the rendered markdown table plus any drift warnings found while assembling it. Read-only, zero-arg — use it to answer "why did this request escalate?" or to review the policy without reading the classifier source.

rss_parseA

Parse an RSS or Atom feed and return its entries.

Fetches and parses RSS 2.0, RSS 1.0, and Atom feeds. Returns feed title, description, link, and a list of entries with title, link, description, publication date, and author.

content_qualityA

Analyze content quality of a web page.

Evaluates readability scores (Flesch-Kincaid, Gunning Fog), keyword density, content structure, metadata quality, and provides actionable suggestions for improvement.

broken_linksA

Check for broken links on a web page.

Extracts all links from the page, checks each one's HTTP status, and returns a report with broken links, redirects, and link statistics.

web_knowledge_graphA

Build a knowledge graph of entities and relationships from a page or text.

Extracts named entities (people, organizations, locations, technologies) and the relationships between them, then ranks entities by centrality.

Provide either url (fetched first) or text (used directly). This is useful for mapping who/what a document is about and how they connect, rather than reading it linearly.

Relationship types are currently co_occurs_with only: relationships come from proximity, not from parsing sentence semantics. Entity types are rule-based, so some are reported as unknown rather than guessed.

web_rag_contextA

Retrieve the passages of a page most relevant to a query, compressed to fit a context window.

Splits the content into sentence-aligned passages, ranks them against query, and returns the best ones together with a compressed context block. Use this when a page is too long to hand to a model whole, or when you need to find the specific passages that answer a question.

Provide either url (fetched first) or text (used directly). query is required: this tool retrieves, it does not summarise.

Ranking is lexical (stop-word-filtered IDF over content terms), not embedding based, so a query phrased with different vocabulary than the document will miss. Use compressed_context as the text to pass to a model; chunks are for inspection and citation.

web_compare_sitesA

Compare several websites side by side on SEO, performance, content and links.

Fetches each URL, computes metrics, and returns a ranked comparison with a per-metric matrix and prioritised recommendations. Use this to benchmark a page against competitors, or to check whether your own pages regress.

Scores are 0-100 per axis and a weighted overall. Comparisons are only as good as the markup available to a plain HTTP fetch: a JavaScript-rendered page may look thin even when it is not, and that will be visible as a low content score.

A URL that cannot be fetched is reported as an entry with an error and its recovery reason, rather than failing the whole comparison.

web_monitorA

Check a URL for changes since the last check, and return what changed.

Compares the page against the last snapshot stored on disk and reports text changes, price moves, and keyword appearances or disappearances. Useful for tracking a page across sessions: docs, prices, availability, job postings.

This checks once per call. It does not schedule itself - an MCP tool is passive. To monitor continuously, call it again from your MCP client or an external scheduler. The first call for a URL only records a baseline and returns no changes.

Use ignore_text (regexes) for values that change on every request such as timestamps or session ids, and ignore_selector (CSS) for regions that churn independently such as ad slots. Without them most pages report a change every time, which is why they exist.

Price watching is independent of min_change_size: a one-character move like $100.00 to $900.00 is exactly what a text threshold would swallow, so it is compared as a percentage when price_selector is set.

json_path watches one value inside a JSON response instead of the whole body: json:$.product.price uses JSONPath, jq:.product.price uses jq (needs the [json] extra; JSONPath needs it too). Reducing an API response to one field is what makes it stable enough to monitor - markup and prose churn, a price field does not. Without json_path the whole body is compared as before.

notify takes target URLs such as webhook://example.com/hook, dingtalk://oapi.dingtalk.com/robot/send?access_token=X, telegram://TOKEN@CHAT or smtp://user:pass@host:587?to=a@b.c. When a change is found it is pushed to those channels, and the send result is reported in notifications. Credentials stay in the URL and are never echoed back.

Prompts

Interactive templates invoked by user choice

NameDescription

No prompts

Resources

Contextual data attached and managed by the client

NameDescription

No resources

TDQS

A3.6/5.0

Scored across 17 tools

Disambiguation4/5

Most tools target a clearly distinct operation: fetch vs crawl vs search vs monitor vs RSS vs cache management. The main overlap cluster is the page-analysis family (web_extract vs web_extract_schema vs metadata_extract), but descriptions explicitly disambiguate them (exact selectors vs schema inference vs metadata dump). recovery_policy and search_health both serve diagnostics, though their scopes are stated as policy table vs backend status.

Naming Consistency3/5

There is a recognizable `web_` prefix convention for the core scraping tools, but eight tools (cache_stats, cache_clear, recovery_policy, broken_links, search_health, metadata_extract, rss_parse, content_quality) drop it. Verb ordering is also mixed between noun_verb (web_search, rss_parse) and noun_noun (cache_stats, content_quality, search_health), so the pattern is readable but not predictable.

Tool Count4/5

At 17 tools this is on the heavier side, but the domain (web fetching, extraction, analysis, monitoring) genuinely spans that many operations and each tool has a defensible role. The analysis/audit cluster (broken_links, content_quality, web_compare_sites, metadata_extract) is where trimming could be argued.

Completeness4/5

The surface covers the full pipeline: search, fetch with continuation, crawl, selector-based and schema-based extraction, metadata, RSS, content analysis, monitoring, plus cache and health introspection. Minor gaps exist (no sitemap/robots introspection tool, no explicit pagination for search beyond what descriptions imply), but agents can work around these.

Maintenance

ActivityActive
ResponsivenessNo recent community issues