Skip to main content
Glama
ptrken01

searxng-mcp-scraper

by ptrken01

Server Configuration

Describes the environment variables required to run the server.

NameRequiredDescriptionDefault
MCP_HOSTNoBind host when --transport streamable-http127.0.0.1
MCP_PORTNoBind port when --transport streamable-http8765
LOG_LEVELNoStderr log verbosity: DEBUG/INFO/WARNING/ERRORINFO
USER_AGENTNoUser-Agent header for all requestssearxng-mcp-scraper/0.1
FETCH_MAX_BYTESNoRefuse to read more than this many bytes per fetch5000000
FETCH_TIMEOUT_SNoHTTP timeout (seconds) for fetch20
MCP_CORS_ORIGINSNoComma-separated CORS allow-origins. Use * to allow any origin (not recommended on LAN).http://localhost,http://localhost:*,http://127.0.0.1,http://127.0.0.1:*
SEARCH_TIMEOUT_SNoHTTP timeout (seconds) for search30
SEARXNG_BASE_URLYesBase URL of your SearXNG instance, e.g. http://localhost:8888
FIRECRAWL_API_KEYNoOptional. Empty uses the keyless free tier
FIRECRAWL_BASE_URLNoFirecrawl API base — the local keyless proxy by defaulthttp://127.0.0.1:8788
FETCH_ALLOW_PRIVATENoAllow fetching private/loopback URLs (SSRF guard, default on)false
FETCH_MAX_REDIRECTSNoMax HTTP redirects to follow5
FIRECRAWL_TIMEOUT_SNoTimeout for a fallback scrape (Firecrawl renders before responding)60
BLOG_SCRAPE_MAX_POSTSNoMax post pages per blog scrape200
BLOG_SCRAPE_OUTPUT_DIRNoDirectory for scrape_blog / deep_scrape Markdown output and assets~/scraped-blogs
SEARCH_MAX_RESULTS_CAPNoHard cap on results per call50
BLOG_SCRAPE_CONCURRENCYNoMax parallel post fetches5
DEEP_SCRAPE_CONCURRENCYNoMax parallel document/image downloads during deep_scrape5
SEARXNG_DEFAULT_ENGINESNoDefault engine list when a caller doesn't pass engines. Curated to the keyless engines that return results from this machine's egress IP (Brave/Google/Startpage/DuckDuckGo are rate-limited here). Set to empty to use SearXNG's own default set.wiby,naver,seznam,marginalia,wikipedia,duckduckgo_html
FIRECRAWL_MIN_TEXT_CHARSNoStatic extractions shorter than this are treated as failures and retried500
SEARXNG_DEFAULT_CATEGORYNoDefault search categorygeneral
SEARXNG_DEFAULT_LANGUAGENoDefault language codeen
FIRECRAWL_FALLBACK_ENABLEDNoRetry a failed/thin static fetch through Firecrawl (see below)true
SEARXNG_DEFAULT_SAFESEARCHNoDefault safesearch: 0/1/20
BLOG_SCRAPE_DISCOVERY_PAGESNoMax HTML index pages to scan when no feed is found10
DEEP_SCRAPE_MAX_IMAGE_BYTESNoRefuse a single image above this size15000000
DEEP_SCRAPE_MAX_DOCUMENT_BYTESNoRefuse a single linked document above this size50000000
DEEP_SCRAPE_MAX_IMAGES_PER_BLOGNoMax images downloaded by one deep_scrape call200
DEEP_SCRAPE_MAX_DOCUMENTS_PER_BLOGNoMax linked documents downloaded/extracted by one deep_scrape call100

Capabilities

Features and capabilities supported by this server

CapabilityDetails
tools
{
  "listChanged": false
}
prompts
{
  "listChanged": false
}
resources
{
  "subscribe": false,
  "listChanged": false
}
experimental
{}

Tools

Functions exposed to the LLM to take actions

NameDescription
searchA

Search the web via the configured SearXNG instance.

Args:
    query: The search query string. Required.
    categories: SearXNG category list, e.g. ["general"], ["images"].
        Defaults to the server's `SEARXNG_DEFAULT_CATEGORY`.
    engines: Restrict to specific engines, e.g. ["google", "bing"].
    language: Language code, e.g. "en", "de". Defaults to server setting.
    pageno: Page number (1-indexed). Default 1.
    time_range: One of "day", "month", "year", or None.
    safesearch: 0 (none), 1 (moderate), 2 (strict). Defaults to server setting.
    max_results: Cap on results returned (1-50). Default 10.

Returns:
    Dict with `results` (list of {title, url, snippet, engine, score}),
    `suggestions`, `number_of_results`, and on error an `error` field.
fetchA

Fetch a URL and return its cleaned main text content.

Args:
    url: An http:// or https:// URL.

Returns:
    Dict with `url`, `final_url`, `content_type`, `text`, `byte_count`,
    and on error an `error` field with a stable string code.
scrape_blogA

Discover every post on a blog and write a single aggregated Markdown file.

Tries the blog's RSS/Atom feed first (common paths like /rss.xml,
/feed, /atom.xml). If no feed responds, crawls the HTML index
(up to blog_scrape_discovery_pages pages). Each discovered post
URL is fetched in parallel and its cleaned main text is appended
to one Markdown file at:

  {blog_scrape_output_dir}/<safe-host>_<safe-path>_<unix-ts>.md

The file starts with YAML front matter (blog URL, post count,
byte total, duration) followed by one H3 section per post with
its title, original URL, published date (if known from the feed),
content-type, byte count, and the cleaned text. Failures are
noted in-band as `- Fetch error: <code>` bullets, not by aborting
the whole run.

Args:
    blog_url: Root URL of the blog (e.g. "https://blog.example.com/").

Returns:
    Small summary dict — never the file contents:
      {blog_url, output_path, post_count, post_count_ok, byte_count,
       duration_s, discovery}
    On failure: {blog_url, error: <stable_code>, message}.
deep_scrapeA

Discover every post on a blog, fetch each, AND download + extract every linked document (PDF/DOCX/XLSX/PPTX/CSV/JSON/XML/MD/EPUB/...).

Same discovery as `scrape_blog` (RSS/Atom first, HTML index
fallback). For every post fetched, the raw HTML is scanned for
links whose URL ends in a recognized document extension. Each
such document is downloaded (capped by
`deep_scrape_max_documents_per_blog`, 100 by default) and its
text is extracted via markitdown. The original blog post text
and every document's extracted text are all inlined into one
Markdown file at:

  {blog_scrape_output_dir}/<safe-host>_<safe-path>_<unix-ts>_deep.md

Downloaded binaries are saved alongside, under:

  {blog_scrape_output_dir}/deep_assets/

Per-post and per-document failures are noted in-band as
`- Fetch error: <code>` / `- Error: <code>` bullets, not by
aborting the whole run. The returned summary dict only carries
an `error` field when discovery itself failed (no posts).

Args:
    blog_url: Root URL of the blog (e.g. "https://blog.example.com/").

Returns:
    Small summary dict — never the file contents:
      {blog_url, output_path, assets_dir, post_count, post_count_ok,
       documents_found, documents_extracted, post_byte_count,
       document_byte_count, duration_s, discovery}
    On failure: {blog_url, error: <stable_code>, message}.

Prompts

Interactive templates invoked by user choice

NameDescription

No prompts

Resources

Contextual data attached and managed by the client

NameDescription

No resources

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/ptrken01/searxng-mcp-scraper'

If you have feedback or need assistance with the MCP directory API, please join our Discord server