mcp-ddg-research

Lightweight MCP server for DuckDuckGo HTML search with optional ddgs mode, safe webpage fetching, caching, and clean text extraction.

mcp-ddg-research is a self-hosted Python MCP server that exposes deterministic research primitives to MCP clients. By default it searches DuckDuckGo through DuckDuckGo's lightweight HTML endpoint, fetches webpages with SSRF protections, caches search/fetch responses, deduplicates URLs, and extracts readable text from HTML pages. The optional ddgs provider can be enabled explicitly for DuckDuckGo-backed or broader metasearch behavior.

The MCP client or agent is responsible for reasoning over the returned data. This server only returns structured search results and fetched page text.

What This Project Does

• Searches DuckDuckGo through the DuckDuckGo HTML endpoint by default.

• Can optionally use ddgs.DDGS().text(...) when SEARCH_PROVIDER=ddgs or as a fallback when SEARCH_PROVIDER=auto.

• Parses DuckDuckGo HTML results with BeautifulSoup.

• Resolves DuckDuckGo redirect URLs such as /l/?uddg=....

• Deduplicates normalized result URLs.

• Fetches webpages with strict URL and DNS safety checks.

• Follows redirects manually and validates every redirect target.

• Extracts clean text from HTML by removing script, style, navigation, footer, and similar boilerplate.

• Caches search and fetch responses in a file-based JSON cache.

• Prunes expired, corrupt, temporary, old, and oversized cache files.

• Provides a simple deep search tool that searches once and fetches top result pages concurrently.

Related MCP server: webmcp

What This Project Does Not Do

• No LLM integration.

• No summarization.

• No report generation.

• No browser automation.

• No proxy rotation.

• No captcha bypassing.

• No ranking with model endpoints.

• No OpenAI, Anthropic, Ollama, LM Studio, or other model endpoint support.

Why DuckDuckGo HTML Is Default

The project is DuckDuckGo-focused by default. SEARCH_PROVIDER=duckduckgo_html uses only:

https://html.duckduckgo.com/html/

This keeps default behavior predictable and avoids surprising requests to other upstream search services. The HTML provider uses conservative request defaults, browser-like headers, and BeautifulSoup selectors for .result, .result__a, and .result__snippet.

The ddgs package remains available as an opt-in provider. Current ddgs versions are broader metasearch libraries, not only DuckDuckGo clients. With DDGS_BACKEND=auto, ddgs may query multiple upstream services depending on the installed package, such as Brave, Google, Yahoo, Startpage, Yandex, Mojeek, Wikipedia, or Grokipedia. That can increase coverage, but it can also increase captcha or rate-limit noise. Use it only when broader search is intentional.

Search Provider Modes

Default DuckDuckGo-only mode:

SEARCH_PROVIDER: duckduckgo_html
DDGS_BACKEND: duckduckgo

Behavior:

• Uses only DuckDuckGo's HTML endpoint.

• Does not import or call ddgs.

• Returns "provider": "duckduckgo_html".

• This is the default and recommended mode for predictable DuckDuckGo-focused use.

Optional ddgs mode using the DuckDuckGo backend:

SEARCH_PROVIDER: ddgs
DDGS_BACKEND: duckduckgo

Behavior:

• Uses DDGS().text(...).

• Passes backend="duckduckgo" when supported by the installed ddgs package.

• Returns "provider": "ddgs".

Optional broader metasearch mode:

SEARCH_PROVIDER: ddgs
DDGS_BACKEND: auto

Behavior:

• Uses DDGS().text(...) with backend="auto".

• May query multiple upstream search providers depending on the installed ddgs package.

• Can increase coverage, but may also increase captcha/rate-limit noise.

• This mode is always opt-in.

Auto fallback mode:

SEARCH_PROVIDER: auto
DDGS_BACKEND: duckduckgo

Behavior:

• Tries DuckDuckGo HTML first.

• Falls back to ddgs only if DuckDuckGo HTML fails, times out, raises, or returns no results.

• The response provider reports the provider that returned results.

Available MCP Tools

ddg_search

Search DuckDuckGo and return structured results.

Arguments:

{
  "query": "python mcp server fastmcp",
  "max_results": 10,
  "search_window": null,
  "safe_search": "off",
  "time_filter": "month",
  "blocked_domains": [],
  "allowed_domains": [],
  "preferred_domains": []
}

Argument rules:

• query: string, required.

• max_results: integer, default 10, minimum 1, maximum 30.

• search_window: optional integer, minimum 1, maximum 100. If provided, this is the internal search result window, meaning the provider result count requested before dedupe/domain controls/final cap. It is not a time window.

• safe_search: one of off, moderate, strict, default off.

• time_filter: optional, one of day, week, month, year.

• blocked_domains: optional list of domains to remove from results, default [].

• allowed_domains: optional list of domains to keep, default [].

• preferred_domains: optional list of domains to move earlier while preserving stable order, default [].

Response example:

{
  "query": "python mcp server fastmcp",
  "provider": "duckduckgo_html",
  "results": [
    {
      "title": "MCP Python SDK",
      "url": "https://github.com/modelcontextprotocol/python-sdk",
      "snippet": "Python SDK for Model Context Protocol servers and clients."
    }
  ],
  "cached": false,
  "error": null
}

web_fetch

Fetch a single webpage and return clean text.

Arguments:

{
  "url": "https://example.com/article",
  "max_chars": 12000
}

Argument rules:

• url: HTTP or HTTPS URL.

• max_chars: integer, default 12000, minimum 1000, maximum 50000.

Response example:

{
  "url": "https://example.com/article",
  "final_url": "https://example.com/article",
  "title": "Example Article",
  "content": "Readable extracted page text...",
  "content_type": "text/html; charset=utf-8",
  "cached": false,
  "success": true,
  "error": null
}

ddg_deep_search

Search once, fetch top result pages concurrently, and return sources plus page content.

Arguments:

{
  "query": "model context protocol python sdk",
  "max_results": 10,
  "search_window": null,
  "max_pages": 5,
  "max_chars_per_page": 12000,
  "safe_search": "off",
  "time_filter": "year",
  "blocked_domains": [],
  "allowed_domains": [],
  "preferred_domains": [],
  "max_concurrency": null
}

Argument rules:

• query: string, required.

• max_results: integer, default 10, minimum 1, maximum 30.

• search_window: optional integer, minimum 1, maximum 100. Passed through to ddg_search as the internal search result window before final result capping. It is not a time window.

• max_pages: integer, default 5, minimum 1, maximum 10.

• max_chars_per_page: integer, default 12000, minimum 1000, maximum 50000.

• safe_search: one of off, moderate, strict, default off.

• time_filter: optional, one of day, week, month, year.

• blocked_domains: optional list of domains to remove from search results before fetching, default [].

• allowed_domains: optional list of domains to keep before fetching, default [].

• preferred_domains: optional list of domains to move earlier before fetching, default [].

• max_concurrency: optional per-call page fetch concurrency, minimum 1, maximum 12. If omitted, MAX_CONCURRENCY is used.

Response example:

{
  "query": "model context protocol python sdk",
  "search_provider": "duckduckgo_html",
  "sources": [
    {
      "title": "MCP Python SDK",
      "url": "https://github.com/modelcontextprotocol/python-sdk",
      "snippet": "Python SDK for Model Context Protocol servers and clients."
    }
  ],
  "pages": [
    {
      "title": "MCP Python SDK",
      "url": "https://github.com/modelcontextprotocol/python-sdk",
      "final_url": "https://github.com/modelcontextprotocol/python-sdk",
      "content": "Extracted page text..."
    }
  ],
  "failed_pages": [],
  "cached": false
}

cache_stats

Return current cache file counts and byte totals for each cache namespace.

Arguments: none.

Response example:

{
  "cache_dir": "/data/cache",
  "namespaces": {
    "search": {"files": 10, "bytes": 12345},
    "fetch": {"files": 20, "bytes": 98765}
  },
  "total_files": 30,
  "total_bytes": 111110
}

cache_prune

Manually run cache pruning using the same rules as automatic pruning.

Arguments:

{
  "expired_only": false,
  "dry_run": false
}

Argument rules:

• expired_only: boolean, default false. When true, deletes expired, corrupt, and temporary files but skips size-limit pruning.

• dry_run: boolean, default false. When true, reports files that would be deleted without deleting them.

Response example:

{
  "deleted_files": 5,
  "deleted_bytes": 123456,
  "remaining_files": 25,
  "remaining_bytes": 99999,
  "dry_run": false
}

cache_clear

Clear one cache namespace, or all cache namespaces, when explicitly confirmed.

Arguments:

{
  "namespace": "fetch",
  "confirm": true
}

Argument rules:

• namespace: one of search, fetch, or all.

• confirm: must be true to delete files.

Response example:

{
  "namespace": "fetch",
  "deleted_files": 20,
  "deleted_bytes": 98765
}

Domain Controls

Domain controls are opt-in. If you do not pass blocked_domains, allowed_domains, preferred_domains, or search_window, ddg_search requests exactly max_results from DuckDuckGo and preserves DuckDuckGo's default ranking order after URL deduplication. The server does not apply a built-in source bias, source boost, or domain blocklist.

When any domain control is provided, the server requests a larger internal search result window from the provider before applying dedupe and domain controls. The default provider result window is:

min(max_results * 3, 50)

The final response is still capped to max_results. You can override the provider result window with search_window, minimum 1, maximum 100. This is a count of search results requested from DuckDuckGo, not a recency or day range. It is useful when a desired allowed/preferred domain might appear outside the first max_results provider results.

Domain inputs are normalized by lowercasing, removing URL schemes, removing paths and query strings, and stripping a leading www.. Matching supports exact domains and subdomains. For example, docs.example.com matches example.com, but example.com.evil.com does not.

Filtering order:

1. Apply allowed_domains if provided.

2. Apply blocked_domains if provided.

3. Apply preferred_domains if provided.

preferred_domains performs a stable partition: preferred matches move earlier, relative order is preserved inside the preferred and non-preferred groups, and no numeric score is invented.

Block domains:

{
  "query": "self hosted photo backup",
  "blocked_domains": ["example.com", "old-docs.example.org"]
}

Allow only specific domains:

{
  "query": "python mcp server",
  "allowed_domains": ["github.com", "modelcontextprotocol.io"]
}

Prefer domains without excluding others:

{
  "query": "duckduckgo html search endpoint",
  "preferred_domains": ["duckduckgo.com", "github.com"]
}

Search a larger internal search result window before applying domain controls:

{
  "query": "python mcp server",
  "max_results": 10,
  "search_window": 40,
  "allowed_domains": ["github.com", "modelcontextprotocol.io"]
}

Deep search with the same provider result window behavior:

{
  "query": "model context protocol python sdk",
  "max_results": 10,
  "max_pages": 5,
  "search_window": 40,
  "preferred_domains": ["github.com"]
}

Limit deep-search fetch concurrency for one call:

{
  "query": "model context protocol python sdk",
  "max_pages": 5,
  "max_concurrency": 2
}

Docker Stdio Usage

Build the local image:

docker build -t mcp-ddg-research:local .

Run the server over stdio. This mode is auth-free because the MCP client owns stdin/stdout and there is no listening network socket:

docker run --rm -i -v "$PWD/data:/data" mcp-ddg-research:local

Docker Stdio MCP Client Configuration

{
  "mcpServers": {
    "ddg-research": {
      "command": "docker",
      "args": [
        "run",
        "--rm",
        "-i",
        "-v",
        "/opt/mcp-ddg-research/data:/data",
        "mcp-ddg-research:local"
      ]
    }
  }
}

docker-compose Usage

The included compose file starts the server in streamable HTTP mode on /mcp. It maps host port 49317 to container port 8000 and requires Authorization: Bearer change-me-now by default.

Build and start the service:

docker compose up --build ddg-research

The compose file persists cache data at:

~/docker/docker-data/mcp-ddg-research/cache

It also enables conservative cache pruning defaults:

CACHE_PRUNE_ON_START: "true"
CACHE_PRUNE_INTERVAL_SECONDS: "3600"
CACHE_MAX_AGE_SECONDS: "604800"
CACHE_MAX_SIZE_MB: "512"

The compose file keeps search DuckDuckGo-only by default:

SEARCH_PROVIDER: duckduckgo_html
DDGS_BACKEND: duckduckgo

The checked-in compose token is the placeholder change-me-now. It is acceptable for local smoke tests only. Replace MCP_AUTH_TOKEN in docker-compose.yml before using LAN, VPN, reverse-proxy, or Cloudflare Tunnel deployments.

The server defaults MCP_ALLOWED_HOSTS=* and MCP_ALLOWED_ORIGINS=* so the same container can run behind a LAN IP, hostname, domain, reverse proxy, or HTTPS endpoint. The compose file documents these optional variables as commented examples. In MCP SDK 1.27.2, wildcard Host/Origin validation is not supported by the DNS rebinding middleware, so wildcard mode disables the SDK Host/Origin allowlist and relies on the bearer token. To enable strict Host/Origin checks, uncomment the variables in docker-compose.yml and set exact comma-separated values such as:

MCP_ALLOWED_HOSTS="example.com,example.com:443,localhost:49317"
MCP_ALLOWED_ORIGINS="https://example.com,http://localhost:*"

LAN HTTP Example

Set a real token in docker-compose.yml and start the server:

docker compose up -d --build

Use your server's LAN IP in the client URL:

http://YOUR_SERVER_IP:49317/mcp

OpenCode remote MCP configuration for a LAN deployment:

{
  "mcp": {
    "ddg-research": {
      "type": "remote",
      "enabled": true,
      "url": "http://YOUR_SERVER_IP:49317/mcp",
      "oauth": false,
      "headers": {
        "Authorization": "Bearer change-me-now"
      }
    }
  }
}

HTTPS Reverse Proxy Example

Run the container on the server and terminate TLS in a reverse proxy. The proxy should forward /mcp to http://127.0.0.1:49317/mcp and preserve standard upgrade/streaming behavior.

Minimal Nginx-style location:

location /mcp {
    proxy_pass http://127.0.0.1:49317/mcp;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_buffering off;
}

OpenCode configuration for the HTTPS endpoint:

{
  "mcp": {
    "ddg-research": {
      "type": "remote",
      "enabled": true,
      "url": "https://your-domain.example/mcp",
      "oauth": false,
      "headers": {
        "Authorization": "Bearer change-me-now"
      }
    }
  }
}

Cloudflare Tunnel Example

Cloudflare Tunnel lets cloudflared make outbound-only connections from your server to Cloudflare, so you can publish the MCP HTTP endpoint without opening an inbound router/firewall port.

In the Cloudflare dashboard, create a tunnel and add a public hostname such as:

https://mcp.example.com

If cloudflared runs on the host, set the tunnel service URL to:

http://127.0.0.1:49317

If cloudflared runs as another service in the same compose project/network, set the tunnel service URL to the container service name and internal port:

http://ddg-research:8000

Minimal compose service example for token-managed tunnels:

cloudflared:
  image: cloudflare/cloudflared:latest
  restart: unless-stopped
  command: tunnel --no-autoupdate run --token ${CLOUDFLARE_TUNNEL_TOKEN}
  depends_on:
    - ddg-research

Keep CLOUDFLARE_TUNNEL_TOKEN outside version control. In OpenCode, use the public HTTPS URL and keep the MCP bearer token header:

{
  "mcp": {
    "ddg-research": {
      "type": "remote",
      "enabled": true,
      "url": "https://mcp.example.com/mcp",
      "oauth": false,
      "headers": {
        "Authorization": "Bearer change-me-now"
      }
    }
  }
}

For production, replace change-me-now with a long random token. Cloudflare Tunnel protects the network path, but the MCP server should still require its own bearer token.

Do not expose HTTP mode to an untrusted network without HTTPS and a strong MCP_AUTH_TOKEN. If MCP_AUTH_TOKEN is unset in HTTP mode, the server logs a warning and accepts unauthenticated HTTP requests.

For MCP stdio clients, direct docker run -i is usually simpler than compose because the client owns stdin/stdout.

HTTP Smoke Tests

Raw curl is useful for checking HTTP authentication and Host handling, but it does not perform a complete MCP streamable HTTP session. A request with the correct bearer token may therefore return 406 Not Acceptable because curl did not send the MCP client's expected Accept: text/event-stream negotiation headers. That still proves the request passed bearer-token auth and Host validation.

With the compose server running and the default compose token:

curl -i http://127.0.0.1:49317/mcp

Expected: 401 Unauthorized.

curl -i \
  -H "Host: YOUR_SERVER_IP:49317" \
  -H "Authorization: Bearer change-me-now" \
  http://127.0.0.1:49317/mcp

Expected: usually 406 Not Acceptable from raw curl, but not 401 Unauthorized and not 421 Misdirected Request.

With a real MCP client, such as OpenCode configured with the same URL and Authorization header, ListTools and CallTool should work for ddg_search, web_fetch, ddg_deep_search, cache_stats, cache_prune, and cache_clear.

Environment Variables

Cache Behavior

Search results are cached under the search cache namespace. Fetch responses are cached under the fetch cache namespace. Cache keys are SHA256 hashes of stable JSON payloads, so equivalent tool arguments map to the same file path.

Cache files are written atomically by writing a temporary file in the target cache directory and then renaming it into place. Corrupt, malformed, or expired cache files are ignored safely.

TTL checks on read prevent stale cache values from being returned, but TTL on read alone does not remove old files from disk. Long-running deployments can therefore accumulate expired cache files unless pruning deletes them. This server prunes cache files on startup when CACHE_PRUNE_ON_START=true and also opportunistically during cache access/write, throttled by CACHE_PRUNE_INTERVAL_SECONDS.

Pruning covers both known cache namespaces:

• search

• fetch

Pruning deletes:

• corrupt or malformed .json cache files

• leftover .tmp files from interrupted atomic writes

• files expired by DDG_CACHE_TTL_SECONDS or FETCH_CACHE_TTL_SECONDS

• files older than CACHE_MAX_AGE_SECONDS, when set

• oldest cache files first when total cache size exceeds CACHE_MAX_SIZE_MB

If CACHE_MAX_AGE_SECONDS is unset, invalid, or 0, max-age pruning is disabled. If CACHE_MAX_SIZE_MB is unset, invalid, or 0, size-based pruning is disabled. Pruning errors are logged and do not fail search or fetch requests.

Manual cache tools are available:

• cache_stats: inspect current file counts and byte totals.

• cache_prune: run pruning manually, with optional dry_run.

• cache_clear: clear search, fetch, or all after confirm=true.

The default compose configuration persists cache files in /data/cache, with ~/docker/docker-data/mcp-ddg-research mounted into the container.

Rate Limit Notes

Defaults are intentionally conservative:

• ddg_search defaults to 10 results and caps at 30.

• ddg_deep_search defaults to 5 fetched pages and caps at 10.

• Deep search concurrency defaults to 5.

• Search and fetch results are cached to reduce repeated DuckDuckGo and website hits.

• Search defaults to DuckDuckGo HTML only. DDGS_BACKEND=auto is broader metasearch and can generate requests to additional upstream services.

This project does not rotate proxies, bypass captchas, or attempt to evade rate limits. If DuckDuckGo blocks or rate limits requests, the tool returns structured errors instead of retrying aggressively.

SSRF and Security Protections

web_fetch only allows http and https URLs. It blocks known local or internal hostnames, including:

• localhost

• metadata

• metadata.google.internal

• hostnames ending in .local, .localhost, .internal, .lan, .intranet

It also rejects IP addresses in private, loopback, link-local, reserved, multicast, or unspecified ranges, including:

• 0.0.0.0/8

• 10.0.0.0/8

• 127.0.0.0/8

• 169.254.0.0/16

• 172.16.0.0/12

• 192.168.0.0/16

• ::1/128

• fc00::/7

• fe80::/10

DNS is resolved before fetching. If any resolved address is unsafe, the request is rejected. Redirects are followed manually, and every redirect target is validated before the next request.

Unsupported schemes such as file://, ftp://, ssh://, gopher://, and data: are never fetched.

Development Setup

Python 3.12 is required.

Create and activate a virtual environment:

python3.12 -m venv .venv
source .venv/bin/activate

Install the package with development tools:

python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

Run the MCP server locally:

python -m mcp_ddg_research.server

Test Commands

Run tests:

python -m pytest

Run lint:

python -m ruff check .

Build a wheel/sdist using the configured build backend:

python -m pip install build
python -m build

Release Automation

Releases are automated by .github/workflows/release.yml when commits or release tags are pushed. The workflow is Python-native:

1. Install the project with development dependencies.

2. Run Ruff, pytest, compile checks, Python package build, and a Docker build.

3. On main branch pushes, use Python Semantic Release to create the next GitHub release from conventional commits.

4. On v* tag pushes, treat the pushed tag as the release tag.

5. If a release or release tag is present, build and push multi-architecture Docker images for linux/amd64 and linux/arm64.

The workflow publishes these image tags:

DOCKERHUB_USERNAME/mcp-ddg-research:latest
DOCKERHUB_USERNAME/mcp-ddg-research:vX.Y.Z
ghcr.io/isyuricunha/mcp-ddg-research:latest
ghcr.io/isyuricunha/mcp-ddg-research:vX.Y.Z

Required repository secrets:

Use conventional commits to drive release versions:

• fix: ... and perf: ... create patch releases.

• feat: ... creates minor releases while the project is in 0.x.

• Breaking changes are capped to a minor release while the project is in 0.x; after 1.0.0, they create major releases.

• docs:, ci:, chore:, test:, style:, and refactor: do not create a release by default.

The release workflow updates pyproject.toml and src/mcp_ddg_research/__init__.py during semantic-release commits. It does not maintain a changelog file. It is intentionally skipped for documentation-only pushes and compose-file-only pushes.

Manual milestone releases are also supported. Create and push a vX.Y.Z tag that points at the intended release commit, and the tag workflow publishes the same Docker Hub and GHCR tags.

Limitations

• DuckDuckGo HTML mode does not support every option exposed by DuckDuckGo's full web interface.

• time_filter is applied to the ddgs provider. DuckDuckGo HTML mode only sends the query and safe-search parameter.

• PDF parsing is not implemented in v1.

• JavaScript-rendered pages are not rendered because there is no browser automation.

• Some websites block automated HTTP clients or return incomplete content.

• DNS safety checks reduce SSRF risk but cannot make arbitrary third-party fetching risk-free.

Optional Future Roadmap

These are optional future improvements, not current behavior:

• Add configurable per-domain fetch throttling.

• Add optional robots.txt awareness.

• Add additional text extraction heuristics for common article layouts.

• Add more integration tests around redirect chains and text content types.