# Findle

**Local Web Search + Fetch-as-Markdown over MCP**

A fully local stack that gives **LM Studio**, **Claude Code**, **GSD Pi** and **OpenCode**
two capabilities through MCP (Model Context Protocol), behind a **single endpoint**:

1. **Web search** — via a self-hosted [SearXNG](https://github.com/searxng/searxng) metasearch
   engine (no API key, queries stay local), wrapped by
   [mcp-searxng](https://github.com/ihor-sokoliuk/mcp-searxng).
2. **Fetch a page as clean markdown** — two engines:
   - **`fetch`** (lightweight, static HTML) — [zcaceres/fetch-mcp](https://github.com/zcaceres/fetch-mcp), fast, no browser.
   - **`fetch-js`** (JS-rendering) — [Crawl4AI](https://github.com/unclecode/crawl4ai), Playwright/Chromium, for SPAs.

All three capability servers are aggregated by [mcphub](https://github.com/samanhappy/mcphub)
into one merged MCP endpoint. Everything runs in Docker, bound to `127.0.0.1`. Clients
register **one URL** and see all tools.

## Architecture

``` text
                              ┌────────────────────── Docker (localhost) ──────────────────────┐
                              │                                                                │
 MCP clients                  │   ┌──────────┐                                                 │
 ┌───────────────┐            │   │          │  search    ┌─────────────┐     ┌──────────┐     │
 │ LM Studio     │            │   │          │───────────▶│ mcp-searxng │────▶│ SearXNG  │     │
 │ Claude Code   │            │   │  mcphub  │            └─────────────┘     │ (local)  │     │
 │ GSD Pi        │ HTTP/mcp ──┼──▶│  :8800   │  fetch     ┌─────────────┐     └──────────┘     │
 │ OpenCode      │            │   │  /mcp    │───────────▶│    fetch    │  (static HTML)       │
 └───────────────┘            │   │          │            └─────────────┘                      │
                              │   │          │  fetch-js  ┌─────────────┐                      │
                              │   │          │───────────▶│  crawl4ai   │  (Chromium / JS)     │
                              │   └──────────┘            └─────────────┘                      │
                              │                                                                │
                              └────────────────────────────────────────────────────────────────┘
```

## Endpoint

| What                   | URL                         |
| ---------------------- | --------------------------- |
| **MCP (all tools)**    | `http://localhost:8800/mcp` |
| mcphub admin dashboard | `http://localhost:8800/`    |


## Authentication

The `/mcp` endpoint requires a **bearer token** (`enableBearerAuth: true` in
`mcphub/mcp_settings.json`). Every client must send:

```
Authorization: Bearer <token>
```

The token lives in `mcphub/mcp_settings.json` under `bearerKeys[].token`. That file is
**gitignored** because mcphub also writes other secrets into it at runtime (OAuth client
secrets, the admin password hash). A sanitized `mcphub/mcp_settings.example.json` is
committed as the template — copy it and fill in your own values on first run:

```bash
cp mcphub/mcp_settings.example.json mcphub/mcp_settings.json
# then edit mcp_settings.json: set a random bearerKeys[].token, a UUID id,
# and the bcrypt hash of your admin password
```

Rotate the token by editing `mcphub/mcp_settings.json` (or via the admin dashboard's
**Keys** section) and restarting mcphub (`docker compose restart mcphub`). The admin
dashboard at `http://localhost:8800/` uses the separate `MCPHUB_ADMIN_PASSWORD` from
`.env`, not this token.

## Bring it up

```bash
cp .env.example .env          # then edit .env: set SEARXNG_SECRET, MCPHUB_ADMIN_PASSWORD,
                              #   and CRAWL4AI_API_TOKEN
cp mcphub/mcp_settings.example.json mcphub/mcp_settings.json
                              # then edit it: set bearerKeys[].token, the admin bcrypt hash,
                              #   and the fetch-js Authorization header = your CRAWL4AI_API_TOKEN
docker compose up -d --build  # first run builds the fetch image + pulls Chromium (~1.5GB)
docker compose ps             # all five services should be running
```

## Verify

```bash
# 1. SearXNG JSON API (proves the json format is enabled)
docker compose exec searxng wget -qO- "http://localhost:8080/search?q=test&format=json" | head

# 2. Open the mcphub dashboard at http://localhost:8800/ and confirm all three
#    upstream servers (web-search, fetch, fetch-js) show as connected.

# 3. Inspect the merged endpoint's tools
npx @modelcontextprotocol/inspector
#    then connect to http://localhost:8800/mcp and confirm tools from all three appear.
```

End-to-end: in any client below, ask *"search the web for X and fetch the top result as
markdown."* Use the `fetch` tool for docs/blogs/news and `fetch-js` for JS-heavy SPAs.

## Tests

`tests/` holds a pytest suite (Python 3.10+, needs `pytest` and `mcp`):

```bash
pip install pytest mcp           # or: uv run --with pytest --with mcp python -m pytest tests/ -v
pytest tests/ -v
```

- **`test_fetch_container.py`** — exercises every tool of the `fetch` container in isolation
  (`fetch_html/markdown/txt/json/readable/youtube_transcript`). Self-contained: it starts the
  built `findle-fetch` image on a throwaway port and tears it down. Set
  `FETCH_MCP_URL` to test an already-running endpoint instead.
- **`test_stack.py`** — integration test through the mcphub gateway with bearer auth: checks
  the endpoint rejects unauthenticated calls, that all 17 merged tools are advertised, and that
  web search, static fetch, and JS-rendered fetch each return content. Requires the stack to be
  up; reads the bearer token from `mcphub/mcp_settings.json` (override with `MCPHUB_BEARER_TOKEN`
  / `STACK_MCP_URL`).

The YouTube-transcript test is marked `xfail` (non-strict) — YouTube often blocks datacenter
IPs or a video lacks captions, so it passes whether or not the transcript comes back.

## Client configuration — one URL + bearer token each

Replace `<token>` below with the value of `bearerKeys[].token` from
`mcphub/mcp_settings.json`.

### LM Studio — `%USERPROFILE%\.lmstudio\mcp.json`
```json
{
  "mcpServers": {
    "findle": {
      "url": "http://localhost:8800/mcp",
      "headers": { "Authorization": "Bearer <token>" }
    }
  }
}
```

### Claude Code
```bash
claude mcp add --transport http findle http://localhost:8800/mcp \
  --header "Authorization: Bearer <token>"
```

### OpenCode — `opencode.json`
```json
{
  "mcp": {
    "findle": {
      "type": "remote",
      "url": "http://localhost:8800/mcp",
      "headers": { "Authorization": "Bearer <token>" }
    }
  }
}
```

### GSD Pi — project `.mcp.json` (or local-only `.gsd/mcp.json`)
```json
{
  "mcpServers": {
    "findle": {
      "url": "http://localhost:8800/mcp",
      "headers": { "Authorization": "Bearer <token>" }
    }
  }
}
```

## Security notes

- **Network exposure.** Only mcphub's port is published, on `127.0.0.1` only. The three
  capability servers and SearXNG are reachable only on the internal Docker network. Do
  **not** switch mcphub to `0.0.0.0` without adding `Origin`-header validation
  (DNS-rebinding protection).
- **Non-root.** Every container runs as an unprivileged user (searxng→`977`, mcp-searxng &
  fetch→`1000`, crawl4ai→`999`, mcphub→`nobody`) with `no-new-privileges` set, so a
  container escape doesn't start as root. Consider daemon-level `userns-remap` to also
  unmap container-root from host-root.
- **Gateway auth.** The `/mcp` endpoint requires a bearer token (see *Authentication*).
  OAuth dynamic registration is disabled and client secrets + `state` are required, so the
  endpoint can't be used to self-register clients.
- **crawl4ai (`fetch-js`) is token-gated.** crawl4ai binds loopback-only unless a credential
  is set; `CRAWL4AI_API_TOKEN` both exposes it on the docker network *and* makes it require
  that token on every request. mcphub sends it via the `fetch-js` `headers` entry in
  `mcp_settings.json` — the two values must match. crawl4ai drives a real Chromium against
  arbitrary pages, so this is the highest-risk component; the non-root + token gating limit
  the blast radius.
- **mcp-searxng (`web-search`)** binds loopback by default (`MCP_HTTP_HOST=0.0.0.0` exposes
  it on the docker net). It's unauthenticated like the static `fetch` server — fine while
  internal-only. To require a token, set `MCP_HTTP_HARDEN=true` + `MCP_HTTP_AUTH_TOKEN` +
  `MCP_HTTP_ALLOWED_ORIGINS` and add a matching header to its `mcp_settings.json` entry.
- **SSRF.** The fetch engines can be aimed at internal URLs. zcaceres/fetch-mcp and Crawl4AI
  block private/loopback targets — keep that on. (The official Python `mcp-server-fetch`
  was deliberately **not** used: no SSRF protection, CVE-2025-65513.)
- **Secrets.** `.env` and `mcphub/mcp_settings.json` hold all secrets and are gitignored;
  commit the `*.example` templates instead. Set a real `MCPHUB_ADMIN_PASSWORD` so the
  dashboard isn't left open.

## Notes / things to adjust

- mcphub also exposes per-server paths (`/mcp/web-search`, etc.) and an AI "smart routing"
  endpoint (`/mcp/$smart`, needs an embedding model). We use the merged `/mcp` for all tools.
- The `fetch` image builds zcaceres/fetch-mcp from source and assumes the build output is
  `dist/index.js`. If upstream changes the output path, update `fetch/Dockerfile`.
- SearXNG's request limiter is disabled (single-user local). For heavier use, add a
  `valkey` sidecar and enable the limiter per the SearXNG docs.
- Prefer no gateway? An earlier three-endpoint layout (each server published directly) also
  works; mcphub just collapses it to one URL per client. See git history / the plan file.

## License

This project's own files (compose, Dockerfile, configs, tests, docs) are licensed under the
**MIT License** — see [LICENSE](LICENSE).

## Third-party components

This repo orchestrates third-party services pulled at runtime/build; it does **not** bundle or
redistribute their source. Each keeps its own license:

| Component | License |
| --- | --- |
| [SearXNG](https://github.com/searxng/searxng) | **AGPL-3.0-or-later** |
| [mcp-searxng](https://github.com/ihor-sokoliuk/mcp-searxng) | MIT |
| [Crawl4AI](https://github.com/unclecode/crawl4ai) | Apache-2.0 |
| [zcaceres/fetch-mcp](https://github.com/zcaceres/fetch-mcp) | MIT |
| [supergateway](https://github.com/supercorp-ai/supergateway) | MIT |
| [Bun](https://github.com/oven-sh/bun) (build-time only) | MIT |
| [mcphub](https://github.com/samanhappy/mcphub) | Apache-2.0 |
| [Node.js](https://github.com/nodejs/node) base image | MIT |

**SearXNG is AGPL-3.0.** It runs as the **stock upstream image, unmodified** — only a
`settings.yml` is supplied via volume mount, which is not a derivative of SearXNG's source. This
repo therefore does not inherit AGPL. If you offer this stack as a network service to others,
AGPL entitles those users to SearXNG's corresponding source, which is the public upstream image.
Forking and modifying SearXNG's own code would bring AGPL obligations onto that modified code.

The Python test dependencies (see `requirements.txt`) are all permissive — MIT, BSD, Apache-2.0,
MPL-2.0 (certifi), and PSF.