# MCP Local FastMCP Server

A lightweight, Dockerized [FastMCP](https://gofastmcp.com/) server that exposes web search, scraping, weather, market-data, news, road-condition, and placeholder mail tools through a single MCP Streamable HTTP endpoint.

The stack keeps the repository's existing Docker and GitHub Actions deployment/rollback workflow while replacing the original demo web application with a Python MCP service. SearXNG runs as the primary self-hosted search provider, optional external providers can extend or back up selected tools, and scraped references can be persisted in a Docker volume.

## Highlights

- One MCP endpoint: `/mcp`
- Streamable HTTP with JSON-RPC 2.0 and JSON/SSE response support
- Self-hosted SearXNG as the primary search provider
- Optional Tavily fallback for web search
- Static and JavaScript-assisted web scraping
- Open-Meteo and U.S. National Weather Service weather support
- Stock quote, stock news, movement-explanation, and general-news tools
- Road-condition research with official-source preference
- JSONL reference persistence
- URL-safety controls for private and local network targets
- Single-file Postman collection with strict functional, quality, boundary, and degradation tests
- Existing local deployment, GitHub Actions, backup, and rollback workflow retained

## Architecture

```text
MCP client / Postman
        |
        | JSON-RPC 2.0 over Streamable HTTP
        v
https://mcp.home.arpa/mcp          Optional Caddy/TLS endpoint
        |
        v
http://127.0.0.1:8002/mcp         Host-bound Docker port
        |
        v
FastMCP application :8000
        |
        +--> SearXNG container     Primary web search
        +--> Tavily                Optional search fallback
        +--> Web pages             Static or JS-assisted scraping
        +--> Open-Meteo / NWS      Weather
        +--> Market/news providers Optional API-key integrations
        +--> /data/references.jsonl
                 |
                 v
             mcp-data volume
```

The application is configured for stateless Streamable HTTP. Clients should still perform the standard MCP initialization sequence and should preserve an `Mcp-Session-Id` response header when the server returns one.

## Available MCP tools

The server currently registers 15 tools.

### Runtime and health

| Tool | Purpose |
|---|---|
| `health_check` | Returns service status, active provider configuration, limits, and supported free-API capabilities. |

### Search and scraping

| Tool | Purpose |
|---|---|
| `web_search` | Searches through SearXNG, with optional Tavily fallback. Supports language, category, recency, and result-count controls. |
| `web_search_and_scrape` | Searches and then extracts content from selected results, with optional official-source preference and image extraction. |
| `scrape_url` | Extracts readable content and metadata from a specific public URL. |
| `extract_image_urls` | Extracts candidate image URLs and related metadata from a public page. |

### Weather, markets, news, and roads

| Tool | Purpose |
|---|---|
| `weather_lookup` | Resolves a location and returns current conditions and forecast data. Uses Open-Meteo generally and can use NWS data for U.S. locations. |
| `stock_quote` | Returns a stock quote through configured providers, with provider fallback behavior. |
| `stock_news` | Returns recent news associated with a stock symbol. |
| `explain_stock_move` | Combines quote and news evidence to summarize plausible reasons for a stock movement. It is informational, not financial advice. |
| `news_search` | Searches general news through configured news APIs, with search fallback where supported. |
| `road_condition_search` | Builds a current road-condition query and prioritizes official transportation, police, and public-agency sources. |

### Mail abstraction placeholders

| Tool | Purpose |
|---|---|
| `mail_search` | Placeholder contract for a future mail-provider search implementation. |
| `mail_read` | Placeholder contract for reading a message. |
| `mail_create_draft` | Placeholder contract for creating a draft. |
| `mail_send_draft` | Placeholder send contract requiring explicit confirmation; no real mail backend is currently connected. |

> The four mail tools intentionally expose stable MCP contracts but do not currently connect to Gmail, Microsoft Graph, IMAP, SMTP, or another production mail provider.

## Requirements

- Docker Engine
- Docker Compose v2 (`docker compose`)
- Outbound HTTPS access for search, scraping, weather, market, and news providers
- Optional: Postman desktop/web app or Newman for collection execution
- Optional: Caddy or another reverse proxy for HTTPS and external access

## Quick start

### 1. Clone the repository

```bash
git clone https://github.com/appNucleus/mcp.local.git
cd mcp.local
```

To test the branch containing the Postman collection before it is merged:

```bash
git switch postman
```

### 2. Create the runtime environment file

```bash
cp .env.example .env
```

Review `.env` before starting the stack. Replace optional provider values with your own credentials or leave optional integrations disabled.

**Never commit `.env` or real credentials.** The committed `.env.example` must contain placeholders only.

### 3. Start the stack

```bash
docker compose --env-file .env up --build -d
```

### 4. Verify containers and logs

```bash
docker compose ps
docker compose logs -f hello
```

The default host endpoint is:

```text
http://127.0.0.1:8002/mcp
```

Stop the stack with:

```bash
docker compose --env-file .env down
```

## MCP request format

The server uses JSON-RPC 2.0 over a single Streamable HTTP endpoint.

Required request headers:

```http
Content-Type: application/json
Accept: application/json, text/event-stream
MCP-Protocol-Version: 2025-06-18
```

Initialize the MCP connection:

```bash
curl -i \
  -X POST http://127.0.0.1:8002/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -H 'MCP-Protocol-Version: 2025-06-18' \
  --data-raw '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "initialize",
    "params": {
      "protocolVersion": "2025-06-18",
      "capabilities": {},
      "clientInfo": {
        "name": "curl-mcp-client",
        "version": "1.0.0"
      }
    }
  }'
```

If the response includes `Mcp-Session-Id`, send that value in later requests:

```http
Mcp-Session-Id: <returned-session-id>
```

List registered tools:

```bash
curl -sS \
  -X POST http://127.0.0.1:8002/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -H 'MCP-Protocol-Version: 2025-06-18' \
  --data-raw '{
    "jsonrpc": "2.0",
    "id": 2,
    "method": "tools/list",
    "params": {}
  }'
```

Add the `Mcp-Session-Id` header to the command when the initialization response supplied one.

## Configuration

Configuration is loaded from environment variables. Keep environment-specific values in `.env`; keep `.env.example` limited to safe placeholders and documentation defaults.

### Core server

| Variable | Purpose | Typical value |
|---|---|---|
| `HOST_BIND` | Host interface exposed by Docker. | `127.0.0.1` |
| `APP_PORT` | Host port mapped to container port `8000`. | `8002` |
| `MCP_NAME` | MCP server name reported to clients. | `local-tools` |
| `MCP_TRANSPORT` | FastMCP transport mode. | `streamable-http` |
| `MCP_HOST` | Bind address inside the application container. | `0.0.0.0` |
| `MCP_PORT` | Internal FastMCP port. | `8000` |
| `MCP_PATH` | Streamable HTTP endpoint path. | `/mcp` |

### Search and scraping configuration

| Variable | Purpose |
|---|---|
| `SEARXNG_URL` | Internal or external SearXNG base URL. |
| `TAVILY_API_KEY` | Optional Tavily fallback credential. |
| `MAX_SEARCH_RESULTS` | Upper bound for returned search results. |
| `MAX_SCRAPE_PAGES_PER_CALL` | Maximum pages processed by combined search-and-scrape requests. |
| `MAX_CHARS_PER_PAGE` | Maximum extracted characters retained per page. |
| `MAX_IMAGES_PER_PAGE` | Maximum image candidates returned. |
| `MAX_CONCURRENT_SCRAPES` | Maximum concurrent page extraction operations. |
| `JS_RENDER_MODE` | JavaScript rendering policy, such as `auto`, `always`, or `never`. |
| `ALLOW_PRIVATE_IP_URLS` | Controls whether scraping can target private, loopback, or local-network addresses. Keep disabled for untrusted callers. |

### Weather, stock, and news providers

Some capabilities work without an API key; others improve when optional provider credentials are configured.

| Variable | Integration |
|---|---|
| `ALPHA_VANTAGE_API_KEY` | Optional stock quote/news provider. |
| `FINNHUB_API_KEY` | Optional stock quote/news provider. |
| `NEWSAPI_API_KEY` | Optional general or stock-news provider. |
| `GNEWS_API_KEY` | Optional general or stock-news provider. |
| `USER_AGENT` | Identifies the application to web and public-API services, including NWS requests. |

Open-Meteo and supported Yahoo endpoints can provide no-key fallback behavior. Provider availability, quotas, response latency, and terms remain external dependencies.

### Reference persistence

| Variable | Purpose |
|---|---|
| `REFERENCE_STORE_BACKEND` | Reference persistence mode, such as JSONL or disabled/no-op. |
| `REFERENCE_STORE_JSONL_PATH` | JSONL file path inside the container. |

The standard container path is:

```text
/data/references.jsonl
```

It is backed by the named Docker volume:

```text
mcp-data
```

## Postman collection

The `postman` branch adds a single-file strict-quality collection:

```text
docs/postman_collection/mcp-home-arpa-single-file.postman_collection.json
```

The collection is self-contained and does not require a separate Postman environment file. It includes its endpoint, MCP protocol version, session state, sample symbols, locations, road name, response scratch variables, and collection-level header logic.

### Import and run in Postman

1. Open Postman and select **Import**.
2. Import `docs/postman_collection/mcp-home-arpa-single-file.postman_collection.json`.
3. Open the collection's **Variables** tab.
4. Set `mcp_url` to the endpoint being tested:

   Local Docker endpoint:

   ```text
   http://127.0.0.1:8002/mcp
   ```

   Reverse-proxied endpoint:

   ```text
   https://mcp.home.arpa/mcp
   ```

5. Keep `mcp_protocol_version` at `2025-06-18` unless the deployed server and collection are deliberately upgraded together.
6. Run folders in numeric order, beginning with folder `00`.

The collection automatically sends the MCP content-negotiation headers and reuses `Mcp-Session-Id` when initialization returns one. Its response parser supports both plain JSON and JSON carried in SSE `data:` frames.

### Test-suite organization

| Folder | Priority and coverage |
|---|---|
| `00` | P0 handshake, initialized notification, tool inventory, and runtime health/configuration. |
| `01` | P0 web-search relevance, language, recency, category handling, official-source pressure, Unicode, and bounds. |
| `02` | P0 search-plus-scrape extraction, official-document preference, images, multilingual content, rendering fallback, and limits. |
| `03` | Direct scraping, image extraction, text limits, and unsafe URL-scheme/private-target rejection. |
| `04` | Weather behavior for U.S. and global locations, forecast bounds, and invalid-location handling. |
| `05` | P0 stock quotes, dotted symbols, stock news, stock-movement explanations, invalid symbols, general news, and item limits. |
| `06` | P1 road-condition query quality, official-source preference, future-commute wording, and page limits. |
| `07` | P2 abstract mail placeholder contracts. |
| `08` | P2 extreme input, normalization, and graceful-degradation cases. |

A final standalone request lists all tools again for manual inspection.

### Run with Newman

With Node.js available, the collection can also be executed from the repository root:

```bash
npx newman run \
  docs/postman_collection/mcp-home-arpa-single-file.postman_collection.json \
  --env-var 'mcp_url=http://127.0.0.1:8002/mcp' \
  --env-var 'mcp_protocol_version=2025-06-18'
```

Run the handshake folder first when diagnosing a deployment:

```bash
npx newman run \
  docs/postman_collection/mcp-home-arpa-single-file.postman_collection.json \
  --folder '00 - P0 CRITICAL - MCP handshake, inventory, runtime config' \
  --env-var 'mcp_url=http://127.0.0.1:8002/mcp'
```

### Interpreting collection failures

- A folder `00` failure usually indicates endpoint, protocol, header, deployment, or tool-registration problems. Resolve it before evaluating provider quality.
- Search, weather, stock, and news tests depend on external sources. A failure can reflect upstream downtime, rate limits, regional differences, or changed search rankings rather than an MCP transport defect.
- Relevance assertions are intentionally strict. Review the structured response and Postman Console before weakening a quality test.
- Placeholder mail tests should validate the abstract contract, not successful real-world mail delivery.

## Local deployment and rollback flow

Use the existing deployment wrapper:

```bash
./scripts/deploy-local.sh
```

For self-hosted GitHub Actions runner preparation, see:

```text
docs/RUNNER_SETUP.md
```

The deployment workflow should continue to preserve runtime configuration and persistent Docker volumes during normal upgrades and rollbacks.

## Reverse proxy and production access

Keep the Docker host binding private unless direct network exposure is intentional:

```dotenv
HOST_BIND=127.0.0.1
APP_PORT=8002
```

A minimal Caddy route can proxy the HTTPS hostname to the local container port:

```caddyfile
mcp.home.arpa {
    reverse_proxy 127.0.0.1:8002
}
```

For any endpoint accessible beyond a trusted private network:

- Terminate TLS at Caddy or another trusted reverse proxy.
- Add authentication and authorization at the proxy or application layer.
- Restrict source networks when practical.
- Apply request-size, concurrency, and rate limits.
- Keep `ALLOW_PRIVATE_IP_URLS` disabled for untrusted clients.
- Avoid exposing provider credentials or verbose internal errors in responses.
- Protect or disable tools that can trigger costly or sensitive operations.

## Security notice for repository maintainers

Before merging or publishing configuration changes:

1. Ensure `.env.example` contains placeholders only.
2. If any committed value was ever a valid credential, revoke and rotate it immediately.
3. Remove exposed credentials from Git history when required; deleting them only from the latest commit is not sufficient.
4. Resolve duplicate environment-variable declarations so each setting has one unambiguous documented value.
5. Keep `.env`, local runtime files, logs containing secrets, and generated credentials out of Git.

## Project layout

```text
.
├── app/                         FastMCP application, settings, providers, and tools
├── docs/
│   ├── RUNNER_SETUP.md          Self-hosted GitHub Actions runner guidance
│   └── postman_collection/      Single-file MCP Postman quality suite
├── examples/                    Example client or integration material
├── scripts/                     Deployment, verification, backup, and rollback scripts
├── searxng/                     SearXNG configuration
├── tests/                       Automated application tests
├── compose.yaml                 FastMCP and SearXNG services
├── Dockerfile                   MCP application image
├── requirements.txt             Runtime Python dependencies
├── requirements-dev.txt         Development/test dependencies
└── .env.example                 Safe configuration template; never store real secrets here
```

## Troubleshooting

### The endpoint returns 404

Use the complete MCP path:

```text
http://127.0.0.1:8002/mcp
```

Do not send MCP requests to only the host root.

### The endpoint returns a content-type or accept error

Send both headers:

```http
Content-Type: application/json
Accept: application/json, text/event-stream
```

### Requests fail after initialization

Check whether the initialize response returned `Mcp-Session-Id`. When present, send the same value on subsequent requests. The Postman collection captures this automatically.

### Search returns no results

Check the FastMCP and SearXNG containers:

```bash
docker compose ps
docker compose logs --tail=200 hello
docker compose logs --tail=200 searxng
```

Confirm that the application can reach the configured SearXNG URL and that the enabled engines are responding.

### JavaScript-heavy pages do not scrape correctly

Review `JS_RENDER_MODE`, confirm the browser runtime is available in the image, and inspect application logs. Keep page count, text size, image count, and concurrency limits conservative because browser rendering is more resource-intensive than static extraction.

### Stock or news providers fail

Verify that optional credentials are configured only in `.env`, review provider quotas and account status, and confirm that fallback providers are enabled. Do not paste credentials into issues, logs, Postman examples, or committed files.

## Known limitations

- Mail tools are abstract placeholders and do not send or retrieve real messages.
- Road conditions are research/search based; they are not a normalized, guaranteed real-time traffic incident feed.
- Search relevance can vary with SearXNG engine availability and regional results.
- External weather, market, and news providers can impose quotas, delays, schema changes, or availability restrictions.
- Scraping success depends on site structure, robots/policy constraints, anti-bot controls, and JavaScript requirements.
- `explain_stock_move` produces an evidence-based summary, not investment advice or a definitive causal determination.

## Operational checklist

Before promoting a release:

- [ ] `docker compose config` completes successfully.
- [ ] No real secrets exist in tracked files or Git history.
- [ ] FastMCP and SearXNG containers are healthy.
- [ ] The local `/mcp` endpoint completes initialization.
- [ ] Postman folder `00` passes.
- [ ] The exact 15-tool inventory is present.
- [ ] Relevant provider folders pass or any external-provider exceptions are documented.
- [ ] Persistent reference data survives a container recreation.
- [ ] Reverse-proxy TLS and access controls are verified.
- [ ] Backup and rollback scripts are tested for the target host.