# discourse-mcp

A Model Context Protocol (MCP) server that lets any MCP-compatible AI app search, fetch, and synthesize what tech communities are discussing — across **Hacker News**, **Lobsters**, and **Reddit** (optional).

[![CI](https://github.com/anushapundir/discourse-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/anushapundir/discourse-mcp/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE)
![Python](https://img.shields.io/badge/python-3.11%2B-blue)

Ask your assistant *"what's Hacker News saying about MCP this week?"* and it answers from the actual threads — no browser, no copy-paste.

**11 tools + 1 resource**, a LangGraph orchestrator, and an A/B model eval harness.

---

## Quick start

Requires **Python ≥ 3.11**. Works on Windows, macOS, and Linux.

```bash
git clone https://github.com/anushapundir/discourse-mcp.git
cd discourse-mcp

python -m venv .venv
source .venv/bin/activate           # Windows (PowerShell): .venv\Scripts\Activate.ps1

pip install -e .
```

**No API keys needed.** Hacker News and Lobsters are free, no-auth APIs, so search, threads, cross-source synthesis, and the watchlist all work immediately. Keys are only for the *optional* Reddit source and the *optional* orchestrator.

> `pip install -e .` matters: installing as a package is what makes `python -m server.main` resolve no matter which directory your MCP host launches it from.

Check that it runs:

```bash
fastmcp dev inspector server/main.py
```

The Inspector should discover 11 tools and the `watchlist://topics` resource.

---

## Connect it to your AI app

Your MCP host launches the server as a subprocess, so point it at the **virtual-env Python** and use the module form `-m server.main`.

### Claude Desktop

Edit `claude_desktop_config.json` — on macOS at `~/Library/Application Support/Claude/`, on Windows at `%APPDATA%\Claude\`:

```json
{
  "mcpServers": {
    "discourse-mcp": {
      "command": "/absolute/path/to/discourse-mcp/.venv/bin/python",
      "args": ["-m", "server.main"]
    }
  }
}
```

On Windows, `"command"` is `C:\\absolute\\path\\to\\discourse-mcp\\.venv\\Scripts\\python.exe`.

### Cursor

Same block, in **Settings → MCP → Add new server** or `.cursor/mcp.json`.

Restart the app, and you're done.

### Then ask it things

> *"What's Hacker News saying about MCP this week?"*
>
> *"Synthesize what tech communities are discussing about AI agents."*
>
> *"Pull the top comments from that HN thread and tell me the main objection."*
>
> *"Add 'rust async' to my watchlist."*

---

## What it gives the model

### Tools (11)

| Tool | What it does |
|---|---|
| `search_hackernews(query, time_range="week", min_points=0)` | Relevance-ranked HN story search, filterable by recency and minimum score. |
| `get_hn_thread(story_id, max_comments=20)` | A single HN thread with top comments, indented by reply depth. |
| `get_hn_top_stories(category="top", limit=10)` | The current front-page lists: `top` / `new` / `best` / `ask` / `show`. |
| `search_lobsters(query, limit=15)` | Search recent Lobsters stories and tag feeds (Lobsters has no full-text search API). |
| `get_lobsters_post(short_id, max_comments=20)` | A single Lobsters story with its comments. |
| `search_reddit(query, sort="relevance", time_range="all", limit=15)` | Search Reddit across all subreddits. **Optional source** — see the note below. |
| `get_reddit_post(post_id, max_comments=20)` | A single Reddit post with its comment thread. **Optional source.** |
| `get_trending_synthesis(topic_filter="", sources="all", timeframe="week", limit_per_source=10)` | Cross-source digest: fetches HN + Lobsters concurrently, dedupes by article URL (a cross-post is labeled "HN + Lobsters"), ranks by points. Tolerates one source being down. |
| `add_watchlist_topic(topic, sources="all")` | Save a topic to track over time (unique; re-adding is a no-op). |
| `remove_watchlist_topic(topic)` | Remove a tracked topic. |
| `list_watchlist_topics()` | List everything currently on the watchlist. |

### Resources (1)

| Resource | What it exposes |
|---|---|
| `watchlist://topics` | Read-only view of the saved watchlist (markdown). |

> **Reddit is optional and key-gated.** It is read through [ScrapeBadger](https://scrapebadger.com) as a **temporary stopgap** while official Reddit API access is pending. That is a paid service on trial credits, so **the Reddit tools stop working when the credits expire** — the official Reddit API is the intended long-term backend, and swapping to it is confined to `server/clients/reddit.py`. Without a key, the Reddit tools return a clean "not configured" message and *everything else is unaffected*.

---

## How it works

`discourse-mcp` is a single local Python process speaking MCP over stdio. It does not listen on a port. The design is strictly layered — **tools format, clients fetch, and the layers never reach sideways**:

- **`server/tools/`** — MCP-aware. Validates inputs (Pydantic), calls a client, formats markdown. Knows nothing about HTTP.
- **`server/clients/`** — pure `httpx`. Talks to external APIs, returns typed models. Knows nothing about MCP.
- **`server/main.py`** — the FastMCP app. Registers tools and resources; implements none of them.
- **`server/db.py` + `server/resources/`** — SQLite-backed watchlist state.
- **`orchestrator/`** — a LangGraph agent that consumes this server *as a real MCP client* over stdio, with a provider-swappable model layer.

Two rules make it robust: **stdout is reserved for the protocol** (all logs go to stderr), and **errors are data** — a failed upstream call returns an `"Error: …"` string the model can reason about, never an exception that kills the server.

![Architecture diagram](docs/architecture.png)

<details>
<summary>Diagram source (Mermaid)</summary>

```mermaid
flowchart TB
    subgraph consumers["Consumers (MCP hosts)"]
        direction LR
        desktop["Claude Desktop / Cursor<br/>(interactive)"]
        orch["LangGraph orchestrator<br/>(programmatic)"]
    end

    consumers -- "stdio + JSON-RPC (MCP)" --> main

    subgraph server["discourse-mcp server (single Python process)"]
        direction TB
        main["server/main.py<br/>FastMCP app · registration"]

        subgraph tools["server/tools/ — MCP-aware"]
            direction LR
            hn_t["hackernews.py"]
            lob_t["lobsters.py"]
            reddit_t["reddit.py (optional)"]
            syn_t["synthesis.py"]
            wl_t["watchlist.py"]
        end

        subgraph clients["server/clients/ — pure HTTP (httpx)"]
            direction LR
            base["base.py<br/>retry + backoff"]
            hn_c["hackernews.py"]
            lob_c["lobsters.py"]
            reddit_c["reddit.py (optional)"]
        end

        subgraph state["State + resources"]
            direction LR
            db["db.py<br/>SQLite"]
            res["resources/watchlist.py<br/>watchlist://topics"]
        end

        fmt["formatting/markdown.py<br/>shared renderer"]

        main --> hn_t & lob_t & reddit_t & syn_t & wl_t
        main --> res
        hn_t --> hn_c
        lob_t --> lob_c
        reddit_t --> reddit_c
        syn_t --> hn_c & lob_c
        hn_c & lob_c & reddit_c --> base
        wl_t --> db
        res --> db
        hn_t & lob_t & reddit_t & syn_t -.-> fmt
    end

    subgraph external["External APIs"]
        direction LR
        algolia["Algolia HN Search<br/>(free)"]
        firebase["HN Firebase<br/>(free)"]
        lobsters_api["Lobsters JSON API<br/>(free)"]
        scrapebadger["ScrapeBadger<br/>(Reddit · paid, API key)"]
    end

    base -- HTTPS --> algolia & firebase & lobsters_api & scrapebadger
```

</details>

For the full design rationale — every decision, and what broke along the way — read [`docs/PROJECT_DEEP_DIVE.md`](docs/PROJECT_DEEP_DIVE.md). For the engineering standards, see [`CLAUDE.md`](./CLAUDE.md).

---

## The orchestrator (optional)

A LangGraph ReAct agent that consumes this server **the same way Claude Desktop does** — launching `python -m server.main` over stdio via `langchain-mcp-adapters`. The model provider is swappable by environment variable alone, with zero code changes.

```bash
pip install -e ".[orchestrator]"
cp .env.example .env                # Windows: copy .env.example .env

# Ask a one-off question (the agent picks the tools)
python -m orchestrator.main --ask "what is HN saying about rust this month"

# Run the autonomous daily-digest workflow over your watchlist
python -m orchestrator.main --workflow daily-digest
```

Set `MODEL_PROVIDER` in `.env` to `anthropic` (needs `ANTHROPIC_API_KEY`) or `ollama` (needs Ollama running locally — no key, no cost).

---

## Configuration

Everything is environment variables. The orchestrator reads `.env`; the server reads the process environment (and falls back to `.env` for local runs, without overriding anything a host injects).

| Variable | Used by | Default | Purpose |
|---|---|---|---|
| `DISCOURSE_DB_PATH` | server | `~/.discourse-mcp/watchlist.db` | Where the watchlist SQLite file lives. |
| `DISCOURSE_HTTP_TIMEOUT` | server | `10` | Per-request HTTP timeout (seconds). |
| `SCRAPE_BADGER_REDDIT_API_KEY` | server | — | Enables the optional Reddit tools. |
| `MODEL_PROVIDER` | orchestrator | `anthropic` | `anthropic` or `ollama`. |
| `MODEL_NAME` | orchestrator | `claude-haiku-4-5-20251001` | Model id for the chosen provider. |
| `MODEL_MAX_TOKENS` | orchestrator | `2048` | Output-token cap per call (budget guard). |
| `ANTHROPIC_API_KEY` | orchestrator | — | Required when `MODEL_PROVIDER=anthropic`. |
| `OLLAMA_HOST` | orchestrator | `http://localhost:11434` | Ollama endpoint. |

---

## Evaluation: hosted vs. local model

The same agent and the same MCP server run under both providers — only `MODEL_PROVIDER` changes. A fixed judge (Claude Haiku, `temperature=0`) scores **groundedness** and **relevance** 1–5. Full methodology in [`evals/model_comparison.md`](evals/model_comparison.md).

| Provider | Prompt | Latency | Groundedness | Relevance |
|---|---|---|---|---|
| anthropic | search | 22.2s | 4 | 5 |
| anthropic | synthesis | 14.9s | 4 | 5 |
| ollama (`qwen2.5:7b`, CPU) | search | 1174.8s | 4 | 5 |
| ollama (`qwen2.5:7b`, CPU) | synthesis | 858.6s | **1** | 5 |

Both providers genuinely drive the MCP tools, so the provider swap works end to end. But the local 7B model is ~50–200× slower on CPU, and on the synthesis prompt the judge caught it **fabricating** — invented URLs and future dates — where the hosted model stayed grounded.

```bash
python -m evals.run_eval            # free structural gate (14 cases, no API cost)
python -m evals.run_eval --compare  # paid A/B comparison → regenerates the table
```

---

## Project layout

```
server/
├── main.py              # FastMCP app — tool + resource registration only
├── db.py                # SQLite watchlist storage
├── clients/             # pure HTTP (httpx) — base.py adds retry/backoff
├── tools/               # MCP tools — validate, call a client, format markdown
├── resources/           # read-only MCP resources
└── formatting/          # shared source-neutral StoryCard renderer

orchestrator/            # LangGraph agent (consumes the server over stdio)
evals/                   # hybrid eval harness (structural gate + judge)
docs/                    # architecture diagram + design deep dive
```

**Engineering notes:** async throughout (`asyncio.gather` for cross-source fan-out) · Pydantic at every boundary · a shared HTTP layer that retries *only* transient failures (connection errors, timeouts, 5xx, 429) and **never 4xx**, so a bad story id fails fast in ~0.7s instead of after three slow retries · errors returned as data so a tool call can never kill the server.

---

## Roadmap

- [ ] Reddit via the **official** Reddit API, replacing the ScrapeBadger stopgap
- [ ] Reddit folded into `get_trending_synthesis` as a third source
- [ ] Streamable HTTP transport with OAuth 2.1
- [ ] Published to the public MCP registry
- [ ] Prompt-injection guardrails on tool outputs

---

## Contributing

Contributions welcome — new sources especially. See [CONTRIBUTING.md](./CONTRIBUTING.md) for setup and the layering rules, and [CLAUDE.md](./CLAUDE.md) for the full engineering standards. Pull requests need one approving review before merging.

## License

MIT — see [LICENSE](./LICENSE).