# web-research-mcp

An MCP server that exposes web search and page reading as standard
tools, resources, and a prompt template — so any MCP client (Claude
Desktop, Claude Code, or another agent) can research a topic without
each one reimplementing search/fetch/extract from scratch.

## Why this exists

I'd already built [factcheck-agent](https://github.com/orackle/factcheck-agent),
a LangGraph agent with its own search-fetch-extract loop hardcoded into
its graph. That logic is genuinely reusable — any agent that needs to
look something up on the web needs roughly the same thing. This project
pulls that capability out into a standard [MCP](https://modelcontextprotocol.io)
server instead of leaving it locked inside one agent's internals.

## What it exposes

**Tools**
- `search(query, max_results)` — title/url/snippet results from DuckDuckGo (or Tavily, if configured).
- `fetch(url)` — fetches a page and extracts its readable text. Returns an `error` field instead of raising on failure (PDFs, JS-rendered pages, and blocked requests are all reported, not silently swallowed).
- `research(topic, depth)` — search, then fetch the top results in one call, returning a source list with excerpts. Also appends everything gathered to the current session.

**Resources**
- `research://session` — everything gathered by `research` calls so far, readable without re-searching.

**Prompts**
- `fact_check(claim)` — a template that frames a claim as a research task and requires citations for any verdict.

## Install

```bash
git clone https://github.com/orackle/web-research-mcp.git
cd web-research-mcp
python -m venv .venv
.venv\Scripts\activate   # or source .venv/bin/activate on macOS/Linux
pip install -r requirements.txt
```

## Run

Local inspector (for poking at tools/resources by hand):
```bash
mcp dev web_research_mcp/server.py
```

Register with Claude Desktop:
```bash
mcp install web_research_mcp/server.py
```

## Configuration

All optional — defaults work with no API keys, using DuckDuckGo:

| Env var | Default | Purpose |
|---|---|---|
| `SEARCH_PROVIDER` | `duckduckgo` | `duckduckgo` or `tavily` |
| `TAVILY_API_KEY` | — | required if using Tavily |
| `RESULTS_PER_QUERY` | `5` | results returned per `search` call |
| `MAX_DOCS_TO_FETCH` | `5` | cap on pages `research` fetches |
| `FETCH_TIMEOUT_SECONDS` | `10` | per-page fetch timeout |
| `MAX_DOC_CHARS` | `4000` | truncation limit for extracted text |

## Stack

- [MCP Python SDK](https://github.com/modelcontextprotocol/python-sdk) (`mcp` 2.x — `MCPServer`, formerly `FastMCP`)
- httpx + BeautifulSoup for fetching/extraction
- DuckDuckGo (`ddgs`) / Tavily for search
- Pydantic for typed tool schemas

## Eval

`eval/run_eval.py` exercises all three tools against the real network (live search engines, real pages) — not mocked — and checks each result against what that case expects, including two deliberate failure cases (an unreachable domain, an image URL) to confirm errors surface as data instead of crashing the tool.

```bash
python -m eval.run_eval
```

Latest run, 9/9 passed:

```
[1/9] PASS search   (1.35s)  5 results
[2/9] PASS search   (1.65s)  3 results
[3/9] PASS search   (1.18s)  3 results (any count ok)
[4/9] PASS fetch    (1.10s)  text=4000 chars, error=None
[5/9] PASS fetch    (0.52s)  text=2288 chars, error=None
[6/9] PASS fetch    (0.54s)  error='fetch failed: [Errno 11001] getaddrinfo failed'
[7/9] PASS fetch    (0.99s)  error='unsupported content-type: image/jpeg'
[8/9] PASS research (4.24s)  2 sources, 2 fetched
[9/9] PASS research (3.01s)  3 sources, 2 fetched

9/9 cases passed

Per-tool metrics:
  fetch      4/4 passed  latency: mean=0.79s median=0.77s max=1.10s
  research   2/2 passed  latency: mean=3.62s median=3.62s max=4.24s
  search     3/3 passed  latency: mean=1.39s median=1.35s max=1.65s
```

Full output: [`eval/eval_results.log`](eval/eval_results.log).

### Verified against the MCP Inspector

The server connects cleanly to the official [MCP Inspector](https://github.com/modelcontextprotocol/inspector) over stdio, and the `research` tool was run live from it end to end (real Wikipedia fetch, 5.5s round trip):

![research tool called live from the MCP Inspector](eval/screenshots/inspector-research-tool.png)

The `research://session` resource correctly reflects the accumulated state from that call:

![research session resource showing accumulated sources](eval/screenshots/inspector-resource-session.png)

## Known limits

- The session resource is a single in-memory session per server process — fine for one local client at a time, not built for multi-tenant use.
- `fetch` only handles HTML pages; PDFs and JS-rendered pages come back as an `error`, not a partial/garbled result.
- DuckDuckGo has no API key and is occasionally rate-limited or flaky — that's the tradeoff for zero-setup search; swap to Tavily via env vars if you need more reliability.