# Pocketpedia

Offline English Wikipedia for local LLMs: a 14 GB index on your own disk, a command line
tool and an MCP server. Search runs locally (re-ranking uses a local embedding model by
default), so nothing leaves your machine.

**A 4B model with Pocketpedia beats a 120B model answering from memory.** On the same 500
held-out PopQA questions, Gemma 3 4B with Pocketpedia scored **49.6%** (95% CI 45.2–54.0).
GPT-OSS 120B without it scored **34.4%** (30.4–38.7), even on the scoring most favourable to it.

| Model, 1,000 held-out PopQA questions | From memory | With Pocketpedia |
|---|---|---|
| Gemma 3 4B | 16.0% | **52.8%** |
| Qwen 3.5 35B-A3B (≈3B active) | 30.5% | **61.3%** |
| DeepSeek-flash (hosted) | 57.1% | **64.7%** |

![PopQA accuracy by popularity quintile](bench/results-popqa-quintiles.png)

The gain is largest on obscure entities, the long tail a small model cannot memorise. On
SimpleQA (300 questions), Gemma 3 4B goes from 3.3% to 40.3% correct.

How the numbers were made, and what they don't show:

- The plan and the bar for "good enough to publish" were [written down before any results](BENCHMARK_PLAN.md).
- Retrieval was tuned on 300 development questions. All headline numbers come from 1,000
  different test questions (200 per popularity quintile, seed 20261031).
- A fresh clone reproduced the Gemma numbers exactly ([bar check](bench/bar-check.md)).
- GPT-OSS 120B and Qwen 3.8 27B ran on a stratified half (500 questions); the headline compares
  the same 500. Mistral Medium 3.5 128B could not be loaded and was not run.
- GPT-OSS often put narrow no-break spaces inside names. Its plain-match score was 19.8%; the
  34.4% above is after Unicode normalisation, which was added after registration in its favour.
- PopQA is entity-centric and Wikipedia-native, close to the best case for this approach.
  SimpleQA was graded by a local GPT-OSS 120B with the official template, not the official
  grader, so it is not comparable with published SimpleQA scores.

Full tables and limitations: [bench/results.md](bench/results.md). Benchmark instructions:
[bench/README.md](bench/README.md).

## Quickstart

Requires Python 3.12 or later with SQLite FTS5. From a clone of this repository:

```bash
python3 -m venv .venv
. .venv/bin/activate
pip install -e '.[build]'
pocketpedia download
pocketpedia build
pocketpedia doctor
pocketpedia search "Who directed Pixels" --retrieval v1
pocketpedia read "Pixels (2010 film)" --max-chars 3000
```

The download is the 41 English `wikimedia/wikipedia` **20231101.en** parquet
shards, about **11.6 GB**. Building took about **five minutes** on the benchmark
machine; allow roughly **25 GB** for the download and indexes, plus room for
caches (the benchmark directory reached about 26 GB). Build time depends on your
CPU and disk. No Wikipedia text or index is shipped with this package.

Downloads run three shards at a time. Re-running resumes `.part` files and skips
completed shards. Use an empty download directory when switching from the old
shell downloader, which did not distinguish partial from complete files.
`build` refuses to overwrite an existing index; choose a new `--out` to rebuild.

Data defaults to `~/.pocketpedia`. `--data-dir` overrides `$POCKETPEDIA_HOME`,
which overrides that default. Options work before or after the command.

```bash
pocketpedia download --dir /path/to/parquet
pocketpedia build --parquet /path/to/parquet --out /path/to/data/index
pocketpedia search "Who directed Pixels" -k 5 --json --retrieval v1 --data-dir /path/to/data
pocketpedia doctor --data-dir /path/to/data
```

`index/index.sqlite` holds titles and leads; `index/fulltext.sqlite` holds
compressed full articles. Read and v1 search need no service. PyArrow is imported
only by `build`; `pip install -e .` is enough to use an existing index.
Python 3.14 uses stdlib zstd; Python 3.12–3.13 install `zstandard` automatically.

## Retrieval

V1 is the benchmark's **boosted** retrieval: the longest question span matching
an article title goes first, followed by SQLite FTS5 BM25 results over titles
and leads. A lead is the first two nonempty paragraphs, capped at 1,500
characters. V2 adds parenthesised title candidates, uses embeddings to select
five articles, splits their first 8,000 words into roughly 120–200-word passages,
then reranks the top 30 BM25 passages with embeddings. The packaged
[v2 configuration](src/pocketpedia/retrieval_v2.json) is frozen to the benchmark.

Search defaults to v2, using an OpenAI-compatible embeddings service at
`http://localhost:1234/v1` with `text-embedding-nomic-embed-text-v1.5` (as served
by LM Studio in the benchmark). Override the service with `--embed-url` or
`$POCKETPEDIA_EMBED_URL`, and the model with `--embed-model`:

```bash
pocketpedia search "Who directed Pixels" --retrieval v2 \
  --embed-url http://localhost:1234/v1 \
  --embed-model text-embedding-nomic-embed-text-v1.5
```

Vectors are cached in `embcache.sqlite` in the data directory. Changing the
endpoint or model uses distinct cache keys. V2 sends questions and candidate
Wikipedia passages to that endpoint; use a local service to keep processing
offline. Changing models can change results, and the frozen prefixes are for
the Nomic model. If embeddings are unavailable, search warns once on stderr
and uses v1 for the rest of that command or MCP session. `doctor` probes the
embedding endpoint without writing a cache; an unavailable optional endpoint
does not make a healthy index fail its check.

## MCP

The stdlib server uses newline-delimited JSON-RPC over stdio, protocol version
`2025-06-18`. Diagnostics go to stderr. It provides:

- `wikipedia_search(query, k=5)`: titled passages with v2 and the warned v1 fallback.
- `wikipedia_read(title, max_chars=20000, offset=0)`: full-text paging, with
  `total_chars` and `next_offset`. Offsets count characters; `null` means the end.
- `wikipedia_lookup(title)`: existence, canonical title and lead, case-insensitively.

Claude Code setup (activate the environment containing `pocketpedia` first):

```bash
claude mcp add pocketpedia -- pocketpedia mcp --data-dir "$HOME/.pocketpedia"
```

For a generic MCP client that accepts `mcpServers` configuration:

```json
{
  "mcpServers": {
    "pocketpedia": {
      "command": "pocketpedia",
      "args": ["mcp", "--data-dir", "~/.pocketpedia"]
    }
  }
}
```

The client must be able to find the installed `pocketpedia` executable. If it
does not inherit your shell's PATH, set `command` to the executable in your
virtual environment. Tool failures return `isError: true` and leave the server running.

## Limits and attribution

This is **English Wikipedia from November 2023**. It cannot provide current
facts, and retrieval can select the wrong entity or miss information outside
the lead or passage window. `read` accesses the full stored article. There is
no generative answer model in Pocketpedia; your client interprets the passages.
V2 needs an embedding endpoint on a cold cache. V1 works entirely offline.

Code is [MIT licensed](LICENSE), copyright Jon Hammant. Wikipedia text is
downloaded by the user and is licensed separately under
[CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/), with some text
also available under the GFDL; see
[Wikipedia's copyright notice](https://en.wikipedia.org/wiki/Wikipedia:Copyrights).
When quoting or redistributing text, credit Wikipedia and its contributors,
link the source article and licence, indicate changes, and comply with
share-alike requirements. Returned canonical titles identify the source:
for example, `Pixels (2010 film)` corresponds to
[its Wikipedia article](https://en.wikipedia.org/wiki/Pixels_(2010_film)).
The live article may differ from the 2023 snapshot. Preserve provenance when
building answers from retrieved passages.

## Development

```bash
pip install -e '.[build]'
python -m unittest discover -s tests
ruff check src tests
ruff format --check src tests
```

Tests build twenty invented articles in temporary directories and need no
network or LM Studio. The subprocess test exercises the installed MCP command.
An additional test compares 20 development questions with stored boosted
passages if `bench/data` contains the existing index
and PopQA files; it skips otherwise. The parquet build test runs when the build
extra is installed. See [bench/README.md](bench/README.md) for benchmark reproduction.