Skip to main content
Glama
io.github.Maverick0351a/rapidbm25

io.github.Maverick0351a/rapidbm25

rapidbm25

wheels

Drop-in, Rust-native replacement for rank_bm25's BM25Okapi, BM25L, and BM25Plus. Same constructor, same attributes, bit-for-bit identical scores, about 1,470× faster get_scores, plus a fast top_n, save/load, an incremental index, batch search, optional tokenizers, hybrid fusion, a ready-to-run MCP server, and an agent skill.

Install

pip install rapidbm25

Prebuilt wheels: CPython 3.9–3.15 on Linux (manylinux_2_28 and musllinux_1_2, x86_64 and aarch64), macOS (x86_64 10.13+ for CPython 3.9–3.13 and 10.15+ for 3.14–3.15, arm64 11.0+), and Windows (x86_64). Other platforms build from the sdist and need a Rust toolchain (1.85+).

Related MCP server: MemoryClaw

Use

# from rank_bm25 import BM25Okapi
from rapidbm25 import BM25Okapi

docs = ["Hello there good man!", "It is quite windy in London", "How is the weather today?"]
bm25 = BM25Okapi([d.split(" ") for d in docs])        # also: tokenizer=..., k1, b, epsilon

query = "windy London".split(" ")
scores = bm25.get_scores(query)                        # numpy float64, identical to rank_bm25
top = bm25.get_top_n(query, docs, n=1)                 # ['It is quite windy in London']
ids, best = bm25.top_n(query, n=2)                     # new: Rust partial select, no full sort

Which rank_bm25?

pip install rank_bm25 gives 0.2.2, and its BM25L and BM25Plus differ from the project's later git main (47aa3dd): 0.2.2's BM25L multiplies by term frequency (docs without the term score 0), and its BM25Plus idf is ln((N+1)/df) instead of ln(N+1) - ln(df). BM25Okapi is identical. rapidbm25 matches 0.2.2 by default; pass compat="47aa3dd" to BM25L / BM25Plus for git-main behavior. Both are tested bit for bit.

from rapidbm25 import BM25L
BM25L([d.split(" ") for d in docs], compat="47aa3dd")

What is the same as rank_bm25

  • Classes BM25Okapi(corpus, tokenizer=None, k1=1.5, b=0.75, epsilon=0.25), BM25L(..., delta=0.5), BM25Plus(..., delta=1).

  • Attributes corpus_size, avgdl, doc_len, doc_freqs, idf, average_idf (Okapi), k1, b, epsilon/delta.

  • get_scores, get_batch_scores, get_top_n with the same arguments and asserts.

  • Scores match bit for bit, including repeated query terms, unknown terms, empty documents, the Okapi negative-idf floor, and the NaN cases (k1=0, b=1 with empty docs).

What is different

rank_bm25

rapidbm25

get_top_n tie order

reverse of numpy's unstable argsort (varies by numpy build)

score descending, then lower doc id (deterministic)

Tokens

any hashable

str (non-str corpus tokens raise TypeError)

Editing .idf / .doc_freqs after build

changes later scores

no effect (scores come from the native index)

tokenizer=

runs in a multiprocessing pool

runs in-process

compat= keyword on BM25L / BM25Plus

n/a

"0.2.2" (default) or "47aa3dd"

Pickling / save / load

pickle only

pickle, plus a versioned file (load copies; mmap=True is opt-in)

fixes=

n/a

compat classes default False (bit-exact). LiveIndex defaults True

Lone surrogate in a token

accepted (str can hold it)

raises UnicodeEncodeError

Save, load, and pickle

okapi_ix = BM25Okapi([d.split(" ") for d in docs])
okapi_ix.save("/tmp/rapidbm25-okapi.rbm25")
okapi_loaded = BM25Okapi.load("/tmp/rapidbm25-okapi.rbm25")   # copies into memory

save writes format 1: magic RBM25, a JSON header, then little-endian numeric columns. Floats are stored as raw IEEE-754 bits, so get_scores after load matches the index that was saved, bit for bit. The default load copies the columns into memory, so later truncation or in-place edits of the file cannot change scores or kill the process. load(mmap=True) maps the caller's file and is unsafe if that file changes: truncation can raise SIGBUS, and an in-place write can change scores. Pickle keeps a picklable tokenizer; the file format stores the tokens already in the index, not an arbitrary callable.

Live index

LiveIndex is a new class (the compat classes stay immutable and bit-exact). It assigns stable string ids, updates N, document frequency, avgdl, and idf on every add, delete, and update, and patches posting lists instead of rebuilding them. Score-vector order is insertion order; update keeps a document's place; delete closes the gap. get_scores matches BM25Okapi / BM25L / BM25Plus built on live.corpus() with the same k1, b, epsilon or delta, compat, and fixes. An empty live index is valid. fixes defaults to True on LiveIndex and to False on the compat classes. Pass fixes=False for rank_bm25's exact bugs as well as its formulas. The folder index used by bm25_index_path is a LiveIndex, so it uses that default.

from rapidbm25 import LiveIndex, Tokenizer, fuse
words = Tokenizer(lowercase=True, stopwords="english", stemmer="english")
live_ix = LiveIndex(tokenizer=words)
live_ix.add("dogs", "The running dogs chased the ball")
live_ix.add("cats", "A sleeping cat", metadata={"kind": "pet"})
live_ix.add("weather", "It is quite windy in London")
pet_hits = live_ix.search("sleeping cats", k=1, where={"kind": "pet"})
live_ix.save("/tmp/rapidbm25-live.rbm25")
reloaded = LiveIndex.load("/tmp/rapidbm25-live.rbm25")
batch_scores = reloaded.get_scores_many(["dogs", "london wind"])
fused = fuse([["dogs", "cats"], [("cats", 0.9), ("weather", 0.2)]], method="rrf")

where is a dict ($eq, $ne, $in, $nin, $gt, $gte, $lt, $lte, $exists) or a callable. $in and $nin take a list, not a string. A boolean does not match 0 or 1. The callable runs outside the index lock. add / delete / update take a lock, so several threads can share one LiveIndex.

variant is "okapi" (default), "l", or "plus". compat is "0.2.2" or "47aa3dd", same as the static classes. delta defaults to 0.5 for L and 1 for Plus.

Tokenizers

Tokenizer is opt-in. Unicode words (UAX #29), optional lowercase, an original English stop list (or your own), a Snowball stemmer (stemmer="english" is Porter2; other Snowball languages are accepted), and cjk=True for overlapping character bigrams over Han, Hiragana, Katakana, and Hangul. Passing tokenizer= to a compat class is the only way it tokenizes; omitting it leaves those classes unchanged.

get_scores_many(queries) returns a (n_queries, corpus_size) float64 array. top_n_many(queries, n) returns one (ids, scores) pair per query. Both release the GIL and score queries on a thread pool. Row i is bit-identical to the single-query call. LiveIndex has the same two methods.

fixes=True

On BM25Okapi, BM25L, and BM25Plus, fixes=False is the default and matches rank_bm25, bugs included. LiveIndex defaults to fixes=True. The Okapi IDF formula stays ln(N - df + 0.5) - ln(df + 0.5) either way. There is no smoothed replacement.

issue

fixes=False

fixes=True

#19 repeated query tokens

each repeat adds the term again

the term is counted once, first-seen order

#36 empty corpus

ZeroDivisionError on the compat classes

empty static index, get_scores returns shape (0,)

#39 idf exactly 0

Okapi idf is 0 when a term is in exactly half the documents

that idf uses the existing epsilon * average_idf floor when the floor is positive

#39 does not change any other idf. It does not apply when epsilon * average_idf is zero or negative. That is the usual 2-document case: each term that appears in one document is in half the corpus, the average idf is not positive, and the term still scores 0. BM25L and BM25Plus do not have this zero. The fix is Okapi-only. L and Plus idf never hit exactly 0 for a real document frequency.

Hybrid fusion

fuse(rankings, method="rrf", k=60) is reciprocal rank fusion: weight / (k + rank), ranks starting at 1. k=60 is the default (the usual Cormack constant). Input scores are ignored, so a BM25 list and a vector-store list do not need a shared scale. method="weighted" min-max normalizes each list to [0, 1] and returns the weighted sum. A list of equal positive scores contributes its weight; a list of zeros contributes nothing.

Integrations

Optional extras. Both are thin retrievers over LiveIndex, with from_texts, k, and save / load. Tests skip when the dependency is not installed.

pip install "rapidbm25[langchain]"      # langchain-core
pip install "rapidbm25[llama-index]"    # llama-index-core
from rapidbm25.integrations.langchain import BM25Retriever
retriever = BM25Retriever.from_texts(["sunny Paris", "windy London"], k=1)
assert retriever.invoke("windy")[0].page_content == "windy London"

BM25Retriever.from_documents accepts LangChain Document objects. rapidbm25.integrations.llama_index.BM25Retriever implements _retrieve and returns NodeWithScore nodes.

Benchmarks

Workload: 16,000 chunks of Python stdlib sources and docs (mean 193 tokens, vocab 58,073), 200 held-out queries (mean 8.7 tokens), BM25Okapi. Machine: Intel Xeon (Sapphire Rapids) KVM guest, 8 vCPU, Linux x86_64, CPython 3.13.5, numpy 2.5.4. Shared VM, so treat small gaps as noise.

5 rounds; in each round every implementation runs once in a fresh process, in rotated order. Medians; rapidbm25 rows time 20 batches per process.

Implementation

Index build

get_scores × 200

End to end × 200 (tokenize + score + top-10)

rank_bm25 0.2.2

0.433 s

4.225 s

4.322 s

rapidbm25, top-10 via np.lexsort

0.119 s (3.6×)

2.87 ms (1,470×)

53.6 ms (81×)

rapidbm25, top_n(q, 10)

0.124 s

2.90 ms

16.4 ms (263×)

End to end with np.lexsort is dominated by sorting 16,000 scores per query; top_n does a partial select in Rust and is 3.3× faster.

python bench/bench_v02.py, one process, median of 5 after a warmup. Machine: Intel Xeon, 4 vCPU, Linux x86_64, CPython 3.12.3, release build. Synthetic corpus unless noted: 8,000 documents, 48 tokens each, vocabulary 4,000. The saved file is 7,924,566 bytes.

operation

n

median

static BM25Okapi build

8,000

10.7 ms

save

8,000

15.9 ms

load(mmap=True) + one query

8,000

2.01 ms

load(mmap=False) + one query

8,000

2.81 ms

LiveIndex.add of every document

8,000

85.3 ms

LiveIndex.delete

800

6.82 ms

static rebuild of the documents still left

7,200

8.53 ms

LiveIndex.add of 200 more, index already built

200

1.57 ms

static rebuild of that whole set

8,200

9.58 ms

Building 8,000 documents in one BM25Okapi(...) call is faster than 8,000 individual adds. Once the live index exists, adding 200 documents is 1.57 ms against 9.58 ms to rebuild all 8,200, and deleting 800 is 6.82 ms against 8.53 ms to rebuild the rest.

Batch search is the same comparison, end to end (get_scores in a loop vs get_scores_many), median of 5:

workload

sequential

get_scores_many

speedup

Okapi, 8,000×48, vocab 4,000, 64 queries (short postings)

0.331 ms

1.021 ms

0.32×

Okapi, 30,000×40, vocab 200, 64 queries (long postings)

4.94 ms

3.64 ms

1.36×

BM25Plus, 30,000×32, vocab 4,000, 48 queries (every document is scored)

7.59 ms

3.66 ms

2.08×

On this 4-vCPU guest the sparse Okapi kernel is cheap enough that the thread pool loses when each query only walks short postings. It wins once a query walks long postings, and by about 2× for BM25Plus, which adds a term's contribution to every document. Rows match the single-query scores bit for bit either way.

Reproduce the 0.1 comparison:

pip install rapidbm25 rank-bm25==0.2.2
python bench/gen_workload.py --out /tmp/bm25-wl          # chunks your Python stdlib (numbers will differ by machine and corpus)
python bench/bench_interleaved.py --workloads /tmp/bm25-wl --out /tmp/bench.json

Reproduce the benchmarks

Fixed-seed synthetic corpora of 500, 4,000, and 8,000 documents. The command times index build, get_scores, and top_n against rank_bm25 0.2.2 (batch median, plus per-query p50 and p99), records the RSS added by the index and the saved index size, smoke-tests the MCP server, and writes bench.json and bench.md. CI runs the 500- and 4,000-document sizes and fails if the 4,000-document get_scores speedup drops below the floor in bench/bench.py.

pip install rapidbm25 rank-bm25==0.2.2
python bench/bench.py --out /tmp/rapidbm25-bench

Retrieval quality (nDCG@10 and recall@100) reads a local BEIR tree: one dataset directory, or a directory of them (corpus.jsonl, queries.jsonl, qrels/test.tsv). The script does not download datasets. bm25s is scored when it is installed and skipped when it is not. CI caches scifact and also fails if rapidbm25 and rank_bm25 differ in nDCG@10 by more than the tolerance in bench/bench.py.

pip install bm25s   # optional
python bench/bench.py --sizes '' --quality /path/to/beir-datasets --out /tmp/rapidbm25-bench

--sizes '' skips the synthetic timing.

Correctness evidence

  • Contract replay: contract/bm25.contract.json holds 407 input/output vectors observed by running rank_bm25 0.2.2 from PyPI; contract/bm25-47aa3dd.contract.json holds 407 from git 47aa3dd. tests/replay_contract.py replays them with exact comparison (floats as hex): 407/407 each. A control build that nudges scores by 1 ulp (tests/control_ulp_off.py) fails 263 of them, so the replay is not blind.

  • Fuzz (tests/fuzz_equivalence.py): 1,000 random cases against each reference, across all three variants and edge parameters, compared bit for bit on scores, batch scores, idf, doc_freqs, avgdl, average_idf, and top-n: 0 failures.

  • Every wheel in CI is installed on its own platform and runs the test suite, a 300-case fuzz against rank_bm25 0.2.2, and both contract replays.

  • Wheels are built with cargo-auditable (dependency list embedded in the .dep-v0 section).

MCP server

rapidbm25-mcp is a stdio MCP server shipped in the wheel (no MCP SDK dependency): bm25_index_texts, bm25_index_path, bm25_search, bm25_list_indexes, bm25_drop_index. Text indexes live in memory. bm25_index_path caches the folder index under RAPIDBM25_CACHE (default ~/.cache/rapidbm25, directory mode 0700, files mode 0600). A cached file is reused only when its size, mtime, and the SHA-256 of the bytes that were indexed still match; the path is statted again after the read. Symlinks are not followed. Files with another hard link (st_nlink > 1) are skipped. Dotfiles are skipped unless the glob's filename starts with .. Reads stay inside RAPIDBM25_ROOTS (default: the working directory). Nothing leaves the machine.

{ "mcpServers": { "rapidbm25": { "command": "uvx", "args": ["--from", "rapidbm25", "rapidbm25-mcp"],
  "env": { "RAPIDBM25_ROOTS": "/path/to/repo" } } } }

Details: mcp/README.md. Registry metadata: server.json.

Agent skill

skill/rapidbm25-search/ (SKILL.md + scripts/bm25_search.py) for agents that run shell commands:

python skill/rapidbm25-search/scripts/bm25_search.py --path ./src --query "parse_header timeout" -k 10

Agent eval

eval/agent/ asks 20 code-search questions against the urllib3 commit pinned in eval/agent/tasks.json (release 2.8.0). Each answer is one file and one symbol, checked automatically. The harness asks the same model the same prompt twice: once with the rapidbm25 MCP tools, once with grep and read-file. It writes runs.json and summary.md (correctness rate and medians) under eval/agent/out/.

Dry-run, with a scripted model and no API key:

python -m venv .venv && . .venv/bin/activate
pip install "maturin>=1.7,<2" && maturin develop
python eval/agent/harness.py --dry-run

Live run, against an https OpenAI-compatible chat completions endpoint (http is accepted only for localhost and 127.0.0.1):

export EVAL_API_KEY=...          # secret; never commit it
export EVAL_MODEL=...
export EVAL_BASE_URL=https://api.openai.com/v1   # root that serves /chat/completions
python eval/agent/harness.py

The GitHub workflow agent-eval runs the live comparison only from Actions → Run workflow on main, and only if the EVAL_API_KEY repository secret is set. A dispatch from any other ref stops before that secret is loaded. The form supplies the model and base URL. Pull requests run the dry-run and do not call a model.

Build from source

pip install maturin && maturin build --release      # or scripts/build_auditable.sh

License

Apache-2.0. See LICENSE and NOTICE (API compatibility with rank_bm25 is credited there).

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables AI agents to store and retrieve persistent memories using BM25 search, allowing them to remember past conversations and context across sessions.
      8
      MIT
    • A
      license
      A
      quality
      D
      maintenance
      Enables coding agents to search local notes via BM25 keyword ranking, returning relevant passages with file paths and line numbers, fully offline with zero dependencies.
      3
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables agents to search and retrieve past workspace conversations, dialogue transcripts, and markdown artifacts through local BM25 retrieval, providing cross-conversation architectural recall with low-latency, offline operation.
      MIT