io.github.Maverick0351a/rapidbm25
Provides a BM25Retriever integration for LangChain, backed by LiveIndex. LangChain applications can build retrievers from texts or Document objects and retrieve top-k BM25 matches.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
rapidbm25
Drop-in, Rust-native replacement for rank_bm25's BM25Okapi, BM25L, and BM25Plus.
Same constructor, same attributes, bit-for-bit identical scores, about 1,470× faster
get_scores, plus a fast top_n, save/load, an incremental index, batch search,
optional tokenizers, hybrid fusion, a ready-to-run MCP server, and an agent skill.
Install
pip install rapidbm25Prebuilt wheels: CPython 3.9–3.15 on Linux (manylinux_2_28 and musllinux_1_2, x86_64 and aarch64), macOS (x86_64 10.13+ for CPython 3.9–3.13 and 10.15+ for 3.14–3.15, arm64 11.0+), and Windows (x86_64). Other platforms build from the sdist and need a Rust toolchain (1.85+).
Related MCP server: MemoryClaw
Use
# from rank_bm25 import BM25Okapi
from rapidbm25 import BM25Okapi
docs = ["Hello there good man!", "It is quite windy in London", "How is the weather today?"]
bm25 = BM25Okapi([d.split(" ") for d in docs]) # also: tokenizer=..., k1, b, epsilon
query = "windy London".split(" ")
scores = bm25.get_scores(query) # numpy float64, identical to rank_bm25
top = bm25.get_top_n(query, docs, n=1) # ['It is quite windy in London']
ids, best = bm25.top_n(query, n=2) # new: Rust partial select, no full sortWhich rank_bm25?
pip install rank_bm25 gives 0.2.2, and its BM25L and BM25Plus differ from the project's
later git main (47aa3dd): 0.2.2's BM25L multiplies by term frequency (docs without the term score 0),
and its BM25Plus idf is ln((N+1)/df) instead of ln(N+1) - ln(df). BM25Okapi is identical.
rapidbm25 matches 0.2.2 by default; pass compat="47aa3dd" to BM25L / BM25Plus for git-main
behavior. Both are tested bit for bit.
from rapidbm25 import BM25L
BM25L([d.split(" ") for d in docs], compat="47aa3dd")What is the same as rank_bm25
Classes
BM25Okapi(corpus, tokenizer=None, k1=1.5, b=0.75, epsilon=0.25),BM25L(..., delta=0.5),BM25Plus(..., delta=1).Attributes
corpus_size,avgdl,doc_len,doc_freqs,idf,average_idf(Okapi),k1,b,epsilon/delta.get_scores,get_batch_scores,get_top_nwith the same arguments and asserts.Scores match bit for bit, including repeated query terms, unknown terms, empty documents, the Okapi negative-idf floor, and the NaN cases (
k1=0,b=1with empty docs).
What is different
rank_bm25 | rapidbm25 | |
| reverse of numpy's unstable argsort (varies by numpy build) | score descending, then lower doc id (deterministic) |
Tokens | any hashable |
|
Editing | changes later scores | no effect (scores come from the native index) |
| runs in a multiprocessing pool | runs in-process |
| n/a |
|
Pickling / | pickle only | pickle, plus a versioned file (load copies; |
| n/a | compat classes default |
Lone surrogate in a token | accepted ( | raises |
Save, load, and pickle
okapi_ix = BM25Okapi([d.split(" ") for d in docs])
okapi_ix.save("/tmp/rapidbm25-okapi.rbm25")
okapi_loaded = BM25Okapi.load("/tmp/rapidbm25-okapi.rbm25") # copies into memorysave writes format 1: magic RBM25, a JSON header, then little-endian numeric columns.
Floats are stored as raw IEEE-754 bits, so get_scores after load matches the index that
was saved, bit for bit. The default load copies the columns into memory, so later truncation or
in-place edits of the file cannot change scores or kill the process.
load(mmap=True) maps the caller's file and is unsafe if that file changes:
truncation can raise SIGBUS, and an in-place write can change scores. Pickle
keeps a picklable tokenizer; the file
format stores the tokens already in the index, not an arbitrary callable.
Live index
LiveIndex is a new class (the compat classes stay immutable and bit-exact). It assigns
stable string ids, updates N, document frequency, avgdl, and idf on every add,
delete, and update, and patches posting lists instead of rebuilding them. Score-vector
order is insertion order; update keeps a document's place; delete closes the gap.
get_scores matches BM25Okapi / BM25L / BM25Plus built on live.corpus() with the
same k1, b, epsilon or delta, compat, and fixes. An empty live index is valid.
fixes defaults to True on LiveIndex and to False on the compat classes. Pass
fixes=False for rank_bm25's exact bugs as well as its formulas. The folder index used
by bm25_index_path is a LiveIndex, so it uses that default.
from rapidbm25 import LiveIndex, Tokenizer, fuse
words = Tokenizer(lowercase=True, stopwords="english", stemmer="english")
live_ix = LiveIndex(tokenizer=words)
live_ix.add("dogs", "The running dogs chased the ball")
live_ix.add("cats", "A sleeping cat", metadata={"kind": "pet"})
live_ix.add("weather", "It is quite windy in London")
pet_hits = live_ix.search("sleeping cats", k=1, where={"kind": "pet"})
live_ix.save("/tmp/rapidbm25-live.rbm25")
reloaded = LiveIndex.load("/tmp/rapidbm25-live.rbm25")
batch_scores = reloaded.get_scores_many(["dogs", "london wind"])
fused = fuse([["dogs", "cats"], [("cats", 0.9), ("weather", 0.2)]], method="rrf")where is a dict ($eq, $ne, $in, $nin, $gt, $gte, $lt, $lte, $exists)
or a callable. $in and $nin take a list, not a string. A boolean does not match
0 or 1. The callable runs outside the index lock. add / delete / update
take a lock, so several threads can share one LiveIndex.
variant is "okapi" (default), "l", or "plus". compat is "0.2.2" or
"47aa3dd", same as the static classes. delta defaults to 0.5 for L and 1 for Plus.
Tokenizers
Tokenizer is opt-in. Unicode words (UAX #29), optional lowercase, an original English
stop list (or your own), a Snowball stemmer (stemmer="english" is Porter2; other
Snowball languages are accepted), and cjk=True for overlapping character bigrams over
Han, Hiragana, Katakana, and Hangul. Passing tokenizer= to a compat class is the only
way it tokenizes; omitting it leaves those classes unchanged.
Batch search
get_scores_many(queries) returns a (n_queries, corpus_size) float64 array.
top_n_many(queries, n) returns one (ids, scores) pair per query. Both release the
GIL and score queries on a thread pool. Row i is bit-identical to the single-query call.
LiveIndex has the same two methods.
fixes=True
On BM25Okapi, BM25L, and BM25Plus, fixes=False is the default and matches
rank_bm25, bugs included. LiveIndex defaults to fixes=True. The Okapi IDF formula
stays ln(N - df + 0.5) - ln(df + 0.5) either way. There is no smoothed replacement.
issue |
|
|
#19 repeated query tokens | each repeat adds the term again | the term is counted once, first-seen order |
#36 empty corpus |
| empty static index, |
#39 idf exactly 0 | Okapi idf is 0 when a term is in exactly half the documents | that idf uses the existing |
#39 does not change any other idf. It does not apply when epsilon * average_idf is
zero or negative. That is the usual 2-document case: each term that appears in one
document is in half the corpus, the average idf is not positive, and the term still
scores 0. BM25L and BM25Plus do not have this zero. The fix is Okapi-only. L and
Plus idf never hit exactly 0 for a real document frequency.
Hybrid fusion
fuse(rankings, method="rrf", k=60) is reciprocal rank fusion: weight / (k + rank),
ranks starting at 1. k=60 is the default (the usual Cormack constant). Input scores
are ignored, so a BM25 list and a vector-store list do not need a shared scale.
method="weighted" min-max normalizes each list to [0, 1]
and returns the weighted sum. A list of equal positive scores contributes its weight;
a list of zeros contributes nothing.
Integrations
Optional extras. Both are thin retrievers over LiveIndex, with from_texts, k,
and save / load. Tests skip when the dependency is not installed.
pip install "rapidbm25[langchain]" # langchain-core
pip install "rapidbm25[llama-index]" # llama-index-corefrom rapidbm25.integrations.langchain import BM25Retriever
retriever = BM25Retriever.from_texts(["sunny Paris", "windy London"], k=1)
assert retriever.invoke("windy")[0].page_content == "windy London"BM25Retriever.from_documents accepts LangChain Document objects.
rapidbm25.integrations.llama_index.BM25Retriever implements _retrieve and returns
NodeWithScore nodes.
Benchmarks
Workload: 16,000 chunks of Python stdlib sources and docs (mean 193 tokens, vocab 58,073), 200
held-out queries (mean 8.7 tokens), BM25Okapi. Machine: Intel Xeon (Sapphire Rapids) KVM guest,
8 vCPU, Linux x86_64, CPython 3.13.5, numpy 2.5.4. Shared VM, so treat small gaps as noise.
5 rounds; in each round every implementation runs once in a fresh process, in rotated order. Medians; rapidbm25 rows time 20 batches per process.
Implementation | Index build |
| End to end × 200 (tokenize + score + top-10) |
rank_bm25 0.2.2 | 0.433 s | 4.225 s | 4.322 s |
rapidbm25, top-10 via | 0.119 s (3.6×) | 2.87 ms (1,470×) | 53.6 ms (81×) |
rapidbm25, | 0.124 s | 2.90 ms | 16.4 ms (263×) |
End to end with np.lexsort is dominated by sorting 16,000 scores per query; top_n does a
partial select in Rust and is 3.3× faster.
0.2 save, incremental updates, batch search
python bench/bench_v02.py, one process, median of 5 after a warmup. Machine: Intel Xeon,
4 vCPU, Linux x86_64, CPython 3.12.3, release build. Synthetic corpus unless noted:
8,000 documents, 48 tokens each, vocabulary 4,000. The saved file is 7,924,566 bytes.
operation | n | median |
static | 8,000 | 10.7 ms |
| 8,000 | 15.9 ms |
| 8,000 | 2.01 ms |
| 8,000 | 2.81 ms |
| 8,000 | 85.3 ms |
| 800 | 6.82 ms |
static rebuild of the documents still left | 7,200 | 8.53 ms |
| 200 | 1.57 ms |
static rebuild of that whole set | 8,200 | 9.58 ms |
Building 8,000 documents in one BM25Okapi(...) call is faster than 8,000 individual
adds. Once the live index exists, adding 200 documents is 1.57 ms against 9.58 ms to
rebuild all 8,200, and deleting 800 is 6.82 ms against 8.53 ms to rebuild the rest.
Batch search is the same comparison, end to end (get_scores in a loop vs get_scores_many),
median of 5:
workload | sequential |
| speedup |
Okapi, 8,000×48, vocab 4,000, 64 queries (short postings) | 0.331 ms | 1.021 ms | 0.32× |
Okapi, 30,000×40, vocab 200, 64 queries (long postings) | 4.94 ms | 3.64 ms | 1.36× |
BM25Plus, 30,000×32, vocab 4,000, 48 queries (every document is scored) | 7.59 ms | 3.66 ms | 2.08× |
On this 4-vCPU guest the sparse Okapi kernel is cheap enough that the thread pool loses when each query only walks short postings. It wins once a query walks long postings, and by about 2× for BM25Plus, which adds a term's contribution to every document. Rows match the single-query scores bit for bit either way.
Reproduce the 0.1 comparison:
pip install rapidbm25 rank-bm25==0.2.2
python bench/gen_workload.py --out /tmp/bm25-wl # chunks your Python stdlib (numbers will differ by machine and corpus)
python bench/bench_interleaved.py --workloads /tmp/bm25-wl --out /tmp/bench.jsonReproduce the benchmarks
Fixed-seed synthetic corpora of 500, 4,000, and 8,000 documents. The command times index
build, get_scores, and top_n against rank_bm25 0.2.2 (batch median, plus per-query
p50 and p99), records the RSS added by the index and the saved index size, smoke-tests the MCP server,
and writes bench.json and bench.md. CI runs the 500- and 4,000-document sizes and
fails if the 4,000-document get_scores speedup drops below the floor in bench/bench.py.
pip install rapidbm25 rank-bm25==0.2.2
python bench/bench.py --out /tmp/rapidbm25-benchRetrieval quality (nDCG@10 and recall@100) reads a local BEIR tree: one dataset directory,
or a directory of them (corpus.jsonl, queries.jsonl, qrels/test.tsv). The script does
not download datasets. bm25s is scored when it is installed and skipped when it is not.
CI caches scifact and also fails if rapidbm25 and rank_bm25 differ in nDCG@10 by more than
the tolerance in bench/bench.py.
pip install bm25s # optional
python bench/bench.py --sizes '' --quality /path/to/beir-datasets --out /tmp/rapidbm25-bench--sizes '' skips the synthetic timing.
Correctness evidence
Contract replay:
contract/bm25.contract.jsonholds 407 input/output vectors observed by running rank_bm25 0.2.2 from PyPI;contract/bm25-47aa3dd.contract.jsonholds 407 from git 47aa3dd.tests/replay_contract.pyreplays them with exact comparison (floats as hex): 407/407 each. A control build that nudges scores by 1 ulp (tests/control_ulp_off.py) fails 263 of them, so the replay is not blind.Fuzz (
tests/fuzz_equivalence.py): 1,000 random cases against each reference, across all three variants and edge parameters, compared bit for bit on scores, batch scores, idf, doc_freqs, avgdl, average_idf, and top-n: 0 failures.Every wheel in CI is installed on its own platform and runs the test suite, a 300-case fuzz against rank_bm25 0.2.2, and both contract replays.
Wheels are built with
cargo-auditable(dependency list embedded in the.dep-v0section).
MCP server
rapidbm25-mcp is a stdio MCP server shipped in the wheel (no MCP SDK dependency):
bm25_index_texts, bm25_index_path, bm25_search, bm25_list_indexes, bm25_drop_index.
Text indexes live in memory. bm25_index_path caches the folder index under
RAPIDBM25_CACHE (default ~/.cache/rapidbm25, directory mode 0700, files mode 0600).
A cached file is reused only when its size, mtime, and the SHA-256 of the bytes that
were indexed still match; the path is statted again after the read. Symlinks are not
followed. Files with another hard link (st_nlink > 1) are skipped. Dotfiles are
skipped unless the glob's filename starts with .. Reads stay inside RAPIDBM25_ROOTS
(default: the working directory). Nothing leaves the machine.
{ "mcpServers": { "rapidbm25": { "command": "uvx", "args": ["--from", "rapidbm25", "rapidbm25-mcp"],
"env": { "RAPIDBM25_ROOTS": "/path/to/repo" } } } }Details: mcp/README.md. Registry metadata: server.json.
Agent skill
skill/rapidbm25-search/ (SKILL.md + scripts/bm25_search.py) for agents that run shell commands:
python skill/rapidbm25-search/scripts/bm25_search.py --path ./src --query "parse_header timeout" -k 10Agent eval
eval/agent/ asks 20 code-search questions against the urllib3 commit pinned in
eval/agent/tasks.json (release 2.8.0). Each answer is one file and one symbol, checked
automatically. The harness asks the same model the same prompt twice: once with the
rapidbm25 MCP tools, once with grep and read-file. It writes runs.json and summary.md
(correctness rate and medians) under eval/agent/out/.
Dry-run, with a scripted model and no API key:
python -m venv .venv && . .venv/bin/activate
pip install "maturin>=1.7,<2" && maturin develop
python eval/agent/harness.py --dry-runLive run, against an https OpenAI-compatible chat completions endpoint
(http is accepted only for localhost and 127.0.0.1):
export EVAL_API_KEY=... # secret; never commit it
export EVAL_MODEL=...
export EVAL_BASE_URL=https://api.openai.com/v1 # root that serves /chat/completions
python eval/agent/harness.pyThe GitHub workflow agent-eval runs the live comparison only from Actions → Run workflow
on main, and only if the EVAL_API_KEY repository secret is set. A dispatch from any
other ref stops before that secret is loaded. The form supplies the model and base URL.
Pull requests run the dry-run and do not call a model.
Build from source
pip install maturin && maturin build --release # or scripts/build_auditable.shLicense
Apache-2.0. See LICENSE and NOTICE (API compatibility with rank_bm25 is credited there).
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Route each agent task to the few tools it needs (BM25 shortlist). Free tier; x402 or credits.
Persistent memory and knowledge management for AI agents with semantic search and 50+ tools.
Private, multimodal knowledge store for agents: hybrid search with cited, provenance-backed results.
Universal persistent memory and knowledge retrieval layer for AI agents and LLMs.
Related MCP Servers
- FlicenseAqualityDmaintenanceEnables AI agents to store, search, and retrieve long-term memories with BM25 full-text search, auto-tagging, and importance scoring.7-
- AlicenseNot gradedqualityDmaintenanceEnables AI agents to store and retrieve persistent memories using BM25 search, allowing them to remember past conversations and context across sessions.8MIT
- AlicenseAqualityDmaintenanceEnables coding agents to search local notes via BM25 keyword ranking, returning relevant passages with file paths and line numbers, fully offline with zero dependencies.3MIT
- AlicenseNot gradedqualityCmaintenanceEnables agents to search and retrieve past workspace conversations, dialogue transcripts, and markdown artifacts through local BM25 retrieval, providing cross-conversation architectural recall with low-latency, offline operation.MIT