rulebook-ai — An NBA Rulebook Assistant, Measured

Ask a question about the official 2026-27 NBA Playing Rules and get a short answer that cites the rule section it comes from, or an honest "the rulebook does not address this". The project has three parts:

• Retrieval over the rulebook, split into 122 citable sections (196 chunks): keyword (BM25), local embeddings, and a hybrid of the two

• An MCP server (search_rules, get_section, list_sections), so Claude Code, Claude Desktop or any MCP client can use the rulebook as a tool

• An evaluation harness: 50 gold questions, five answering strategies, and accuracy, citation, abstention, latency and token-cost measurements

Everything runs locally: Gemma 4 (8B) and nomic-embed-text through Ollama on a 16 GB MacBook. Running the evaluation costs nothing, and token counts show what a hosted model would cost.

Results

42 answerable questions, phrased the way staff or fans ask them ("How long can a defender camp in the paint?"), plus 8 the rulebook cannot answer (salary cap, draft lottery). Answers are graded automatically against facts that a test checks appear in the cited rule text (evals/questions.yaml, every answer in results/answers.jsonl).

Retrieval alone (is a correct section among the top results?):

What I learned

• Grounding is the whole game. Alone, the model gets 29% right. It doesn't know 2026-27 changes, such as the second challenge awarded after a successful first one, and it invents citations to sections that don't exist. With the right sections in front of it, the same model is right 95% of the time and never cited a fake section.

• Keywords lose to embeddings on how people actually ask. "Get the ball past half court" never says "midcourt line" or "backcourt", so BM25 misses it. Fusing keywords with embeddings made things worse here: 86% vs 95% at top 3.

• An agent was not the right tool for this job. Letting the model decide when to search scored below a fixed pipeline. Version 1 searched once, read short excerpts, and gave up on 8 answerable questions. Version 2 added an instruction to search again before giving up, which seemed obviously helpful, but the longer prompt made the model skip its tools entirely on 9 questions and scored lower. Version 3 runs the first search for it and reaches 90.5%. Since it almost never searches twice, it is effectively RAG with extra steps. Agents earn their keep on multi-step tasks; single-lookup questions want a pipeline.

• A bug that only the agents had. Every agent wrote the backboard as 6' by 3'; it's 3½', and RAG got it right. The tools returned JSON with Python's default ensure_ascii, so the model saw 3 \u00bd\u2019 instead of "3 ½’". With the fix and a test, agent v3 went from 88.1% to 90.5% and gets the backboard right. Versions 1 and 2 ran with the bug.

• Where it still misses. Several strategies said the rulebook doesn't state the foul-out limit. The six-foul rule sits in Rule 3 (Team), not in the foul rules, so a search for "fouls" surfaces the wrong sections. That argues for adding glossary-style aliases to sections.

Grading, and what changed

The grader checks required facts on word boundaries, so "4" doesn't match inside "14". It counts a citation as correct if it names a gold section, and it counts an abstention as correct only on out-of-scope questions. While reading results I fixed grader bugs, re-scored every strategy's saved answers the same way, and did not re-run models:

• "twirl" vs "twirling" and "eject" vs "ejected" (whole-word matching)

• "after the third team foul" as a correct phrasing

• markdown bold breaking "will not"

• paraphrased refusals ("the rulebook does not contain...")

The three agent versions were developed against these same 50 questions, so a held-out question set is the next step before trusting the 90.5%.

Related MCP server: Aegis

Use it

uv sync
ollama pull gemma4 && ollama pull nomic-embed-text
uv run python -c "from rulebook_ai.ingest import build, fetch; build(fetch('data/rules.pdf'), 'data/sections.json')"
uv run python -m rulebook_ai.evaluate          # about 20 minutes on a laptop; resumes if interrupted
uv run python -m rulebook_ai.evaluate --regrade  # re-score saved answers without calling the model
uv run pytest                                  # parser, search, grader, gold-set integrity, MCP tools

As an MCP server (for example in Claude Code):

claude mcp add nba-rules -- uv --directory /path/to/rulebook-ai run rulebook-mcp

Then ask "What happens after a successful coach's challenge?" and the model searches the official text itself.

How it's built

• ingest.py: downloads the official PDF and verifies its SHA-256. It splits the text into sections with stable ids: R7-IV is Rule 7, Section IV; Rule 12's parts become R12A and R12B; the Comments on the Rules become C-II-L. Long sections are split at their lettered items and keep the parent citation. Sub-items lettered I, V or X were first misread as new Roman-numeral parts, which a test now covers.

• search.py: BM25, cosine search over nomic-embed-text vectors, and reciprocal rank fusion.

• answer.py: model alone, RAG, and the three agent versions with search_rules and get_section tools. Every answer cites section ids in brackets.

• mcp_server.py: the same tools over the Model Context Protocol (Python SDK 2.x), tested with a real stdio client.

• evaluate.py: retrieval recall, graded answers, Wilson confidence intervals, latency and tokens. Answers are saved as they are produced: the first full run died when the local model exhausted 16 GB of memory at a 16k context, so the harness now retries, resumes, and runs at 8k.

Limitations

• One 8B local model; a larger model would likely close the reading errors. The harness takes any Ollama model (RULEBOOK_MODEL=...).

• 50 questions give wide confidence intervals: 95% and 93% are not distinguishable, while 29% vs 95% clearly is.

• The rulebook PDF is the NBA's and is not redistributed here; the build downloads it from nba.com.

License and AI disclosure

Code under MIT (see LICENSE). Built with help from Claude (Anthropic) via Claude Code; I reviewed, tested and verified the code and results.