Skip to main content
Glama
rulebook-mcp

rulebook-mcp

rulebook-ai — An NBA Rulebook Assistant, Measured

Ask a question about the official 2026-27 NBA Playing Rules and get a short answer that cites the rule section it comes from, or an honest "the rulebook does not address this". The project has three parts:

  • Retrieval over the rulebook, split into 122 citable sections (196 chunks): keyword (BM25), local embeddings, and a hybrid of the two

  • An MCP server (search_rules, get_section, list_sections), so Claude Code, Claude Desktop or any MCP client can use the rulebook as a tool

  • An evaluation harness: 50 gold questions, five answering strategies, and accuracy, citation, abstention, latency and token-cost measurements

Everything runs locally: Gemma 4 (8B) and nomic-embed-text through Ollama on a 16 GB MacBook. Running the evaluation costs nothing, and token counts show what a hosted model would cost.

Results

42 answerable questions, phrased the way staff or fans ask them ("How long can a defender camp in the paint?"), plus 8 the rulebook cannot answer (salary cap, draft lottery). Answers are graded automatically against facts that a test checks appear in the cited rule text (evals/questions.yaml, every answer in results/answers.jsonl).

Strategy

Correct (95% CI)

Cites the right section

Declines out-of-scope questions

Answers citing a section that doesn't exist

Median time

Tokens in / out

Model alone

28.6% (17–44)

0%

5 / 8

4

0.9 s

113 / 40

RAG, hybrid search

92.9% (81–97)

81%

8 / 8

0

5.3 s

1,656 / 45

RAG, embedding search

95.2% (84–99)

88%

8 / 8

0

4.8 s

1,584 / 44

Agent v1: tools, short excerpts

76.2% (62–87)

83%

8 / 8

0

4.8 s

1,409 / 68

Agent v2: full text, "search again" rule

71.4% (56–83)

60%

8 / 8

0

5.5 s

1,400 / 51

Agent v3: search done before it starts

90.5% (78–96)

86%

8 / 8

0

5.9 s

1,742 / 52

Retrieval alone (is a correct section among the top results?):

Top 1

Top 3

Top 6

BM25 keywords

52%

76%

86%

Embeddings

71%

95%

95%

Hybrid (reciprocal rank fusion)

67%

86%

95%

What I learned

  • Grounding is the whole game. Alone, the model gets 29% right. It doesn't know 2026-27 changes, such as the second challenge awarded after a successful first one, and it invents citations to sections that don't exist. With the right sections in front of it, the same model is right 95% of the time and never cited a fake section.

  • Keywords lose to embeddings on how people actually ask. "Get the ball past half court" never says "midcourt line" or "backcourt", so BM25 misses it. Fusing keywords with embeddings made things worse here: 86% vs 95% at top 3.

  • An agent was not the right tool for this job. Letting the model decide when to search scored below a fixed pipeline. Version 1 searched once, read short excerpts, and gave up on 8 answerable questions. Version 2 added an instruction to search again before giving up, which seemed obviously helpful, but the longer prompt made the model skip its tools entirely on 9 questions and scored lower. Version 3 runs the first search for it and reaches 90.5%. Since it almost never searches twice, it is effectively RAG with extra steps. Agents earn their keep on multi-step tasks; single-lookup questions want a pipeline.

  • A bug that only the agents had. Every agent wrote the backboard as 6' by 3'; it's 3½', and RAG got it right. The tools returned JSON with Python's default ensure_ascii, so the model saw 3 \u00bd\u2019 instead of "3 ½’". With the fix and a test, agent v3 went from 88.1% to 90.5% and gets the backboard right. Versions 1 and 2 ran with the bug.

  • Where it still misses. Several strategies said the rulebook doesn't state the foul-out limit. The six-foul rule sits in Rule 3 (Team), not in the foul rules, so a search for "fouls" surfaces the wrong sections. That argues for adding glossary-style aliases to sections.

Grading, and what changed

The grader checks required facts on word boundaries, so "4" doesn't match inside "14". It counts a citation as correct if it names a gold section, and it counts an abstention as correct only on out-of-scope questions. While reading results I fixed grader bugs, re-scored every strategy's saved answers the same way, and did not re-run models:

  • "twirl" vs "twirling" and "eject" vs "ejected" (whole-word matching)

  • "after the third team foul" as a correct phrasing

  • markdown bold breaking "will not"

  • paraphrased refusals ("the rulebook does not contain...")

The three agent versions were developed against these same 50 questions, so a held-out question set is the next step before trusting the 90.5%.

Related MCP server: lector

Use it

uv sync
ollama pull gemma4 && ollama pull nomic-embed-text
uv run python -c "from rulebook_ai.ingest import build, fetch; build(fetch('data/rules.pdf'), 'data/sections.json')"
uv run python -m rulebook_ai.evaluate          # about 20 minutes on a laptop; resumes if interrupted
uv run python -m rulebook_ai.evaluate --regrade  # re-score saved answers without calling the model
uv run pytest                                  # parser, search, grader, gold-set integrity, MCP tools

As an MCP server (for example in Claude Code):

claude mcp add nba-rules -- uv --directory /path/to/rulebook-ai run rulebook-mcp

Then ask "What happens after a successful coach's challenge?" and the model searches the official text itself.

How it's built

  • ingest.py: downloads the official PDF and verifies its SHA-256. It splits the text into sections with stable ids: R7-IV is Rule 7, Section IV; Rule 12's parts become R12A and R12B; the Comments on the Rules become C-II-L. Long sections are split at their lettered items and keep the parent citation. Sub-items lettered I, V or X were first misread as new Roman-numeral parts, which a test now covers.

  • search.py: BM25, cosine search over nomic-embed-text vectors, and reciprocal rank fusion.

  • answer.py: model alone, RAG, and the three agent versions with search_rules and get_section tools. Every answer cites section ids in brackets.

  • mcp_server.py: the same tools over the Model Context Protocol (Python SDK 2.x), tested with a real stdio client.

  • evaluate.py: retrieval recall, graded answers, Wilson confidence intervals, latency and tokens. Answers are saved as they are produced: the first full run died when the local model exhausted 16 GB of memory at a 16k context, so the harness now retries, resumes, and runs at 8k.

Limitations

  • One 8B local model; a larger model would likely close the reading errors. The harness takes any Ollama model (RULEBOOK_MODEL=...).

  • 50 questions give wide confidence intervals: 95% and 93% are not distinguishable, while 29% vs 95% clearly is.

  • The rulebook PDF is the NBA's and is not redistributed here; the build downloads it from nba.com.

License and AI disclosure

Code under MIT (see LICENSE). Built with help from Claude (Anthropic) via Claude Code; I reviewed, tested and verified the code and results.

Available Tools

3 tools
get_sectionA

Full text of one rulebook section by id, e.g. 'R14-V' (Rule 14, Section V) or 'C-II-L' (Comments).

Inputs: section_id (required).

list_sectionsA

The rulebook's table of contents: every section id with its title.

Inputs: no named parameters. Returns: result

search_rulesB

Search the 2026-27 NBA playing rules. Returns the most relevant sections with ids and an excerpt.

Inputs: query (required), k. Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4/5.0

    Scored across 3 tools

    Disambiguation5/5

    The three tools have clearly distinct roles: search for relevance-based discovery, get_section for fetching full text by known id, and list_sections for browsing the table of contents. An agent can easily choose between them based on whether it needs search, direct retrieval, or an index.

    Naming Consistency5/5

    All tool names follow a consistent snake_case verb_noun pattern: search_rules, get_section, list_sections. Singular/plural usage is also coherent with each operation's object.

    Tool Count5/5

    Three tools are well-scoped for a read-only rulebook reference server. Each tool supports a distinct access pattern—search, browse, and retrieve—without unnecessary surface area.

    Completeness5/5

    The surface covers the full read-only lifecycle for this domain: discover sections via list_sections, find relevant content via search_rules, and retrieve full text via get_section. No obvious operational gaps exist for a rulebook reference tool.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables AI agents to query and manage a document knowledge base via MCP, with RAG-powered search and grounded answers with citations.
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables MCP clients to search and retrieve cited passages from the EU AI Act in English or Italian, including full articles and cross-references, so agents can ground answers in verified regulatory text.
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables MCP clients to list, search, and read SEC 10-K filings, returning relevant passages with company, fiscal year, section path, and SEC URL citations so answers can be grounded in source documents.
      MIT