A
licenseNot graded
qualityD
maintenanceEnables AI agents to query and manage a document knowledge base via MCP, with RAG-powered search and grounded answers with citations.
MIT
rulebook-mcp
Provides tools (search_rules, get_section, list_sections) for retrieving, searching and listing citable sections of the official 2026-27 NBA Playing Rules (downloaded from nba.com), so an agent can answer questions about playing rules with answers grounded in and citing the correct rule sections.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@rulebook-mcphow long can a defender camp in the paint?"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
rulebook-ai — An NBA Rulebook Assistant, Measured
Ask a question about the official 2026-27 NBA Playing Rules and get a short answer that cites the rule section it comes from, or an honest "the rulebook does not address this". The project has three parts:
Retrieval over the rulebook, split into 122 citable sections (196 chunks): keyword (BM25), local embeddings, and a hybrid of the two
An MCP server (
search_rules,get_section,list_sections), so Claude Code, Claude Desktop or any MCP client can use the rulebook as a toolAn evaluation harness: 50 gold questions, five answering strategies, and accuracy, citation, abstention, latency and token-cost measurements
Everything runs locally: Gemma 4 (8B) and nomic-embed-text through Ollama on a 16 GB MacBook. Running the evaluation costs nothing, and token counts show what a hosted model would cost.
Results
42 answerable questions, phrased the way staff or fans ask them ("How long can a defender camp in the paint?"),
plus 8 the rulebook cannot answer (salary cap, draft lottery). Answers are graded automatically against facts that a
test checks appear in the cited rule text (evals/questions.yaml, every answer in results/answers.jsonl).
Strategy | Correct (95% CI) | Cites the right section | Declines out-of-scope questions | Answers citing a section that doesn't exist | Median time | Tokens in / out |
Model alone | 28.6% (17–44) | 0% | 5 / 8 | 4 | 0.9 s | 113 / 40 |
RAG, hybrid search | 92.9% (81–97) | 81% | 8 / 8 | 0 | 5.3 s | 1,656 / 45 |
RAG, embedding search | 95.2% (84–99) | 88% | 8 / 8 | 0 | 4.8 s | 1,584 / 44 |
Agent v1: tools, short excerpts | 76.2% (62–87) | 83% | 8 / 8 | 0 | 4.8 s | 1,409 / 68 |
Agent v2: full text, "search again" rule | 71.4% (56–83) | 60% | 8 / 8 | 0 | 5.5 s | 1,400 / 51 |
Agent v3: search done before it starts | 90.5% (78–96) | 86% | 8 / 8 | 0 | 5.9 s | 1,742 / 52 |
Retrieval alone (is a correct section among the top results?):
Top 1 | Top 3 | Top 6 | |
BM25 keywords | 52% | 76% | 86% |
Embeddings | 71% | 95% | 95% |
Hybrid (reciprocal rank fusion) | 67% | 86% | 95% |
What I learned
Grounding is the whole game. Alone, the model gets 29% right. It doesn't know 2026-27 changes, such as the second challenge awarded after a successful first one, and it invents citations to sections that don't exist. With the right sections in front of it, the same model is right 95% of the time and never cited a fake section.
Keywords lose to embeddings on how people actually ask. "Get the ball past half court" never says "midcourt line" or "backcourt", so BM25 misses it. Fusing keywords with embeddings made things worse here: 86% vs 95% at top 3.
An agent was not the right tool for this job. Letting the model decide when to search scored below a fixed pipeline. Version 1 searched once, read short excerpts, and gave up on 8 answerable questions. Version 2 added an instruction to search again before giving up, which seemed obviously helpful, but the longer prompt made the model skip its tools entirely on 9 questions and scored lower. Version 3 runs the first search for it and reaches 90.5%. Since it almost never searches twice, it is effectively RAG with extra steps. Agents earn their keep on multi-step tasks; single-lookup questions want a pipeline.
A bug that only the agents had. Every agent wrote the backboard as 6' by 3'; it's 3½', and RAG got it right. The tools returned JSON with Python's default
ensure_ascii, so the model saw3 \u00bd\u2019instead of "3 ½’". With the fix and a test, agent v3 went from 88.1% to 90.5% and gets the backboard right. Versions 1 and 2 ran with the bug.Where it still misses. Several strategies said the rulebook doesn't state the foul-out limit. The six-foul rule sits in Rule 3 (Team), not in the foul rules, so a search for "fouls" surfaces the wrong sections. That argues for adding glossary-style aliases to sections.
Grading, and what changed
The grader checks required facts on word boundaries, so "4" doesn't match inside "14". It counts a citation as correct if it names a gold section, and it counts an abstention as correct only on out-of-scope questions. While reading results I fixed grader bugs, re-scored every strategy's saved answers the same way, and did not re-run models:
"twirl" vs "twirling" and "eject" vs "ejected" (whole-word matching)
"after the third team foul" as a correct phrasing
markdown bold breaking "will not"
paraphrased refusals ("the rulebook does not contain...")
The three agent versions were developed against these same 50 questions, so a held-out question set is the next step before trusting the 90.5%.
Related MCP server: lector
Use it
uv sync
ollama pull gemma4 && ollama pull nomic-embed-text
uv run python -c "from rulebook_ai.ingest import build, fetch; build(fetch('data/rules.pdf'), 'data/sections.json')"
uv run python -m rulebook_ai.evaluate # about 20 minutes on a laptop; resumes if interrupted
uv run python -m rulebook_ai.evaluate --regrade # re-score saved answers without calling the model
uv run pytest # parser, search, grader, gold-set integrity, MCP toolsAs an MCP server (for example in Claude Code):
claude mcp add nba-rules -- uv --directory /path/to/rulebook-ai run rulebook-mcpThen ask "What happens after a successful coach's challenge?" and the model searches the official text itself.
How it's built
ingest.py: downloads the official PDF and verifies its SHA-256. It splits the text into sections with stable ids:R7-IVis Rule 7, Section IV; Rule 12's parts becomeR12AandR12B; the Comments on the Rules becomeC-II-L. Long sections are split at their lettered items and keep the parent citation. Sub-items lettered I, V or X were first misread as new Roman-numeral parts, which a test now covers.search.py: BM25, cosine search over nomic-embed-text vectors, and reciprocal rank fusion.answer.py: model alone, RAG, and the three agent versions withsearch_rulesandget_sectiontools. Every answer cites section ids in brackets.mcp_server.py: the same tools over the Model Context Protocol (Python SDK 2.x), tested with a real stdio client.evaluate.py: retrieval recall, graded answers, Wilson confidence intervals, latency and tokens. Answers are saved as they are produced: the first full run died when the local model exhausted 16 GB of memory at a 16k context, so the harness now retries, resumes, and runs at 8k.
Limitations
One 8B local model; a larger model would likely close the reading errors. The harness takes any Ollama model (
RULEBOOK_MODEL=...).50 questions give wide confidence intervals: 95% and 93% are not distinguishable, while 29% vs 95% clearly is.
The rulebook PDF is the NBA's and is not redistributed here; the build downloads it from nba.com.
License and AI disclosure
Code under MIT (see LICENSE). Built with help from Claude (Anthropic) via Claude Code; I reviewed, tested and
verified the code and results.
Available Tools
3 toolsget_sectionA
Full text of one rulebook section by id, e.g. 'R14-V' (Rule 14, Section V) or 'C-II-L' (Comments).
| Name | Required | Description | Default |
|---|---|---|---|
| section_id | Yes |
TDQS
A3.7/5.0
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden; it does usefully disclose that the return value is the complete section text rather than a summary or excerpt. It says nothing about behavior on an unknown id, whether large sections are truncated, or any access constraints.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
One sentence, front-loaded with what is returned, then the id examples. Nothing redundant and no wasted framing.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-required-parameter read tool with no output schema and no annotations, this covers the essentials: what comes back and what the id looks like. Only edge-case behavior (invalid ids, oversized sections) is absent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must supply parameter meaning, and it does: it gives two worked examples and decodes their structure ('R14-V' = Rule 14, Section V; 'C-II-L' = Comments). A full grammar for the id format would earn a 5.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific resource and scope: the full text of exactly one rulebook section, retrieved by id. This implicitly separates it from list_sections and search_rules, though it never names them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'by id' implies an id must already be known (presumably from list_sections or search_rules), but the description never states when to prefer this tool over those siblings or what to do without an id. Usage is inferable rather than specified.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_sectionsA
The rulebook's table of contents: every section id with its title.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
A3.7/5.0
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are supplied, so the description bears the full burden. It does convey that the call is a pure enumeration returning id/title pairs, which signals a safe read, but it says nothing about cost, pagination, or whether the section list is exhaustive across rulebook versions. For a zero-parameter read, the gap is modest but real.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence, front-loaded with the mental model and immediately followed by the returned fields. No redundant restatement of the name, no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no parameters, an output schema present, and a simple read-only enumeration, the description covers essentially everything an agent needs to decide to call it. Only an explicit pointer to get_section as the follow-up would make it complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so there is nothing for the description to disambiguate. The baseline for a parameterless tool applies, and the description correctly avoids inventing filter options that do not exist.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a concrete metaphor ('table of contents') tied to a specific resource ('the rulebook') and states the exact payload: every section id with its title. It implicitly distinguishes itself from search_rules (which finds by query) and get_section (which fetches one section), though it never names either sibling explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: an agent can infer this is the discovery step before get_section, but the description never says when to use it, when not to, or that search_rules is the alternative for targeted lookups. The 'table of contents' framing carries the guidance rather than explicit routing language.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_rulesB
Search the 2026-27 NBA playing rules. Returns the most relevant sections with ids and an excerpt.
| Name | Required | Description | Default |
|---|---|---|---|
| k | No | ||
| query | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
B3.3/5.0
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and only partially meets it: it discloses the corpus (2026-27 rules) and that ranking is by relevance, returning ids plus excerpts. It says nothing about how many results come back, whether results are truncated, or how excerpts relate to full sections.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, no filler, purpose stated first and return shape second. Every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so the description needn't explain return values in detail and it covers the essentials of what the tool searches. However, a search tool with an unexplained result-count parameter and unnamed sibling alternatives is only minimally complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the parameters 'query' and especially 'k' are undocumented structurally. The description implies relevance ranking but never explains that k controls the number of results or its default of 5, leaving the main non-obvious parameter unaddressed.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb and resource ('Search the 2026-27 NBA playing rules') and states the return shape (relevant sections with ids and excerpts). It implicitly contrasts with get_section by focusing on retrieval rather than lookup, though it never names the siblings explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no statement of when to use this versus list_sections or get_section, no mention of prerequisites, and no note that results are relevance-ranked or limited. The agent must infer that search is for unknown-section queries and that ids feed get_section.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
3 tool updates
v0.1.0- First observed
get_section - First observed
list_sections - First observed
search_rules
TDQS
A4/5.0
Scored across 3 tools
Disambiguation5/5
The three tools have clearly distinct roles: search for relevance-based discovery, get_section for fetching full text by known id, and list_sections for browsing the table of contents. An agent can easily choose between them based on whether it needs search, direct retrieval, or an index.
Naming Consistency5/5
All tool names follow a consistent snake_case verb_noun pattern: search_rules, get_section, list_sections. Singular/plural usage is also coherent with each operation's object.
Tool Count5/5
Three tools are well-scoped for a read-only rulebook reference server. Each tool supports a distinct access pattern—search, browse, and retrieve—without unnecessary surface area.
Completeness5/5
The surface covers the full read-only lifecycle for this domain: discover sections via list_sections, find relevant content via search_rules, and retrieve full text via get_section. No obvious operational gaps exist for a rulebook reference tool.
Maintenance
ActivityMaintained
ResponsivenessNo recent community issues
Related MCP Connectors
Resolve, search and verify legal citations against the official sources, with provenance.
Read-only hosted MCP over CanonicAI's cited Answers corpus on canonicai.com.
Judged, citation-checked policy corpus over MCP. Keyless public reads; API key for AI tools.
Query your own PDFs and documents from any MCP client. Every answer cites the page it came from.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables MCP clients to search and retrieve cited passages from the EU AI Act in English or Italian, including full articles and cross-references, so agents can ground answers in verified regulatory text.MIT
- AlicenseNot gradedqualityCmaintenanceEnables MCP clients to ask questions over the indexed document corpus and search handbook passages, returning cited answers and ranked results.MIT
- AlicenseNot gradedqualityCmaintenanceEnables MCP clients to list, search, and read SEC 10-K filings, returning relevant passages with company, fiscal year, section path, and SEC URL citations so answers can be grounded in source documents.MIT