MCP Spark Documentation Server
Provides tools for searching and retrieving Apache Spark documentation, enabling full-text keyword searches with section filtering and access to the full content of documentation pages.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@MCP Spark Documentation Serversearch for window functions in the sql-ref section"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
MCP Spark Documentation Server
An MCP (Model Context Protocol) server that provides search and retrieval tools for Apache Spark documentation. This server enables AI assistants like Claude to search and read Spark documentation directly.
Features
Full-text search using SQLite FTS5 with BM25 ranking and Porter stemming
Section filtering to narrow search results by documentation category
Sparse checkout for efficient cloning of only the docs directory from apache/spark
Docker support for portable deployment across projects
STDIO transport for seamless MCP client integration
Related MCP server: anydocs
Quick Start
Using Docker (Recommended)
# Build the Docker image (includes pre-indexed documentation)
make docker-build
# Test the server
make docker-runUsing uv (Local Development)
# Initialise the environment
make init
# Build the documentation index
make index
# Run the server
make runConfiguration
Claude Code / Claude Desktop
Add to your .mcp.json or global settings:
{
"mcpServers": {
"spark-documentation": {
"command": "docker",
"args": ["run", "-i", "--rm", "martoc/mcp-spark-documentation:latest"]
}
}
}For a locally built Docker image:
{
"mcpServers": {
"spark-documentation": {
"command": "docker",
"args": ["run", "-i", "--rm", "mcp-spark-documentation"]
}
}
}For local development without Docker:
{
"mcpServers": {
"spark-documentation": {
"command": "uv",
"args": ["run", "mcp-spark-documentation"],
"cwd": "/path/to/mcp-spark-documentation"
}
}
}MCP Tools
Tool | Description |
| Search Spark documentation by keyword query with optional section filtering |
| Retrieve the full content of a specific documentation page |
search_documentation
Search Apache Spark documentation using full-text search with stemming support.
Parameter | Type | Required | Default | Description |
| string | Yes | - | Search terms (supports stemming) |
| string | No | None | Filter by section (e.g., sql-ref, streaming, mllib) |
| integer | No | 10 | Maximum results (1-50) |
Common Sections: sql-ref, api, streaming, mllib, graphx, structured-streaming, configuration, tuning
read_documentation
Retrieve the full content of a documentation page.
Parameter | Type | Required | Description |
| string | Yes | Relative path to document (from search results) |
CLI Commands
# Build/rebuild the documentation index
uv run spark-docs-index index
uv run spark-docs-index index --rebuild
uv run spark-docs-index index --branch master
# Show index statistics
uv run spark-docs-index statsDevelopment
make init # Initialise development environment
make build # Run full build (lint, typecheck, test)
make test # Run tests with coverage
make format # Format code
make lint # Run linter
make typecheck # Run type checkerDocumentation
USAGE.md - Detailed usage instructions
CODESTYLE.md - Code style guidelines
CLAUDE.md - Claude Code instructions
Licence
This project is licensed under the MIT Licence - see the LICENSE file for details.
Available Tools
2 toolsread_documentationA
Read the full content of a specific Spark documentation page.
| Name | Required | Description | Default |
|---|---|---|---|
| path | Yes | The relative path to the documentation file (e.g., 'sql-ref/sql-syntax.md' or 'api/python/index.md'). This path is returned in search results. |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden, and it does disclose one useful trait: it returns the FULL page content rather than a snippet. It does not address page size or truncation, whether an invalid path errors or returns empty, or any cost/latency characteristics of fetching whole documents.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One sentence, front-loaded with the verb and resource, with zero redundant wording. Nothing to trim and nothing buried.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter read tool with an output schema that covers return values, the description is nearly sufficient; the remaining gap is the missing explicit hand-off from search_documentation to this tool. The schema compensates for path semantics, so completeness is high.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the single `path` parameter is documented with concrete examples and its provenance (returned in search results), so the schema already does the work. The description adds nothing about path format or validation beyond that, which is the correct baseline for a fully described single parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
"Read the full content of a specific Spark documentation page" gives a specific verb (read) and resource (documentation page) and scopes it to a single page, which implicitly separates it from search_documentation. It stops short of naming the sibling outright, so an agent must infer the contrast.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: the phrase "a specific page" plus the schema note that paths are returned in search results suggests this is the follow-up call after a search. There is no explicit statement of when to use this versus search_documentation, nor any precondition about obtaining a path first.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_documentationC
Search Apache Spark documentation by keyword query.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Maximum number of results to return (default: 10, max: 50). | |
| query | Yes | Search terms to find in the documentation. Supports full-text search with stemming (e.g., "stream" matches "streaming", "streams"). | |
| section | No | Optional section to filter results. Common sections include: 'sql-ref', 'api', 'streaming', 'mllib', 'graphx', 'structured-streaming', etc. |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations supplied, the description carries the full behavioral burden and discloses almost nothing: no ranking behavior, result shape, pagination, or scope limits. The only behavioral signal is 'keyword query', which is thin for a tool with no annotation coverage.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no filler, so it is efficient and easy to scan. It is arguably too terse for the tool's role, but that gap is captured under other dimensions rather than as verbosity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple: three fully documented parameters and an output schema, so return values need no explanation here. The remaining gap is routing guidance against read_documentation and any note on search behavior, which the description omits.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the query, limit, and section parameters are already fully documented, including stemming behavior and common section values. The description adds no parameter meaning beyond that, which meets the baseline for fully covered schemas.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description gives a specific verb and resource ('Search Apache Spark documentation by keyword query'), so an agent knows exactly what the tool does. It does not differentiate from the sibling read_documentation, leaving the search-vs-read choice to the name alone.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no statement of when to use this tool versus read_documentation, nor any prerequisite or exclusion. The only hint is the phrase 'keyword query', which implies lookup-by-term usage but is never made explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
2 tool updates
v0.3.7- Changed
read_documentation1 field changed- added
Input schema / properties / path / descriptionAdded value: +"The relative path to the documentation file (e.g.,\n 'sql-ref/sql-syntax.md' or 'api/python/index.md'). This path\n is returned in search results."
- Changed
search_documentation3 fields changed- added
Input schema / properties / limit / descriptionAdded value: +"Maximum number of results to return (default: 10, max: 50)." - added
Input schema / properties / query / descriptionAdded value: +"Search terms to find in the documentation. Supports\n full-text search with stemming (e.g., \"stream\" matches\n \"streaming\", \"streams\")." - added
Input schema / properties / section / descriptionAdded value: +"Optional section to filter results. Common sections include:\n 'sql-ref', 'api', 'streaming', 'mllib', 'graphx',\n 'structured-streaming', etc."
2 tool updates
v0.2.6- Changed
read_documentation1 field changed- added
Input schema / additionalPropertiesAdded value: +false
- Changed
search_documentation1 field changed- added
Input schema / additionalPropertiesAdded value: +false
2 tool updates
v0.1.0- First observed
read_documentation - First observed
search_documentation
TDQS
Scored across 2 tools
search_documentation and read_documentation have clearly complementary, non-overlapping purposes: one finds pages, the other fetches a specific page's content. An agent can trivially pick the right one based on whether it has a page identifier or only a query.
Both tools follow the same snake_case verb_noun pattern (search_documentation, read_documentation), with consistent verb style and a shared noun suffix.
Two tools is a tight, purposeful surface for a documentation-lookup server, with no redundancy. It is slightly thin — discovery aids like listing pages or versions are absent — but nothing feels gratuitous.
Search-then-read covers the core documentation workflow end to end, with no dead ends for locating and consuming content. Minor gaps exist around browsing structure (listing available pages, versions, or sections), which the search tool partially compensates for.
Maintenance
Related MCP Connectors
Search and read Rust documentation for the standard library and any crate on crates.io
Search and read Vector Panda docs: API operations, pricing, storage tiers, measured benchmarks.
Versioned documentation registry and semantic search for AI tools and coding assistants.
Docs Q&A: search 169 data and AI guides, fetch any page as markdown. Read-only, keyless.
Related MCP Servers
- AlicenseAqualityDmaintenanceProvides tools to search and retrieve Google Gemini API documentation with full-text search capabilities and automatic documentation updates stored in a local SQLite database.3101MIT
- AlicenseAqualityBmaintenanceProvides fast, token-efficient search over coding agent documentation (e.g., Claude Code, Cursor) using local SQLite FTS5 indexing, with tools for searching snippets, reading pages, and grepping markdown.5MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI assistants to run incremental SQLite FTS5 full-text search over explicitly configured local workspace roots, covering Markdown, code, PDF, DOCX, XLSX, CSV and HTML with BM25 ranking, Chinese bigrams, and phrase, prefix, path, type and date filters. Exposes search, safe root-confined reads with line/page/sheet locations, recent-file listing, and reindexing as MCP tools so assistants can locate and quote source content without network or account access.MIT
- AlicenseAqualityCmaintenanceEnables AI agents to search a local folder of Markdown, AsciiDoc, and text files through a SQLite FTS5 keyword index, returning the most relevant sections with file paths, line ranges, and heading breadcrumbs instead of grepping through a repo. It also exposes tools to list indexed documents and refresh the index incrementally, all offline with no dependencies, network calls, or embeddings.3MIT