golden-dataset-mcp
This server provides version-controlled golden dataset management and semantic evaluation for RAG/LLM pipelines, exposing tools so an LLM agent can manage datasets conversationally. Every tool requires an explicit dataset_path, allowing management of multiple datasets from a single server instance.
Initialize a dataset (
init_dataset): Create a new golden dataset at a specified path with a name and optional description.Add entries (
add_entry): Add question-answer pairs (with optional tags, contexts, and metadata) to the working tree.Update entries (
update_entry): Edit fields of an existing working-tree entry by its ID.Delete entries (
delete_entry): Remove an entry from the working tree without affecting committed versions.List entries (
list_entries): View entries in the uncommitted working tree or a specific committed version.Commit versions (
commit_version): Snapshot the current working tree as a new immutable, auto-incremented version.Diff versions (
diff_versions): Compare two committed versions to see entries that were added, removed, or changed.Evaluate answers (
evaluate_answers): Score actual LLM/RAG-generated answers against a golden dataset version using TF-IDF cosine similarity — no LLM API key required — returning per-entry scores, average similarity, and pass/fail results.Check dataset status (
dataset_status): View the dataset's name, description, current version, working tree size, and full version history.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@golden-dataset-mcp Initialize a new golden dataset for RAG evaluation."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
mcp-name: io.github.nipunkhanderia/golden-dataset-mcp
golden-dataset-mcp
An MCP server wrapping golden-dataset-studio — version-controlled golden dataset management and semantic evaluation for RAG/LLM pipelines.
This is a thin protocol layer over the existing golden_dataset library (DatasetStore, Evaluator). It does not reimplement any logic — it exposes the library's existing Python API as MCP tools so an agent (Claude Desktop, Claude Code, or any MCP client) can manage golden datasets conversationally.
No LLM API key required. Evaluation uses TF-IDF cosine similarity (scikit-learn), not an LLM call.
Why a separate package from golden-dataset-studio?
golden-dataset-studio is a CLI tool — designed for a human typing golden add, golden commit, etc. in a terminal. golden-dataset-mcp exposes the same underlying operations as MCP tools so an LLM agent can drive them programmatically, e.g. as part of an automated RAG evaluation pipeline. Keeping them as separate PyPI packages means CLI users aren't forced to pull in fastmcp as a dependency, and MCP users get a clean, protocol-focused package.
Related MCP server: AgentOps EvalBench MCP
Tools
Tool | What it does |
| Initialise a new dataset at a given path |
| Add a question/answer pair to the working tree |
| Edit fields of an existing working-tree entry |
| Remove an entry from the working tree |
| List working-tree or committed-version entries |
| Snapshot the working tree as a new immutable version |
| Show entries added/removed/changed between two versions |
| Score actual answers against a version via TF-IDF cosine similarity |
| Show current version, working tree size, and version history |
Design: every tool takes an explicit dataset_path
Unlike the CLI (which operates on the current working directory), every tool here requires an explicit dataset_path parameter. This keeps the server fully stateless between calls — no hidden "current dataset" session state to lose track of, and safe for one server instance to manage multiple datasets or serve multiple concurrent clients.
Installation
pip install golden-dataset-mcpThis pulls in golden-dataset-studio and scikit-learn automatically as dependencies.
Usage with Claude Desktop / Claude Code
{
"mcpServers": {
"golden-dataset": {
"command": "golden-dataset-mcp"
}
}
}No environment variables needed — no API key, no config.
Example flow
1. init_dataset(dataset_path="./my-rag-eval", name="support-bot-eval")
2. add_entry(dataset_path="./my-rag-eval", question="...", answer="...")
[repeat for each golden Q&A pair]
3. commit_version(dataset_path="./my-rag-eval", description="initial 50 questions")
4. [run your RAG pipeline, collect actual answers]
5. evaluate_answers(dataset_path="./my-rag-eval", actual_answers=[...])
-> avg_semantic_similarity, per-entry scores, pass/failAs your RAG pipeline changes over time, commit_version again after edits and use diff_versions to see exactly what changed in your golden set between releases.
Relationship to the underlying library
|
| |
Interface | CLI ( | MCP tools |
Driven by | A human typing commands | An LLM agent / MCP client |
Path handling | Current working directory | Explicit |
Dependency direction | — | Depends on |
If you want the human-driven CLI, use golden-dataset-studio directly. If you want an agent to drive it, use this package.
Development
git clone https://github.com/nipunkhanderia/golden-dataset-mcp
cd golden-dataset-mcp
pip install -e ".[dev]"
pytest -vValidate the MCP-facing contract:
npx @modelcontextprotocol/inspector golden-dataset-mcpLimitations
evaluate_answersuses TF-IDF cosine similarity, which captures lexical overlap better than deep semantic meaning. For embedding-based or RAGAS-style metrics, call the underlying library'sEvaluator.ragas_evaluate()directly (requirespip install "golden-dataset-studio[ragas]"— not exposed as an MCP tool in this version).Very short or stop-word-only answers will raise an error. scikit-learn's TF-IDF vectorizer raises
ValueError: empty vocabularyon inputs like a bare"4"or"the a an". Avoid single-token golden answers, or expectevaluate_answersto fail on them.All state is filesystem-backed JSON/JSONL under
<dataset_path>/.golden_dataset/; this server does no remote storage or syncing.
License
MIT
Available Tools
9 toolsadd_entryA
Add a question-answer pair to the working tree of a golden dataset.
Entries added here are NOT yet versioned — call commit_version to snapshot them. dataset_path must already be initialised.
Inputs: input (required). Returns: id, status
commit_versionB
Snapshot the current working tree as a new immutable dataset version.
Versions auto-increment (1.0 -> 1.1 -> 1.2...). Fails if the working tree is empty.
Inputs: input (required). Returns: version, entry_count, sha256, parent_version
dataset_statusB
Show the current state of a golden dataset: name, current version, and working tree size.
Inputs: input (required). Returns: name, description, current_version, working_tree_entry_count, +1 more
delete_entryA
Remove an entry from the working tree by its id. Does not affect already-committed versions.
Inputs: input (required). Returns: id, status
diff_versionsB
Show entries added, removed, or changed between two committed versions.
Inputs: input (required). Returns: added, removed, changed
evaluate_answersA
Score actual LLM/RAG-generated answers against the golden dataset using TF-IDF cosine similarity (no LLM call, no API key needed).
actual_answers must be supplied in the same order as the entries in
the target version. Omit version to evaluate against the current
committed version.
Inputs: input (required). Returns: dataset_name, version, total_entries, avg_semantic_similarity, +2 more
init_datasetA
Initialise a new version-controlled golden dataset at dataset_path.
Creates a .golden_dataset/ directory there. Fails if one already exists at that path — delete .golden_dataset/ manually to start fresh.
Inputs: input (required). Returns: name, description, created_at
list_entriesA
List entries in a dataset. Omit version to see the uncommitted working tree; pass a version (e.g. '1.0') to see a committed snapshot.
Inputs: input (required). Returns: entries, count
update_entryA
Update fields of an existing working-tree entry by its id.
Only fields you provide are changed; omitted fields are left as-is. Raises an error if entry_id is not found in the working tree.
Inputs: input (required). Returns: id, question, answer, contexts, +3 more
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 9 tools
Each tool has a clearly distinct purpose: init, add, delete, update, list, commit, status, diff, and evaluate. No two tools overlap in functionality.
All tool names follow the verb_noun pattern in snake_case, e.g., add_entry, commit_version, list_entries. Consistent and predictable.
9 tools is well-scoped for a version-controlled golden dataset server. It covers all essential operations without being bloated or sparse.
The tool surface covers init, CRUD for working tree, versioning, listing, status, diff, and evaluation. Minor gap: no dedicated get-single-entry tool, but list_entries likely returns full entries.
Maintenance
Related MCP Connectors
LLM & RAG evals: hallucination, eval contracts, semantic tests, tokens & model info. Free, no key.
Ingest, manage, and retrieve documents for RAG-powered AI applications
Versioned documentation registry and semantic search for AI tools and coding assistants.
Semantic nearest-neighbour search over your datasets, plus indices and projections.
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceCreates deterministic, auditable vector databases from any content source with deployable RAG applications. Supports multiple embedding providers and vector databases with fine-grained pipeline control or project-based workflows.7-
- AlicenseNot gradedqualityCmaintenanceEnables LLM evaluation and observability by uploading documents, building test sets, running RAG pipelines, and automatically scoring answers for groundedness, hallucination risk, retrieval quality, latency, and cost, with tools exposed to MCP-compatible clients.1MIT
- AlicenseAqualityDmaintenanceEnables ingestion and semantic search over text documents using PostgreSQL + pgvector and OpenAI-compatible embeddings, allowing any LLM agent to retrieve relevant chunks for grounded answers.4AGPL 3.0
- FlicenseNot gradedqualityDmaintenanceEnables document Q&A and knowledge retrieval through hybrid semantic and keyword search, with tools for document ingestion, chunking, summarization, PII redaction, and RAGAS-based evaluation.-