Skip to main content
Glama
golden-dataset-mcp

golden-dataset-mcp

mcp-name: io.github.nipunkhanderia/golden-dataset-mcp

golden-dataset-mcp

An MCP server wrapping golden-dataset-studio — version-controlled golden dataset management and semantic evaluation for RAG/LLM pipelines.

This is a thin protocol layer over the existing golden_dataset library (DatasetStore, Evaluator). It does not reimplement any logic — it exposes the library's existing Python API as MCP tools so an agent (Claude Desktop, Claude Code, or any MCP client) can manage golden datasets conversationally.

No LLM API key required. Evaluation uses TF-IDF cosine similarity (scikit-learn), not an LLM call.

Why a separate package from golden-dataset-studio?

golden-dataset-studio is a CLI tool — designed for a human typing golden add, golden commit, etc. in a terminal. golden-dataset-mcp exposes the same underlying operations as MCP tools so an LLM agent can drive them programmatically, e.g. as part of an automated RAG evaluation pipeline. Keeping them as separate PyPI packages means CLI users aren't forced to pull in fastmcp as a dependency, and MCP users get a clean, protocol-focused package.

Related MCP server: AgentOps EvalBench MCP

Tools

Tool

What it does

init_dataset

Initialise a new dataset at a given path

add_entry

Add a question/answer pair to the working tree

update_entry

Edit fields of an existing working-tree entry

delete_entry

Remove an entry from the working tree

list_entries

List working-tree or committed-version entries

commit_version

Snapshot the working tree as a new immutable version

diff_versions

Show entries added/removed/changed between two versions

evaluate_answers

Score actual answers against a version via TF-IDF cosine similarity

dataset_status

Show current version, working tree size, and version history

Design: every tool takes an explicit dataset_path

Unlike the CLI (which operates on the current working directory), every tool here requires an explicit dataset_path parameter. This keeps the server fully stateless between calls — no hidden "current dataset" session state to lose track of, and safe for one server instance to manage multiple datasets or serve multiple concurrent clients.

Installation

pip install golden-dataset-mcp

This pulls in golden-dataset-studio and scikit-learn automatically as dependencies.

Usage with Claude Desktop / Claude Code

{
  "mcpServers": {
    "golden-dataset": {
      "command": "golden-dataset-mcp"
    }
  }
}

No environment variables needed — no API key, no config.

Example flow

1. init_dataset(dataset_path="./my-rag-eval", name="support-bot-eval")
2. add_entry(dataset_path="./my-rag-eval", question="...", answer="...")
   [repeat for each golden Q&A pair]
3. commit_version(dataset_path="./my-rag-eval", description="initial 50 questions")
4. [run your RAG pipeline, collect actual answers]
5. evaluate_answers(dataset_path="./my-rag-eval", actual_answers=[...])
   -> avg_semantic_similarity, per-entry scores, pass/fail

As your RAG pipeline changes over time, commit_version again after edits and use diff_versions to see exactly what changed in your golden set between releases.

Relationship to the underlying library

golden-dataset-studio

golden-dataset-mcp

Interface

CLI (golden ...)

MCP tools

Driven by

A human typing commands

An LLM agent / MCP client

Path handling

Current working directory

Explicit dataset_path per call

Dependency direction

—

Depends on golden-dataset-studio

If you want the human-driven CLI, use golden-dataset-studio directly. If you want an agent to drive it, use this package.

Development

git clone https://github.com/nipunkhanderia/golden-dataset-mcp
cd golden-dataset-mcp
pip install -e ".[dev]"
pytest -v

Validate the MCP-facing contract:

npx @modelcontextprotocol/inspector golden-dataset-mcp

Limitations

  • evaluate_answers uses TF-IDF cosine similarity, which captures lexical overlap better than deep semantic meaning. For embedding-based or RAGAS-style metrics, call the underlying library's Evaluator.ragas_evaluate() directly (requires pip install "golden-dataset-studio[ragas]" — not exposed as an MCP tool in this version).

  • Very short or stop-word-only answers will raise an error. scikit-learn's TF-IDF vectorizer raises ValueError: empty vocabulary on inputs like a bare "4" or "the a an". Avoid single-token golden answers, or expect evaluate_answers to fail on them.

  • All state is filesystem-backed JSON/JSONL under <dataset_path>/.golden_dataset/; this server does no remote storage or syncing.

License

MIT

Available Tools

9 tools
add_entryA

Add a question-answer pair to the working tree of a golden dataset.

Entries added here are NOT yet versioned — call commit_version to snapshot them. dataset_path must already be initialised.

Inputs: input (required). Returns: id, status

commit_versionB

Snapshot the current working tree as a new immutable dataset version.

Versions auto-increment (1.0 -> 1.1 -> 1.2...). Fails if the working tree is empty.

Inputs: input (required). Returns: version, entry_count, sha256, parent_version

dataset_statusB

Show the current state of a golden dataset: name, current version, and working tree size.

Inputs: input (required). Returns: name, description, current_version, working_tree_entry_count, +1 more

delete_entryA

Remove an entry from the working tree by its id. Does not affect already-committed versions.

Inputs: input (required). Returns: id, status

diff_versionsB

Show entries added, removed, or changed between two committed versions.

Inputs: input (required). Returns: added, removed, changed

evaluate_answersA

Score actual LLM/RAG-generated answers against the golden dataset using TF-IDF cosine similarity (no LLM call, no API key needed).

actual_answers must be supplied in the same order as the entries in the target version. Omit version to evaluate against the current committed version.

Inputs: input (required). Returns: dataset_name, version, total_entries, avg_semantic_similarity, +2 more

init_datasetA

Initialise a new version-controlled golden dataset at dataset_path.

Creates a .golden_dataset/ directory there. Fails if one already exists at that path — delete .golden_dataset/ manually to start fresh.

Inputs: input (required). Returns: name, description, created_at

list_entriesA

List entries in a dataset. Omit version to see the uncommitted working tree; pass a version (e.g. '1.0') to see a committed snapshot.

Inputs: input (required). Returns: entries, count

update_entryA

Update fields of an existing working-tree entry by its id.

Only fields you provide are changed; omitted fields are left as-is. Raises an error if entry_id is not found in the working tree.

Inputs: input (required). Returns: id, question, answer, contexts, +3 more

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.9/5.0

    Scored across 9 tools

    Disambiguation5/5

    Each tool has a clearly distinct purpose: init, add, delete, update, list, commit, status, diff, and evaluate. No two tools overlap in functionality.

    Naming Consistency5/5

    All tool names follow the verb_noun pattern in snake_case, e.g., add_entry, commit_version, list_entries. Consistent and predictable.

    Tool Count5/5

    9 tools is well-scoped for a version-controlled golden dataset server. It covers all essential operations without being bloated or sparse.

    Completeness4/5

    The tool surface covers init, CRUD for working tree, versioning, listing, status, diff, and evaluation. Minor gap: no dedicated get-single-entry tool, but list_entries likely returns full entries.

    Maintenance

    ActivityInactive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      Not graded
      quality
      D
      maintenance
      Creates deterministic, auditable vector databases from any content source with deployable RAG applications. Supports multiple embedding providers and vector databases with fine-grained pipeline control or project-based workflows.
      7
      -
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables LLM evaluation and observability by uploading documents, building test sets, running RAG pipelines, and automatically scoring answers for groundedness, hallucination risk, retrieval quality, latency, and cost, with tools exposed to MCP-compatible clients.
      1
      MIT
    • A
      license
      A
      quality
      D
      maintenance
      Enables ingestion and semantic search over text documents using PostgreSQL + pgvector and OpenAI-compatible embeddings, allowing any LLM agent to retrieve relevant chunks for grounded answers.
      4
      AGPL 3.0