mcp-promptdiff

Zero-dependency stdio MCP server and CLI for token-aware prompt version diffs.

Compare two prompt strings or files and get a machine-readable token delta report — no external APIs, no native dependencies, runs anywhere.

Quick Start

pip install git+https://github.com/prasad-a-abhishek/mcp-promptdiff.git

from mcp_promptdiff import diff_prompts, estimate_tokens

# Token-aware diff between two prompts
report = diff_prompts(
    "You are a helpful assistant.",
    "You are a helpful, concise assistant.",
)
print(f"Token delta: {report.token_delta:+d}")   # e.g. +4
print(f"Changed: {report.changed}")              # True

# Estimate token count for any string
est = estimate_tokens("Hello 🤖 world")
print(f"Tokens: {est.token_count}")             # e.g. 5
print(f"Confidence: {est.confidence.value}")     # low (emoji)

Performance & Benchmarks

Token estimation is a fast heuristic (byte-length ÷ 4), making it suitable for CI pipelines and high-throughput MCP tooling without network latency or native dependencies.

Local benchmark reproduction:

python3 benchmarks/run_benchmark.py

Why mcp-promptdiff?

The problem: LLM application developers version-control prompts but get no token-level signal from git diff. You can't answer "how many tokens were added?" or "which section grew most?" from a character-level diff.

Existing tools are too heavy: tiktoken and tokenizers require native data files and per-model downloads. Git diff has no concept of token boundaries. The MCP ecosystem had no dedicated prompt diff tool.

mcp-promptdiff is the minimal building block: Zero dependencies, pure stdlib, runs in any Python 3.11+ environment. Estimates are clearly labeled approximate with confidence/method fields so callers decide how much to trust them.

Key Features

• Zero dependencies — pure Python stdlib, no pip installs beyond the package itself

• Both CLI and library API — use in CI, code review, or embedded MCP tooling

• MCP stdio server — drop into any MCP client (Claude Desktop, Cursor, etc.)

• Batch diff — compare many prompt pairs in one call

• Encoding presets — cl100k_approx, o200k_base, p50k_base

• Confidence labels — every estimate reports LOW/MEDIUM/HIGH confidence so clients can decide trust level

API Reference

estimate_tokens(text, encoding="cl100k_approx")

Returns a TokenEstimate:

@dataclass(frozen=True)
class TokenEstimate:
    text: str
    token_count: int         # estimated token count
    confidence: Confidence   # HIGH | MEDIUM | LOW
    method: str              # e.g. "cl100k_approx_byte_ratio"
    encoding: str            # the encoding used

diff_prompts(prompt_a, prompt_b, encoding="cl100k_approx")

Returns a TokenReport:

@dataclass(frozen=True)
class TokenReport:
    token_delta: int                     # tokens added (negative = removed)
    token_count_a: int
    token_count_b: int
    section_deltas: List[SectionDelta]   # per-paragraph breakdown
    changed: bool                        # True if content differs
    encoding: str
    method: str
    estimated_cost_a: float              # USD at ~$0.01/1k tokens
    estimated_cost_b: float             # USD at ~$0.01/1k tokens

diff_files(path_a, path_b, encoding="cl100k_approx")

Same shape as diff_prompts, loaded from disk paths.

batch_diff(pairs, encoding="cl100k_approx")

Takes a list of PromptPair(prompt_a, prompt_b, label=None) and returns a list of TokenReport.

MCP Server

# Run as stdio MCP server (stays alive on stdin/stdout)
from mcp_promptdiff.mcp_server import run
run()

Tools exposed: diff_prompts, diff_files, estimate_tokens, batch_diff

CLI Reference

mcp-promptdiff diff --a "original" --b "revised"
mcp-promptdiff diff --file-a prompt_v1.md --file-b prompt_v2.md
mcp-promptdiff diff --file-a a.md --file-b b.md --format json
mcp-promptdiff estimate --text "Hello world"
mcp-promptdiff estimate --text "Hello 🤖" --encoding o200k_base

Limitations

• Estimates, not exact counts. This is a byte-length heuristic (byte_length ÷ 4), not exact tokenization. Every result includes a confidence field so callers can decide how much to trust it.

• No multi-turn conversation diffs. Compares two flat prompt strings or two files only.

• No HTTP/SSE transport. stdio JSON-RPC only — the MCP server reads stdin and writes stdout.

• No prompt optimization or templating. This is a diff tool, not a prompt engineering tool.

Non-Goals

• Exact tokenizer parity with any specific provider (OpenAI, Anthropic, etc.)

• Calling any hosted LLM API for evaluation

• Prompt optimization, variable substitution, or templating

• Persistent state or history

• HTTP/SSE transport

License

MIT License — Abhishek Prasad