CompletionKit
Server Details
Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.
- Status
- Healthy
- Uptime
- 100.0% over the last 24 hours
- Last Tested
- Transport
- Streamable HTTP · MCP 2025-06-18
- URL
- Repository
- homemade-software-inc/completion-kit
- GitHub Stars
- 1
- ChatGPT
- –
- Claude
- –
TDQS
Scored across 54 tools
Most tools target distinct resources and actions, and descriptions help differentiate similar operations. However, the large number of run-related tools (create, generate, rerun, regrade, retry_failures) and parallel CRUD sets for metrics, metric_versions, and metric_groups create some potential for misselection.
All tool names consistently follow a snake_case resource_action pattern (e.g., datasets_create, prompts_list, runs_generate). Resource names are consistently plural and verbs are clear, with only trivial deviations like judges_compare/replay and prompts_suggest_improvement.
54 tools is far above the 25-tool threshold where scope becomes heavy for an agent to navigate reliably. Even for a broad evaluation platform, the surface is large and includes many thin CRUD wrappers that risk redundancy and selection difficulty.
The server covers the core evaluation lifecycle: datasets, prompts, runs, scoring, metrics, versions, agreements, and tags. Minor gaps exist, such as no explicit tags-assign or metric-group-linking operations, but main workflows are well supported.
Available Tools
54 toolsagreements_createAInspect
Upsert an agreement for (run, response, metric, created_by). Verdict is one of agree, disagree, borderline. corrected_score (1..5) is required when verdict is 'disagree'.
Inputs: run_id (required), verdict (required), metric_id (required), response_id (required), +3 more.
agreements_listAInspect
List agreements. Filter by run_id, response_id, metric_id, or created_by.
Inputs: run_id, metric_id, created_by, response_id.
datasets_createAInspect
Create a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer key (ground truth) given to the judge and to checks that compare against the row's expected value, and "actual_output" is a pre-made output to score in a prompt-less run. Both are overridable per run (expected_column / output_column). Every column is also available to the prompt as a variable.
Inputs: name (required), csv_data (required), tag_names.
datasets_create_from_urlAInspect
Create a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data never has to pass through the tool-call arguments. The URL is SSRF-checked and the download is capped at 10MB. First row is the header; the "expected_output" (answer key) and "actual_output" (pre-made output) columns are recognized specially, overridable per run.
Inputs: url (required), name (required), tag_names.
datasets_deleteDInspect
Delete a dataset
Inputs: id (required).
datasets_getCInspect
Get a dataset by ID
Inputs: id (required).
datasets_listAInspect
List all datasets
Inputs: no named parameters.
datasets_updateCInspect
Update a dataset
Inputs: id (required), name, csv_data, tag_names.
judges_compareAInspect
Compare two versions of one metric's agreement stats side by side. Requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). Unavailable for check metrics.
Inputs: metric_id (required), metric_version_a_id (required), metric_version_b_id (required).
judges_replayAInspect
Create a scoring run for the current judge over a dataset's existing outputs (wraps runs_create with prompt_id omitted and output_column supplied). This only sets up the run; call runs_generate to actually re-judge the outputs so you can compare against human verdicts.
Inputs: name (required), metric_id (required), dataset_id (required), judge_model (required), +1 more.
metric_groups_createCInspect
Create a metric group
Inputs: name (required), tag_names, metric_ids, description.
metric_groups_deleteCInspect
Delete a metric group
Inputs: id (required).
metric_groups_getAInspect
Get a metric group by ID
Inputs: id (required).
metric_groups_listBInspect
List all metric groups
Inputs: no named parameters.
metric_groups_updateCInspect
Update a metric group
Inputs: id (required), name, tag_names, metric_ids, +1 more.
metrics_createAInspect
Create a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/numeric_equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds/numeric_bounds); valid_json takes no extra keys. target_path is required when target is json_path; a target_path sent with no target sets target to json_path, and one sent with any other target is refused. target_path walks array indices, as in items.0.name; any path may start at the root with $., as in $.items.0.name. list_overlap grades a list-valued field and stores a fraction rather than a verdict: score_by is recall, precision, f1 or jaccard (recall by default), and min is the fraction that counts as a pass (1, a perfect score, by default). numeric_bounds and numeric_equals read a number, ignoring the thousands commas and currency symbol a figure is written with; a comma counts only as a thousands separator, with three digits after it and no decimal point before it, so 1,5 is not a number; numeric_equals takes a tolerance, either a plain number for a fixed amount or a percentage string such as 2% for a share of the expected value. A check with no case_sensitive matches case-insensitively, regex and list_overlap included; send case_sensitive: true to match case exactly. Before 0.29.0 a regex check read a missing case_sensitive as case-sensitive. Set compare_to:"expected" on contains, not_contains, equals, json_path_equals, list_overlap, numeric_equals to grade against each row's own expected_output (ground truth) instead of a constant; drop value (or expected, for json_path_equals) and add expected_path to dig into the answer key when it is JSON.
Inputs: name (required), tag_names, instruction, metric_type, +2 more.
metrics_deleteBInspect
Delete a metric
Inputs: id (required).
metrics_getBInspect
Get a metric by ID
Inputs: id (required).
metrics_listAInspect
List all metrics
Inputs: no named parameters.
metrics_suggest_variantsAInspect
Ask the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source="suggestion". Returns the persisted drafts. Stripe-metering hooks fire via ActiveSupport::Notifications under completion_kit.judge_suggestion.generated.
Inputs: metric_id (required), count, model.
metrics_updateBInspect
Update a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/numeric_equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds/numeric_bounds); valid_json takes no extra keys. target_path is required when target is json_path; a target_path sent with no target sets target to json_path, and one sent with any other target is refused. target_path walks array indices, as in items.0.name; any path may start at the root with $., as in $.items.0.name. list_overlap grades a list-valued field and stores a fraction rather than a verdict: score_by is recall, precision, f1 or jaccard (recall by default), and min is the fraction that counts as a pass (1, a perfect score, by default). numeric_bounds and numeric_equals read a number, ignoring the thousands commas and currency symbol a figure is written with; a comma counts only as a thousands separator, with three digits after it and no decimal point before it, so 1,5 is not a number; numeric_equals takes a tolerance, either a plain number for a fixed amount or a percentage string such as 2% for a share of the expected value. A check with no case_sensitive matches case-insensitively, regex and list_overlap included; send case_sensitive: true to match case exactly. Before 0.29.0 a regex check read a missing case_sensitive as case-sensitive. Set compare_to:"expected" on contains, not_contains, equals, json_path_equals, list_overlap, numeric_equals to grade against each row's own expected_output (ground truth) instead of a constant; drop value (or expected, for json_path_equals) and add expected_path to dig into the answer key when it is JSON.
Inputs: id (required), name, tag_names, instruction, +3 more.
metric_versions_dismissAInspect
Destroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused. To demote a published version, publish a different one as current instead.
Inputs: metric_version_id (required).
metric_versions_listAInspect
List every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps.
Inputs: metric_id (required).
metric_versions_publishAInspect
Publish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, demotes peers, and writes the version's instruction + rubric_bands back onto the metric so the judge grades against it.
Inputs: metric_version_id (required).
promptfoo_importAInspect
Import a promptfooconfig.yaml. Creates a prompt, a dataset from the test vars, and metrics from the assert blocks (llm-rubric/g-eval become judge metrics; contains/equals/regex/is-json become deterministic check metrics). Returns a summary of what mapped and what was skipped and why; nothing is dropped silently.
Inputs: config (required).
prompts_createCInspect
Create a prompt
Inputs: name (required), template (required), llm_model (required), tag_names, +1 more.
prompts_deleteBInspect
Delete a prompt
Inputs: id (required).
prompts_getBInspect
Get a prompt by ID
Inputs: id (required).
prompts_listBInspect
List all prompts
Inputs: no named parameters.
prompts_publishCInspect
Publish a prompt version, making it the current version
Inputs: id (required).
prompts_suggest_improvementAInspect
Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewritten template (preserving {{variables}}) and persists it as a Suggestion. Requires a run that has a prompt (not a scoring-only run).
Inputs: run_id (required).
prompts_updateAInspect
Update a prompt. If the prompt already has runs, this creates a new DRAFT version (current=false) rather than editing in place or publishing. Promote it with prompts_publish, so an agent's edits don't go live without a gate. If it has no runs, it is updated in place.
Inputs: id (required), name, template, llm_model, +2 more.
provider_credentials_createCInspect
Create a provider credential
Inputs: api_key (required), provider (required), api_version, api_endpoint.
provider_credentials_deleteCInspect
Delete a provider credential
Inputs: id (required).
provider_credentials_getAInspect
Get a provider credential by ID (API key is not exposed)
Inputs: id (required).
provider_credentials_listAInspect
List all provider credentials (API keys are not exposed)
Inputs: no named parameters.
provider_credentials_updateAInspect
Update a provider credential. Changing provider or api_endpoint on a credential with a stored key requires api_key in the same call, so a stored key never moves to a new provider or endpoint.
Inputs: id (required), api_key, provider, api_version, +1 more.
responses_getCInspect
Get a specific response
Inputs: id (required), run_id (required).
responses_listAInspect
List responses for a run, in row order. Returns {total, limit, offset, returned, responses}. Defaults to 50 rows because full payloads are large: use "fields" to drop the bodies, "min_score"/"max_score" to isolate low scorers, and sort "score_asc" to read the worst rows first. For per-metric averages of the whole run use runs_get instead of aggregating here.
Inputs: run_id (required), sort, limit, fields, +4 more.
runs_createAInspect
Create a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones.
Inputs: name (required), prompt_id, tag_names, dataset_id, +8 more.
runs_deleteDInspect
Delete a run
Inputs: id (required).
runs_generateAInspect
Start a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded dataset column and grades it.
Inputs: id (required).
runs_getAInspect
Get a run by ID, including "metric_averages": a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and how many scored low. Use this to find the metric dragging a prompt down without listing responses.
Inputs: id (required).
runs_regradeAInspect
Re-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated run.
Inputs: id (required).
runs_rerunAInspect
Create and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixing versions.
Inputs: id (required).
runs_retry_failuresAInspect
Re-run only the failed responses of a run, optionally limited to specific response ids via "only".
Inputs: id (required), only.
runs_updateDInspect
Update a run
Inputs: id (required), name, tag_names, dataset_id, +8 more.
tags_createBInspect
Create a tag. Color is auto-assigned.
Inputs: name (required).
tags_deleteAInspect
Delete a tag. Removes the tag from every linked metric, prompt, run, and dataset.
Inputs: id (required).
tags_updateCInspect
Rename a tag.
Inputs: id (required), name.
usage_getAInspect
Get this organization's plan usage and limits for the current billing period: runs and prompt fetches used, their limits, how many remain, and when the period resets. Call this to pre-check quota before starting runs. Runs are hard-blocked once the run limit is reached (with a small grace band), so a run over the limit will fail with run_limit_reached.
Inputs: no named parameters.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Related MCP Connectors
MCP-native AI evaluation: rubric audits, eval suites, and proof reports for AI/LLM output.
Remote MCP for Gemini upgrade evals, prompt regressions, output diffs, and eval receipts.
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
Pay-per-call AI evaluation MCP server. Score LLM outputs against benchmark rubrics via Workers AI.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceA local MCP server for prompt evaluation, enabling users to define test cases, run prompts against multiple LLM providers, score outputs with deterministic and LLM-graded assertions, and track quality over time, all within an AI coding environment.MIT
- AlicenseAqualityBmaintenanceEnables MCP clients to score retrieval quality with recall, precision, nDCG, hit rate and MRR, compare 2-3 configurations with seeded paired-bootstrap confidence intervals, audit agent tool calls against expected traces, and detect prompt regressions against golden sets. It also generates Markdown, JSON or JUnit reports with pass/fail gates, all deterministically and offline without paid API keys.673 npm1MIT
- AlicenseNot gradedqualityDmaintenanceMCP-based code evaluation harness — sandboxed execution + LLM quality scoring.MIT
- AlicenseNot gradedqualityCmaintenanceScores AI outputs for faithfulness, relevancy, and hallucination inside any MCP client, with custom metrics, golden sets, and run history with dashboards.Apache 2.0
Glama MCP Gateway
Add one secure layer between your agents and this server.