groundcheck_run_suite
Read-only
Run faithfulness and retrieval quality evaluation over a batch of RAG pipeline cases, producing a summary report with mean metrics and worst-performing cases.
Instructions
Run faithfulness (+ retrieval, where gold labels exist) over a batch of cases.
Use this to evaluate a whole RAG pipeline run rather than one answer at a
time. Supply exactly one of `cases` or `dataset_path`.
Args:
cases: inline list of {id, query, answer, sources, relevant_ids?}.
dataset_path: path to a JSONL file of the same case objects, one per
line. Must resolve inside the allowlisted data directory (env
GROUNDCHECK_DATA_DIR, default cwd) -- paths outside it are rejected.
k_values: retrieval cutoffs (default [3, 5, 10]).
Persists a full report and returns a summary (mean faithfulness, mean
NDCG, worst 5 cases, report_id). Fetch the full report with
groundcheck_get_report(report_id). Cost scales with case count: ~2 model
calls per case via your client's sampling.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| cases | No | ||
| k_values | No | ||
| dataset_path | No |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| mean_ndcg | No | ||
| report_id | Yes | ||
| case_count | Yes | ||
| worst_cases | Yes | Ids of the 5 lowest-faithfulness cases. | |
| mean_faithfulness | Yes |