Use this to A/B two RAG configurations (prompts, retrievers, models) on
the same query. Position bias is mitigated automatically: the judge sees
(A,B) and (B,A) in separate calls, and any criterion whose verdict flips
with presentation order is reported as a tie rather than a pick.

Args:
    query: the shared query both answers respond to.
    answer_a: first candidate answer.
    answer_b: second candidate answer.
    sources: optional list of {id, text} chunks, used to judge faithfulness.
    criteria: judged criteria (default ["faithfulness", "completeness", "relevance"]).

Returns a winner ("a"/"b"/"tie/uncertain"), a verdict per criterion, and a
brief rationale. Costs 2 model calls via your client's sampling -- no API
key needed if your client supports sampling.

Use this in a fix-it loop where you only care about what's wrong, not a
full faithfulness report -- for the full picture with a score and every
claim's verdict, use groundcheck_evaluate_faithfulness instead.

Args:
    answer: the RAG-generated answer text to check.
    sources: list of {id, text} chunks the answer was generated from.

Returns an empty list if the answer is clean. Otherwise, each entry has
the exact answer span, the closest source passage that fails to support
it, and a one-line reason. Costs 2 model calls via your client's
sampling -- no API key needed if your client supports sampling.

Use this when you need a faithfulness score (0-1) and want to see every
claim's verdict, not just the problems -- for that, use
groundcheck_detect_hallucinations instead, which is tighter and cheaper.

Args:
    answer: the RAG-generated answer text to check.
    sources: list of {id, text} chunks the answer was generated from,
        e.g. [{"id": "doc1#chunk3", "text": "..."}].
    response_format: "concise" (default) returns the score, claim counts,
        and only unsupported/contradicted claims. "detailed" returns
        every claim's verdict.

Returns a score (supported/total claims), counts by verdict, and a claims
list (filtered per response_format). Costs 2 model calls via your client's
sampling (decompose, then verify) -- no API key needed if your client
supports sampling.

Two modes, chosen automatically:
- Mode A (pass `relevant_ids`): precision@k, recall@k, MRR, NDCG computed
  by pure math from your gold relevance labels. Instant, no model calls.
- Mode B (omit `relevant_ids`): no gold labels available, so each chunk
  is graded 0-3 for relevance via one sampling call, then the same
  metrics are computed from those grades. Use this when you don't have
  a labeled relevant-docs set for this query.

Args:
    query: the search query the chunks were retrieved for.
    retrieved: ranked list of {id, text} chunks, in retrieval order (rank matters).
    relevant_ids: ids of chunks known to be relevant. Supply this whenever
        you have gold labels -- Mode A is free and exact.
    k_values: cutoffs to compute metrics at (default [3, 5, 10]).

Output states which mode ran. Mode A: instant. Mode B: 1 model call, no
API key needed if your client supports sampling.

Args:
    report_id: the id returned by groundcheck_run_suite.
    response_format: "concise" returns just the summary (scores,
        aggregates, worst cases). "detailed" returns every case's full
        faithfulness and retrieval results.

Raises an error listing available report ids if `report_id` is unknown.
No model calls -- reads from the local report store.

Use this to evaluate a whole RAG pipeline run rather than one answer at a
time. Supply exactly one of `cases` or `dataset_path`.

Args:
    cases: inline list of {id, query, answer, sources, relevant_ids?}.
    dataset_path: path to a JSONL file of the same case objects, one per
        line. Must resolve inside the allowlisted data directory (env
        GROUNDCHECK_DATA_DIR, default cwd) -- paths outside it are rejected.
    k_values: retrieval cutoffs (default [3, 5, 10]).

Persists a full report and returns a summary (mean faithfulness, mean
NDCG, worst 5 cases, report_id). Fetch the full report with
groundcheck_get_report(report_id). Cost scales with case count: ~2 model
calls per case via your client's sampling.