lbb_evals
Manage ground-truth evaluations for app questions and searches: label traces, freeze golden queries, run commit checks, and compare workflow experiments.
Instructions
Managed evals: ground truth for the questions an app asks and the searches it runs. A query run with request (lbb_query, also mode=question) records a trace: the question, the query that answered it, the type of that query (query_type: sparql, hybrid = search by meaning with conditions, search = search by meaning alone) and one item per result (hit or row); trace reads it back with the item ids. label has two forms. On a question's trace, valid alone judges the whole answer: valid=true makes the trace's query the golden query of the question; valid=false with sparql gives the right query, which becomes the golden query; valid=false alone marks the answer wrong; route with valid=true (or valid=false with sparql) names the route the question expects, and valid=true alone takes the trace's route. item + valid (or items) judges one result, a citation of the answer: the ground truth of the hits of a search by meaning, and a result marked wrong must not come back for any type. golden freezes a stored query or a search (every result it returns now is relevant). run checks every golden at the current commit: a question is asked again through the query rewriter, then the type of the query it wrote is compared with the expected type, its rows with the rows of the golden query at the same commit (query_check), and its results with the judged results, and the router's pick with the golden's route (route_check, which does not change the verdict); a search is done again; new results open a review trace. run_answers checks your own app's answers instead: ask each question golden's request with your own loop, then send answers=[{golden, sparql, route}] (route = the route of lbb_query mode=question with route mode, sparql absent for an unanswerable question, error for a failure); the server runs each query and checks it the same way, and the runs form the app's own series (runner; results with runner reads them). judge lets the platform's judge model (the hosted frontier model) label unlabeled results. summary, traces, goldens, results, and settings read state. Model checks: a judge model checks a sample of the model calls LBB makes for the graph (rerank, route, ask = the answer of a question, rewrite, fit, propose, label). checks_summary reads a month per job and model (checks, score, right, partly, wrong, reviews) and the judge's agreement with people; checks lists the month's checks, newest first, with the judge's verdict, score and reason and the ground truth (truth). review_check records a person's review of one check (call_id): agree=true keeps the judge's verdict; agree=false with verdict (and an optional score, reference and note) corrects it. The review becomes the call's ground truth, so show the check to the user and review only what the user confirmed. Datasets: named sets of examples with the outputs a correct run gives, which workflow tests and template evals share. The kind values holds per item an input text (name, text, iri), the property values a correct run writes (values) and those it must leave out (leave_out). datasets lists the graph's datasets, newest first, without their items (one page with next_cursor); datasets_get reads one (id) with its items; datasets_export returns it as a template's samples, records (records.json) and expected (expected.json). datasets_create makes one with name and items (at most 500), install (the settings it is graded with: each property's value type, units, context) and about, or from a workflow test's answers with source={kind: test, id}. datasets_items adds items to a dataset (id); an item with the name of one it holds replaces it. Delete a dataset with lbb_evals_delete, which asks the person first. A dataset ends with its graph. Experiments: two configurations of a workflow, A and B, on the items of a dataset, with the difference of B against A and its 95% interval. A is the workflow as it runs now; b changes one part: {model: {use: extract or decide, model: {provider, model, effort}}} (a model from lbb_model_choice action=choices options) or {revision: wrev_…} (a kept settings revision), not both. Each arm runs as a workflow test that writes nothing to the graph and spends model budget; each item is graded against the dataset's labels. experiments_estimate shows the cost first (items, the cost of both arms, the limit, the budget left): call it, show the user the figures, and start only what the user confirmed. experiments_create starts one (workflow, dataset, b, optionally max_items up to 200 and max_cost_usd up to 100, default 5); poll experiment (id) until its status is done, failed or cancelled, and read report.delta with ci_low and ci_high (outcome too_few under 10 items, else better, worse or same) and the item rows. experiments lists them (workflow, dataset, status, one page with next_cursor); experiments_cancel stops one, keeping what it compared. None of them changes the workflow or the graph. To use B, make the change with the experiment's id: lbb_model_choice_manage action=put_choice (or lbb_workflows_manage action=activate) with experiment, after the user confirmed; it fails with experiment_outdated when the workflow moved since.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| b | No | experiments_estimate, experiments_create: what B changes in A: {model: {use, model: {provider, model, effort}}} or {revision}, not both. | |
| by | No | label: who labels (an agent name). | |
| id | No | datasets_get, datasets_export, datasets_items: the dataset's id (ds_…). experiment, experiments_cancel: the experiment's id (exp_…). | |
| job | No | checks: only the checks of one job. | |
| item | No | label: the result id (a hit's `id`, or an item id from trace) to label with `valid`. | |
| kind | No | datasets_create: what the items hold: values (the default). | |
| name | No | datasets_create: the dataset's name. | |
| note | No | label / review_check: why. review_check: at most 2,000 characters. | |
| about | No | datasets_create: what the dataset is, in sentences. | |
| after | No | checks: the next_after of the previous page. | |
| agree | No | review_check: true = the judge is right, false = correct it with verdict. | |
| graph | No | Graph to target; defaults to the connection's graph | |
| items | No | label: several results at once, each {id, valid, note}. datasets_create, datasets_items: the examples, at most 500, each {name, text, values, leave_out, iri, case}; values and leave_out hold {property, value, accept, stated, quote, reason, source}; source says which person's answer gave a label (kind test or review, id, at): send it back as read, or leave it out. | |
| limit | No | ||
| month | No | checks_summary / checks: the month, yyyy-mm (UTC). Defaults to the current month. | |
| route | No | label: with valid=true, or valid=false with sparql, the route the question expects. | |
| score | No | review_check with agree=false: the right score, 0 to 1. Omit for the judge's score of that verdict. | |
| top_k | No | golden: the top_k of a search golden. | |
| valid | No | label: true = the result answers the request (thumbs up), false = it does not. Without item or items, on a question's trace: the whole answer is right (true) or wrong (false). | |
| action | Yes | ||
| cursor | No | datasets, experiments: the next_cursor of the page before. | |
| detail | No | Response detail level. Defaults to compact. | |
| runner | No | run_answers / results: your app's name, lower-case letters, digits and -, at most 40 characters (default app). results without runner reads the server's runs. | |
| source | No | datasets_create: {"kind": "test", "id": <test id>}: the answers of a workflow test become the items, and its settings the install. | |
| sparql | No | golden: the query to freeze (the search text for surface=search). label: with valid=false and no item, the right query for the question; the server checks and runs it once. | |
| status | No | experiments: only the experiments with this status. | |
| answers | No | run_answers: one answer per question golden your app asked: golden (its id), sparql (the query whose rows hold the answer; leave it out when the question is unanswerable), entailment, as_of_commit_seq, route (as the router returned it), reply, error (your app failed), cost_micro_usd. | |
| call_id | No | review_check: the check's `call` id, from checks. | |
| dataset | No | experiments_estimate, experiments_create: the dataset (ds_…) whose items both arms run. experiments: only the experiments on this dataset. | |
| install | No | datasets_create: the settings it is graded with. | |
| request | No | golden: the user's words the query answers. | |
| surface | No | golden: which surface the query runs on (default sparql). | |
| verdict | No | checks: only the checks whose ground truth has this verdict. review_check with agree=false: the right verdict. | |
| reviewed | No | checks: true = only the checks a person reviewed, false = only the others. | |
| trace_id | No | trace / label / judge: the trace id from lbb_query. | |
| workflow | No | experiments_estimate, experiments_create: the workflow to compare. experiments: only the experiments of this workflow. | |
| embedding | No | golden: the embedding, for surface=search. | |
| golden_id | No | accept / delete: the golden id. | |
| max_items | No | experiments_estimate, experiments_create: run the dataset's first items only (default 200). | |
| reference | No | review_check with agree=false: the right answer. A rerank or label check: {"grades": {"<hit id>": 0..3}}; a route or fit check: {"picks": {"<question id>": "<option>"}}. | |
| unlabeled | No | traces: only traces without a label. | |
| consistency | No | run / run_answers / accept: strong reads the head, eventual the last published commit. | |
| max_cost_usd | No | experiments_estimate, experiments_create: the cost limit of both arms in US dollars (default 5). |