Skip to main content
Glama
littlebigbrains

@littlebigbrain/mcp

by littlebigbrains
OverviewSchemaSourceRelated ServersDiscussionsAdmin
TypeScript
Hybrid

lbb_evals

Label query results, freeze golden sets, replay evaluations, and review judge checks to validate and correct knowledge graph answers.

Instructions

Managed evals: the thumbs up / thumbs down of the graph, per result. A query run with request (lbb_query) records a trace with one item per result (hit or row); trace reads it back with the item ids. label marks one result (item + valid) or several (items) relevant or not; the labels become the golden's ground truth. golden freezes a query (every result it returns now is relevant). run replays every golden at the current commit: pass when every relevant result is back and no wrong one is; new results open a review trace. judge lets the platform's judge model (the hosted frontier model) label unlabeled results. summary, traces, goldens, results, and settings read state. Model checks: a judge model checks a sample of the model calls LBB makes for the graph (rerank, route, rewrite, fit, propose, label). checks_summary reads a month per job and model (checks, score, right, partly, wrong, reviews) and the judge's agreement with people; checks lists the month's checks, newest first, with the judge's verdict, score and reason and the ground truth (truth). review_check records a person's review of one check (call_id): agree=true keeps the judge's verdict; agree=false with verdict (and an optional score, reference and note) corrects it. The review becomes the call's ground truth, so show the check to the user and review only what the user confirmed.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
byNolabel: who labels (an agent name).
jobNochecks: only the checks of one job.
itemNolabel: the result id (a hit's `id`, or an item id from trace) to label with `valid`.
noteNolabel / review_check: why. review_check: at most 2,000 characters.
afterNochecks: the next_after of the previous page.
agreeNoreview_check: true = the judge is right, false = correct it with verdict.
graphNoGraph to target; defaults to the connection's graph
itemsNolabel: several results at once.
limitNo
monthNochecks_summary / checks: the month, yyyy-mm (UTC). Defaults to the current month.
scoreNoreview_check with agree=false: the right score, 0 to 1. Omit for the judge's score of that verdict.
top_kNogolden: the top_k of a search golden.
validNolabel: true = the result answers the request (thumbs up), false = it does not.
actionYes
detailNoResponse detail level. Defaults to compact.
sparqlNogolden: the query to freeze (the search text for surface=search).
call_idNoreview_check: the check's `call` id, from checks.
requestNogolden: the user's words the query answers.
surfaceNogolden: which surface the query runs on (default sparql).
verdictNochecks: only the checks whose ground truth has this verdict. review_check with agree=false: the right verdict.
reviewedNochecks: true = only the checks a person reviewed, false = only the others.
trace_idNotrace / label / judge: the trace id from lbb_query.
embeddingNogolden: the embedding, for surface=search.
golden_idNoaccept / delete: the golden id.
referenceNoreview_check with agree=false: the right answer. A rerank or label check: {"grades": {"<hit id>": 0..3}}; a route or fit check: {"picks": {"<question id>": "<option>"}}.
unlabeledNotraces: only traces without a label.
consistencyNorun / accept: strong reads the head, eventual the last published commit.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed11 schema fields changedv0.9.0
    • changedInput schema / properties / action / enum
      Previous value: -[
-  "summary",
-  "traces",
-  "trace",
-  "label",
-  "judge",
-  "goldens",
-  "golden",
-  "accept",
-  "delete",
-  "run",
-  "results",
-  "settings"
-]New value: +[
+  "summary",
+  "traces",
+  "trace",
+  "label",
+  "judge",
+  "goldens",
+  "golden",
+  "accept",
+  "delete",
+  "run",
+  "results",
+  "settings",
+  "checks_summary",
+  "checks",
+  "review_check"
+]
    • addedInput schema / properties / after
      Added value: +{
+  "description": "checks: the next_after of the previous page.",
+  "type": "string"
+}
    • addedInput schema / properties / agree
      Added value: +{
+  "description": "review_check: true = the judge is right, false = correct it with verdict.",
+  "type": "boolean"
+}
    • addedInput schema / properties / call_id
      Added value: +{
+  "description": "review_check: the check's `call` id, from checks.",
+  "type": "string"
+}
    • addedInput schema / properties / job
      Added value: +{
+  "description": "checks: only the checks of one job.",
+  "enum": [
+    "rerank",
+    "route",
+    "rewrite",
+    "fit",
+    "propose",
+    "label"
+  ],
+  "type": "string"
+}
    • addedInput schema / properties / month
      Added value: +{
+  "description": "checks_summary / checks: the month, yyyy-mm (UTC). Defaults to the current month.",
+  "pattern": "^\\d{4}-(0[1-9]|1[0-2])$",
+  "type": "string"
+}
    • addedInput schema / properties / note / description
      Added value: +"label / review_check: why. review_check: at most 2,000 characters."
    • addedInput schema / properties / reference
      Added value: +{
+  "additionalProperties": {},
+  "description": "review_check with agree=false: the right answer. A rerank or label check: {\"grades\": {\"<hit id>\": 0..3}}; a route or fit check: {\"picks\": {\"<question id>\": \"<option>\"}}.",
+  "type": "object"
+}
    • addedInput schema / properties / reviewed
      Added value: +{
+  "description": "checks: true = only the checks a person reviewed, false = only the others.",
+  "type": "boolean"
+}
    • addedInput schema / properties / score
      Added value: +{
+  "description": "review_check with agree=false: the right score, 0 to 1. Omit for the judge's score of that verdict.",
+  "maximum": 1,
+  "minimum": 0,
+  "type": "number"
+}
    • addedInput schema / properties / verdict
      Added value: +{
+  "description": "checks: only the checks whose ground truth has this verdict. review_check with agree=false: the right verdict.",
+  "enum": [
+    "right",
+    "partly",
+    "wrong"
+  ],
+  "type": "string"
+}
  2. Addedv0.5.2

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With annotations only covering readOnly/destructive/openWorld, the description carries real behavioral weight: labels 'become the golden's ground truth', a golden 'freezes a query (every result it returns now is relevant)', a run 'passes when every relevant result is back and no wrong one is', and 'the review becomes the call's ground truth'. Those are state-effect disclosures the annotations cannot express. It omits, however, any mention of the accept/delete actions or their effects, and gives no auth or rate-limit context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Roughly 330 words for 15 actions is defensible in size and the core concept is front-loaded, but it is delivered as a single dense block of semicolon-chained clauses with no per-action structure. Scanning for a specific action such as review_check requires reading through unrelated eval prose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 27-param, 15-action multiplexer with no output schema, the description covers the substantive workflow and even names the fields checks_summary returns (checks, score, right, partly, wrong, reviews). Gaps remain: accept and delete are never described beyond the schema's 'the golden id', and five read actions are compressed into one clause ('summary, traces, goldens, results, and settings read state').

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 93%, so the baseline is 3, but the prose adds meaning the schema does not: it explains that 'label marks one result (item + valid) or several (items)' — clarifying the single-vs-batch alternative — and that trace_id comes 'from lbb_query'. It also frames verdict/score/agree as a correction triple for review_check. Remaining params (limit, detail, graph, golden_id) get no description-level treatment.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a concrete verb+resource for nearly every action: 'label marks one result ... relevant or not', 'golden freezes a query', 'run replays every golden at the current commit', 'review_check records a person's review of one check'. An agent can map the 15 enum values to distinct purposes. Sibling differentiation is thin, though: lbb_query is named only as the source of trace ids, and no contrast is drawn with lbb_models despite the overlapping 'Model checks' section.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is mostly implied through the described eval workflow (label -> golden -> run -> review), rather than stated as when-to-use. Only one explicit operational directive is given: 'show the check to the user and review only what the user confirmed.' There are no when-not conditions and no routing guidance against the sibling tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools