Skip to main content
Glama
Sign In

Human Creative AI Workflow & Eval Lab

by io.github.jdurey
OverviewDiscussionsAdmin

calibrate_judge_rubric_4family

Eliminates single-model self-preference bias and stochastic verdict flips by decomposing subjective quality criteria into deterministic Python checks + a 4-family Judge-Trust rubric panel verified across byte-identical repeats ($149 Stripe / $2.00 x402 USDC, or set preview_mode=true for a free calibrated rubric preview).

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "client_name": "<client_name>",
      "task_domain": "<task_domain>",
      "current_rubric": "<current_rubric>"
    },
    "name": "calibrate_judge_rubric_4family"
  }
}

Input Schema

NameRequiredDescriptionDefault
client_nameYesOrganization or team name.
task_domainYesOutput type being graded (e.g. 'Customer refund drafts', 'SQL agent traces').
preview_modeNoSet true for an instant free binary pass/fail rubric decomposition on Edge.
contact_emailNoDelivery email for the calibrated rubric & repeatability report.
payment_tokenNoOptional Stripe or x402 payment token.
current_rubricYesCurrent grading prompt, Likert scale, or QA checklist.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

B3.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations flag readOnlyHint=false, idempotentHint=false and openWorldHint=true, and the description substantially adds to that: concrete pricing ($149 Stripe / $2.00 x402 USDC), a free preview pathway, email delivery of the result, and repeat verification across byte-identical runs. It still does not say how long calibration takes or what happens if payment fails, but the cost and mode profile is unusually well disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

It is a single sentence with no filler, but it is front-loaded with a benefit claim ('Eliminates single-model self-preference bias') rather than the action, and the pricing parenthetical is wedged into the middle of the clause. Dense jargon ('stochastic verdict flips', 'byte-identical repeats') costs parsing effort for an agent skimming to decide.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a non-idempotent, open-world, paid tool with no output schema, the description covers the important operational facts: cost in two payment rails, the free preview escape hatch, and that the deliverable is a calibrated rubric plus a repeatability report sent to contact_email. Only turnaround time and failure/refund behavior remain unstated.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already explains all six parameters including preview_mode's free binary pass/fail behavior. The description only reinforces preview_mode and implies that payment_token corresponds to the stated Stripe/x402 prices, adding little syntax or constraint detail beyond the structured fields.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description conveys the mechanism (decomposing subjective criteria into deterministic Python checks plus a 4-family Judge-Trust rubric panel) but never states a plain action verb such as 'calibrate a grading rubric'. A reader must reverse-engineer that the tool takes an existing rubric and returns a calibrated version, and jargon like '4-family Judge-Trust rubric panel' obscures the resource being acted on.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no when-to-use / when-not-to-use guidance and no reference to the sibling tools (e.g. diagnose_ai_workflow or audit_workflow_guardrails_50v) that an agent might otherwise pick. The only conditional guidance is intra-tool: set preview_mode=true for a free preview, which is a mode switch rather than tool-selection help.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All Human Creative AI Workflow & Eval Lab toolsInspect
Try in Browser
Claude Code, inside the Godot 4 editorFlockbay puts an AI agent in the real Godot 4 editor: it writes GDScript, runs your game and fixes what breaks. Free, or bring your own Claude Code or Codex.flockbay.comAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources