Skip to main content
Glama

nittim

by com.nittim
OverviewDiscussionsAdmin

Judge Arbitrary Output (independent cross-vendor model)

judge_output

Run a cross-vendor judge model over any text you post: code, a document, another model's output, anything. Returns findings + rationale ONLY — never a score, never a pass/fail verdict. Costs 5.03 credits. The judge always comes from a different vendor family than whatever produced the content, and the answer says which one ran. modelUnderTest names that family. NEEDS A KEY: mint one at https://nittim.com/keys. COSTS MONEY in three calls: the first quotes a price and charges nothing; a second with confirmedCost set to that exact price returns a one-time link for the ACCOUNT OWNER to confirm — their yes, not the assistant's; a plain third call with the same arguments runs once at that price. A moved price is re-quoted, never charged.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
contentYesThe text to judge — code, a document, another model's output. Up to ~100KB.
contextNoOptional — background the judge should know, e.g. what this content is for.
criteriaNoOptional — what to judge it against, e.g. 'correctness and security'.
authorizationNoHUMAN CONFIRMATION. The `authorization` id from the confirm-link answer, once the account owner has pressed Confirm. Single-use, short TTL.
confirmedCostNoCOST CONFIRMATION. Omit on the first call; then send back the exact quoted `cost`.
modelUnderTestNoOptional — which vendor family produced `content`, if it is itself a model's output. The judge that runs is always a different family than this names. Use 'unspecified' for anything that is not model output, or when the family is unknown.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed6 schema fields changed
    • changedInput schema / properties / authorization / description
      Previous value: -"HUMAN CONFIRMATION. Omit until the account owner has pressed Confirm on the link a paid tool answered with; then set to that answer's `authorization` id (single-use, short TTL). Protocol: server instructions."New value: +"HUMAN CONFIRMATION. The `authorization` id from the confirm-link answer, once the account owner has pressed Confirm. Single-use, short TTL."
    • changedInput schema / properties / confirmedCost / description
      Previous value: -"COST CONFIRMATION. Omit on the first call (it only quotes); then set to the exact `cost` object quoted. A moved price is re-quoted, never charged. Protocol: server instructions."New value: +"COST CONFIRMATION. Omit on the first call; then send back the exact quoted `cost`."
    • changedInput schema / properties / confirmedCost / properties / centicredits / description
      Previous value: -"Copy the `centicredits` integer from the cost this tool quoted you, if it has one. Omit it if the quote did not carry one."New value: +"The `centicredits` integer from the quoted cost, if it carried one."
    • changedInput schema / properties / confirmedCost / properties / credits / description
      Previous value: -"Copy the `credits` number from the cost this tool quoted you."New value: +"The `credits` number from the quoted cost."
    • changedInput schema / properties / confirmedCost / properties / kind / description
      Previous value: -"Copy the `kind` from the cost this tool quoted you, e.g. 'credits'."New value: +"The `kind` from the quoted cost, e.g. 'credits'."
    • changedInput schema / properties / modelUnderTest / description
      Previous value: -"Optional — which vendor family produced `content`, if it's itself a model's output. The judge that runs is always a different vendor family than this names, so the grader is never the same vendor family as what it's grading. Set 'unspecified' for anything that isn't model output (code, a document) or when the family is genuinely unknown. Passing it explicitly is most useful whenever the content did not come from you asking about your own output."New value: +"Optional — which vendor family produced `content`, if it is itself a model's output. The judge that runs is always a different family than this names. Use 'unspecified' for anything that is not model output, or when the family is unknown."
  2. Changed1 schema field changed
    • changedInput schema / properties / modelUnderTest / description
      Previous value: -"Optional — which vendor family produced `content`, if it's itself a model's output. 'openai' routes the judge to Anthropic instead of the default OpenAI judge, so the grader is never the same vendor family as what it's grading. Set 'unspecified' for anything that isn't model output (code, a document) or when the family is genuinely unknown — that pins the judge to the default (OpenAI). OMITTING this is different from 'unspecified': the family is then inferred from your MCP client's declared name, so an OpenAI-based client posting its own output gets an Anthropic judge without having to ask. Any explicit value here overrides that inference."New value: +"Optional — which vendor family produced `content`, if it's itself a model's output. The judge that runs is always a different vendor family than this names, so the grader is never the same vendor family as what it's grading. Set 'unspecified' for anything that isn't model output (code, a document) or when the family is genuinely unknown. Passing it explicitly is most useful whenever the content did not come from you asking about your own output."
  3. Added

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description fully discloses the multi-call, cost-confirming, human-authorization behavior, including that the first call charges nothing, the third runs once at the quoted price, and the authorization is single-use with a short TTL. This goes well beyond the annotations and accurately reflects the side-effecting nature.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured and front-loaded with the core purpose, but it is somewhat repetitive (e.g., 'never a score, never a pass/fail verdict' repeated, and the cost flow spelled out twice in different words). Still, the multi-step cost flow justifies the extra detail.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has no output schema, the description usefully states the return shape (findings + rationale only). It also covers prerequisites, cost confirmation, authorization, and vendor-family selection, making the tool usable without external documentation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, and each parameter carries meaningful guidance: content limits, optional context/criteria roles, confirmedCost usage, authorization human-confirmation semantics, and modelUnderTest vendor-family behavior. The descriptions add practical usage context beyond the JSON schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool runs a cross-vendor judge model over arbitrary text and returns findings and rationale, never a score or pass/fail verdict. This distinguishes it from sibling audit/scan/verify tools, which evaluate with verdicts or structured findings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit usage guidance: first call quotes a price, second call confirms cost with the authorization id, third call runs the judge. It also explains the prerequisite of minting a key at the provided URL and clarifies that the judge always comes from a different vendor family than the content producer.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All nittim toolsInspect
Try in Browser
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources