Skip to main content
Glama

nittim

by com.nittim

Judge Arbitrary Output (independent cross-vendor model)

judge_output

Run a cross-vendor judge model over any text you post: code, a document, another model's output, anything. Returns findings + rationale ONLY — never a numeric score, never a pass/fail verdict. Costs 5.03 credits. The one paid tool a publishable (nittim_pk_…) key may call, subject to its own spend cap. The judge is always a model from a different vendor family than whatever produced the content — grading a model with a model from its own vendor family shares training data and blind spots, the exact thing an independent judge exists to avoid. modelUnderTest names which vendor family produced content, and is most useful whenever that content did not come from the client itself asking. Every answer states which judge vendor ran. NEEDS A KEY: mint one at https://nittim.com/keys. COSTS MONEY: the first call only returns a price quote and runs nothing. A second call, with confirmedCost set to that exact price, runs it — a key that is not marked autonomous also needs the account owner's own approval before that second call runs.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
contentYesThe text to judge — code, a document, another model's output. Up to ~100KB.
contextNoOptional — background the judge should know, e.g. what this content is for.
criteriaNoOptional — what to judge it against, e.g. 'correctness and security'.
authorizationNoHUMAN CONFIRMATION. Omit until the account owner has pressed Confirm on the link a paid tool answered with; then set to that answer's `authorization` id (single-use, short TTL). Protocol: server instructions.
confirmedCostNoCOST CONFIRMATION. Omit on the first call (it only quotes); then set to the exact `cost` object quoted. A moved price is re-quoted, never charged. Protocol: server instructions.
modelUnderTestNoOptional — which vendor family produced `content`, if it's itself a model's output. The judge that runs is always a different vendor family than this names, so the grader is never the same vendor family as what it's grading. Set 'unspecified' for anything that isn't model output (code, a document) or when the family is genuinely unknown. Passing it explicitly is most useful whenever the content did not come from you asking about your own output.

Schema Changelog

Changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. Changed1 schema field changed
    • changedInput schema / properties / modelUnderTest / description
      Before
      "Optional — which vendor family produced `content`, if it's itself a model's output. 'openai' routes the judge to Anthropic instead of the default OpenAI judge, so the grader is never the same vendor family as what it's grading. Set 'unspecified' for anything that isn't model output (code, a document) or when the family is genuinely unknown — that pins the judge to the default (OpenAI). OMITTING this is different from 'unspecified': the family is then inferred from your MCP client's declared name, so an OpenAI-based client posting its own output gets an Anthropic judge without having to ask. Any explicit value here overrides that inference."
      After
      "Optional — which vendor family produced `content`, if it's itself a model's output. The judge that runs is always a different vendor family than this names, so the grader is never the same vendor family as what it's grading. Set 'unspecified' for anything that isn't model output (code, a document) or when the family is genuinely unknown. Passing it explicitly is most useful whenever the content did not come from you asking about your own output."
  2. Added

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations are uniformly false, so the description carries the full burden — and it over-delivers: cost (5.03 credits), key requirement with minting URL, two-phase execution (first call quotes and runs nothing, second runs on confirmedCost), owner-approval requirement for non-autonomous keys, the always-cross-vendor guarantee, and the output-shape restriction. No contradiction with the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Long (~170 words) but dense and front-loaded: the core function and return contract lead, followed by cost and key facts before the protocol detail. Minor redundancy exists — the confirmedCost two-call flow is restated in the schema descriptions — and the vendor-family rationale sentence runs slightly long, so not a 5.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a paid, two-call, key-gated tool with no output schema and no informative annotations, the description covers every operational essential: purpose, return format, cost, key requirements, approval step, cross-vendor behavior, and judge-vendor disclosure. Nothing an agent needs to drive the protocol correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3. The description adds genuine value on top: it explains the behavioral effect of modelUnderTest (the judge is always a different vendor family than the one named) and why that matters, plus the overall two-call protocol that gives operational meaning to confirmedCost and authorization.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: 'Run a cross-vendor judge model over any text you post: code, a document, another model's output, anything.' It also pins down the return contract ('findings + rationale ONLY — never a numeric score, never a pass/fail verdict') and the cross-vendor constraint, distinguishing it clearly from the repo/source audit siblings (audit_repo, scan_source, run_module, etc.).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear usage context: applicable to any text, the two-call cost-confirmation flow, and when modelUnderTest is most useful ('whenever that content did not come from the client itself asking'). It does not explicitly name sibling alternatives or state when not to use it, but the siblings are functionally distant (repo/source audits, module runs), so routing risk is low.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All nittim toolsInspect
Try in Browser
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

TDQS

A4.3/5.0
Disambiguation4/5

Most tools map to distinct workflow stages: estimate, preview, submit audit, retrieve audit, run module, dispute finding, and judge output. The closest pairs are audit_repo vs audit_source and scan_source vs run_module, but the descriptions provide enough boundary detail for an agent to choose correctly.

Naming Consistency5/5

Every tool follows the same snake_case verb_noun pattern: audit_repo, audit_source, estimate_audit, get_audit, list_modules, run_module, scan_source, and so on. There is no mixing of naming conventions or vague standalone verbs.

Tool Count5/5

11 tools is well-scoped for a paid AI audit service with async delivery, approval flows, and both repo-based and source-based inputs. Each tool covers a distinct step in the audit pipeline without feeling redundant or bloated.

Completeness4/5

The core lifecycle is covered: estimate cost, submit audits, run single modules, check status, retrieve results, and dispute findings. Minor gaps include no way to list past audits, cancel a pending audit, or manage credits directly, though estimate_audit does show the balance.

Resources