Skip to main content
Glama

jev_evaluate

Read-only

Ask a calibrated judgment model multiple typed questions about one shared state and get probabilities plus an auto/review/escalate gate; use it for decisions no other jev_* tool covers.

Instructions

Ask Jev — a fast, calibrated judgment model — many typed questions about one shared state; returns probabilities plus a gate computed in code. Use it when no other jev_* tool fits. Writing questions (Jev reads literally):

  • State the exact condition in instructions; put boundary cases in criteria.

  • Criteria accept JSON: Choice options and Noul sides as {what, not_for, examples}; Score levels as {summary, signals}. Put lookalike cases under not_for on the side they would wrongly land on.

  • One judgment per question; split compound ones and combine in code.

  • Batch every question sharing a state into ONE call. They run in parallel and cost only their own tokens, so speculative questions are nearly free.

  • Send only the state the question needs; point at parts by path, e.g. ticket.messages[0].text.

  • Never ask it to count, do arithmetic, or compare dates/numbers — compute those in code and pass the result in. Answers: it selects from your options and never generates text; noul is P(yes), ~0.5 means unsure; choice/score carry confidence; gate is auto/review/escalate. Budget: ~64k tokens state + all questions, ~32k state + longest question.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "state": "<state>",
      "questions": "<questions>"
    },
    "name": "jev_evaluate"
  }
}

Input Schema

NameRequiredDescriptionDefault
modelNoOverride the configured model, e.g. `jev-1.13.0` to pin a version.
stateYesThe content to judge: a plain string, or structured data that questions reference by path.
questionsYesMap of question id -> question. Ids are yours; answers come back under the same ids.
thresholdsNoOverride the server's configured gating thresholds for this call only.

Output Schema

NameRequiredDescriptionDefault
modelYesThe versioned model id that actually answered.
usageYesToken usage. Jev charges for input tokens only.
answersYesOne answer per question id, each augmented with a `gate` computed in code.
latency_msYesWall-clock time for the underlying API call(s), including retries.
thresholdsYesThe thresholds actually applied.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv0.7.0
    • changedInput schema / properties / questions / additionalProperties / oneOf
      Previous value: -[
      -  {
      -    "properties": {
      -      "criteria": {
      -        "additionalProperties": {
      -          "type": [
      -            "string",
      -            "null"
      -          ]
      -        },
      -        "description": "option -> rubric. Use null when the option name says it all. At least 2 options; add an 'other'/'none' escape hatch.",
      -        "propertyNames": {
      -          "type": "string"
      -        },
      -        "type": "object"
      -      },
      -      "instructions": {
      -        "anyOf": [
      -          {
      -            "type": "string"
      -          },
      -          {
      -            "items": {},
      -            "type": "array"
      -          },
      -          {
      -            "additionalProperties": {},
      -            "propertyNames": {
      -              "type": "string"
      -            },
      -            "type": "object"
      -          }
      -        ],
      -        "description": "The question itself. Write it in full; question ids are never sent to the model."
      -      },
      -      "type": {
      -        "const": "choice",
      -        "type": "string"
      -      }
      -    },
      -    "required": [
      -      "type",
      -      "instructions",
      -      "criteria"
      -    ],
      -    "type": "object"
      -  },
      -  {
      -    "properties": {
      -      "criteria": {
      -        "description": "Ordered level descriptions, lowest first. At least 2.",
      -        "items": {
      -          "type": "string"
      -        },
      -        "type": "array"
      -      },
      -      "instructions": {
      -        "anyOf": [
      -          {
      -            "type": "string"
      -          },
      -          {
      -            "items": {},
      -            "type": "array"
      -          },
      -          {
      -            "additionalProperties": {},
      -            "propertyNames": {
      -              "type": "string"
      -            },
      -            "type": "object"
      -          }
      -        ],
      -        "description": "The question itself. Write it in full; question ids are never sent to the model."
      -      },
      -      "type": {
      -        "const": "score",
      -        "type": "string"
      -      }
      -    },
      -    "required": [
      -      "type",
      -      "instructions",
      -      "criteria"
      -    ],
      -    "type": "object"
      -  },
      -  {
      -    "properties": {
      -      "criteria": {
      -        "description": "Optional clarification of what yes and no mean. Keep them aligned with the instructions.",
      -        "properties": {
      -          "false": {
      -            "type": "string"
      -          },
      -          "true": {
      -            "type": "string"
      -          }
      -        },
      -        "type": "object"
      -      },
      -      "instructions": {
      -        "anyOf": [
      -          {
      -            "type": "string"
      -          },
      -          {
      -            "items": {},
      -            "type": "array"
      -          },
      -          {
      -            "additionalProperties": {},
      -            "propertyNames": {
      -              "type": "string"
      -            },
      -            "type": "object"
      -          }
      -        ],
      -        "description": "The question itself. Write it in full; question ids are never sent to the model."
      -      },
      -      "type": {
      -        "const": "noul",
      -        "type": "string"
      -      }
      -    },
      -    "required": [
      -      "type",
      -      "instructions"
      -    ],
      -    "type": "object"
      -  }
      -]New value: +[
      +  {
      +    "properties": {
      +      "criteria": {
      +        "additionalProperties": {
      +          "anyOf": [
      +            {
      +              "type": "string"
      +            },
      +            {
      +              "items": {},
      +              "type": "array"
      +            },
      +            {
      +              "additionalProperties": {},
      +              "propertyNames": {
      +                "type": "string"
      +              },
      +              "type": "object"
      +            },
      +            {
      +              "type": "null"
      +            }
      +          ],
      +          "description": "A rubric entry: a string, or JSON structure such as {what, not_for, examples} / {summary, signals}."
      +        },
      +        "description": "option -> rubric. Use null when the option name says it all. At least 2 options; add an 'other'/'none' escape hatch.",
      +        "propertyNames": {
      +          "type": "string"
      +        },
      +        "type": "object"
      +      },
      +      "instructions": {
      +        "anyOf": [
      +          {
      +            "type": "string"
      +          },
      +          {
      +            "items": {},
      +            "type": "array"
      +          },
      +          {
      +            "additionalProperties": {},
      +            "propertyNames": {
      +              "type": "string"
      +            },
      +            "type": "object"
      +          }
      +        ],
      +        "description": "The question itself. Write it in full; question ids are never sent to the model."
      +      },
      +      "type": {
      +        "const": "choice",
      +        "type": "string"
      +      }
      +    },
      +    "required": [
      +      "type",
      +      "instructions",
      +      "criteria"
      +    ],
      +    "type": "object"
      +  },
      +  {
      +    "properties": {
      +      "criteria": {
      +        "description": "Ordered level descriptions, lowest first. At least 2.",
      +        "items": {
      +          "anyOf": [
      +            {
      +              "type": "string"
      +            },
      +            {
      +              "items": {},
      +              "type": "array"
      +            },
      +            {
      +              "additionalProperties": {},
      +              "propertyNames": {
      +                "type": "string"
      +              },
      +              "type": "object"
      +            },
      +            {
      +              "type": "null"
      +            }
      +          ],
      +          "description": "A rubric entry: a string, or JSON structure such as {what, not_for, examples} / {summary, signals}."
      +        },
      +        "minItems": 2,
      +        "type": "array"
      +      },
      +      "instructions": {
      +        "anyOf": [
      +          {
      +            "type": "string"
      +          },
      +          {
      +            "items": {},
      +            "type": "array"
      +          },
      +          {
      +            "additionalProperties": {},
      +            "propertyNames": {
      +              "type": "string"
      +            },
      +            "type": "object"
      +          }
      +        ],
      +        "description": "The question itself. Write it in full; question ids are never sent to the model."
      +      },
      +      "type": {
      +        "const": "score",
      +        "type": "string"
      +      }
      +    },
      +    "required": [
      +      "type",
      +      "instructions",
      +      "criteria"
      +    ],
      +    "type": "object"
      +  },
      +  {
      +    "properties": {
      +      "criteria": {
      +        "description": "Optional clarification of what yes and no mean. Keep them aligned with the instructions.",
      +        "properties": {
      +          "false": {
      +            "anyOf": [
      +              {
      +                "type": "string"
      +              },
      +              {
      +                "items": {},
      +                "type": "array"
      +              },
      +              {
      +                "additionalProperties": {},
      +                "propertyNames": {
      +                  "type": "string"
      +                },
      +                "type": "object"
      +              },
      +              {
      +                "type": "null"
      +              }
      +            ],
      +            "description": "A rubric entry: a string, or JSON structure such as {what, not_for, examples} / {summary, signals}."
      +          },
      +          "true": {
      +            "anyOf": [
      +              {
      +                "type": "string"
      +              },
      +              {
      +                "items": {},
      +                "type": "array"
      +              },
      +              {
      +                "additionalProperties": {},
      +                "propertyNames": {
      +                  "type": "string"
      +                },
      +                "type": "object"
      +              },
      +              {
      +                "type": "null"
      +              }
      +            ],
      +            "description": "A rubric entry: a string, or JSON structure such as {what, not_for, examples} / {summary, signals}."
      +          }
      +        },
      +        "type": "object"
      +      },
      +      "instructions": {
      +        "anyOf": [
      +          {
      +            "type": "string"
      +          },
      +          {
      +            "items": {},
      +            "type": "array"
      +          },
      +          {
      +            "additionalProperties": {},
      +            "propertyNames": {
      +              "type": "string"
      +            },
      +            "type": "object"
      +          }
      +        ],
      +        "description": "The question itself. Write it in full; question ids are never sent to the model."
      +      },
      +      "type": {
      +        "const": "noul",
      +        "type": "string"
      +      }
      +    },
      +    "required": [
      +      "type",
      +      "instructions"
      +    ],
      +    "type": "object"
      +  }
      +]
  2. First observedv0.1.4

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish read-only and open-world, and the description adds substantial context beyond them: questions run in parallel and cost only their own tokens, the model selects from supplied options and never generates free text, ~0.5 means unsure, and the gate is computed in code. It also discloses budget limits (~64k tokens total, ~32k state plus longest question), which is exactly the kind of constraint an agent needs before calling.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with a one-line summary, then organized as tight bullets covering writing questions, answers, and budget. A few lines restate what the schema already documents (criteria accepting JSON), but overall the density is justified for a tool with this much surface area.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with nested question types and an output schema, the description supplies everything an agent needs beyond structured fields: how to size state, how to phrase and split questions, what the answer fields mean, and the token budget. Return-value shape is covered by the output schema, so no required context is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3, but the description adds real meaning: instructions are read literally, criteria accept structured {what, not_for, examples} and {summary, signals} shapes, boundary and lookalike cases belong in not_for on the side they would wrongly land on, and state is referenced by path such as ticket.messages[0].text. The thresholds parameter is only implied via the auto/review/escalate gate wording.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence states a specific verb and resource: ask a judgment model many typed questions about one shared state, returning probabilities plus a code-computed gate. The 'one shared state, many questions' scope cleanly distinguishes it from single-answer siblings like jev_pick or jev_verify, though the explicit sibling routing is a generic wildcard rather than naming what each alternative does.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a clear selection rule ('use it when no other jev_* tool fits') plus concrete do/don't guidance: batch all questions sharing a state into one call, one judgment per question, and never ask it to count or compare dates. It does not say which specific sibling to use for the cases this tool is wrong for, so routing is only half specified.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.