Skip to main content
Glama
ismaelsoilet

jev-harness

by ismaelsoilet
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Remote

Triage Test Failure

jev_triage_test_failure
Read-onlyIdempotent

Triages test failures, compilation errors, or runtime exceptions to classify as env_missing, flaky_transient, deep_logic, or no_failure, and returns skip_llm plus action recommendation.

Instructions

Triages test failure traceback, compilation error, or runtime exception using Jev System One non-autoregressive decision classification (70-300ms, zero LLM generation). Detects missing environment packages, flaky transient glitches, or deep logic defects.

Use when: An agent encounters a test failure, traceback, compiler error (e.g. TS2307, ModuleNotFoundError, E0463), or needs to decide whether to invoke a frontier LLM. Do NOT use when: Tests pass, or for general code review or feature generation.

Returns: JSON object with 'category' (env_missing, flaky_transient, deep_logic, no_failure), 'skip_llm' (boolean: true if resolvable deterministically without frontier LLM), and 'action_recommendation' (string with concrete recovery action).

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
failure_logYesRaw test failure output, stack trace, or compiler error log.
test_commandNoOptional test command that was executed (e.g. 'pytest tests/', 'npm test').

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv0.2.2
    • addedInput schema / properties / test_command
      Added value: +{
+  "description": "Optional test command that was executed (e.g. 'pytest tests/', 'npm test').",
+  "type": "string"
+}
  2. First observedv0.2.0

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive, and the description adds genuinely new behavioral facts: the 70-300ms non-autoregressive classification with zero LLM generation, the three failure categories it can detect, and the JSON shape returned including 'skip_llm'. This goes well beyond what the annotations convey.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core purpose, then cleanly sectioned into Use when / Do NOT use when / Returns. Every sentence carries actionable information with no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, and the description compensates by enumerating the returned JSON fields (category values, skip_llm, action_recommendation). Combined with explicit triggers and exclusions, an agent has everything needed to invoke and interpret the tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so both parameters (failure_log, test_command) are already fully documented in the schema. The description mentions error-code examples that hint at log content but adds no format, size, or constraint guidance for the parameters themselves, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (triages) and resource (test failure traceback / compilation error / runtime exception), and names the mechanism (Jev System One non-autoregressive decision classification). It clearly distinguishes itself from siblings like jev_route_task or jev_check_abort by scoping to failure triage.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit 'Use when' clause lists concrete triggers (test failure, traceback, TS2307/ModuleNotFoundError/E0463) and the decision of whether to invoke a frontier LLM. The 'Do NOT use when' clause explicitly excludes passing tests and general code review or feature generation, leaving nothing to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools