Skip to main content
Glama
Sign In
Kiln

Kiln

by codeofaxel
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python

cross_preset_ab_report

Read-only

Compares print outcomes across presets to report A/B performance differences. Use it to identify which preset version produces better results and exclude presets with too few recorded outcomes.

Instructions

Cross-preset outcome A/B analysis.

Needs Kiln Pro.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "preset_inputs": "<preset_inputs>"
    },
    "name": "cross_preset_ab_report"
  }
}

Input Schema

NameRequiredDescriptionDefault
preset_inputsYesThe presets to compare, one dict each with ``preset_id``, ``preset_version_id``, ``preset_name``, ``fingerprint`` and optionally ``outcomes``. An entry without ``outcomes`` gets the recorded history of its ``preset_version_id``.
min_outcomes_per_presetNoThe fewest outcomes a preset needs to be compared; presets with fewer are excluded. Defaults to ``3``. Default: 3.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed2 schema fields changedv1.5.1
    • changedInput schema / properties / min_outcomes_per_preset / description
      Previous value: -"Default: 3."New value: +"The fewest outcomes a preset needs to be compared; presets with fewer are excluded.  Defaults to ``3``. Default: 3."
    • addedInput schema / properties / preset_inputs / description
      Added value: +"The presets to compare, one dict each with ``preset_id``, ``preset_version_id``, ``preset_name``, ``fingerprint`` and optionally ``outcomes``.  An entry without ``outcomes`` gets the recorded history of its ``preset_version_id``."
  2. Addedv1.5.0

TDQS

C2.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, destructiveHint=false and openWorldHint=false, so the safety profile is covered. The description adds a tier/entitlement requirement (Kiln Pro), which is real behavioral context not present in the annotations, but it says nothing about what the analysis computes, whether it errors on insufficient outcomes, or what the result looks like.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short lines, front-loaded with the operation and followed by the gating requirement — no wasted words. It is arguably too terse rather than padded, but structurally clean.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a report/analysis tool with no output schema, the description should indicate what the comparison yields (metrics, per-preset breakdown, verdict), and it does not. Combined with two near-duplicate A/B siblings and no differentiator, an agent cannot confidently select this tool for a given comparison task.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the preset_inputs and min_outcomes_per_preset descriptions are detailed (entry shape, fallback to recorded history, default of 3, exclusion behavior). The description itself adds nothing beyond the schema, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

"Cross-preset outcome A/B analysis" names the resource (presets) and the operation (outcome A/B comparison), so the general intent is inferable. However, it never says what is actually compared or produced, and it gives no hint of how it differs from the near-identical siblings cross_feature_ab_report and cross_branch_ab_report.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

"Needs Kiln Pro" is a useful prerequisite, but there is no when-to-use guidance at all and no routing away from the parallel cross_feature_ab_report / cross_branch_ab_report tools. An agent choosing among the three A/B report variants gets no discriminating signal.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools