Skip to main content
Glama

CompletionKit

by com.completionkit
OverviewDiscussionsAdmin

metrics_create

Create a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/numeric_equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds/numeric_bounds); valid_json takes no extra keys. target_path is required when target is json_path; a target_path sent with no target sets target to json_path, and one sent with any other target is refused. target_path walks array indices, as in items.0.name; any path may start at the root with $., as in $.items.0.name. list_overlap grades a list-valued field and stores a fraction rather than a verdict: score_by is recall, precision, f1 or jaccard (recall by default), and min is the fraction that counts as a pass (1, a perfect score, by default). numeric_bounds and numeric_equals read a number, ignoring the thousands commas and currency symbol a figure is written with; a comma counts only as a thousands separator, with three digits after it and no decimal point before it, so 1,5 is not a number; numeric_equals takes a tolerance, either a plain number for a fixed amount or a percentage string such as 2% for a share of the expected value. A check with no case_sensitive matches case-insensitively, regex and list_overlap included; send case_sensitive: true to match case exactly. Before 0.29.0 a regex check read a missing case_sensitive as case-sensitive. Set compare_to:"expected" on contains, not_contains, equals, json_path_equals, list_overlap, numeric_equals to grade against each row's own expected_output (ground truth) instead of a constant; drop value (or expected, for json_path_equals) and add expected_path to dig into the answer key when it is JSON.

Input Schema

NameRequiredDescriptionDefault
nameYes
tag_namesNo
instructionNo
metric_typeNo
check_configNo
rubric_bandsNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed9 schema fields changed
    • addedInput schema / properties / check_config / properties / case_sensitive / description
      Added value: +"A check with no case_sensitive matches case-insensitively, regex and list_overlap included; send case_sensitive: true to match case exactly. Before 0.29.0 a regex check read a missing case_sensitive as case-sensitive."
    • changedInput schema / properties / check_config / properties / check_kind / enum
      Previous value: -[
-  "contains",
-  "not_contains",
-  "equals",
-  "regex",
-  "valid_json",
-  "json_path_equals",
-  "length_bounds"
-]New value: +[
+  "contains",
+  "not_contains",
+  "equals",
+  "regex",
+  "valid_json",
+  "json_path_equals",
+  "length_bounds",
+  "list_overlap",
+  "numeric_bounds",
+  "numeric_equals"
+]
    • changedInput schema / properties / check_config / properties / max / type
      Previous value: -"integer"New value: +"number"
    • changedInput schema / properties / check_config / properties / min / type
      Previous value: -"integer"New value: +"number"
    • addedInput schema / properties / check_config / properties / score_by
      Added value: +{
+  "enum": [
+    "recall",
+    "precision",
+    "f1",
+    "jaccard"
+  ],
+  "type": "string"
+}
    • addedInput schema / properties / check_config / properties / tolerance
      Added value: +{
+  "description": "A plain number for a fixed amount, or a percentage string such as \"2%\" for a share of the expected value",
+  "type": [
+    "number",
+    "string"
+  ]
+}
    • addedInput schema / properties / check_config / properties / value / description
      Added value: +"Only list_overlap takes a list, as a JSON array or as comma-separated text; every other rule refuses one"
    • addedInput schema / properties / check_config / properties / value / items
      Added value: +{
+  "type": [
+    "string",
+    "number"
+  ]
+}
    • changedInput schema / properties / check_config / properties / value / type
      Previous value: -"string"New value: +[
+  "string",
+  "number",
+  "array"
+]
  2. First observed

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are supplied, so the description carries the full burden, and it does disclose non-obvious behaviors: target_path sent with a non-json_path target is refused, defaults (score_by=recall, min=1), comma-parsing rules for numbers, and version-dependent case_sensitive behavior before 0.29.0. It omits auth/permission needs and any side effects of persisting a metric, keeping it short of a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose is front-loaded and almost no sentence is filler, but the body is a single dense run-on block of prose without bullets or grouping, making the per-kind rules and version notes hard to parse quickly. Length is defensible given schema coverage, but structure is weak.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no annotations, no output schema, and a nested 6-param schema, the description covers the check_config branch well but says nothing about the llm_judge branch, instruction, rubric_bands, name, or tag_names. An agent configuring a judge metric gets no guidance, so coverage of the tool is only partial.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is effectively 0%, yet the description compensates thoroughly: it maps each check_kind to its required keys, explains target/target_path interaction, list_overlap scoring, tolerance formats, compare_to/expected_path, and case_sensitive semantics. This is meaning well beyond what the bare schema conveys.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The first sentence gives a specific verb+resource ('Create a metric with evaluation criteria') and immediately scopes the deterministic-check path. It is clearly distinguishable from metrics_update/metrics_list/metrics_delete by the create verb, though it never names a sibling tool explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives conditional routing for configuration ('For a deterministic check set metric_type:"check" and check_config') and per-kind required keys, which is real usage guidance. But it never says when to reach for metrics_create versus metrics_update, nor when the llm_judge alternative is appropriate, so tool-level selection is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All CompletionKit toolsInspect
Try in Browser
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources