noisefloor

by beepboop2025

Overview Schema Source Related Servers Discussions Admin

Peek-safe A/B verdict

ab_test

Read-onlyIdempotent

Compare two cumulative binary-outcome experiment arms to decide whether either success rate is higher, using beta-binomial confidence sequences and repeated looks. Use for conversions.

Instructions

Compare two cumulative binary-outcome experiment arms to decide whether either success rate is higher. Use for conversions or other Bernoulli outcomes with constant conditional success probabilities; use did_it_change for an ordered numeric metric, not these counts. The beta-binomial confidence sequences split alpha across arms, allowing repeated looks under the stated sampling assumptions. Do not substitute selected windows, continuous returns or changing-probability outcomes. Returns decided, winner (label or null), reading, and each label's successes/n/rate/interval; rates and bounds are fractions in [0,1]. Zero totals are allowed and leave the test undecided. No winner means overlapping intervals, not equal rates. Invalid counts (including successes > total) or duplicate/reserved labels return a tool error. Example: {"a_successes":40,"a_total":100,"b_successes":55,"b_total":100}.

Input Schema

TableJSON Schema

Name	Required	Description	Default
`alpha`	No	Total error budget shared by both confidence sequences; default 0.05 means 5%, split equally between arms. Must be strictly between 0 and 1.
`a_label`	No	Arm A's output key and possible winner label; default A. Must differ from b_label and not replace result keys such as decided, winner or reading.	A
`a_total`	Yes	Cumulative trials in arm A, including successes and failures. Zero is valid and returns an undecided result.
`b_label`	No	Arm B's output key and possible winner label; default B. Must differ from a_label and not replace result keys such as alpha, method or validity.	B
`b_total`	Yes	Cumulative trials in arm B, including successes and failures. Counts must use the same pre-defined experiment and outcome as arm A.
`a_successes`	Yes	Cumulative successful binary outcomes in arm A, an integer from 0 through a_total; do not supply a percentage.
`b_successes`	Yes	Cumulative successful binary outcomes in arm B, an integer from 0 through b_total; do not supply a percentage.

Output Schema

TableJSON Schema

Name	Required	Description
`alpha`	No	Total error budget split across the arms, as a fraction.
`error`	No	A string on tool error, or a successful arm object when the caller labels that arm error; inspect MCP isError.
`method`	No	Confidence-sequence method and reference.
`winner`	No	Winning caller-supplied arm label; null when undecided. Null means unavailable or not applicable, never zero.
`decided`	No	Whether the two confidence sequences are disjoint.
`reading`	No	Interpretation of the current comparison.
`validity`	No	Guarantees are conditional on the Bernoulli sampling model.
`guarantee`	No	Conditional confidence-sequence guarantee; not a market-performance claim.
`assumptions`	No	Ordered statements or identifiers.
`safe_to_peek`	No	Repeated looks are allowed only under the stated Bernoulli assumptions.
`what_would_decide_it`	No	Qualitative guidance for an undecided comparison with data in both arms.
`observed_difference_pp`	No	B's observed success rate minus A's, in percentage points; present only for undecided nonempty arms.

Schema Changelog

Changes observed during successful MCP inspections.

Changed4 schema fields changedv0.4.3Oct 18, 2026

removedOutput schema / additionalProperties
```
Removed value: -true
```

addedOutput schema / anyOf

Added value: +[
+  {
+    "additionalProperties": {
+      "additionalProperties": false,
+      "description": "One experiment arm, stored under its caller-supplied label.",
+      "properties": {
+        "interval": {
+          "description": "Lower and upper confidence-sequence bounds.",
+          "items": {
+            "description": "Success-probability bound as a fraction.",
+            "type": "number"
+          },
+          "maxItems": 2,
+          "minItems": 2,
+          "type": "array"
+        },
+        "n": {
+          "description": "Cumulative trials, including failures.",
+          "minimum": 0,
+          "type": "integer"
+        },
+        "rate": {
+          "anyOf": [
+            {
+              "description": "Observed success fraction; null when n is zero.",
+              "type": "number"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "description": "Observed success fraction; null when n is zero. Null means unavailable or not applicable, never zero."
+        },
+        "successes": {
+          "description": "Cumulative successful binary outcomes.",
+          "minimum": 0,
+          "type": "integer"
+        }
+      },
+      "required": [
+        "successes",
+        "n",
+        "rate",
+        "interval"
+      ],
+      "type": "object"
+    },
+    "properties": {
+      "alpha": {},
+      "assumptions": {},
+      "decided": {},
+      "guarantee": {},
+      "method": {},
+      "observed_difference_pp": {},
+      "reading": {},
+      "safe_to_peek": {},
+      "validity": {},
+      "what_would_decide_it": {},
+      "winner": {}
+    },
+    "required": [
+      "decided",
+      "winner",
+      "reading",
+      "alpha",
+      "safe_to_peek",
+      "validity",
+      "guarantee",
+      "assumptions",
+      "method"
+    ],
+    "type": "object"
+  },
+  {
+    "additionalProperties": false,
+    "description": "Existing tool-error payload; no successful result fields are implied.",
+    "properties": {
+      "error": {
+        "description": "Validation or computation failure message; inspect MCP isError or the HTTP status.",
+        "type": "string"
+      }
+    },
+    "required": [
+      "error"
+    ],
+    "type": "object"
+  }
+]

changedOutput schema / description

Previous value: -"Tool-specific result object described in each tool's description. Check status/ok/state, reading, validity and limits before using numbers. MCP returns this object in structuredContent and JSON text; invalid input returns isError=true with an error field. No tool fetches sources or executes trades."New value: +"Two dynamically named arm objects plus comparison metadata. A and B are defaults, not fixed output keys. Tool errors retain the separate error-message shape."

addedOutput schema / properties

Added value: +{
+  "alpha": {
+    "description": "Total error budget split across the arms, as a fraction.",
+    "type": "number"
+  },
+  "assumptions": {
+    "description": "Ordered statements or identifiers.",
+    "items": {
+      "description": "One retained statement or identifier.",
+      "type": "string"
+    },
+    "type": "array"
+  },
+  "decided": {
+    "description": "Whether the two confidence sequences are disjoint.",
+    "type": "boolean"
+  },
+  "error": {
+    "anyOf": [
+      {
+        "description": "Validation or computation failure message; inspect MCP isError or the HTTP status.",
+        "type": "string"
+      },
+      {
+        "additionalProperties": false,
+        "description": "One experiment arm, stored under its caller-supplied label.",
+        "properties": {
+          "interval": {
+            "description": "Lower and upper confidence-sequence bounds.",
+            "items": {
+              "description": "Success-probability bound as a fraction.",
+              "type": "number"
+            },
+            "maxItems": 2,
+            "minItems": 2,
+            "type": "array"
+          },
+          "n": {
+            "description": "Cumulative trials, including failures.",
+            "minimum": 0,
+            "type": "integer"
+          },
+          "rate": {
+            "anyOf": [
+              {
+                "description": "Observed success fraction; null when n is zero.",
+                "type": "number"
+              },
+              {
+                "type": "null"
+              }
+            ],
+            "description": "Observed success fraction; null when n is zero. Null means unavailable or not applicable, never zero."
+          },
+          "successes": {
+            "description": "Cumulative successful binary outcomes.",
+            "minimum": 0,
+            "type": "integer"
+          }
+        },
+        "required": [
+          "successes",
+          "n",
+          "rate",
+          "interval"
+        ],
+        "type": "object"
+      }
+    ],
+    "description": "A string on tool error, or a successful arm object when the caller labels that arm error; inspect MCP isError."
+  },
+  "guarantee": {
+    "description": "Conditional confidence-sequence guarantee; not a market-performance claim.",
+    "type": "string"
+  },
+  "method": {
+    "description": "Confidence-sequence method and reference.",
+    "type": "string"
+  },
+  "observed_difference_pp": {
+    "description": "B's observed success rate minus A's, in percentage points; present only for undecided nonempty arms.",
+    "type": "number"
+  },
+  "reading": {
+    "description": "Interpretation of the current comparison.",
+    "type": "string"
+  },
+  "safe_to_peek": {
+    "const": true,
+    "description": "Repeated looks are allowed only under the stated Bernoulli assumptions.",
+    "type": "boolean"
+  },
+  "validity": {
+    "const": "conditional_on_bernoulli_model",
+    "description": "Guarantees are conditional on the Bernoulli sampling model.",
+    "type": "string"
+  },
+  "what_would_decide_it": {
+    "description": "Qualitative guidance for an undecided comparison with data in both arms.",
+    "type": "string"
+  },
+  "winner": {
+    "anyOf": [
+      {
+        "description": "Winning caller-supplied arm label; null when undecided.",
+        "type": "string"
+      },
+      {
+        "type": "null"
+      }
+    ],
+    "description": "Winning caller-supplied arm label; null when undecided. Null means unavailable or not applicable, never zero."
+  }
+}

Changed11 schema fields changedv0.4.3-glama.1Oct 17, 2026

addedInput schema / properties / a_label / default
```
Added value: +"A"
```

addedInput schema / properties / a_label / description

Added value: +"Arm A's output key and possible winner label; default A. Must differ from b_label and not replace result keys such as decided, winner or reading."

addedInput schema / properties / a_successes / description

Added value: +"Cumulative successful binary outcomes in arm A, an integer from 0 through a_total; do not supply a percentage."

addedInput schema / properties / a_total / description

Added value: +"Cumulative trials in arm A, including successes and failures. Zero is valid and returns an undecided result."

addedInput schema / properties / alpha / default
```
Added value: +0.05
```

addedInput schema / properties / alpha / description

Added value: +"Total error budget shared by both confidence sequences; default 0.05 means 5%, split equally between arms. Must be strictly between 0 and 1."

addedInput schema / properties / b_label / default
```
Added value: +"B"
```

addedInput schema / properties / b_label / description

Added value: +"Arm B's output key and possible winner label; default B. Must differ from a_label and not replace result keys such as alpha, method or validity."

addedInput schema / properties / b_successes / description

Added value: +"Cumulative successful binary outcomes in arm B, an integer from 0 through b_total; do not supply a percentage."

addedInput schema / properties / b_total / description

Added value: +"Cumulative trials in arm B, including successes and failures. Counts must use the same pre-defined experiment and outcome as arm A."

addedOutput schema / description

Added value: +"Tool-specific result object described in each tool's description. Check status/ok/state, reading, validity and limits before using numbers. MCP returns this object in structuredContent and JSON text; invalid input returns isError=true with an error field. No tool fetches sources or executes trades."

Changed20 schema fields changedv0.4.1Oct 17, 2026
- addedInput schema / properties / a_label / maxLength
```
Added value: +128
```
- addedInput schema / properties / a_label / minLength
```
Added value: +1
```
- removedInput schema / properties / a_successes / description
```
Removed value: -"conversions in arm A"
```
- addedInput schema / properties / a_successes / maximum
```
Added value: +1000000000000
```
- addedInput schema / properties / a_successes / minimum
```
Added value: +0
```
- removedInput schema / properties / a_total / description
```
Removed value: -"total observations in arm A"
```
- addedInput schema / properties / a_total / maximum
```
Added value: +1000000000000
```
- addedInput schema / properties / a_total / minimum
```
Added value: +0
```
- removedInput schema / properties / alpha / description
```
Removed value: -"error budget, default 0.05"
```
- addedInput schema / properties / alpha / exclusiveMaximum
```
Added value: +1
```
- addedInput schema / properties / alpha / exclusiveMinimum
```
Added value: +0
```
- addedInput schema / properties / b_label / maxLength
```
Added value: +128
```
- addedInput schema / properties / b_label / minLength
```
Added value: +1
```
- removedInput schema / properties / b_successes / description
```
Removed value: -"conversions in arm B"
```
- addedInput schema / properties / b_successes / maximum
```
Added value: +1000000000000
```
- addedInput schema / properties / b_successes / minimum
```
Added value: +0
```
- removedInput schema / properties / b_total / description
```
Removed value: -"total observations in arm B"
```
- addedInput schema / properties / b_total / maximum
```
Added value: +1000000000000
```
- addedInput schema / properties / b_total / minimum
```
Added value: +0
```
- changedOutput schema / (root)
```
Previous value: -nullNew value: +{
+  "additionalProperties": true,
+  "type": "object"
+}
```
First observedv0.1.2Jul 28, 2026

TDQS

A4.8/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only say this is a safe read; the description goes well beyond that by disclosing the statistical method and its assumption (beta-binomial confidence sequences splitting alpha across arms, peek-safe under stated sampling assumptions), the error contract (invalid counts, successes > total, duplicate/reserved labels raise a tool error), and the interpretation caveat that 'no winner' means overlapping intervals rather than equal rates. The one redundancy — listing returned fields — is covered by the output schema, but the rest is genuinely additive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single dense but front-loaded paragraph: purpose first, then routing, assumptions, return shape, edge cases, and an example. Nearly every sentence earns its place, though the return-field enumeration partly duplicates the output schema and adds length without new routing value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 7 parameters, full schema coverage, an output schema, and read-only/idempotent annotations, the description still covers the things structured fields cannot: sampling assumptions, error conditions, zero-total behavior, and how to read an undecided verdict. An agent has everything needed to call this correctly and interpret the result.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3; the description adds a concrete worked example ({"a_successes":40,"a_total":100,"b_successes":55,"b_total":100}) that pins down the expected integer-count form and reinforces 'do not supply a percentage'. It adds little beyond the schema on alpha and labels, so it stays below a full 5.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Compare two cumulative binary-outcome experiment arms to decide whether either success rate is higher') and immediately names the sibling it is not ('use did_it_change for an ordered numeric metric'). An agent can route between ab_test and did_it_change without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when-to-use ('conversions or other Bernoulli outcomes with constant conditional success probabilities'), an explicit alternative with its selecting condition ('did_it_change for an ordered numeric metric'), and explicit exclusions ('do not substitute selected windows, continuous returns or changing-probability outcomes'). Nothing is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server