Skip to main content
Glama
luiacuaniello

PerspectiveGraph

by luiacuaniello
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Go
Hybrid

get_score_trust

Read-onlyIdempotent

Validate whether a score can be quoted as a probability: compare predicted vs. observed red-team or BAS outcomes, check calibration, and test if rankings beat chance.

Instructions

Report how well the engine's probabilities have matched reality, measured against recorded red-team or BAS outcomes: the verdict (well-calibrated / calibrated-on-average / overconfident / underconfident / insufficient-data; calibrated-on-average means only the average matches, so no individual score may be quoted as a probability), the predicted-versus-observed rates, and what to do about the gap. Call this before quoting any score as a probability. If it reports insufficient-data, the numbers are expert estimates and must be presented as the model's own estimate, not as odds. It also reports discrimination - whether the score, and separately the triage Priority order, put confirmed paths above refuted ones (AUC with a 95% interval). Do not present the ranking as evidence of which path is most dangerous unless priorityDiscrimination reads 'discriminates'; 'insufficient-data' or 'indistinguishable-from-chance' means the order has not been shown to beat a coin, and must be said so.

Input Schema

NameRequiredDescriptionDefault

No arguments

Output Schema

NameRequiredDescriptionDefault
validationNoWhat happened to the routes that were tested.
calibrationNoThe scores compared with the recorded red-team and BAS outcomes.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv1.38.1
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
+  "properties": {
+    "calibration": {
+      "description": "The scores compared with the recorded red-team and BAS outcomes.",
+      "properties": {
+        "brier": {
+          "description": "Mean squared gap between score and outcome, 0-1. Lower is better.",
+          "type": [
+            "number",
+            "null"
+          ]
+        },
+        "diagnosis": {
+          "description": "What to do about the gap, in one line.",
+          "type": [
+            "string",
+            "null"
+          ]
+        },
+        "discrimination": {
+          "description": "Whether the score puts confirmed routes above refuted ones.",
+          "properties": {
+            "auc": {
+              "description": "Probability that a confirmed route outranks a refuted one. 0.5 is a coin. Null until there is one of each.",
+              "type": [
+                "number",
+                "null"
+              ]
+            },
+            "aucHigh": {
+              "description": "High end. An interval containing 0.5 cannot be told from chance.",
+              "type": [
+                "number",
+                "null"
+              ]
+            },
+            "aucLow": {
+              "description": "Low end of its approximate 95% interval.",
+              "type": [
+                "number",
+                "null"
+              ]
+            },
+            "hasData": {
+              "description": "False when no outcome has been recorded.",
+              "type": [
+                "boolean",
+                "null"
+              ]
+            },
+            "negatives": {
+              "description": "Refuted outcomes behind the figure.",
+              "type": [
+                "integer",
+                "null"
+              ]
+            },
+            "positives": {
+              "description": "Confirmed outcomes behind the figure.",
+              "type": [
+                "integer",
+                "null"
+              ]
+            },
+            "verdict": {
+              "description": "discriminates, indistinguishable-from-chance, inverted or insufficient-data.",
+              "type": [
+                "string",
+                "null"
+              ]
+            }
+          },
+          "type": [
+            "object",
+            "null"
+          ]
+        },
+        "ece": {
+          "description": "Expected calibration error, 0-1. Lower is better.",
+          "type": [
+            "number",
+            "null"
+          ]
+        },
+        "hasData": {
+          "description": "False when no outcome has been recorded: every score is then an expert estimate.",
+          "type": [
+            "boolean",
+            "null"
+          ]
+        },
+        "meanPredicted": {
+          "description": "The mean score of the tested routes.",
+          "type": [
+            "number",
+            "null"
+          ]
+        },
+        "observedRate": {
+          "description": "The share of them that were confirmed.",
+          "type": [
+            "number",
+            "null"
+          ]
+        },
+        "priorityDiscrimination": {
+          "description": "Whether the triage order puts confirmed routes above refuted ones.",
+          "properties": {
+            "auc": {
+              "description": "Probability that a confirmed route outranks a refuted one. 0.5 is a coin. Null until there is one of each.",
+              "type": [
+                "number",
+                "null"
+              ]
+            },
+            "aucHigh": {
+              "description": "High end. An interval containing 0.5 cannot be told from chance.",
+              "type": [
+                "number",
+                "null"
+              ]
+            },
+            "aucLow": {
+              "description": "Low end of its approximate 95% interval.",
+              "type": [
+                "number",
+                "null"
+              ]
+            },
+            "hasData": {
+              "description": "False when no outcome has been recorded.",
+              "type": [
+                "boolean",
+                "null"
+              ]
+            },
+            "negatives": {
+              "description": "Refuted outcomes behind the figure.",
+              "type": [
+                "integer",
+                "null"
+              ]
+            },
+            "positives": {
+              "description": "Confirmed outcomes behind the figure.",
+              "type": [
+                "integer",
+                "null"
+              ]
+            },
+            "verdict": {
+              "description": "discriminates, indistinguishable-from-chance, inverted or insufficient-data.",
+              "type": [
+                "string",
+                "null"
+              ]
+            }
+          },
+          "type": [
+            "object",
+            "null"
+          ]
+        },
+        "recommendedScale": {
+          "description": "Observed rate over predicted: below 1 the scores run high, above 1 low. Null until the sample is large enough.",
+          "type": [
+            "number",
+            "null"
+          ]
+        },
+        "samples": {
+          "description": "Tested routes that carried a score: the size of the sample.",
+          "type": [
+            "integer",
+            "null"
+          ]
+        },
+        "verdict": {
+          "description": "well-calibrated, calibrated-on-average, overconfident, underconfident or insufficient-data.",
+          "type": [
+            "string",
+            "null"
+          ]
+        }
+      },
+      "type": [
+        "object",
+        "null"
+      ]
+    },
+    "validation": {
+      "description": "What happened to the routes that were tested.",
+      "properties": {
+        "confirmed": {
+          "description": "Tested routes an attacker could really follow.",
+          "type": [
+            "integer",
+            "null"
+          ]
+        },
+        "missed": {
+          "description": "Real routes a tester found that the engine had not reported: false negatives.",
+          "type": [
+            "integer",
+            "null"
+          ]
+        },
+        "partial": {
+          "description": "Tested routes that worked in part.",
+          "type": [
+            "integer",
+            "null"
+          ]
+        },
+        "precision": {
+          "description": "confirmed / (confirmed + refuted). Null until one of the two exists.",
+          "type": [
+            "number",
+            "null"
+          ]
+        },
+        "recall": {
+          "description": "confirmed / (confirmed + missed). Null until one of the two exists.",
+          "type": [
+            "number",
+            "null"
+          ]
+        },
+        "refuted": {
+          "description": "Tested routes that turned out not to work: false positives.",
+          "type": [
+            "integer",
+            "null"
+          ]
+        },
+        "tested": {
+          "description": "Routes the engine reported that were then tested.",
+          "type": [
+            "integer",
+            "null"
+          ]
+        }
+      },
+      "type": [
+        "object",
+        "null"
+      ]
+    }
+  },
+  "type": "object"
+}
  2. First observed

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover read-only/idempotent/destructive status, so the safety profile is settled. The description adds substantial behavioral context beyond that: the meaning of each verdict, the calibrated-on-average caveat, and the rule that insufficient-data implies expert estimates. It is rich, though it frames much of this as output interpretation rather than tool behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The primary rule (call before quoting a probability) is front-loaded, and most clauses carry genuinely load-bearing interpretation rules. However, the verdict enumeration is packed into one dense sentence with nested parentheticals, and the ranking caveat is restated, which hurts scannability.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be explained, and the description instead supplies the critical interpretation layer: verdict meanings, discrimination/AUC reporting, and explicit warnings about how to present insufficient or chance-level results. For a no-parameter interpretation tool this is complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With zero parameters and 100% schema coverage, there are no parameter semantics to document; the baseline is 4. The description correctly focuses its words on interpretation rather than inventing parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb+resource: it reports how well the engine's probabilities matched reality against red-team/BAS outcomes, and enumerates the verdicts it returns. This is unmistakably distinct from siblings like get_posture or explain_attack_path, which cover different concerns.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives explicit when-to-use ('Call this before quoting any score as a probability') and conditional handling for downstream states ('If it reports insufficient-data, ... must be presented as the model's own estimate'), plus a when-not-to-use rule for the ranking. Nothing is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools