Skip to main content
Glama
anish632

ground-truth-mcp

by anish632
OverviewSchemaSourceRelated ServersDiscussionsAdmin
TypeScript
Remote

Multi-step Hypothesis Test

test_hypothesis
Read-onlyIdempotent

Execute a verification plan of live checks to determine if a hypothesis is supported, partially supported, or refuted.

Instructions

Run a small verification plan made of concrete live checks and summarize whether a hypothesis is supported. Use this when one conclusion depends on multiple simple checks such as endpoint reachability, npm search counts, or whether a page contains an exact substring. This is a coordination tool, not an open-ended research agent: every test must be explicitly defined in advance, and tests run in order with no branching or early exit. The final verdict is mechanical: all tests passing => SUPPORTED, zero passing => REFUTED, otherwise PARTIALLY SUPPORTED. Use verify_claim when you already have evidence URLs, estimate_market for category sizing, and compare_competitors when you already know exact package names.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
hypothesisYesClaim to test, for example 'there are fewer than 50 MCP email servers on npm'.
testsYesOrdered list of one to ten checks to run. Each test object uses only the fields required by its type.

Output Schema

TableJSON Schema
NameRequiredDescriptionDefault
hypothesisYesHypothesis that was evaluated.
testsYesPer-test execution results in input order.
verdictYesHigh-level verdict for the hypothesis.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv0.5.1
    • changedInput schema / properties / tests / items / anyOf
      Previous value: -[
-  {
-    "additionalProperties": false,
-    "properties": {
-      "description": {
-        "description": "Short explanation of what this endpoint check is meant to prove.",
-        "minLength": 3,
-        "type": "string"
-      },
-      "type": {
-        "const": "endpoint_exists",
-        "description": "Perform one unauthenticated GET request and pass when the endpoint returns a 2xx HTTP status.",
-        "type": "string"
-      },
-      "url": {
-        "description": "Public URL to probe, for example https://api.github.com.",
-        "format": "uri",
-        "type": "string"
-      }
-    },
-    "required": [
-      "description",
-      "type",
-      "url"
-    ],
-    "type": "object"
-  },
-  {
-    "additionalProperties": false,
-    "properties": {
-      "description": {
-        "description": "Short explanation of what this npm lower-bound count check is meant to prove.",
-        "minLength": 3,
-        "type": "string"
-      },
-      "query": {
-        "description": "npm search phrase to count, for example 'mcp email server'.",
-        "minLength": 2,
-        "type": "string"
-      },
-      "threshold": {
-        "description": "Lower bound that the npm search result count must exceed.",
-        "minimum": 0,
-        "type": "integer"
-      },
-      "type": {
-        "const": "npm_count_above",
-        "description": "Search npm and pass when the reported result count is strictly greater than the threshold.",
-        "type": "string"
-      }
-    },
-    "required": [
-      "description",
-      "type",
-      "query",
-      "threshold"
-    ],
-    "type": "object"
-  },
-  {
-    "additionalProperties": false,
-    "properties": {
-      "description": {
-        "description": "Short explanation of what this npm upper-bound count check is meant to prove.",
-        "minLength": 3,
-        "type": "string"
-      },
-      "query": {
-        "description": "npm search phrase to count, for example 'business verification mcp'.",
-        "minLength": 2,
-        "type": "string"
-      },
-      "threshold": {
-        "description": "Upper bound that the npm search result count must stay below.",
-        "minimum": 0,
-        "type": "integer"
-      },
-      "type": {
-        "const": "npm_count_below",
-        "description": "Search npm and pass when the reported result count is strictly less than the threshold.",
-        "type": "string"
-      }
-    },
-    "required": [
-      "description",
-      "type",
-      "query",
-      "threshold"
-    ],
-    "type": "object"
-  },
-  {
-    "additionalProperties": false,
-    "properties": {
-      "description": {
-        "description": "Short explanation of what this response-content check is meant to prove.",
-        "minLength": 3,
-        "type": "string"
-      },
-      "substring": {
-        "description": "Exact case-sensitive text to search for in the fetched response body.",
-        "minLength": 1,
-        "type": "string"
-      },
-      "type": {
-        "const": "response_contains",
-        "description": "Fetch a public URL and pass when the response body contains the exact substring using case-sensitive matching. The tool does not parse DOM structure or execute JavaScript before matching.",
-        "type": "string"
-      },
-      "url": {
-        "description": "Public URL whose response body should contain the expected text.",
-        "format": "uri",
-        "type": "string"
-      }
-    },
-    "required": [
-      "description",
-      "type",
-      "url",
-      "substring"
-    ],
-    "type": "object"
-  }
-]New value: +[
+  {
+    "additionalProperties": false,
+    "properties": {
+      "description": {
+        "description": "Short explanation of what this endpoint check is meant to prove.",
+        "minLength": 3,
+        "type": "string"
+      },
+      "type": {
+        "const": "endpoint_exists",
+        "description": "Perform one unauthenticated GET request and pass when the endpoint returns a 2xx HTTP status.",
+        "type": "string"
+      },
+      "url": {
+        "description": "Public URL to probe, for example https://example.com.",
+        "format": "uri",
+        "type": "string"
+      }
+    },
+    "required": [
+      "description",
+      "type",
+      "url"
+    ],
+    "type": "object"
+  },
+  {
+    "additionalProperties": false,
+    "properties": {
+      "description": {
+        "description": "Short explanation of what this npm lower-bound count check is meant to prove.",
+        "minLength": 3,
+        "type": "string"
+      },
+      "query": {
+        "description": "npm search phrase to count, for example 'mcp email server'.",
+        "minLength": 2,
+        "type": "string"
+      },
+      "threshold": {
+        "description": "Lower bound that the npm search result count must exceed.",
+        "minimum": 0,
+        "type": "integer"
+      },
+      "type": {
+        "const": "npm_count_above",
+        "description": "Search npm and pass when the reported result count is strictly greater than the threshold.",
+        "type": "string"
+      }
+    },
+    "required": [
+      "description",
+      "type",
+      "query",
+      "threshold"
+    ],
+    "type": "object"
+  },
+  {
+    "additionalProperties": false,
+    "properties": {
+      "description": {
+        "description": "Short explanation of what this npm upper-bound count check is meant to prove.",
+        "minLength": 3,
+        "type": "string"
+      },
+      "query": {
+        "description": "npm search phrase to count, for example 'business verification mcp'.",
+        "minLength": 2,
+        "type": "string"
+      },
+      "threshold": {
+        "description": "Upper bound that the npm search result count must stay below.",
+        "minimum": 0,
+        "type": "integer"
+      },
+      "type": {
+        "const": "npm_count_below",
+        "description": "Search npm and pass when the reported result count is strictly less than the threshold.",
+        "type": "string"
+      }
+    },
+    "required": [
+      "description",
+      "type",
+      "query",
+      "threshold"
+    ],
+    "type": "object"
+  },
+  {
+    "additionalProperties": false,
+    "properties": {
+      "description": {
+        "description": "Short explanation of what this response-content check is meant to prove.",
+        "minLength": 3,
+        "type": "string"
+      },
+      "substring": {
+        "description": "Exact case-sensitive text to search for in the fetched response body.",
+        "minLength": 1,
+        "type": "string"
+      },
+      "type": {
+        "const": "response_contains",
+        "description": "Fetch a public URL and pass when the response body contains the exact substring using case-sensitive matching. The tool does not parse DOM structure or execute JavaScript before matching.",
+        "type": "string"
+      },
+      "url": {
+        "description": "Public URL whose response body should contain the expected text.",
+        "format": "uri",
+        "type": "string"
+      }
+    },
+    "required": [
+      "description",
+      "type",
+      "url",
+      "substring"
+    ],
+    "type": "object"
+  }
+]
  2. Changed3 schema fields changedv0.4.0
    • changedInput schema / properties / tests / items / anyOf
      Previous value: -[
-  {
-    "additionalProperties": false,
-    "properties": {
-      "description": {
-        "description": "Short explanation of what this endpoint check is meant to prove.",
-        "minLength": 3,
-        "type": "string"
-      },
-      "type": {
-        "const": "endpoint_exists",
-        "description": "Perform one unauthenticated GET request and pass when the endpoint returns a 2xx HTTP status.",
-        "type": "string"
-      },
-      "url": {
-        "description": "Public URL to probe, for example https://api.github.com.",
-        "format": "uri",
-        "type": "string"
-      }
-    },
-    "required": [
-      "description",
-      "type",
-      "url"
-    ],
-    "type": "object"
-  },
-  {
-    "additionalProperties": false,
-    "properties": {
-      "description": {
-        "description": "Short explanation of what this npm lower-bound count check is meant to prove.",
-        "minLength": 3,
-        "type": "string"
-      },
-      "query": {
-        "description": "npm search phrase to count, for example 'mcp email server'.",
-        "minLength": 2,
-        "type": "string"
-      },
-      "threshold": {
-        "description": "Lower bound that the npm search result count must exceed.",
-        "minimum": 0,
-        "type": "integer"
-      },
-      "type": {
-        "const": "npm_count_above",
-        "description": "Search npm and pass when the reported result count is strictly greater than the threshold.",
-        "type": "string"
-      }
-    },
-    "required": [
-      "description",
-      "type",
-      "query",
-      "threshold"
-    ],
-    "type": "object"
-  },
-  {
-    "additionalProperties": false,
-    "properties": {
-      "description": {
-        "description": "Short explanation of what this npm upper-bound count check is meant to prove.",
-        "minLength": 3,
-        "type": "string"
-      },
-      "query": {
-        "description": "npm search phrase to count, for example 'business verification mcp'.",
-        "minLength": 2,
-        "type": "string"
-      },
-      "threshold": {
-        "description": "Upper bound that the npm search result count must stay below.",
-        "minimum": 0,
-        "type": "integer"
-      },
-      "type": {
-        "const": "npm_count_below",
-        "description": "Search npm and pass when the reported result count is strictly less than the threshold.",
-        "type": "string"
-      }
-    },
-    "required": [
-      "description",
-      "type",
-      "query",
-      "threshold"
-    ],
-    "type": "object"
-  },
-  {
-    "additionalProperties": false,
-    "properties": {
-      "description": {
-        "description": "Short explanation of what this response-content check is meant to prove.",
-        "minLength": 3,
-        "type": "string"
-      },
-      "substring": {
-        "description": "Exact case-sensitive text to search for in the fetched response body.",
-        "minLength": 1,
-        "type": "string"
-      },
-      "type": {
-        "const": "response_contains",
-        "description": "Fetch a public URL and pass when the response body contains the exact substring using case-sensitive matching.",
-        "type": "string"
-      },
-      "url": {
-        "description": "Public URL whose response body should contain the expected text.",
-        "format": "uri",
-        "type": "string"
-      }
-    },
-    "required": [
-      "description",
-      "type",
-      "url",
-      "substring"
-    ],
-    "type": "object"
-  }
-]New value: +[
+  {
+    "additionalProperties": false,
+    "properties": {
+      "description": {
+        "description": "Short explanation of what this endpoint check is meant to prove.",
+        "minLength": 3,
+        "type": "string"
+      },
+      "type": {
+        "const": "endpoint_exists",
+        "description": "Perform one unauthenticated GET request and pass when the endpoint returns a 2xx HTTP status.",
+        "type": "string"
+      },
+      "url": {
+        "description": "Public URL to probe, for example https://api.github.com.",
+        "format": "uri",
+        "type": "string"
+      }
+    },
+    "required": [
+      "description",
+      "type",
+      "url"
+    ],
+    "type": "object"
+  },
+  {
+    "additionalProperties": false,
+    "properties": {
+      "description": {
+        "description": "Short explanation of what this npm lower-bound count check is meant to prove.",
+        "minLength": 3,
+        "type": "string"
+      },
+      "query": {
+        "description": "npm search phrase to count, for example 'mcp email server'.",
+        "minLength": 2,
+        "type": "string"
+      },
+      "threshold": {
+        "description": "Lower bound that the npm search result count must exceed.",
+        "minimum": 0,
+        "type": "integer"
+      },
+      "type": {
+        "const": "npm_count_above",
+        "description": "Search npm and pass when the reported result count is strictly greater than the threshold.",
+        "type": "string"
+      }
+    },
+    "required": [
+      "description",
+      "type",
+      "query",
+      "threshold"
+    ],
+    "type": "object"
+  },
+  {
+    "additionalProperties": false,
+    "properties": {
+      "description": {
+        "description": "Short explanation of what this npm upper-bound count check is meant to prove.",
+        "minLength": 3,
+        "type": "string"
+      },
+      "query": {
+        "description": "npm search phrase to count, for example 'business verification mcp'.",
+        "minLength": 2,
+        "type": "string"
+      },
+      "threshold": {
+        "description": "Upper bound that the npm search result count must stay below.",
+        "minimum": 0,
+        "type": "integer"
+      },
+      "type": {
+        "const": "npm_count_below",
+        "description": "Search npm and pass when the reported result count is strictly less than the threshold.",
+        "type": "string"
+      }
+    },
+    "required": [
+      "description",
+      "type",
+      "query",
+      "threshold"
+    ],
+    "type": "object"
+  },
+  {
+    "additionalProperties": false,
+    "properties": {
+      "description": {
+        "description": "Short explanation of what this response-content check is meant to prove.",
+        "minLength": 3,
+        "type": "string"
+      },
+      "substring": {
+        "description": "Exact case-sensitive text to search for in the fetched response body.",
+        "minLength": 1,
+        "type": "string"
+      },
+      "type": {
+        "const": "response_contains",
+        "description": "Fetch a public URL and pass when the response body contains the exact substring using case-sensitive matching. The tool does not parse DOM structure or execute JavaScript before matching.",
+        "type": "string"
+      },
+      "url": {
+        "description": "Public URL whose response body should contain the expected text.",
+        "format": "uri",
+        "type": "string"
+      }
+    },
+    "required": [
+      "description",
+      "type",
+      "url",
+      "substring"
+    ],
+    "type": "object"
+  }
+]
    • changedOutput schema / properties / tests / items / properties / actual / description
      Previous value: -"Observed value or diagnostic string that explains the result."New value: +"Observed value or diagnostic string that explains the result. The format varies by test type and is meant for human interpretation, not strict machine parsing."
    • changedOutput schema / properties / verdict / properties / summary / description
      Previous value: -"Aggregate verdict across the full test plan."New value: +"Aggregate verdict across the full test plan: all pass => SUPPORTED, none pass => REFUTED, otherwise PARTIALLY SUPPORTED."
  3. Changed11 schema fields changedv0.1.1
    • changedInput schema / properties / hypothesis / description
      Previous value: -"The claim to test"New value: +"Claim to test, for example 'there are fewer than 50 MCP email servers on npm'."
    • addedInput schema / properties / hypothesis / minLength
      Added value: +5
    • addedInput schema / properties / tests / description
      Added value: +"Ordered list of one to ten checks to run. Each test object uses only the fields required by its type."
    • removedInput schema / properties / tests / items / additionalProperties
      Removed value: -false
    • addedInput schema / properties / tests / items / anyOf
      Added value: +[
+  {
+    "additionalProperties": false,
+    "properties": {
+      "description": {
+        "description": "Short explanation of what this endpoint check is meant to prove.",
+        "minLength": 3,
+        "type": "string"
+      },
+      "type": {
+        "const": "endpoint_exists",
+        "description": "Perform one unauthenticated GET request and pass when the endpoint returns a 2xx HTTP status.",
+        "type": "string"
+      },
+      "url": {
+        "description": "Public URL to probe, for example https://api.github.com.",
+        "format": "uri",
+        "type": "string"
+      }
+    },
+    "required": [
+      "description",
+      "type",
+      "url"
+    ],
+    "type": "object"
+  },
+  {
+    "additionalProperties": false,
+    "properties": {
+      "description": {
+        "description": "Short explanation of what this npm lower-bound count check is meant to prove.",
+        "minLength": 3,
+        "type": "string"
+      },
+      "query": {
+        "description": "npm search phrase to count, for example 'mcp email server'.",
+        "minLength": 2,
+        "type": "string"
+      },
+      "threshold": {
+        "description": "Lower bound that the npm search result count must exceed.",
+        "minimum": 0,
+        "type": "integer"
+      },
+      "type": {
+        "const": "npm_count_above",
+        "description": "Search npm and pass when the reported result count is strictly greater than the threshold.",
+        "type": "string"
+      }
+    },
+    "required": [
+      "description",
+      "type",
+      "query",
+      "threshold"
+    ],
+    "type": "object"
+  },
+  {
+    "additionalProperties": false,
+    "properties": {
+      "description": {
+        "description": "Short explanation of what this npm upper-bound count check is meant to prove.",
+        "minLength": 3,
+        "type": "string"
+      },
+      "query": {
+        "description": "npm search phrase to count, for example 'business verification mcp'.",
+        "minLength": 2,
+        "type": "string"
+      },
+      "threshold": {
+        "description": "Upper bound that the npm search result count must stay below.",
+        "minimum": 0,
+        "type": "integer"
+      },
+      "type": {
+        "const": "npm_count_below",
+        "description": "Search npm and pass when the reported result count is strictly less than the threshold.",
+        "type": "string"
+      }
+    },
+    "required": [
+      "description",
+      "type",
+      "query",
+      "threshold"
+    ],
+    "type": "object"
+  },
+  {
+    "additionalProperties": false,
+    "properties": {
+      "description": {
+        "description": "Short explanation of what this response-content check is meant to prove.",
+        "minLength": 3,
+        "type": "string"
+      },
+      "substring": {
+        "description": "Exact case-sensitive text to search for in the fetched response body.",
+        "minLength": 1,
+        "type": "string"
+      },
+      "type": {
+        "const": "response_contains",
+        "description": "Fetch a public URL and pass when the response body contains the exact substring using case-sensitive matching.",
+        "type": "string"
+      },
+      "url": {
+        "description": "Public URL whose response body should contain the expected text.",
+        "format": "uri",
+        "type": "string"
+      }
+    },
+    "required": [
+      "description",
+      "type",
+      "url",
+      "substring"
+    ],
+    "type": "object"
+  }
+]
    • removedInput schema / properties / tests / items / properties
      Removed value: -{
-  "description": {
-    "type": "string"
-  },
-  "query": {
-    "type": "string"
-  },
-  "substring": {
-    "type": "string"
-  },
-  "threshold": {
-    "type": "number"
-  },
-  "type": {
-    "enum": [
-      "endpoint_exists",
-      "npm_count_above",
-      "npm_count_below",
-      "response_contains"
-    ],
-    "type": "string"
-  },
-  "url": {
-    "type": "string"
-  }
-}
    • removedInput schema / properties / tests / items / required
      Removed value: -[
-  "description",
-  "type"
-]
    • removedInput schema / properties / tests / items / type
      Removed value: -"object"
    • addedInput schema / properties / tests / maxItems
      Added value: +10
    • addedInput schema / properties / tests / minItems
      Added value: +1
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "additionalProperties": false,
+  "properties": {
+    "hypothesis": {
+      "description": "Hypothesis that was evaluated.",
+      "type": "string"
+    },
+    "tests": {
+      "description": "Per-test execution results in input order.",
+      "items": {
+        "additionalProperties": false,
+        "properties": {
+          "actual": {
+            "description": "Observed value or diagnostic string that explains the result.",
+            "type": [
+              "string",
+              "number",
+              "null"
+            ]
+          },
+          "description": {
+            "description": "Human-readable explanation of the check.",
+            "type": "string"
+          },
+          "passed": {
+            "description": "True when the test condition was satisfied.",
+            "type": "boolean"
+          },
+          "type": {
+            "description": "Test type that was executed.",
+            "enum": [
+              "endpoint_exists",
+              "npm_count_above",
+              "npm_count_below",
+              "response_contains"
+            ],
+            "type": "string"
+          }
+        },
+        "required": [
+          "description",
+          "type",
+          "passed",
+          "actual"
+        ],
+        "type": "object"
+      },
+      "type": "array"
+    },
+    "verdict": {
+      "additionalProperties": false,
+      "description": "High-level verdict for the hypothesis.",
+      "properties": {
+        "failed": {
+          "description": "Number of tests that failed.",
+          "minimum": 0,
+          "type": "integer"
+        },
+        "passed": {
+          "description": "Number of tests that passed.",
+          "minimum": 0,
+          "type": "integer"
+        },
+        "summary": {
+          "description": "Aggregate verdict across the full test plan.",
+          "enum": [
+            "SUPPORTED",
+            "REFUTED",
+            "PARTIALLY SUPPORTED"
+          ],
+          "type": "string"
+        }
+      },
+      "required": [
+        "passed",
+        "failed",
+        "summary"
+      ],
+      "type": "object"
+    }
+  },
+  "required": [
+    "hypothesis",
+    "tests",
+    "verdict"
+  ],
+  "type": "object"
+}
  4. First observedv0.1.0

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, openWorldHint, idempotentHint, and destructiveHint false, so the safety profile is covered. The description adds substantial behavioral context beyond annotations: tests run in order with no branching or early exit, the verdict mapping is fully mechanical (all passing => SUPPORTED, zero passing => REFUTED, otherwise PARTIALLY SUPPORTED), and the tool is not an open-ended research agent. This goes well beyond what annotations provide.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and well-organized: first sentence defines the action, second covers typical use cases, third imposes constraints, fourth explains the verdict algorithm, fifth names alternatives. Every sentence earns its place, and there is no redundant or filler content. It is appropriately sized for a tool of this complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate-to-high complexity (multiple test types, ordered execution, verdict logic), the description covers all key aspects: what it does, when to use it, behavioral constraints, verdict determination, and sibling alternatives. An output schema exists for return values, and the description already explains the verdict mapping, so nothing critical is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description adds some context by listing example check types (endpoint reachability, npm search counts, substring presence) and emphasizing that tests must be explicitly defined, but this largely restates information already in the schema. The hypothesis parameter is self-explanatory. No additional syntax or format details beyond the schema are provided, so a 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool 'Run a small verification plan made of concrete live checks and summarize whether a hypothesis is supported' with a specific verb and resource. It also distinguishes itself from siblings by explicitly naming alternatives (verify_claim, estimate_market, compare_competitors) and framing itself as a coordination tool, not an open-ended research agent.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit when-to-use guidance ('Use this when one conclusion depends on multiple simple checks...') and when-not-to-use guidance with named alternatives ('Use verify_claim when you already have evidence URLs...'). It also restricts usage by stating 'every test must be explicitly defined in advance' and 'no branching or early exit', which clarifies the boundary versus more flexible research tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools