Santismm Knowledge — Harness Engineering, Agentic AI & Governance

Calculate an agent evaluation sample size

calculate_evaluation_sample_size

Read-onlyIdempotent

Calculate two different samples: how many independent evaluations are needed to detect at least one failure, and how many are needed to estimate its rate at a chosen margin. Use this when a user asks how many tests are enough; do not interpret zero observed failures as proof of zero risk.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "failureRate": "<failureRate>",
      "confidence": "<confidence>",
      "margin": "<margin>",
      "population": "<population>"
    },
    "name": "calculate_evaluation_sample_size"
  }
}

Input Schema

Name	Required	Description
`locale`	No	Language for interpretations, assumptions, formulas and warnings. The Labs execution service currently resolves fr/de/ja/zh to English and reports that fallback.
`margin`	Yes	Margin for estimating the failure rate, in percentage points.
`confidence`	Yes	Confidence level, in percent.
`population`	Yes	Number of distinct evaluable cases.
`failureRate`	Yes	Failure rate to detect, in percent.

Output Schema

Name	Required	Description
`slug`	Yes
`units`	Yes
`inputs`	Yes
`source`	Yes
`api_url`	Yes
`license`	Yes
`results`	Yes
`updated`	Yes
`version`	Yes
`fallback`	Yes
`formulas`	Yes
`language`	Yes
`warnings`	Yes
`assumptions`	Yes
`canonical_url`	Yes	Cite this URL.
`interpretation`	Yes
`schema_version`	Yes
`methodology_url`	Yes
`resolved_locale`	Yes
`requested_locale`	Yes

Schema Changelog

Changes observed during successful MCP inspections.

First observedSep 6, 2026

TDQS

A4.3/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish readOnly, openWorld, idempotent, and destructiveHint=false, so the description does not need to cover mutation safety. It adds value beyond annotations by warning against interpreting zero observed failures as proof of zero risk and clarifying that the tool returns two separate sample-size calculations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences with no filler: the first states what the tool computes, and the second gives the use trigger plus a critical statistical warning. It is front-loaded with the most decision-relevant information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the rich input schema, clear annotations, and the presence of an output schema, the description covers what the tool does, when to invoke it, and the most important interpretation pitfall. Nothing essential is missing for correct selection and invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, and every parameter has its own description with constraints or enums, so the schema carries the semantic weight. The description references failure rate, margin, and the estimate context at a high level but does not need to duplicate the schema's parameter details.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb and resource ('Calculate two different samples' for agent evaluation sample size) and distinguishes two concrete outputs: detecting at least one failure versus estimating the failure rate at a chosen margin. This level of specificity separates it from the sibling calculate tool (calculate_agent_economics) even without naming it.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly states when to use the tool: 'when a user asks how many tests are enough.' It also supplies a key interpretive caveat about zero observed failures. It does not formally state when not to use it or name an alternative sibling, but the trigger condition is clear and actionable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All Santismm Knowledge — Harness Engineering, Agentic AI & Governance tools Inspect

Try in Browser

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.