Skip to main content
Glama
iris-eval

iris-eval/mcp-server

by iris-eval

Iris — stop shipping agents on vibes

Glama Score Install in Cursor npm version npm downloads GitHub stars CI OpenSSF Scorecard OpenSSF Best Practices License: MIT Docker PulseMCP mcp.so

Iris scores every agent run for quality, safety, and cost — on your machine, with no SDK and no account. Most agent projects check quality by running a few remembered prompts and eyeballing the output. Iris replaces that with numbers you can audit: your agent's runs land in a SQLite database on your disk, 13 built-in rules score them deterministically — PII, prompt injection, hallucination markers, cost thresholds — free, with no LLM calls, and an optional LLM judge with a hard per-eval cost cap handles the semantic questions. Every rule is inspectable and editable, because a judge you can't audit is just vibes with a number on it. MIT licensed, no telemetry; your traces never leave your machine.

Requires Node.js 20 or later. Check with node --version.

Iris Dashboard

A failure on screen in 60 seconds

No agent wiring, no config — one command:

npx @iris-eval/mcp-server --demo

This seeds a demo database — a handful of small agents with a week of runs — and serves the dashboard against it at http://localhost:6920 (your browser opens automatically on first run). The dashboard lands on Failures: what failed, worst and newest first. Worth clicking into — a PII leak caught by the safety rules, a flagged prompt-injection attempt, and a failed LLM-judge score with its rationale.

Demo data lives in its own database (demo.db in your Iris home directory — ~/.iris on macOS/Linux, %USERPROFILE%\.iris on Windows) and never mixes with your real traces. Remove all of it with one command:

npx @iris-eval/mcp-server --demo-clear

Related MCP server: runmeter

Hook up your own agent

Add Iris to your MCP config. Works with Claude Desktop, Claude Code, Cursor, Windsurf, Continue, VS Code, Cline, Zed, Codex CLI, Gemini CLI — and any other MCP-compatible agent. One block, dashboard included:

{
  "mcpServers": {
    "iris-eval": {
      "command": "npx",
      "args": ["@iris-eval/mcp-server", "--dashboard"]
    }
  }
}

Your agent discovers Iris's nine tools on connect, and the dashboard serves at http://localhost:6920. Now paste this to your agent:

Log that last task to Iris and evaluate the output.

The trace lands on the dashboard with its scores. Prefer the MCP server headless? Drop --dashboard from the args — you can open the same dashboard any time with npx @iris-eval/mcp-server --dashboard.

One thing worth knowing up front: MCP tools are called when the model decides to call them. Iris doesn't intercept your agent, so traces are logged when your agent asks it to log them — either because you told it to, or because your code calls the tools directly. Ask your agent to "log this to Iris and evaluate it" and it will. If you want capture that doesn't depend on the model choosing, POST /api/v1/traces does exactly that — your code sends the trace over plain HTTP, no model in the loop (see docs/http-ingest.md). The CLI and SDKs on the roadmap will be thin clients over the same endpoint.

Capture over HTTP (no model in the loop)

With the dashboard running, anything that can send an HTTP request can log a trace — and optionally run the deterministic evals in the same request:

curl -s -X POST "http://127.0.0.1:6920/api/v1/traces" \
  -H "Content-Type: application/json" \
  -d '{
    "agent_name": "support-bot",
    "input": "What is the refund policy?",
    "output": "Refunds are available within 30 days of purchase.",
    "evaluate": true,
    "eval_type": "safety"
  }'

Returns 201 with the stored trace_id and the evaluation result. The endpoint accepts the same body as the log_trace tool and sits behind the same loopback-only middleware stack as the rest of the dashboard. Full contract, field reference, and error semantics: docs/http-ingest.md.

Check the install

npx @iris-eval/mcp-server --self-test

An offline install diagnostic: storage round-trip, deterministic evals, dashboard + DNS-rebinding guard — all inside an isolated temp home, so your real database is never opened. Exit code 0 = healthy, 1 = a check failed.

Claude Desktop

Edit your MCP config file:

  • macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

  • Windows: %APPDATA%\Claude\claude_desktop_config.json

Add the JSON config above, then restart Claude Desktop.

Claude Code

claude mcp add --transport stdio iris-eval -- npx @iris-eval/mcp-server

Then restart the session (/clear or relaunch) for tools to load.

Windows note: Do not use cmd /c wrapper — it causes path parsing issues. The npx command works directly.

Cursor / Windsurf

Add to your workspace .cursor/mcp.json or global MCP settings using the JSON config above.

VS Code (native MCP)

Add to .vscode/mcp.json in your workspace (note: VS Code uses servers, not mcpServers):

{
  "servers": {
    "iris-eval": {
      "command": "npx",
      "args": ["@iris-eval/mcp-server"]
    }
  }
}

Cline

Open Cline's MCP Servers panel → Configure MCP Servers, and add the mcpServers JSON config above to cline_mcp_settings.json.

Zed

Add to Zed settings.json:

{
  "context_servers": {
    "iris-eval": {
      "command": {
        "path": "npx",
        "args": ["@iris-eval/mcp-server"]
      }
    }
  }
}

OpenAI Codex CLI

Add to ~/.codex/config.toml:

[mcp_servers.iris-eval]
command = "npx"
args = ["@iris-eval/mcp-server"]

Gemini CLI

Add the mcpServers JSON config above to ~/.gemini/settings.json.

Anything else that speaks MCP

Iris is a standard stdio MCP server — one npx @iris-eval/mcp-server command, no SDK, no code changes. If your client supports MCP, it supports Iris. Client config formats change; when in doubt, check your client's MCP docs and point it at that command.

Other Install Methods

# Global install (recommended for persistent data and faster startup)
npm install -g @iris-eval/mcp-server
iris-mcp --dashboard

# Docker — two servers, two ports: 3000 = MCP HTTP transport,
# 6920 = dashboard (which also serves the POST /api/v1/traces ingest endpoint)
docker run -p 3000:3000 -p 6920:6920 -v iris-data:/data ghcr.io/iris-eval/mcp-server

Tip: Global install (npm install -g) stores traces persistently at ~/.iris/iris.db. With npx, traces persist in the same location, but startup is slower due to package resolution.

What You Get

Trace Logging

Hierarchical span trees with per-tool-call latency, token usage, and cost in USD. Stored in SQLite, queryable instantly.

Output Evaluation

13 built-in rules across 4 categories: completeness, relevance, safety, cost. PII detection (19 patterns: SSN, credit card, phone, email, IBAN, DOB, MRN, IP, API key, passport, plus AWS/Slack/SendGrid/GitHub/Google/npm/DigitalOcean tokens, PEM private-key blocks and seed phrases), prompt injection (37 patterns, phrase + structural), stub-output detection, hallucination detection (25 context-grounded fabrication/contradiction signals — pass input to ground them against the agent's source material). Add custom rules with Zod schemas.

LLM-as-Judge

Optional semantic scoring via Anthropic or OpenAI — bring your own API key. Five templates. Hard per-eval cost cap (IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL, default $0.25), per-eval pricing disclosed in the result.

Cost Visibility

Aggregate cost across all agents over any time window. Set budget thresholds. Get flagged when agents overspend.

Web Dashboard

Real-time dark-mode UI that lands on the failures, worst and newest first — trace visualization, eval results, cost breakdowns, and a command palette (⌘K) that searches your own rules, traces, and evals.

Local-first

Everything lives in SQLite on your disk. No account, no sign-up, no telemetry. Outbound HTTP happens only where you opt in: your own LLM-judge key, citation fetching, or an OTel exporter you configure.

Where this is going next: the roadmap.

MCP Tools

Iris registers nine tools that any MCP-compatible agent can invoke — full rule + trace lifecycle + LLM-as-judge + semantic citation verification:

  • log_trace — Log an agent execution with spans, tool calls, token usage, and cost

  • evaluate_output — Score output quality against completeness, relevance, safety, and cost rules (heuristic, deterministic, free)

  • get_traces — Query stored traces with filtering, pagination, and time-range support

  • list_rules — Enumerate deployed custom eval rules (read-only)

  • deploy_rule — Register a new custom eval rule so it fires on every evaluate_output of that category

  • delete_rule — Remove a deployed custom rule (destructive, idempotent)

  • delete_trace — Remove a single stored trace by ID (destructive, tenant-scoped)

  • evaluate_with_llm_judge — Semantic eval via LLM (Anthropic or OpenAI). Five templates: accuracy, helpfulness, safety, correctness, faithfulness. Cost-capped, per-eval pricing disclosed. Bring your own API key (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY) — Iris doesn't proxy or relay LLM calls.

  • verify_citations — Extract citations from output (numbered, author-year, URLs, DOIs), fetch sources behind an SSRF-guarded + domain-allowlisted resolver, and use an LLM judge to check whether each source actually supports the cited claim. Opt-in outbound HTTP. Same BYOK requirement as evaluate_with_llm_judge.

When IRIS_OTEL_ENDPOINT is configured, log_trace calls also emit a best-effort OTLP/HTTP JSON export to any OpenTelemetry collector (Jaeger, Grafana Tempo, Datadog OTLP, Honeycomb, etc). See docs/otel-integration.md.

How passed is decided

evaluate_output returns both a score and a passed flag — they answer different questions:

  • score (0..1) is the weighted average across the rules that ran — a quality gradient.

  • passed is the ship/no-ship verdict: true only when the score clears the pass threshold (default 0.7) and no critical rule failed.

Genuine safety violations hard-fail. no_pii, no_injection_patterns, and no_blocklist_words are critical rules: if one fails, the eval reports passed: false no matter how well the other rules scored, and the response names the culprits in critical_failures. A leaked SSN can't be averaged away. Custom rules deployed with severity: "high" or "critical" hard-fail the same way; low/medium severities only affect the score. One boundary to know: a critical rule that skipped (missing context, or any other cause of a skip) has not judged the output and does not veto — rule_results shows every skip and its reason, so a gate that must fail closed on non-verdicts can.

One gotcha for CI gates: if you omit eval_type, the default completeness bundle runs — safety rules don't. The response echoes eval_type (plus a note when it was defaulted) so your gate can verify which bundle actually ran. Key on passed for the verdict and eval_type: "safety" for coverage.

Full tool schemas and configuration: iris-eval.com

Hosted features

Iris runs entirely on your machine today, and everything it does is free and MIT licensed with no limits and no account.

Hosted storage, shared team history and alerting are under consideration, not under construction. There is no pricing, and nothing to buy. If shared history would be useful to you, the waitlist is how we find out whether it's worth building — it commits you to nothing.

Two commitments hold regardless: nothing that is free today will move behind a paywall, and no compliance certification will be claimed before it is held.

Examples

Community

CLI Arguments

Flag

Default

Description

--transport

stdio

Transport type: stdio or http

--port

3000

HTTP transport port

--db-path

~/.iris/iris.db

SQLite database path

--config

~/.iris/config.json

Config file path

--api-key

API key for HTTP authentication

--dashboard

false

Enable web dashboard

--dashboard-port

6920

Dashboard port

--dashboard-host

127.0.0.1

Dashboard bind address. Loopback by default — the dashboard is unauthenticated unless --api-key is set, so binding beyond loopback exposes your full trace history

--demo

false

Seed a demo database (separate from your real traces) and serve the dashboard against it

--demo-clear

false

Delete the demo database and exit

--self-test

false

Run the offline install diagnostic in an isolated temp home, then exit (0 = healthy, 1 = a check failed)

Environment Variables

Variable

Description

IRIS_TRANSPORT

Transport type (stdio or http)

IRIS_PORT

HTTP transport port

IRIS_HOST

HTTP transport host (default 127.0.0.1)

IRIS_HOME

Directory for all per-user files: config.json, iris.db, custom-rules.json, audit.log, preferences.json (default ~/.iris)

IRIS_DB_PATH

SQLite database path (overrides IRIS_HOME for the DB only)

IRIS_LOG_LEVEL

Log level: debug, info, warn, error

IRIS_DASHBOARD

Enable web dashboard (true/false; false also overrides dashboard.enabled in config.json)

IRIS_DASHBOARD_PORT

Dashboard port (default 6920)

IRIS_DASHBOARD_HOST

Dashboard bind address (default 127.0.0.1)

IRIS_API_KEY

API key for HTTP authentication

IRIS_ALLOWED_ORIGINS

Comma-separated allowed CORS origins

CLI flags take precedence over environment variables when both are set.

Security

When using HTTP transport, Iris includes:

  • API key authentication with timing-safe comparison

  • CORS restricted to localhost by default

  • Rate limiting (600 req/min dashboard API, 20 req/min MCP)

  • Helmet security headers

  • Zod input validation on all routes

  • ReDoS-safe regex for custom eval rules

  • 1MB request body limits

# Production deployment
iris-mcp --transport http --port 3000 --api-key "$(openssl rand -hex 32)" --dashboard

First move: run the self-test

npx @iris-eval/mcp-server --self-test

It checks storage, the deterministic evals, and the dashboard in an isolated temp home and prints a per-step verdict — the failure output names the broken step. Exit code 0 means the install is healthy.

Iris won't start / ERR_MODULE_NOT_FOUND

You may have a cached older version. Clear the npx cache and retry:

npx --yes @iris-eval/mcp-server@latest

Or install globally to avoid cache issues entirely:

npm install -g @iris-eval/mcp-server@latest

Tools not showing up in Claude Code

MCP tools only load at session start. After adding iris-eval, restart the session with /clear or relaunch the terminal.

Version check

Iris logs its version on the first startup line:

npx @iris-eval/mcp-server --dashboard
# First log line: "Starting Iris MCP server vX.Y.Z"

For a global install, npm ls -g @iris-eval/mcp-server shows the installed version.

Updating

# If using npx (clears cache and fetches latest)
npx --yes @iris-eval/mcp-server@latest

# If installed globally
npm update -g @iris-eval/mcp-server

Node.js version

Iris requires Node.js 20 or later. Node 18 reached EOL in April 2025 and is not supported.

node --version  # Must be v20.x or v22.x+

Windows: cmd /c not needed

Claude Code's /doctor may suggest wrapping npx with cmd /c. This is not needed and causes path parsing issues. Use npx directly:

# Correct
claude mcp add --transport stdio iris-eval -- npx @iris-eval/mcp-server

# Wrong (causes /c to be parsed as a path)
claude mcp add --transport stdio iris-eval -- cmd /c "npx @iris-eval/mcp-server"

If Iris is useful to you, consider starring the repo — it helps others find it.

Star on GitHub

MIT Licensed.

Available Tools

9 tools
delete_ruleDelete Custom RuleA
Destructive

Remove a deployed custom evaluation rule. The rule stops firing on future evaluate_output calls; past eval_results that referenced it are preserved.

Sibling tools — deploy_rule adds custom rules, list_rules enumerates them, evaluate_output runs them. delete_trace handles trace deletion (separate concern); log_trace / get_traces handle trace I/O. delete_rule is the DESTRUCTIVE remove path for the custom-rule store; it does NOT touch traces, eval_results, or built-in (non-custom) rules.

Behavior. DESTRUCTIVE — rewrites /.iris/custom-rules.json without the deleted row and appends a rule.delete entry to the audit log (/.iris/audit.log). Not idempotent: deleting an already-deleted rule returns deleted: false rather than re-emitting the audit row. The rule stops firing immediately on the live process. Historical eval_results that reference this rule_id stay in the database — drift analytics + audit trail remain valid. Tenant-scoped in Cloud tier; OSS operates on LOCAL_TENANT. Rate-limited to 20 req/min on HTTP MCP.

Output shape. Returns JSON: { "deleted": boolean, "rule_id": string }. deleted=true if a row was removed; deleted=false if no rule with that id existed.

Use when a custom rule is obsolete (behavior changed, false positives unacceptable, replaced by a better rule). Typical flow: list_rules → identify the stale one → delete_rule(id). Combine with deploy_rule to replace: delete_rule(oldId) + deploy_rule(newDefinition). To temporarily disable a rule WITHOUT deletion, use the dashboard's toggle affordance instead — delete is permanent in intent (rule is gone; re-adding requires a new id).

Don't use to pause a rule (toggle in the dashboard preserves history better). Don't use on built-in (non-custom) rules — the rule_id format checks for rule-<hex> custom ids; built-ins aren't in the store. Don't use to delete a trace or eval result (use delete_trace for traces; eval_results deletion is not exposed in v0.4 — they fall under data retention).

Parameters. rule_id is the only parameter; must match rule-<lowercase-hex> format (Zod regex). Format mismatch fails Zod with 400 BEFORE the store is touched. Cross-tenant rule_ids return deleted: false silently — they're invisible to the caller's tenant rather than producing a not-found error (prevents enumeration attacks). The rule_id you pass is exactly what list_rules returned in id or what deploy_rule returned in rule.id.

Error modes. Throws 400 on malformed rule_id (wrong prefix). Returns {deleted: false} if rule_id doesn't match any deployed rule (not an error — idempotent-ish). Returns 429 on HTTP rate limit. File-write failures propagate as 500.

ParametersJSON Schema
NameRequiredDescriptionDefault
rule_idYesRule id to delete (format: rule-<hex>); obtained from list_rules or deploy_rule response

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already mark destructiveHint=true and idempotentHint=false, but the description goes far beyond by detailing exactly what happens: rewrites ~/.iris/custom-rules.json, appends to audit log, returns deleted:false on already-deleted rules, stops firing immediately, preserves historical eval_results, tenant-scoped behavior, and rate limits. No contradiction with annotations; the description adds substantial behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is appropriately structured with labeled sections (Behavior, Output shape, Use when, Don't use, Parameters, Error modes), making it scannable. Every sentence provides value, and it is front-loaded with the core action. Despite its length, it is concise for the complexity it covers.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers all relevant aspects: behavior, output shape, usage conditions, parameter details, error modes, and relationship to sibling tools. Given the tool's destructive nature and lack of output schema, this level of detail is complete for an agent to invoke it correctly. No gaps are apparent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

While the schema already covers rule_id with its pattern and description, the description enriches parameter semantics by explaining the Zod format mismatch (400 before store touch) and cross-tenant silent false. It also clarifies that the rule_id is exactly what list_rules or deploy_rule returns. This adds meaning beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with 'Remove a deployed custom evaluation rule,' a specific verb+resource that clearly states the action. It further distinguishes itself from siblings by explicitly positioning delete_rule as the destructive remove path for the custom-rule store, noting it does NOT touch traces, eval_results, or built-in rules. This fully disambiguates it from delete_trace and other management tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit when-to-use guidance ('Use when a custom rule is obsolete...') and a typical flow (list_rules → delete_rule). It also gives clear exclusions: don't use to pause (use dashboard toggle), don't use on built-in rules, and don't use for traces/eval_results. This is exemplary usage direction.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

delete_traceDelete TraceA
Destructive

Remove a single trace by id. Cascades to spans; eval_results keep the score history with trace_id NULLed.

Sibling tools — log_trace creates traces, get_traces queries them, evaluate_output / evaluate_with_llm_judge / verify_citations score them. delete_rule handles custom-rule deletion (separate concern); list_rules / deploy_rule manage the custom-rule lifecycle. delete_trace is the DESTRUCTIVE single-row remove for traces; it does NOT touch eval_results (preserved for audit + drift analytics), spans cascade automatically.

Behavior. DESTRUCTIVE — SQL DELETE scoped to the caller's tenant_id. Cascades: spans belonging to this trace are deleted (FK ON DELETE CASCADE); eval_results that referenced this trace have their trace_id set to NULL (FK ON DELETE SET NULL) so aggregate dashboards + historical scores remain valid even after the trace is gone. Not idempotent: deleting an already-deleted trace returns deleted: false. Does not emit an audit log entry in v0.4 — traces are user-scope data, not policy changes. Rate-limited to 20 req/min on HTTP MCP.

Output shape. Returns JSON: { "deleted": boolean, "trace_id": string }. deleted=true if a row was removed; deleted=false if no trace with that id existed (or it belonged to a different tenant — cross-tenant deletes silently fail).

Use when a trace was captured in error, contains sensitive data that must be removed for compliance (e.g., a customer exercises GDPR right-to-erasure), or when cleaning up test data. Combine with get_traces to find candidates: query with filters → review → delete_trace(id) per target. For bulk time-window deletion, use deleteTracesOlderThan via the CLI / retention config — delete_trace is the single-row surgical path.

Don't use to clean up OLD data in bulk (use retention config with --retention-days). Don't use to PAUSE a trace — traces are immutable once stored; there's nothing to pause. Don't use to delete eval_results — eval_results survive their trace's deletion intentionally (for audit + drift analysis); they're pruned only by retention.

Parameters. trace_id is the only parameter; must match 32-char lowercase hex (Zod regex). The trace_id you pass is exactly what log_trace returned in its response, or what get_traces returned per row. Format mismatch fails Zod with 400 BEFORE the storage layer is touched. Cross-tenant trace_ids return deleted: false silently — they're invisible to the caller's tenant (prevents enumeration attacks; matches delete_rule's tenant-isolation contract).

Error modes. Throws 400 on malformed trace_id (wrong format: not 32-char lowercase hex). Returns {deleted: false} when the id doesn't exist in the caller's tenant (not an error — the trace may simply have been deleted already). Returns 429 on HTTP rate limit. Storage failures propagate as 500.

ParametersJSON Schema
NameRequiredDescriptionDefault
trace_idYesTrace id to delete (32-hex lowercase; obtained from log_trace response or get_traces)

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description extensively details behavior beyond the annotations: SQL DELETE scoped to tenant, cascade to spans, SET NULL for eval_results, non-idempotent behavior (`deleted: false`), lack of audit logging, rate limit of 20 req/min, and cross-tenant silent failures. These are valuable, non-obvious traits disclosed clearly. No contradiction with annotations—destructiveHint=true aligns with the described SQL DELETE.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with clear sections, but it suffers from redundancy. For example, the eval_results SET NULL behavior is repeated in the purpose sentence, the behavior section, and the 'Don't use' list. The cross-tenant 'deleted: false' appears in both behavior and parameter sections. While front-loaded and organized, the verbosity exceeds what a concise description needs.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite no output schema, the description fully explains the output shape ('{ "deleted": boolean, "trace_id": string }') and enumerates error modes (400, 429, 500). It covers security (tenant isolation), rate limits, audit behavior, and the nuances of cascade vs. nullify. For a destructive tool with complex side effects, this is complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Although the schema already covers trace_id with a description and regex pattern, the description adds crucial context: the Zod validation fails with 400 before storage, trace_id provenance (from log_trace/get_traces), and cross-tenant behavior returning `deleted: false`. This goes well beyond the schema's raw type and pattern.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with 'Remove a single trace by id' — a specific verb and resource — and immediately distinguishes it from siblings by noting cascading behavior and explicitly contrasting with delete_rule: 'delete_trace is the DESTRUCTIVE single-row remove for traces.' This leaves no ambiguity about what the tool does.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit when-to-use scenarios ('captured in error, contains sensitive data... compliance') and when-not-to-use ('Don't use to clean up OLD data in bulk... use retention config', 'Don't use to PAUSE a trace', 'Don't use to delete eval_results'). Also suggests combining with get_traces for candidate discovery, giving the agent clear decision guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

deploy_ruleDeploy Custom RuleA

Deploy a new custom evaluation rule that will fire on every future evaluate_output call of its eval category.

Sibling tools — list_rules enumerates deployed rules, delete_rule removes them, evaluate_output runs them. log_trace / get_traces / delete_trace handle the trace lifecycle separately; evaluate_with_llm_judge / verify_citations run semantic scoring (not heuristic-rule-driven). deploy_rule is the WRITE path that grows the custom-rule library.

Behavior. Writes a row to /.iris/custom-rules.json (atomic write via temp file + rename) and appends a rule.deploy entry to the audit log (/.iris/audit.log). The rule activates immediately for the running process and persists across restarts. Each call mints a fresh rule_id; not idempotent (deploying twice creates two rules). Tenant-scoped in Cloud tier; OSS rules are owned by LOCAL_TENANT. Rate-limited to 20 req/min on HTTP MCP.

Output shape. Returns JSON: { "rule": { "id": "rule-XXXX", "name", "description", "evalType", "severity", "definition", "enabled": true, "createdAt", "updatedAt", "version": 1, "sourceMomentId?" } }. The returned rule is the canonical persisted form; save the id if you plan to update or delete later.

Use when an agent observes a recurring failure pattern and decides to enforce it as a standing rule. The sourceMomentId field preserves provenance — downstream audit can trace the rule back to the moment that inspired it. Combine with evaluate_output + get_traces: 1) evaluate_output surfaces failures; 2) get_traces filters to the failure set; 3) analyze the pattern; 4) deploy_rule bakes it into the default eval path.

Don't use to VALIDATE a rule before committing — deploy writes immediately. Use the dashboard's preview endpoint (POST /api/v1/rules/custom/preview) for dry-run validation against sample output. Don't use to EDIT an existing rule — this call only creates; edits require a dedicated flow (coming in v0.5). To update a rule today: delete_rule then deploy_rule with the new definition.

Parameters. name is 1-120 chars (Zod-enforced min/max); appears in eval_result rule_results so make it human-readable. description is optional, max 500 chars (used in dashboard tooltips). evalType determines WHEN the rule fires (must match the eval_type your evaluate_output calls use; e.g., a "completeness" rule fires on every evaluate_output where eval_type="completeness" OR eval_type="custom"). severity affects dashboard sort + audit log signal but does NOT affect scoring (scoring uses the rule's weight). definition.type and definition.config must match (e.g., regex_match needs config.pattern; cost_threshold needs config.max_cost; min_length needs config.min_length; max_length needs config.max_length; contains_keywords/excludes_keywords need config.keywords). Invalid configs are now REJECTED at deploy time with the offending field named, instead of deploying and then failing every evaluation. sourceMomentId is optional but recommended (preserves workflow-inversion provenance from Make-This-A-Rule composer). Defaults: severity="medium".

Error modes. Throws 400 on invalid definition (Zod rejects — e.g., regex that fails safe-regex2 ReDoS check, or length > 1000 chars). Throws 400 on empty name. Throws 400 if the eval category mismatches the definition type. Returns 429 when HTTP rate limit exceeded. File-write failures (disk full, read-only fs) propagate as 500; the audit log is best-effort and does not block deploy.

ParametersJSON Schema
NameRequiredDescriptionDefault
nameYesHuman-readable rule name (used in eval results)
evalTypeYesEval category this rule belongs to; determines when it fires
severityNoSeverity used for dashboard sort + audit alertsmedium
definitionYesCheck definition (regex, length, keyword, cost, or schema)
descriptionNoWhat this rule checks for and why it matters
sourceMomentIdNoOptional Decision Moment ID the rule was derived from (preserves workflow-inversion provenance)

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only declare readOnlyHint=false, idempotentHint=false, destructiveHint=false. The description goes far beyond by disclosing that the tool writes to ~/.iris/custom-rules.json via atomic write, appends to audit log, activates immediately, persists across restarts, is not idempotent (mints fresh rule_id), is tenant-scoped, rate-limited to 20 req/min, and surfaces specific error modes (400, 429, 500). No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but every sentence earns its place. It is front-loaded with purpose, then differentiates siblings, explains behavior, output shape, usage guidance, parameter semantics, and error modes. The structure is logical and scannable, with no repetition or filler. For a tool with 6 parameters and nested objects, this complexity warrants the length.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description is fully complete for a write-path tool with no output schema. It covers purpose, exact file write behavior, non-idempotency, tenant scoping, rate limits, output JSON shape, parameter semantics, when to use/avoid, and all error modes. Even without an output schema, the agent knows exactly what to expect and how to handle failures.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Although schema coverage is 100%, the description adds substantial meaning: name length constraints and its appearance in eval_result rule_results, evalType must match evaluate_output eval_type with concrete examples, severity affects sorting/audit but not scoring, definition type/config pairing examples (regex_match needs config.pattern, cost_threshold needs config.max_cost, etc.), and invalid configs rejected at deploy time. This is far beyond the baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource+scope: 'Deploy a new custom evaluation rule that will fire on every future evaluate_output call of its eval category.' It explicitly distinguishes from siblings by naming them (list_rules, delete_rule, evaluate_output, log_trace, get_traces, delete_trace, evaluate_with_llm_judge, verify_citations) and states deploy_rule is the 'WRITE path that grows the custom-rule library.' This is unambiguous and clearly differentiated.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit when-to-use guidance: 'Use when an agent observes a recurring failure pattern and decides to enforce it as a standing rule.' It also gives explicit when-not-to-use guidance: don't use to validate (use the preview endpoint) and don't use to edit (use delete_rule + deploy_rule). It further maps a concrete workflow combining evaluate_output + get_traces + deploy_rule. Alternatives are explicitly named and contextually placed.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluate_outputEvaluate OutputA
Idempotent

Score agent output against configurable eval rules and return a 0..1 score + per-rule breakdown.

Sibling tools — evaluate_with_llm_judge runs semantic LLM-based scoring (slower, costs money; this tool is heuristic, free, deterministic), verify_citations checks citation grounding specifically, log_trace records executions, get_traces queries them, list_rules / deploy_rule / delete_rule manage the custom-rule lifecycle. evaluate_output is the FAST PATH for length / keyword / PII / injection / cost-threshold checks where rules are sufficient.

Behavior. Deterministic, in-process scoring — same inputs always produce the same result. Writes one eval_result row to Iris storage (linked to trace_id if provided; unlinked otherwise). No external network calls in heuristic mode (v0.4 adds an llm_as_judge eval_type that DOES call LLM APIs; see the separate evaluate_with_llm_judge tool for that). Rate-limited to 20 req/min on HTTP MCP, unlimited on stdio. Runs in ~5-50ms for rule-based evaluation.

Output shape. Returns JSON: { "id": "<uuid>", "score": 0..1, "passed": boolean, "rule_results": [{ "ruleName", "passed", "score", "message", "skipped?" }], "suggestions": string[], "rules_evaluated": number, "rules_skipped": number, "insufficient_data": boolean }. insufficient_data=true means no applicable rules fired (e.g., safety eval with only cost data).

Use when you want a quality score on a specific output — typically after log_trace records the execution. Pass eval_type to route to the right rule bundle: completeness (length, sentence count, relevance to input), relevance (keyword overlap, topic consistency), safety (PII leak, prompt injection, hallucination markers, stub-output detection), cost (budget threshold), or custom (bring your own rules via custom_rules).

Don't use when the output is empty or has no applicable rules — the eval_type decides which rules apply, and invalid combinations return score=0 + insufficient_data=true (not an error, but not actionable). Don't use to VALIDATE JSON schemas directly (use your language's JSON Schema validator — Iris's json_schema custom rule type is for output-shape assertions, not arbitrary validation).

Parameters. expected is REQUIRED when eval_type="relevance" (used as the comparison target for keyword overlap + topic consistency); ignored for other eval_types. cost_usd + token_usage are ONLY consulted when eval_type="cost" (ignored otherwise). custom_rules ALWAYS fires regardless of eval_type — pass eval_type="custom" if you want ONLY your rules to run (otherwise both your rules AND the eval_type bundle run together). trace_id is optional but recommended (linking the eval to its trace surfaces it in the dashboard's drill-through). input adds context to keyword-overlap relevance checks; ignored otherwise. Defaults: eval_type="completeness".

Error modes. Throws on malformed custom_rules (Zod rejects). Returns 400 on regex patterns that fail safe-regex2 ReDoS check or exceed 1000-char limit. Returns 429 when HTTP rate limit exceeded. Storage failures propagate as 500. The eval itself never throws — failing rules report passed: false with a message, they don't bubble exceptions.

ParametersJSON Schema
NameRequiredDescriptionDefault
inputNoOriginal input for context — improves relevance scoring (keyword overlap vs input)
outputYesThe output text to evaluate (the agent's response that gets scored against rules)
cost_usdNoCost in USD — only consulted when eval_type="cost" (compared against cost_threshold rules)
expectedNoExpected output for comparison — REQUIRED when eval_type="relevance" (used as keyword-overlap target)
trace_idNoLink evaluation to a trace — surfaces this eval in the dashboard's trace drill-through
eval_typeNoRule bundle to apply: completeness | relevance | safety | cost | custom — picks which built-in rules firecompleteness
token_usageNoToken usage breakdown — only consulted when eval_type="cost" (used for token-budget rules)
custom_rulesNoCustom evaluation rules — fires REGARDLESS of eval_type; pass eval_type="custom" if you want ONLY these

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Despite annotations already marking idempotentHint and non-destructive, the description adds valuable context: writes one eval_result row to Iris storage, deterministic in-process scoring, no external network calls in heuristic mode, rate limits (20 req/min HTTP, unlimited stdio), runtime ~5-50ms, and detailed output shape including insufficient_data semantics. No contradictions with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but well-organized with clear sections (siblings, behavior, output shape, usage, parameters, error modes). It front-loads the purpose, and each paragraph provides distinct value. Some minor redundancy exists between the output shape prose and the JSON example, but overall it's efficiently structured for a tool with this complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 8 parameters, nested custom_rules, no output schema, and multiple sibling tools, this description is exceptionally complete: covers return shape, error modes (Zod, ReDoS, rate limits, storage failures), parameter interactions, and performance characteristics. It leaves few open questions for an agent deciding to invoke the tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, but the description adds crucial conditional semantics: expected is required only for relevance, cost_usd/token_usage only for cost, custom_rules always fires regardless of eval_type, and eval_type defaults to completeness. This clarifies parameter interactions beyond the schema's per-field descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource: 'Score agent output against configurable eval rules and return a 0..1 score + per-rule breakdown.' It clearly distinguishes from siblings by naming evaluate_with_llm_judge, verify_citations, log_trace, etc., and positions evaluate_output as the fast heuristic path.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit 'Use when' and 'Don't use when' sections provide direct guidance: use for quality scoring after log_trace, avoid for empty outputs or JSON schema validation. It also names alternatives for semantic scoring and citation verification, giving clear decision criteria.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluate_with_llm_judgeEvaluate With LLM JudgeA

Score agent output using an LLM as the judge (Anthropic or OpenAI). Returns a calibrated 0..1 score with rationale, per-dimension breakdown, and exact cost.

Sibling tools — evaluate_output runs heuristic rules (free, deterministic, ~ms latency, no API key needed); this tool runs LLM-based semantic scoring (paid, 1-10s latency, requires API key). verify_citations is a SPECIALIZED form of LLM judging that focuses on citation grounding only. log_trace / get_traces handle trace I/O; list_rules / deploy_rule / delete_rule manage heuristic-rule lifecycle. evaluate_with_llm_judge is the GENERAL semantic-scoring path.

Behavior. Calls an external LLM API (Anthropic or OpenAI) — costs money per call, takes 1-10 seconds, respects an IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL cap. Non-deterministic at temperature > 0; default temperature=0 gives near-deterministic scores. Writes one eval_result row to Iris storage (linked to trace_id if provided) plus captures provider response id + latency + token counts + cost in the rule_results payload. Rate-limited to 20 req/min on HTTP MCP; your LLM provider also enforces its own rate limits (we transparently retry once on 429).

Output shape. Returns JSON: { "id": "<uuid>", "score": 0..1, "passed": boolean, "rationale": string, "dimensions": {...}, "model": string, "provider": "anthropic"|"openai", "template": string, "input_tokens": number, "output_tokens": number, "cost_usd": number, "latency_ms": number }. dimensions has per-dimension sub-scores (e.g., accuracy template returns {factual_claims, citations, internal_consistency}).

Use when heuristic rules (via evaluate_output) are too coarse for the quality signal you need — semantic correctness, factual accuracy vs a reference, RAG faithfulness to sources, nuanced safety/helpfulness. Pick the template that matches: accuracy (hallucination detection), helpfulness (does it address the ask), safety (harm potential beyond regex PII), correctness (vs reference answer — pass expected), faithfulness (RAG grounding — pass source_material).

Don't use for simple regex/length/keyword checks (use evaluate_output with heuristic rules — they're free, deterministic, 1000x faster). Don't use without an API key set (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY). Don't use on very large outputs (>8K tokens) without raising max_cost_usd — the pre-check will refuse the call.

Parameters. model is required (no default — pick consciously since cost varies 100x across models). provider is auto-detected from the model name; override only for ambiguous IDs. expected is REQUIRED when template="correctness" (the reference answer to compare against); ignored for other templates. source_material is REQUIRED when template="faithfulness" (the RAG sources to ground against); ignored otherwise. input is optional but improves scoring on helpfulness/safety templates (gives the judge the user prompt that produced the output). max_cost_usd defaults to env var IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL or $0.25 — the worst-case cost is computed BEFORE the call (input_tokens × prompt_price + max_output_tokens × completion_price); call refused upfront if it would exceed. max_output_tokens caps the judge response (default 512, max 4096); higher = more rationale detail + more cost. temperature default 0 (deterministic). timeout_ms default 60000. trace_id optional but recommended (links eval to trace in dashboard). Defaults: temperature=0, max_output_tokens=512, max_cost_usd=$0.25, timeout_ms=60000.

Error modes. Throws when the required API key env var is missing. Throws when the estimated worst-case cost exceeds max_cost_usd (raise the cap or trim prompts). Throws LLMJudgeError on provider errors — kind=auth on 401/403, rate_limit on 429 (auto-retried once), server_error on 5xx, timeout on abort, malformed_response when the judge fails to emit valid JSON on both attempts. Throws "Unknown model" for unsupported model IDs — update src/eval/llm-judge/pricing.ts first.

ParametersJSON Schema
NameRequiredDescriptionDefault
inputNoUser question / prompt that produced the output (improves accuracy for helpfulness/safety)
modelYesModel ID. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini.
outputYesThe agent output text to evaluate
expectedNoReference answer (required for correctness template)
providerNoAuto-detected from model when omitted
templateYesJudge dimension: accuracy (factual correctness), helpfulness (does it address the ask), safety (harm potential), correctness (vs reference answer — requires `expected`), faithfulness (RAG grounding — requires `source_material`).
trace_idNoLink this evaluation to a trace
timeout_msNoPer-request timeout; default 60_000
temperatureNoSampling temperature; default 0 (deterministic)
max_cost_usdNoCost cap in USD; defaults to IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL or 0.25
source_materialNoProvided RAG sources (required for faithfulness template)
max_output_tokensNoJudge output token cap; default 512

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Even though annotations already declare readOnlyHint=false and openWorldHint=true, the description adds substantial behavioral context: external API costs money, 1-10s latency, cost cap via IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL, non-determinism, writes an eval_result row, rate limits, and one retry on 429. Error modes are enumerated in detail. No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but highly structured with clear section headers (sibling tools, behavior, output shape, use when, parameters, error modes). Every paragraph serves a distinct purpose, and it front-loads the core function. No redundancy with schema or annotations was found.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 12-parameter tool with no output schema, the description fully compensates: it details the exact output JSON structure, template-specific requirements, cost guardrails, rate limits, prerequisites, and all error modes. It leaves no operational ambiguity and is self-contained for an agent to invoke correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, but the description goes far beyond individual field descriptions by explaining cross-parameter constraints: expected is required only for correctness template, source_material only for faithfulness, model has no default and cost varies 100x, max_cost_usd is pre-computed before the call, and explicit defaults for temperature, max_output_tokens, and timeout_ms. This is high-value semantic enrichment.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description opens with a specific verb+resource: 'Score agent output using an LLM as the judge (Anthropic or OpenAI).' It immediately distinguishes itself from siblings by naming evaluate_output as heuristic and verify_citations as a specialized form, making its unique role unmistakable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit when-to-use guidance ('Use when heuristic rules are too coarse...'), explicit don't-use cases ('Don't use for simple regex...', 'Don't use without an API key', 'Don't use on very large outputs'), and names alternatives (evaluate_output, verify_citations). This is exemplary usage guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_tracesGet TracesA
Read-onlyIdempotent

Query stored agent-execution traces with filters, pagination, and optional dashboard summary.

Sibling tools — log_trace creates traces, delete_trace removes a single trace, evaluate_output / evaluate_with_llm_judge / verify_citations score them, list_rules / deploy_rule / delete_rule manage the custom-rule lifecycle. get_traces is the READ path for historical agent executions — never mutates anything.

Behavior. Read-only: never mutates storage, never calls external services. Idempotent: repeated calls with the same args return consistent results (new traces logged after the call obviously show up on subsequent calls). Tenant-scoped: queries only the caller's tenant rows (LOCAL_TENANT in OSS). Paginates results (default limit 50, max 1000). Rate-limited to 20 req/min on HTTP MCP, unlimited on stdio.

Output shape. Returns JSON: { "traces": [{...traceRow}], "total": number, "limit": number, "offset": number, "summary"?: { total_traces, avg_latency_ms, total_cost_usd, error_rate, eval_pass_rate, traces_per_hour, top_agents } }. Each trace row includes trace_id, agent_name, framework, input, output, tool_calls, latency_ms, token_usage, cost_usd, metadata, timestamp. summary only included when include_summary: true.

Use when you need historical data: investigating a past failure, computing quality trends, comparing agents, or feeding an analytics job. Set agent_name / framework / since / until to narrow the query. Set min_score / max_score to surface outliers. Set sort_by: "cost_usd" + sort_order: "desc" to find the most expensive traces. Set include_summary: true when you want dashboard-style aggregates in one round-trip.

Don't use to score a trace (use evaluate_output). Don't use to create a trace (use log_trace). Don't use as a live event stream — it's a query, not a subscription; poll with exponential backoff or use the dashboard's SSE endpoint for real-time.

Parameters. limit defaults to 50, max 1000 (anything higher returns 400). offset is zero-based pagination. min_score / max_score filter on the LATEST eval per trace, not all evals (so a trace with one failing + one passing eval may or may not match depending on which landed last). Combining since + sort_by="latency_ms" + sort_order="desc" is the canonical "find slow recent traces" query. include_summary returns dashboard-style aggregates in the SAME response (saves a round-trip; use true for dashboard ingest, false for analytics queries that don't need them). agent_name and framework are exact-match (no wildcards in v0.4). Defaults: limit=50, offset=0, sort_by="timestamp", sort_order="desc", include_summary=false.

Error modes. Returns 400 on invalid sort_by / sort_order (Zod enum). Returns 400 if limit > 1000. Returns 429 when HTTP rate limit exceeded. Storage failures propagate as 500. Empty result with total: 0 on no matches (not an error).

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoResults per page (default 50, max 1000 — values >1000 return 400)
sinceNoISO timestamp lower bound — return traces with timestamp >= this
untilNoISO timestamp upper bound — return traces with timestamp < this
offsetNoZero-based pagination offset — skip first N results
sort_byNoSort by timestamp | latency_ms | cost_usd (default timestamp)timestamp
frameworkNoFilter by agent framework — exact match (e.g., langchain, autogen)
max_scoreNoMaximum eval score filter (0..1) — applied to LATEST eval per trace
min_scoreNoMinimum eval score filter (0..1) — applied to LATEST eval per trace, not all evals
agent_nameNoFilter by agent name — exact match (no wildcards in v0.4)
sort_orderNoSort order: asc | desc (default desc — most recent / highest first)desc
include_summaryNoInclude dashboard summary stats in same response — saves a round-trip when ingesting for dashboards

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false. The description adds substantial context beyond those: rate limits ('Rate-limited to 20 req/min on HTTP MCP, unlimited on stdio'), tenant scoping ('queries only the caller's tenant rows'), pagination limits, and the subtle idempotency nuance ('new traces logged after the call obviously show up on subsequent calls'). No contradictions with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long, but appropriately structured with headers (Behavior, Output shape, Use when, Don't use, Parameters, Error modes) and front-loaded with purpose. There is minor redundancy (read-only stated twice, limits repeated in schema and text), but every sentence earns its place given 11 parameters and no output schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite no output schema, the description fully documents the return shape ('{"traces": [...], "total": number, "limit": number, "offset": number, "summary"?: ...}'), error modes (400 for invalid enum/limit, 429 rate limit, 500 storage failures), pagination behavior, and empty-result semantics. This is complete for a complex 11-parameter read tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3. The description elevates this by adding behavioral semantics not in the schema: the min_score/max_score apply to 'the LATEST eval per trace', agent_name/framework are exact-match with 'no wildcards in v0.4', limit >1000 returns 400, and canonical combined queries like 'since + sort_by="latency_ms" + sort_order="desc"' are demonstrated. This far exceeds what the schema provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific verb+resource: 'Query stored agent-execution traces with filters, pagination, and optional dashboard summary.' It differentiates itself from siblings explicitly by stating 'get_traces is the READ path for historical agent executions — never mutates anything' and lists what each sibling does for contrast.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit when-to-use guidance ('Use when you need historical data: investigating a past failure, computing quality trends, comparing agents...'), concrete query patterns (e.g., 'Set min_score / max_score to surface outliers'), and clear alternatives: 'Don't use to score a trace (use evaluate_output). Don't use to create a trace (use log_trace). Don't use as a live event stream.' This is exemplary usage guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_rulesList Custom RulesA
Read-onlyIdempotent

Enumerate deployed custom evaluation rules from the local rule store.

Sibling tools — deploy_rule adds custom rules, delete_rule removes them, evaluate_output runs them against agent output. log_trace / get_traces / delete_trace handle the trace lifecycle separately. list_rules is the READ path for the custom-rule store; nothing else exposes the inventory.

Behavior. Pure read of ~/.iris/custom-rules.json (in-memory cached; no disk read per call after server boot). No mutation, no external network. Tenant-scoped in Cloud tier; OSS returns all rules for the single local tenant. Rate-limited to 20 req/min on HTTP MCP, unlimited on stdio. Returns in <5ms.

Output shape. Returns JSON: { "rules": [{ "id": "rule-XXXX", "name", "description?", "evalType", "severity", "definition": { type, config, weight? }, "enabled": boolean, "deployedAt": ISO timestamp, "sourceMomentId?": string }], "total": number, "enabled_count": number }. Empty array + total=0 when no rules deployed.

Use when you need to know what custom rules are currently live (before calling evaluate_output, before deploying a similar rule to avoid duplicates, or when building a dashboard view). Filter with eval_type to scope to a specific category, or enabled_only: true to exclude disabled rules. Use get_traces to see trace data; use evaluate_output to run scoring; use list_rules only when you need the RULE INVENTORY.

Don't use to count traces or evals (that's get_traces). Don't use to inspect built-in (non-custom) rules — those ship with the iris binary and are listed in docs/api-reference.md, not in the rule store. Don't use to deploy a rule (use deploy_rule); don't use to remove one (use delete_rule).

Parameters. eval_type filter is exact-match against each rule's evalType field (no wildcards). enabled_only excludes rules that are deployed-but-disabled (toggled via the dashboard's rule-list affordance — there's no MCP toggle tool in v0.4). Both filters are AND-combined when both are set. Both are optional; with no filter, all rules return. Defaults: eval_type=undefined (no filter), enabled_only=false (returns all rules including disabled).

Error modes. Returns empty list if the rule store file doesn't exist (first run). Returns 429 if HTTP rate limit exceeded. Never throws on valid input.

ParametersJSON Schema
NameRequiredDescriptionDefault
eval_typeNoFilter to rules of a specific eval category
enabled_onlyNoReturn only enabled rules (excludes disabled ones)

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and destructiveHint=false, but the description adds substantial behavioral context: pure read, in-memory caching, no disk read per call after boot, no external network, tenant scoping, rate limits (20 req/min HTTP, unlimited stdio), and <5ms response time. It also discloses error modes (429, empty list for missing file). No contradictions with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but well-structured with clear sections (Sibling tools, Behavior, Output shape, Use when, Don't use, Parameters, Error modes). It front-loads the core purpose and each section earns its place by adding non-redundant information. No fluff or repetition of schema field names beyond what is necessary for clarity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity and the absence of an output schema, the description fully compensates by documenting the output shape, empty-array behavior, rate limits, error modes, and scoping nuances. It covers all contextual dimensions an agent would need for correct invocation and result interpretation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Although the schema covers both parameters (100% coverage), the description adds crucial semantics: eval_type is exact-match with no wildcards, enabled_only excludes deployed-but-disabled rules, both filters are AND-combined, and defaults are explicit (undefined / false). This goes well beyond the schema's field descriptions, providing operational detail that an agent needs to invoke correctly.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Enumerate deployed custom evaluation rules from the local rule store.' It further distinguishes the tool from siblings by explicitly stating 'list_rules is the READ path for the custom-rule store; nothing else exposes the inventory,' making its unique role unmistakable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit when-to-use guidance: 'Use when you need to know what custom rules are currently live' followed by concrete scenarios. It also provides strong exclusions and alternatives: 'Don't use to count traces or evals (that's get_traces)', 'Don't use to inspect built-in rules', and names deploy_rule/delete_rule for mutations. This is textbook usage guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

log_traceLog TraceA

Persist a single agent execution trace (input, output, spans, tool calls, cost, latency, token usage).

Sibling tools — evaluate_output runs heuristic scoring on the trace; evaluate_with_llm_judge runs semantic LLM-based scoring; verify_citations checks citation grounding; get_traces queries stored traces; delete_trace removes a single trace; list_rules / deploy_rule / delete_rule manage custom evaluation rules. log_trace is the WRITE path that records executions; everything else reads, scores, or manages around it.

Behavior. Writes one row to Iris storage (SQLite by default; Postgres in Cloud tier). When IRIS_OTEL_ENDPOINT is set, ALSO fires a best-effort async export to the configured OTLP/HTTP collector (Jaeger, Tempo, Datadog OTLP, OTEL Collector). The OTel export is fire-and-forget — its success does not affect the tool response; failures are logged but the trace is still stored locally. No authentication in stdio mode; HTTP mode requires Bearer token. Rate-limited to 20 req/min on HTTP MCP, unlimited on stdio. Not idempotent: each call mints a fresh trace_id, so resubmitting the same payload creates a duplicate trace.

Output shape. Returns a JSON string: { "trace_id": "<32-hex>", "status": "stored" }. The trace_id is the key you pass to evaluate_output or get_traces afterwards.

Use when you want to record an agent execution for later evaluation, analysis, or audit. Call it AFTER the agent has produced output; call evaluate_output afterwards to score it; call get_traces to query historical traces. Store rich context: spans (span tree), tool_calls (which tools were invoked with latency/errors), token_usage, cost_usd, metadata (arbitrary key-value). All optional except agent_name.

Don't use when you only need a transient log (use console logging). Don't use to update an existing trace — there is no update path in v0.4 (traces are immutable once stored).

Parameters. agent_name is required; everything else is optional. token_usage and cost_usd are summary fields — if you ALSO pass spans with per-tool-call costs, the summary fields are treated as authoritative (no auto-aggregation). spans without an explicit start_time fall back to the trace timestamp; spans with an end_time get a duration_ms derived. metadata is opaque key-value (queryable in the dashboard, not via get_traces filters). tool_calls record per-tool latency + errors; missing latency_ms means "not reported," not "zero." Defaults: span.kind="INTERNAL", span.status_code="UNSET", timestamp=now() if omitted.

Error modes. Throws on missing agent_name. Throws on malformed span or tool_call objects (Zod rejects). Returns 500 on storage failure (disk full, DB locked). Never blocks on the agent — returns within ~50ms for typical payloads.

ParametersJSON Schema
NameRequiredDescriptionDefault
inputNoAgent input text — the user prompt or upstream input that produced this output
spansNoDetailed execution spans (hierarchical span tree with timings, attributes, events)
outputNoAgent output text — what the agent produced (pass to evaluate_output for scoring)
cost_usdNoTotal cost in USD — overrides per-span aggregation when provided (treated as authoritative)
metadataNoOpaque key-value tags (e.g. {requestId, userId, env}) — queryable in dashboard, not via get_traces filters
frameworkNoAgent framework identifier (e.g., langchain, autogen, custom)
timestampNoTrace timestamp (ISO 8601); defaults to now() when omitted
agent_nameYesAgent name — used for filtering in get_traces (e.g., "customer-support-bot")
latency_msNoTotal execution time in milliseconds (end-to-end agent latency)
tool_callsNoTool calls made during execution (per-call latency, errors, input/output)
token_usageNoToken usage breakdown (prompt/completion/total — used for cost analysis)

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes far beyond annotations by detailing persistence backends (SQLite/Postgres), best-effort OTel export with fire-and-forget semantics, auth/rate limits by mode, non-idempotency, output shape, and error modes (Zod rejects, 500 on storage failure, ~50ms latency). These details are not present in the annotations and materially affect agent behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but efficiently organized with labeled sections (Behavior, Output shape, Use when, Parameters, Error modes). It front-loads the core purpose with no filler; each section addresses a distinct question an agent would have, and every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a highly complex tool with nested objects, OTel export, and 11 parameters, the description covers purpose, usage timing, output format, error behavior, parameter nuances, and immutability constraint. The lack of an output schema is compensated by the explicit output shape description. This is a comprehensive and actionable specification.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Although schema covers all 11 parameters, the description adds rich semantics not in the schema: summary fields are authoritative (no auto-aggregation), span start_time fallback and duration derivation, metadata is opaque and not filterable via get_traces, missing latency means 'not reported', and defaults for span.kind/status_code and timestamp. This resolves ambiguities an agent would otherwise face.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with 'Persist a single agent execution trace' — a specific verb and resource. It enumerates the stored data (input, output, spans, tool calls, cost, latency, token usage) and explicitly distinguishes log_trace from siblings as 'the WRITE path' while naming what each other tool does.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Contains a dedicated 'Use when' section listing preconditions (after agent output) and follow-up tools (evaluate_output, get_traces), and a 'Don't use when' section contrasting with console logging and noting the lack of an update path due to immutability. Explicit exclusions and alternatives are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

verify_citationsVerify CitationsA

Extract citations from agent output, fetch the cited sources, and use an LLM judge to check whether each source supports the claim in context. Returns per-citation verdicts + an overall support ratio.

Sibling tools — evaluate_with_llm_judge runs general semantic scoring (accuracy, helpfulness, correctness, faithfulness); this tool is specifically for citation grounding (does the cited source actually support the claim). evaluate_output's no_hallucination_markers heuristic detects FABRICATED-looking citations cheaply (free, no fetch); this tool resolves and verifies them (paid, opt-in fetch, SSRF-guarded). log_trace / get_traces handle trace I/O. verify_citations is the GROUNDING-CHECK path — narrowest in scope, deepest in rigor.

Behavior. Three-phase pipeline: (1) regex extraction of [N] numbered refs, (Author, Year) parentheticals, bare URLs, and DOIs (in-process, no network); (2) SSRF-guarded fetch of URL + DOI citations, with scheme allowlist, private/link-local/cloud-metadata IP blocking, optional domain allowlist (IRIS_CITATION_DOMAINS), 10s timeout, 5MB body cap, manual redirect chase (max 3, re-checked), in-process LRU cache; (3) per-citation LLM judge call asking "does this source support this claim?" with a 256-token verdict. Opt-in via allow_fetch=true or IRIS_CITATION_ALLOW_FETCH=1 — Iris refuses outbound HTTP by default. Cost-capped across the entire call by max_cost_usd_total (default $1.00) — the pipeline stops when the cap would be exceeded. Rate-limited to 20 req/min on HTTP MCP. Writes one eval_result row tagged with per-citation provenance.

Output shape. Returns JSON: { "id": "<uuid>", "overall_score": 0..1|null, "passed": boolean, "total_citations_found": number, "total_resolved": number, "total_supported": number, "total_cost_usd": number, "citations": [{ "citation": { "raw", "kind", "identifier", "offset_start", "offset_end" }, "resolve_status": "ok"|"skipped"|"error", "resolve_error"?, "source"?: { "url", "status", "content_type", "bytes_fetched", "truncated" }, "judge"?: { "supported", "confidence", "rationale", "cost_usd", "latency_ms", "input_tokens", "output_tokens" } }] }. overall_score = supported / resolved; null when nothing resolvable was found.

Use when the output makes factual claims backed by [1]-style references, DOIs, or URLs and you want to separate "cited correctly" from "cited and wrong" from "cited but unresolvable". Particularly useful for research/legal/medical agents where fabricated citations are the dominant failure mode.

Don't use when the agent output has no citations at all (overall_score will be null; the tool degrades gracefully but a heuristic rule is cheaper). Don't use without allow_fetch=true or IRIS_CITATION_ALLOW_FETCH=1 — the tool refuses outbound HTTP unless explicitly enabled. Don't use with an open allowlist + untrusted output on the public internet; you are effectively running a user-directed fetcher. For stricter safety set IRIS_CITATION_DOMAINS to a curated list.

Parameters. model is required; provider auto-detected from model name (override only for ambiguous IDs). allow_fetch=false by default — outbound HTTP is REFUSED unless explicitly true OR IRIS_CITATION_ALLOW_FETCH=1 env. domain_allowlist suffix-matches hostnames (e.g., "wikipedia.org" allows en.wikipedia.org); merged with IRIS_CITATION_DOMAINS env (UNION — either source permits). max_citations defaults 20, hard cap 50 (extras are skipped silently, NOT errored — check total_citations_found in the response if precise). max_cost_usd_total defaults $1.00 — the pipeline stops mid-citation when the next judge call would exceed the cap (returns partial verdicts). per_source_timeout_ms defaults 10000 (10s); per_source_max_bytes defaults 5MB (truncates at boundary, judges still run on truncated content). trace_id optional but recommended. Defaults: max_citations=20, max_cost_usd_total=$1.00, per_source_timeout_ms=10000, per_source_max_bytes=5242880, allow_fetch=false.

Error modes. Throws when the API key env var is missing. Throws "Unknown model" on unsupported model IDs. Per-citation errors are collected (resolve_error.kind = bad_scheme / ssrf / not_allowed_domain / timeout / too_large / bad_status / redirect_loop / not_text / fetch_disabled / malformed_judge_response / cost_cap_reached / unresolvable_kind) and returned in the response rather than thrown. An empty output or output with zero extractable citations returns overall_score=null + passed=true (nothing to fail).

ParametersJSON Schema
NameRequiredDescriptionDefault
modelYesJudge model for per-citation verification. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini.
outputYesThe agent output containing citations to verify
providerNoAuto-detected from model when omitted
trace_idNoLink verification result to a trace
allow_fetchNoPermit outbound HTTP to resolve URLs/DOIs. Defaults to IRIS_CITATION_ALLOW_FETCH=1; false otherwise. SSRF-guarded regardless.
max_citationsNoMax citations to verify (extras skipped); default 20
domain_allowlistNoRestrict fetches to hostnames in this list (suffix match allowed). Merged with IRIS_CITATION_DOMAINS env.
max_cost_usd_totalNoCap TOTAL judge cost across all citations in this call; default $1.00
per_source_max_bytesNoPer-URL body cap; default 5MB
per_source_timeout_msNoPer-URL fetch timeout; default 10_000

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations are minimal (readOnlyHint=false, openWorldHint=true), leaving the description to carry nearly all behavioral context—and it does. It discloses SSRF-guarded fetch mechanics (scheme allowlist, IP blocking, redirect chase with re-checks), default refusal of outbound HTTP unless allow_fetch=true or IRIS_CITATION_ALLOW_FETCH=1, cost caps that stop the pipeline mid-citation, 20 req/min MCP rate limit, and that it writes one eval_result row. These details align with and enrich the open-world hint without contradicting any annotation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but front-loaded and ruthlessly organized: opening summary, then labeled sections for Sibling tools, Behavior, Output shape, Use when, Don't use, Parameters, and Error modes. Each sentence carries distinct operational facts with no filler; the length is proportionate to the tool's 10-parameter, network-facing, judge-calling complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema present, the description supplies a complete JSON response shape, covering every field. It also documents all throw-vs-collect error semantics, per-citation error kinds, graceful degradation (null overall_score), and safety/cost/time limits. For a tool of this complexity, everything needed for correct invocation and interpretation is present.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Although the schema already covers 100% of parameters with descriptions, the tool description adds substantial operational meaning: defaults for every parameter, the iri_citation_allow_fetch env-var interaction, suffix-matching for domain_allowlist with union merge over IRIS_CITATION_DOMAINS, silent skip of citations beyond max_citations (hard cap 50), mid-call cost-cap behavior, and the full resolve_error.kind taxonomy. This goes well beyond the structured schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific three-phase verb chain ('Extract citations from agent output, fetch the cited sources, and use an LLM judge to check...') and names the exact return value (per-citation verdicts + overall support ratio). It then explicitly distinguishes from siblings evaluate_with_llm_judge (general semantic scoring) and evaluate_output (cheap fabricated-citation heuristic), making the tool's unique scope unmistakable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The 'Use when' paragraph gives concrete preconditions (factual claims with [1]-style references, DOIs, or URLs; research/legal/medical domains). The 'Don't use' paragraph lists three explicit exclusions (no citations, no allow_fetch, open allowlist with untrusted output). Alternatives are named directly: evaluate_with_llm_judge for generic scoring, evaluate_output for cheap detection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

TDQS

A4.8/5.0
Disambiguation5/5

Each tool targets a distinct resource or action: traces (log/get/delete), heuristic evaluation (evaluate_output), LLM-based evaluation (evaluate_with_llm_judge), citation verification (verify_citations), and rule CRUD (list/deploy/delete). While multiple evaluation tools exist, their boundaries are explicitly clear by methodology and use case, so agents can reliably select the correct one.

Naming Consistency4/5

Names follow a mostly consistent verb_noun snake_case pattern (log_trace, deploy_rule, get_traces). Minor inconsistencies include mixing 'get' and 'list' for read operations and the longer evaluate_with_llm_judge, which is a bit verbose but still predictable.

Tool Count5/5

With 9 tools, the server is well-scoped, covering trace ingestion/query/deletion and multiple evaluation modes plus rule management without unnecessary bloat. The count is appropriate for the domain and each tool earns its place.

Completeness4/5

The surface covers essential trace lifecycle (create/read/delete) and evaluation (heuristic, LLM, citation) with rule management. Minor gaps exist (no rule update or enable/disable, no bulk trace delete), but these are documented workarounds or deferred to future versions, so they don't severely hinder usage.

Maintenance

ActivityMaintained
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    B
    maintenance
    MCP server for AI agent security guardrails. Provides input validation, prompt injection detection, PII redaction, output filtering, policy enforcement, rate limiting, and comprehensive audit logging.
    76
    1
    MIT
  • A
    license
    A
    quality
    B
    maintenance
    An MCP server that provides cost and reliability observability for LLM and agent workflows. It records model calls and allows querying and aggregating telemetry data through MCP tools.
    6
    MIT
  • F
    license
    Not graded
    quality
    C
    maintenance
    A local-first MCP server that gives AI coding agents runtime visibility and AI-managed debug logging. It replaces blind print() debugging by turning runtime execution into causal chains, allowing agents to instantly locate bugs by finding missing .success events in Python and TypeScript code. Single binary with MCP, CLI, and HTTP interfaces.

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/iris-eval/mcp-server'

If you have feedback or need assistance with the MCP directory API, please join our Discord server