IA-QA Eval & RAG — 38 tools

test_skill

Read-only

Test a SKILL.md in the open Agent Skills format (https://agentskills.io/specification — read by Claude Code, Codex, GitHub Copilot, Cursor, Gemini CLI, OpenCode and others) in two stages. (1) lint: the spec's frontmatter rules checked without a model (name 1-64 chars, lowercase/digits/single hyphens, matches dir_name when given; description 1-1024 chars; compatibility ≤500; metadata string map) — any error returns BLOCK and calls no model; gate on lint.valid, it is deterministic. (2) triggering, measured the way an agent decides: the model sees ONLY the name + description of this skill and of the other installed skills (other_skills, or six built-in neighbours) and answers which one it would load, at temperature 0. trigger_accuracy = share of requests routed correctly (should load / should not); verdict PASS ≥80, FIX ≥50, else BLOCK. Pass your own scenarios [{prompt, should_trigger}] for a reproducible gate; without them requests are generated (half should-load, half near misses) and the result is a signal, flagged by scenarios_source. step_adherence is keyword overlap on the skill body's run and never enters the verdict. A request the model never answered readably (rate limit, empty output) is not counted as a miss: the run is INCONCLUSIVE instead. Uses the server's default Groq model (server key, rate-limited). Pass api_key + model to use your own provider; the SKILL.md and scenarios are sent to ia-qa.com and to that provider.

Input Schema

Name	Required	Description
`model`	No	LLM model ID to use for both scenario generation and testing (e.g. gpt-4o-mini, claude-3-5-haiku-20241022). Defaults to openai/gpt-oss-20b (Groq, server key).
`api_key`	No	API key for the chosen model provider. Not required when using the default Groq model.
`dir_name`	No	Name of the folder holding the SKILL.md. The spec requires name to match it; checked only when given.
`skill_md`	Yes	Full content of the SKILL.md file to test, YAML frontmatter included.
`lint_only`	No	Run the format check only: no model call, no key, deterministic. The CI gate for a SKILL.md — gate on lint.valid.
`scenarios`	No	Fixed user requests for a reproducible gate: [{"prompt": "...", "should_trigger": true}]. Include near misses with should_trigger false. Max 20. When omitted, scenario_count requests are generated.
`other_skills`	No	The other skills installed next to this one, [{"name", "description"}], so triggering is tested against the real neighbours. Default: six built-in generic skills.
`scenario_count`	No	Number of requests to generate when scenarios is omitted: half should-load, half near misses. Default: 6.

Output Schema

Name	Required	Description	Default
`lint`	No
`score`	No
`verdict`	No
`scenarios`	No
`step_adherence`	No
`scenarios_source`	No
`trigger_accuracy`	No

Schema Changelog

Changes observed during successful MCP inspections.

First observedOct 26, 2026

TDQS

A4.6/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations cover safety (readOnlyHint, destructiveHint=false), and the description adds substantial extra context: rate-limiting on the server key, auth via api_key+model, temperature 0, verdict thresholds (PASS ≥80/FIX ≥50/BLOCK), and an explicit data-privacy disclosure that the SKILL.md and scenarios are sent to ia-qa.com and the provider. It also explains that unreadable responses yield INCONCLUSIVE rather than a miss.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the two-stage structure and dense with actionable facts rather than filler, though the single long paragraph is heavy and could be split for readability. Nearly every clause carries operational meaning.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists so return values need no prose, and the description still explains verdicts, step_adherence, scenarios_source, and failure semantics. Given the tool's complexity and 8 parameters, nothing an agent needs to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema carries the parameter definitions (baseline 3). The description still adds meaning: scenarios capped at 20, the should-load/near-miss split, the six built-in default neighbours, and scenario_count defaults.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('Test a SKILL.md in the open Agent Skills format') and enumerates the two stages (lint, triggering). It is instantly distinguishable from sibling linters like mcp_schema_lint or llm_format_check because it names the format and the two-stage process.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Clearly says when to use each mode: lint_only is 'the CI gate for a SKILL.md — gate on lint.valid', and passing your own scenarios yields 'a reproducible gate' versus generated ones being 'a signal'. It does not name sibling alternatives explicitly, so it falls just short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All IA-QA Eval & RAG — 38 tools tools Inspect

Try in Browser

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.