Skip to main content
Glama

Audit the skill catalogue

skill_audit
Read-only

Audit a skill catalogue to measure session context cost, detect overlapping trigger phrases, and report malformed or omitted skills before adding or retiring one.

Instructions

Price the skill catalogue every session pays for, and name the collisions in it. A skill's description: frontmatter is loaded into the agent's context in EVERY session; its body is read only when the skill is invoked, so the descriptions are the standing bill. Call it before adding or retiring a skill, or when two skills keep firing on the same phrase. Walks root for <marketplace>/<plugin>/<version>/skills/<name>/SKILL.md and answers one JSON document: roots, skills (counted), catalogue_bytes (UTF-8 bytes of their description: values), findings and omissions. A degraded server adds the footer as its last key, bantamkit_degraded. Read findings first — each is {kind, severity, skills, detail}: shared-trigger-phrase (high; two skills quote the same literal phrase, so which fires is a coin flip), catalogue-over-budget (high; only when budget is given), never-invoked (low; only when usage is given), frontmatter-malformed and name-mismatch (medium). Then omissions: counted skills plus omissions account for every SKILL.md found, each {subject, count, size, what} with subject one of plugin-not-enabled, duplicate-skill, stale-version, unreadable-file, unparsed-frontmatter. The tool reads nothing but root; the three facts a directory cannot answer come from the caller — enabled (plugin ids the host has switched on, <plugin>@<marketplace>; the cache also holds disabled plugins, so without it the bill is inflated), usage (call counts per skill) and versions (the version directory the host serves per plugin; a plugin absent from it gets the directory name that sorts LAST in byte order, which is wrong for 10.0.0 against 9.0.0). Refused with their own sentence, never as a partial answer: an unknown check, a negative budget, an EMPTY root, a root that is missing or is a file. Full rules in docs/skill-audit.md.

Input Schema

NameRequiredDescriptionDefault
rootYesDirectory to scan for SKILL.md files, absolute or relative to the server's working directory. It must name a directory: the empty string is refused rather than resolved, because resolving it would audit whatever directory the server happens to be standing in
checkNoWhich family of findings to report; default `all`. `skills`, `catalogue_bytes` and `omissions` are reported whatever this says
usageNoSkill id (`<plugin>:<name>`) to call count, the caller's own measurement; a counted skill absent from a supplied map is treated as zero
budgetNoCatalogue byte budget; `catalogue-over-budget` is only reported when this is given
enabledNoPlugin ids that are switched on, `<plugin>@<marketplace>`; when given only skills under those plugins are counted, when omitted every skill under `root` counts
versionsNoPlugin id (`<plugin>@<marketplace>`) to the version directory name the host actually serves; when a plugin is named here that directory is resolved whatever byte order would have said, and a plugin absent from it falls back to byte order

Output Schema

NameRequiredDescriptionDefault
resultYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.36.0

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already mark it read-only and closed-world, and the description adds substantial context beyond that: the exact walk pattern, the JSON keys returned, the degraded-server footer key, and the four refusal conditions (unknown check, negative budget, empty/missing/file root). It stops short of describing pagination or size limits, but the disclosure is far richer than the annotations alone.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Long, but front-loaded with purpose and dense with non-redundant detail (findings kinds, omissions accounting, refusal rules), each sentence carrying weight. Some of the output-schema content (`findings`, `omissions` shapes) is repeated, which trims the score from 5.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 6-parameter tool with nested objects and an output schema, this covers everything: what is scanned, what each optional input contributes, how degraded servers signal themselves, and that refusals are total rather than partial. Nothing an agent needs to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description supplies genuine semantics the schema does not: why omitting `enabled` inflates the bill (the cache holds disabled plugins), and that `versions` exists specifically to fix byte-order misresolution such as `10.0.0` against `9.0.0`. These are caller-facing semantic hints, not restatements.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource — pricing the skill catalogue and naming collisions — with a clear mechanism (the `description:` frontmatter being the standing bill). No sibling tool covers anything similar, so differentiation is implicit but unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit trigger conditions: 'Call it before adding or retiring a skill, or when two skills keep firing on the same phrase.' This tells the agent when the tool is worth invoking rather than leaving it to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.