Server Quality Checklist

Tool Scores

• list_domainsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and non-destructive behavior. The description adds useful context beyond annotations by revealing that the result contains 18 domains, each with trigger keywords and a sample story, which the agent can pass to preview_story or generate_dataset.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the core action in the first sentence. The subsequent sentences add useful behavioral and usage context without unnecessary fluff, though it could have been slightly tighter by merging the first two sentences.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is sufficient for a parameterless list tool: it names the exact output scope, explains what each returned domain contains, gives intended usage triggers, and references downstream consumer tools. The output schema and annotations cover the remaining structural and safety aspects.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool takes zero parameters, and the schema coverage is effectively 100% because there is no input to document. There is nothing for the description to add about parameter syntax or meaning, so the baseline score for a zero-parameter tool applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with a specific verb+resource: 'List the 18 built-in business domains Misata can generate from natural language.' This clearly distinguishes the tool from the generative siblings by indicating it is a read-only catalog of available domains.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states when to use this tool: 'Use this when the user asks "what kinds of data can you generate?" or to suggest a story format.' It also mentions downstream tools like preview_story or generate_dataset, but it does not explicitly address when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_from_schemaA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description fully discloses behavioral traits: it writes CSVs to output_dir, uses a seed for reproducibility ('same seed → byte-identical output'), and notes the role of temp dirs. These align with the annotations (idempotentHint true, destructiveHint false) and add concrete expectations about file creation and determinism.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exceptionally long, containing a full manual with extensive examples, design rules, and directive references. While well-structured with sections and headers, it is far from concise and would be overwhelming for quick reference. Many sentences, though informative, are not essential for a tool summary.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the schema's complexity (nested objects, many optional directives) and the rich output (CSV files and JSON samples), the description is extremely thorough. It covers edge cases, validation via misata.validate_domain, and all schema-level directives, making it essentially a complete reference.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides only types and defaults, but the description's 'Args' section explains each parameter (schema, rows, seed, output_dir, sample_rows) in detail, including the schema's structure and optional directives. This adds meaningful semantics beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Generate a dataset from a schema' and explicitly identifies it as the primary Misata tool. It distinguishes itself by outlining its role in handling exact integrity guarantees, which differentiates it from typical generation tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides extensive guidance on when to use specific features (e.g., outcome curves, exact_incidence, rollup) and design rules for best results. It lacks explicit comparison with sibling tools like generate_dataset or seed_database, but the phrase 'primary Misata tool' gives a strong positional cue. Overall, it effectively communicates usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• inspect_schemaA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already mark this as read-only and non-destructive; the description builds on this by noting it does not generate data and by explaining the type of output (every column with type and distribution params). It stops short of detailing performance implications beyond 'heavier' or clarifying idempotent behavior, but adds meaningful context beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is tightly written, starting with a clear one-sentence purpose, followed by a comparative detail, use cases, and a compact Args list. Every sentence earns its place with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only inspection tool with two parameters and a documented output schema, the description covers the what, when, and parameters, and even notes the lack of data generation. It could go further by describing error scenarios or output format specifics, but these are already captured by the output schema and annotations, making it adequately complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must compensate, and it does via an Args section that explains `story` as 'Plain-English description of the dataset' and `rows` as 'Default row count for the primary table.' This adds clear meaning beyond the bare types, though it could elaborate on how changing `rows` affects its behavior. The high coverage shift justifies a score above baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Return the full schema (tables, columns, relationships) for a story without generating data,' using a specific verb and resource scope. It goes beyond a generic statement by explicitly contrasting with `preview_story` ('Heavier than') and providing concrete use cases, making its purpose unmistakable and distinctive.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly notes this tool is 'Heavier than `preview_story`' and provides two direct use cases: showing the structure the user will get and authoring a `misata.yaml` file from a seed. This gives an agent clear when-to-use guidance and signals an alternative, even if it doesn't mention explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• preview_storyA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false. The description adds context: it clarifies no rows are generated, that rows affects preview only, and lists returned fields (domain, confidence, near-misses, locale, scale, tables). No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: a one-sentence purpose, a list of returned items, a usage note, and an Args section. It is concise with no redundant text and front-loads the core purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 params, output schema exists), the description covers purpose, usage, parameters, and return values. It provides sufficient context for an agent to understand when and how to invoke it, without needing to rely on the output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must clarify parameters. It provides Args: 'story: Plain-English description of the dataset' and 'rows: Default row count for the primary table (affects preview only).' This fully compensates for missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Inspect what Misata would generate from a story — without generating any rows.' This uses a specific verb (inspect) and resource (story interpretation) and differentiates from sibling generation tools like generate_dataset and seed_database.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says when to use: 'Use this to confirm interpretation before committing to a (potentially large) generation.' It implies the alternative is to generate, but does not name the exact sibling tools. Still, the guidance is clear and actionable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_datasetA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds substantial behavioral context beyond the annotations. It states that the tool writes CSV files to disk, returns output directory and file paths, and includes row counts and sample rows. It also discloses determinism: 'same seed → byte-identical output.' These details are not present in the annotations (readOnlyHint false, destructiveHint false) and valuable for the agent to anticipate side effects. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured. The opening sentence states purpose, followed by a clear list of return values, then a bulleted Args section. Every sentence earns its place, and the use of a list for parameters improves readability. There is no redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 5 parameters, an output schema, and complex behavior (file writing, determinism). The description explicitly mentions return values (output directory, file paths, row counts, sample rows), addresses parameter defaults, and explains the sample_rows cap. It is complete for an agent to use effectively without further documentation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has no property descriptions (coverage 0%), so the description carries the full burden. It includes an Args section that explains each of the 5 parameters: story, rows, seed, output_dir, and sample_rows, adding meaning like 'Defaults to a fresh temp dir' and 'max 50'. This fully compensates for the schema's lack of descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action: 'Generate a synthetic dataset from a story and write it to disk as CSV files.' It specifies the resource (dataset from a story) and distinguishes it from siblings like generate_from_schema, which focuses on schema-driven generation. The verb 'generate' and the context 'from a story' are specific and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool: when you have a plain-English story to convert into a dataset. It also explains that it returns sample rows and file paths, which is helpful for the agent to show results. However, it does not explicitly mention alternatives (e.g., 'use generate_from_schema if you have a schema') or state when NOT to use it, leaving a slight gap in decision-making guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_yamlA
  4.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description significantly enriches the annotations (readOnlyHint, idempotentHint) by detailing the two-step execution (structural then semantic), giving examples of semantic rules, and specifying the exact return format including the 'stage' of failure. It doesn't mention any rate limits or auth needs, hence not a 5, but it provides strong context beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely well-structured and concise for the complexity it handles. It uses markdown headers for the two validation levels, a 'Use this when' section for guidance, and clearly formatted Args and Returns blocks. Every sentence provides valuable information without unnecessary verbosity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema and detailed annotations, the description perfectly complements them. It explains the two types of validation, the order of execution, and what happens on failure. This is complete for an agent to understand the tool's behavior, when to use it, and what to expect in return.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Although schema coverage is 0% (no 'description' field in the schema for the parameter), the description explicitly documents the 'yaml_text' parameter: 'The full contents of a misata.yaml file as a string.' This fully compensates for the schema's lack of description, so while it increases transparency, the schema handles the variable type.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool validates a 'misata.yaml' document at two distinct levels (structural and semantic). It uses a specific verb+resource pattern and the detailed explanation of the two checks distinguishes it from sibling tools like 'validate' on other resources or 'inspect_schema'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool: 'Use this when an agent has authored or edited a misata.yaml on the user's behalf...'. It also clearly defines the alternative by mentioning 'before invoking generate_dataset', which is a sibling tool, and contrasts parsing with semantic validation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• seed_databaseA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description goes beyond the annotations by detailing the destructive nature (truncate=True destroys data), the write behavior, and the safety flow (plan first, then apply). It transparently discloses side effects and the conditions under which writes are allowed, adding crucial context that annotations alone do not provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Although lengthy, the description is well-structured with clear sections (summary, safety, args) and every sentence adds value. It avoids redundancy and uses formatting (bold, bullet points) to improve readability, making it efficient despite its length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the 8 parameters and output schema, the description thoroughly covers all aspects: parameter usage, safety considerations, and the return value (plan vs result with integrity verification). It is complete and self-contained, leaving no critical gaps for an agent to misuse the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, but the description's Args section explains every parameter in detail, including examples, defaults, and how they interact (e.g., rows scaling, tables/skip_tables filtering, apply vs truncate/append). This fully compensates for the missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: filling a live Postgres or SQLite database with realistic, connected data based on the database's own schema. It distinguishes itself from sibling tools like generate_from_schema by emphasizing that no schema file or ORM is needed, and it is the only tool that writes to a database.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly explains when to use this tool (to populate a database) and differentiates it from alternatives by noting it is the only one that writes. It also provides clear instructions on the safe default (apply=False) and when to use truncate/append, making it obvious when to call it versus other tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.