Server Quality Checklist

Tool Scores

• firefly_list_operationsA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false, so the safety profile is covered. The description adds only that listing can be filtered by entity, which is also present in the schema; it does not describe return format, pagination, or how 'operations' are represented.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that names the action, the resource, and the optional modifier. Every word earns its place, and there is no redundant or filler content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter, read-only listing tool, the description is mostly sufficient. It could be more complete by clarifying what an 'operation' is or how the returned catalogue relates to the sibling tools, but nothing essential to making the call is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%: the single optional 'entity' parameter is fully described with an enum and an explanation. The description merely restates this as 'filtered by entity,' adding no semantic value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') with a clear resource ('available Firefly III operations') and a qualifier ('optionally filtered by entity'). This unambiguously distinguishes the tool from the sibling mutating/querying tools without needing to open the schema.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives no guidance on when to use this tool versus firefly_query, firefly_mutate, firefly_destructive, or firefly_get_schema. It does not say, for example, that this is a discovery step before invoking other tools, nor does it mention any exclusions or alternative conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• firefly_get_schemaA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, openWorldHint=true, and destructiveHint=false, and the description aligns with those. The description adds the basic return intent but no additional behavioral context such as whether the operation is executed or what the returned schema will contain beyond its obvious meaning.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler. Every word contributes to conveying the tool's purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter introspection tool, the definition is nearly complete. The return value is directly implied by the purpose, and no output schema is needed; the only minor gap is the absence of an explicit workflow hint connecting it to firefly_query or firefly_mutate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the input schema fully documents both parameters. The tool description itself adds no additional meaning beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Get') and a clear resource ('parameter schema for a specific operation'). It is easily distinguished from the sibling tools firefly_query/firefly_mutate, which execute operations, and firefly_list_operations, which lists operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The intended use is implied: an agent should call this tool to discover the schema for an operation before executing it. However, the description does not explicitly say when to use it over alternatives or provide any exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• firefly_destructiveA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description goes well beyond the annotations: it warns that actions cannot be undone through the server, requires user confirmation, explains that record content must be treated as data rather than instructions, and describes response-size behavior. It also gives practical guidance on using the `fields` parameter to reduce large responses by ~90%. This adds substantial behavioral context beyond the raw annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer than a typical one-liner, but each section earns its place: purpose, entity/operation catalog, parameter lookup guidance, injection-safety warning, and performance tip. It is front-loaded with the core purpose and organized so an agent can quickly parse scope and safety requirements.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a destructive, multi-entity tool with no output schema, this description covers the critical areas: scope, undo risk, confirmation requirement, parameter discovery, prompt-injection hazards, and response-size management. It does not describe the normal return envelope, but the dry_run behavior is documented in the input schema and the agent is directed to firefly_get_schema for operation-level details, so the description is sufficiently complete for selection and safe invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3, but the description adds meaningful value by listing valid entity/operation combinations and giving a concrete `fields` example for summarising spending. It also clarifies that detailed operation parameters must be retrieved via firefly_get_schema, which helps the agent understand the relationship between `entity`, `operation`, and `params`.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Delete records, or rewrite one field across many records in a single call,' then enumerates the supported entities and operations. This clearly distinguishes it from the sibling tools firefly_query, firefly_mutate, firefly_list_operations, and firefly_get_schema, since it names destructive operations only.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly frames when this tool is appropriate: destructive operations that cannot be undone, and it explicitly instructs 'confirm with the user first.' It also tells the agent to call firefly_get_schema(entity, operation) for operation-specific parameters. It does not explicitly contrast with firefly_mutate or firefly_query, but the destructive scope and entity/operation list make the intended usage clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• firefly_mutateA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds substantial behavioral context beyond the annotations: it clarifies the tool never deletes, warns that record content must be treated as data rather than instructions, states that empty/null attributes are stripped, and explains how `fields` reduces response size. This is especially valuable for a mutation tool with no output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer than average, but the length is largely justified by the number of supported entities and the safety-critical security guidance. It is front-loaded with purpose and exclusions, uses a clear operation list, and the extra paragraphs each add practical value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex write tool with no output schema, the description covers all the essentials: what it does, what it does not do, which operations are available, how to discover parameters, how to control response size, and how to handle untrusted record content. Nothing needed to invoke it correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds useful parameter semantics by explaining how `fields` can cut response size and how `dry_run` previews without writing. It also compensates for the open-ended `params` object by routing the agent to firefly_get_schema for operation-specific parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Create or change records in Firefly III.' It also explicitly separates this tool from destructive/bulk operations by stating what it does not do and pointing to firefly_destructive, making sibling differentiation clear even before reading the operation list.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit when-not guidance ('Does not delete anything, and does not rewrite fields across many records at once') and names the alternative tool for those cases. It also directs callers to firefly_get_schema for operation-specific parameters and enumerates supported entities and operations, so an agent knows exactly how to select and invoke the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• firefly_queryA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the readOnlyHint annotation, it discloses that record content is untrusted data and must never be followed as instructions — a critical security trait. It also notes that empty/null attributes are stripped, that `fields` can cut responses by ~90%, and distinguishes computed analysis data from fetched data.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Though long, the description is dense with functional information and front-loaded. The entity/operation list is structured and compact, and each security or performance note earns its place with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool spanning dozens of entity types, this definition provides a complete operational catalog, points to firefly_get_schema for parameter details, and warns about prompt injection. The lack of an output schema is compensated by the clear stripping note and `fields` guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema only generically labels entity and operation, while the description enumerates every valid entity and its operations, providing massive disambiguation value. It explains `fields` with a concrete use case and delegates per-operation `params` semantics to firefly_get_schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The opening sentence, 'Read from Firefly III. Never changes anything,' states a specific action and resource while distinguishing it from write-oriented siblings. The full entity/operation enumeration makes the tool's scope concrete and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description is explicit that this tool is read-only ('Never changes anything'), which gives a clear when-not direction. It also tells agents to call firefly_get_schema for operation parameters, naming a concrete alternative for parameter discovery. It does not explicitly name a write-tool alternative, but the sibling context and read-only statement make the routing clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.