Server Quality Checklist

Tool Scores

• chatC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure but only states the action ('ask a question'). It doesn't describe what happens (e.g., sends a message, gets a response, affects session state), permissions needed, rate limits, or error conditions, leaving critical behavioral traits unspecified.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence ('向聊天助手提问') with no wasted words, making it appropriately concise. However, it lacks front-loading of critical details (e.g., session requirement), slightly reducing its effectiveness despite the brevity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a chat interaction tool with no annotations and no output schema, the description is incomplete. It doesn't explain what the tool returns (e.g., a response, status), how it interacts with sessions, or error handling, leaving significant gaps for an AI agent to use it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters ('session_id' and 'question') adequately. The description adds no additional meaning beyond what the schema provides, such as format examples or constraints, meeting the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description '向聊天助手提问' (ask the chat assistant a question) states the general purpose but is vague about what resource or scope it operates on. It doesn't distinguish from sibling tools like 'create_chat' or 'retrieve', leaving ambiguity about whether this initiates a new chat or continues an existing one.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'create_chat' or 'retrieve'. The description implies it's for asking questions to a chat assistant, but it doesn't specify prerequisites (e.g., requires an existing session) or exclusions, leaving usage context unclear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_chatC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. While '创建一个新的聊天助手' clearly indicates a write/mutation operation, the description doesn't address important behavioral aspects: what permissions are required, whether there are rate limits, what happens if creation fails, or what the expected response format is. For a creation tool with zero annotation coverage, this is inadequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise - a single sentence that directly states the tool's purpose. Every word earns its place with no redundancy or unnecessary elaboration. It's front-loaded with the core functionality immediately apparent.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a creation/mutation tool with no annotations and no output schema, the description is insufficient. It doesn't explain what gets created (beyond '聊天助手'), what the creation process entails, what happens on success/failure, or what the agent should expect as a result. The description leaves too many behavioral questions unanswered for a tool that modifies system state.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters completely. The description adds no additional parameter semantics beyond what's in the schema - it mentions '基于指定的数据集' which corresponds to dataset_id, but provides no extra context about dataset requirements, format, or constraints. Baseline 3 is appropriate when schema does all the work.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('创建一个新的聊天助手' - creates a new chat assistant) and the resource ('基于指定的数据集' - based on specified dataset). It's specific about what the tool does, though it doesn't explicitly differentiate from sibling tools like 'chat' which might be for interacting with existing assistants rather than creating new ones.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. There's no mention of prerequisites, when this creation operation is appropriate, or how it differs from sibling tools like 'chat', 'list_datasets', or 'retrieve'. The agent must infer usage context from the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• retrieveC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool retrieves content but doesn't describe how it works (e.g., similarity search, ranking), what the output format is (since no output schema exists), or any constraints like rate limits or authentication needs. For a retrieval tool with zero annotation coverage, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It is front-loaded with the core action and resource, making it easy to understand at a glance. Every part of the sentence earns its place by conveying essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a retrieval operation with no annotations and no output schema, the description is incomplete. It doesn't explain what 'relevant content' means, how results are returned, or any behavioral traits. For a tool that likely involves search logic and result formatting, more context is needed to understand its full functionality and limitations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters ('dataset_ids' and 'question') with descriptions. The tool description adds no additional meaning beyond what the schema provides, such as explaining how the question is used for retrieval or the format of dataset IDs. With high schema coverage, the baseline score of 3 is appropriate as the description doesn't compensate but also doesn't detract.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('检索相关内容' - retrieve relevant content) and the resource ('从RAGFlow中指定的数据集' - from specified datasets in RAGFlow). It distinguishes from sibling tools like 'chat' and 'create_chat' by focusing on retrieval rather than conversation, though it doesn't explicitly mention how it differs from 'list_datasets'. The purpose is specific and actionable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention when to prefer 'retrieve' over 'chat' or 'create_chat' for information needs, nor does it specify prerequisites or exclusions. Usage is implied by the purpose but lacks explicit context for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_datasetsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states it lists datasets but doesn't describe what 'list' entails (e.g., format, pagination, sorting, or any limitations like access controls or rate limits). This leaves significant gaps for a tool with zero annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose with no wasted words. It's appropriately sized and front-loaded, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema, no annotations), the description is minimally adequate. It states what the tool does but lacks behavioral details and usage context. For a list operation, it should ideally mention output format or limitations, but the absence of an output schema increases the need for such information in the description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters with 100% schema description coverage, so the schema fully documents the lack of inputs. The description doesn't need to add parameter details, and it correctly implies no inputs are required by not mentioning any. Baseline is 4 for 0 parameters, as it's complete in this regard.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('列出' meaning 'list') and resource ('RAGFlow中的所有数据集' meaning 'all datasets in RAGFlow'), providing a specific purpose. However, it doesn't differentiate from sibling tools like 'retrieve' which might also involve datasets, so it doesn't reach the highest score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'retrieve' or 'chat'. There's no mention of prerequisites, context, or exclusions, leaving the agent with minimal usage direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.