Server Quality Checklist

Tool Scores

• m365_chat_with_filesA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key behaviors: the tool uses 'M365 Copilot reads the files and answers', implies AI synthesis rather than raw retrieval, and mentions conversation continuity via conversation_id. However, it doesn't address potential limitations like file size constraints, authentication needs, or rate limits that would be helpful for a mutation-like operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core purpose. Each sentence earns its place: the opening statement defines the tool, the bullet points clarify use cases, and the final sections provide clear differentiation from siblings. No wasted words or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (AI-powered document analysis with conversation continuity) and lack of annotations/output schema, the description does well by explaining the tool's behavior, use cases, and alternatives. However, it could benefit from mentioning expected output format or error conditions, especially since there's no output schema provided.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters thoroughly. The description doesn't add significant meaning beyond what's in the schema descriptions, though it reinforces the purpose of message and file_uris through usage examples. This meets the baseline expectation when schema coverage is complete.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Ask questions about specific documents you already have URIs for' with specific verbs like 'summarizing', 'comparing', and 'extracting info'. It explicitly distinguishes from sibling tools m365_search and m365_retrieve, making the scope and differentiation clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('Use for: Summarizing known documents, Comparing specific files, Extracting info from particular docs') and when to use alternatives ('Use m365_search first when: You need to find the files', 'Use m365_retrieve when: You want raw text chunks, not Copilot's synthesis'). This covers both positive and negative use cases with named alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• m365_meetingsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries the full burden of behavioral disclosure. It effectively describes key behavioral traits: the tool returns structured data (notes, decisions, tasks with owners, mentions), has prerequisites (transcription enabled, 4-hour delay), and has exclusions (channel meetings, meetings without transcription). However, it doesn't mention error handling, rate limits, or authentication needs, which could be relevant for an AI agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and concise, with each sentence earning its place. It starts with the core purpose, then details returns, prerequisites, use cases, and exclusions in a logical flow. There's no redundant or verbose language, making it easy for an AI agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (involving meeting insights with prerequisites) and the absence of annotations and output schema, the description does a good job of covering essential context: purpose, usage, behavioral constraints, and exclusions. However, without an output schema, it could benefit from more detail on the structured data format (e.g., sample outputs or data types), which would help the agent understand what to expect.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters (meeting_id, join_url, since) with clear descriptions. The description doesn't add any parameter-specific information beyond what's in the schema, such as explaining interactions between parameters or providing examples. The baseline score of 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get AI-generated meeting summaries, action items, and mentions from Teams.' It specifies the resource (Teams meetings) and the verb (get) with concrete outputs (summaries, action items, mentions). It distinguishes itself from sibling tools like m365_chat or m365_search by focusing specifically on meeting insights rather than chat or general search functionality.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('Post-meeting follow-up', 'Finding action items assigned to you', 'Checking what you missed in meetings') and when not to use it ('Does NOT work for: Channel meetings, Meetings without transcription enabled'). It also mentions prerequisites ('Transcription enabled during meeting', '~4 hours after meeting ends for insights to be ready'), giving clear context for appropriate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• m365_retrieveA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It does well by explaining the tool returns 'relevance-scored excerpts' and that the user controls synthesis. However, it doesn't mention potential limitations like rate limits, authentication requirements, or error conditions. The behavioral context is good but not comprehensive for a tool with no annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly structured and concise. It starts with the core purpose, then provides clear usage guidelines in bullet-point format. Every sentence earns its place by adding value - no repetition or fluff. The information is front-loaded with the most important details first.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (4 parameters, no output schema, no annotations), the description does well but has some gaps. It explains the tool's purpose and usage excellently, but doesn't describe the return format or what 'relevance-scored excerpts' look like. For a retrieval tool with no output schema, more detail about the response structure would be helpful. However, the strong usage guidelines compensate somewhat.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description doesn't add any parameter-specific information beyond what's in the schema. It mentions the tool retrieves from 'SharePoint/OneDrive' which aligns with the data_source parameter, but this is already covered in the schema. Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Retrieve raw text chunks from M365 for YOUR AI to reason over.' It specifies the verb ('retrieve'), resource ('raw text chunks from M365'), and distinguishes from sibling tools by contrasting with m365_chat for different use cases. The description explicitly mentions SharePoint/OneDrive sources and the tool's role in providing source text for analysis.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides excellent usage guidelines with explicit 'Use for' and 'Use m365_chat instead for' sections. It clearly distinguishes when to use this tool (custom analysis, source text needs, deep research) versus alternatives (quick Q&A, calendar/email questions, people lookup). The guidelines are specific and actionable, helping the agent choose between sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• m365_chatA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key traits: it's a Q&A tool that synthesizes answers from multiple sources, supports multi-turn conversations via conversation_id, and includes web search grounding with sensitivity considerations. However, it lacks details on rate limits, authentication needs, or error handling, which are important for a tool interacting with enterprise data.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, followed by bulleted lists for usage guidelines, making it highly scannable and efficient. Every sentence earns its place by providing essential information without redundancy, such as distinguishing from siblings and explaining parameter implications in a structured way.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (interactive Q&A with enterprise data), no annotations, and no output schema, the description does a good job of covering purpose, usage, and behavioral context. However, it could be more complete by mentioning potential limitations (e.g., response format, error cases) or prerequisites, which would help the agent handle edge cases better.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters well. The description adds value by explaining the tool's purpose and usage context, which helps interpret the parameters (e.g., 'message' for questions, 'conversation_id' for follow-ups, 'web_search' for grounding). It doesn't add specific parameter details beyond the schema, but provides meaningful context that compensates for the lack of output schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Quick Q&A with M365 Copilot' that 'Gets synthesized answers from email, calendar, Teams, SharePoint, OneDrive' and 'Supports multi-turn conversation.' It specifies the exact resources (email, calendar, Teams, SharePoint, OneDrive) and distinguishes it from sibling tools like m365_retrieve by emphasizing synthesized answers versus raw source text.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('Use for: - People questions, - Meeting schedules and availability, - Email summaries, - Enterprise facts and policies') and when not to use it ('Use m365_retrieve instead when: - You need raw source text, - You want to control reasoning, - You need cross-document analysis'). This includes clear alternatives and exclusions, helping the agent choose correctly among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• m365_searchA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes what the tool returns ('Returns file metadata, previews, URLs—not full content') and its scope ('OneDrive only'), which are crucial behavioral traits. However, it lacks details on potential rate limits, authentication needs, or error handling, leaving some gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and concise, with clear sections for purpose, usage guidelines, and limitations. Every sentence adds value, such as distinguishing from m365_retrieve and specifying the search scope, without unnecessary repetition or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (3 parameters, no output schema, no annotations), the description is mostly complete. It covers purpose, usage, limitations, and behavioral traits. However, without an output schema, it could benefit from more details on the return format (e.g., structure of metadata), though it does mention what is returned (metadata, previews, URLs).

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the baseline is 3. The description adds value by explaining the hybrid search approach ('semantic + keyword hybrid search') and providing usage examples in the 'Use for' section, which enhances understanding beyond the schema. However, it doesn't explicitly detail parameter interactions or advanced usage scenarios.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Find documents in OneDrive using semantic + keyword hybrid search.' It specifies the resource (documents in OneDrive) and the method (semantic + keyword hybrid search), distinguishing it from sibling tools like m365_retrieve, which retrieves actual content rather than metadata.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool (e.g., 'Discovering relevant files,' 'Finding documents by topic when you don't know exact names') and when to use an alternative ('Use m365_retrieve instead when: You need actual document content, You want text for analysis'). It also mentions a limitation ('OneDrive only (SharePoint search coming)'), offering clear context for usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.