Server Quality Checklist

Tool Scores

• get_thesis_coursesC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description carries full behavioral burden. It reveals the tool is a read operation ('Retrieve') but gives no details on pagination, result limits, error conditions, authentication needs, or what 'all' means in practice. The low effort of simply saying 'retrieve all' does little to disclose nontrivial behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no filler. The core action and scope are immediately understandable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The presence of an output schema covers return values, and the simple 2-parameter read operation is moderately self-explanatory. However, the lack of usage guidance, sibling differentiation, and any behavioral caveats leaves an agent with important gaps when deciding when to call this tool and what to expect for edge cases.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description must compensate. It maps 'department' and 'semester term' to the two parameters, but does not explain expected formats, allowed values, or how they combine to filter results. The description adds only minimal meaning beyond the parameter names themselves.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('Retrieve') and resource ('all thesis work courses'), scoped by department and semester term. However, it does not explicitly differentiate from sibling tools like list_program_courses, leaving some ambiguity about what distinguishes a 'thesis work course' from other course types.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. It does not mention siblings, prerequisites, or any conditions that would make this tool the right choice over list_program_courses or other course retrieval tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_course_replacementsB
  3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations are absent, so the description carries the full burden of behavioral disclosure. 'Retrieve' implies a read-only operation and the parenthetical clarifies the domain concept, but the description does not address what happens when a course has no replacements, edge cases, auth requirements, or rate limits. The burden is only partially met.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, front-loaded sentence with no filler — the verb and resource appear first and the parenthetical adds useful domain context efficiently. It is concise without being tautological.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  An output schema exists, so return values need not be described, and the core purpose is adequately stated. Yet with 0% parameter coverage, no usage exclusions, and no annotations, an agent still lacks guidance on parameter formats and sibling selection — clear gaps for a tool with three required parameters.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description barely compensates: it only hints that the three codes jointly identify a course. It does not explain formats, constraints, or relationships between department_code, semester_code, and course_code (e.g., how a semester code is formatted or whether department_code is an abbreviation). The self-explanatory parameter names carry most of the weight.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Retrieve') and a specific resource ('equivalent and auto-replacement courses (Denk Dersler) for a specific METU course'). The domain term 'Denk Dersler' gives it a distinct identity from siblings like get_course_prerequisites and get_course_info, though it does not explicitly name or contrast any sibling.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  'For a specific METU course' implies the usage context — an agent can infer this applies when a concrete course is identified rather than a whole program or department. However, there is no explicit guidance about when not to use it or which sibling (e.g., list_program_courses, get_course_prerequisites) to prefer instead.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_course_infoB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description bears the full burden and does disclose the type of content returned (sections, instructors, syllabus status, etc.). However, it does not mention edge-case behavior, auth needs, or what happens when the course is not found, leaving those as gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with a clear verb and resource, followed by a list of informative return components. There is no redundant phrasing or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The output schema covers return-value details, and the required parameter flags handle input specification. However, without usage guidance or parameter descriptions, an agent may not know how to format semester_code or where to obtain the codes; sibling tools like get_departments_and_semesters exist but are not pointed to.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not compensate by explaining the parameter formats, relationships, or usage of department_code, semester_code, and course_code. The parameter names are somewhat self-explanatory, but the description itself adds no semantic value beyond calling the target a 'specific METU course'.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Retrieve' and identifies the resource as detailed course information for a specific METU course, listing included components (sections, instructors, syllabus status, announcements, lecture schedules). This clearly distinguishes it from sibling tools like get_course_prerequisites or list_program_courses, but it does not explicitly name them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to choose this tool over alternatives. An agent must infer from the name and sibling list that get_course_info is for broad course details, while get_course_prerequisites or get_course_replacements serve narrower needs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_student_courses_by_categoryB
  3.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It clearly indicates a read-only retrieval operation and gives helpful category_id examples. However, it does not disclose whether both parameters are expected together, what happens when optional parameters are omitted, or any hidden constraints, so behavioral transparency is only partially fulfilled.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single focused sentence that front-loads the core action and resource. The parenthetical examples are useful and not redundant. There is no fluff or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no annotations, 0% schema description coverage, and one entirely unexplained parameter. Even though an output schema exists, the description is not complete enough for an agent to confidently call the tool with correct semantics, especially regarding how program_type interacts with category_id and when each parameter is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains category_id through examples ('1-236' or '2-236'), but program_type is not mentioned at all. With two parameters and no schema descriptions, leaving one parameter completely unexplained is a significant gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Retrieve'), a resource ('list of courses'), and a clear filtering criterion (student's category). It includes concrete examples of category_id formats, making the purpose understandable. It does not explicitly contrast with sibling tools like list_program_courses or get_student_course_categories, so it misses full sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The intended use is implied: call this when you need courses for a student's category. However, there is no explicit guidance on when not to use it, nor any mention of alternative sibling tools such as get_student_course_categories or list_program_courses. The usage context is inferable but not spelled out.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_program_coursesA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description owns the behavioral disclosure. It communicates a read-only listing intent and the returned columns, which is useful, but it is silent on expected code formats, error behavior for unknown department or semester codes, and whether very large catalogs are truncated or paginated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with action and scope, with no filler or repetition of the tool name. The second sentence's field list gives the agent quick expectations even though an output schema exists.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter tool with an output schema, the core purpose and inputs are covered. The gaps are missing code-format conventions and routing guidance against eight siblings, both of which could cause first-try invocation errors.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With schema description coverage at 0%, the description must compensate, and it does partially by confirming that both parameters are METU-specific codes and jointly select the course list. It stops short of format hints (e.g., 'CENG' vs '571' for department_code, or how semester_code is encoded), so the agent may guess the wrong shape.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb plus resource: 'List all offered courses for a given METU department code and semester code,' and enumerates the exact return fields (Course Code, Course Name, ECTS Credits, Credits, Level, and Type). This scope inherently separates it from siblings such as get_thesis_courses, get_course_prerequisite, and get_student_courses_by_category, which target different course subsets or detail levels.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  When to use the tool is inferable from the stated purpose — anytime an agent needs the full course catalog for one department in one semester — but nothing in the text states when not to use it or names alternatives like get_course_info for single-course lookups. An agent must infer the boundary from sibling names rather than being told.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_course_prerequisitesA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations present, the description carries the full burden of behavioral disclosure. It correctly conveys a read-only retrieval and enumerates the data returned, but does not address error behavior, input format expectations, authentication requirements, or effects for missing courses. The verb 'Retrieve' implies no side effects, but the description is otherwise on the minimal side.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence. It front-loads the verb and object, lists the specific data types retrieved, and ends with the scope ('for a specific METU course'). There is no unnecessary information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given an output schema exists, the return values do not need to be explained. However, with no annotations and minimal parameter semantics, the description leaves input format and edge-case behavior unspecified. It is adequate for a simple retrieval tool but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, but it only says 'for a specific METU course' without explaining the role or format of any individual parameter. department_code, semester_code, and course_code are self-explanatory in name, but no additional meaning is added to help an agent construct valid arguments.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Retrieve') and names a clear resource: prerequisite course requirements, prerequisite set groups, and minimum required letter grades such as DD. It also scopes the operation to a specific METU course, making it easy to distinguish from siblings like get_course_info or list_program_courses.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool: whenever prerequisite details for a METU course are needed. It does not explicitly exclude alternatives or mention when not to use it, but the purpose is specific enough that an agent can infer the appropriate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_departmentsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It does convey that the tool performs a keyword-based search and returns department codes, and the examples indicate partial text and numeric codes are accepted. However, it does not explain matching semantics such as case sensitivity, substring vs exact match, result limits, or empty-result behavior, so the behavioral picture is incomplete.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one concise sentence that leads with the core purpose and immediately supplies illustrative examples. No words are wasted, and the structure makes the tool's function easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with one required parameter and an output schema available, so the description does not need to detail return values. The description covers the core purpose and parameter usage sufficiently. It lacks only explicit guidance on when to prefer alternatives, which is a minor gap for a single-purpose search tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for the schema's bare 'query' string property. The description does add meaning by explaining that the query is a keyword for METU department codes and supplies examples, but it leaves the exact expected syntax and matching rules unspecified. It is helpful but not fully detailed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('Search'), a resource ('METU department codes'), and a mechanism ('by keyword'). Concrete examples ('Computer', 'Economics', 'Aerospace', '571') clarify that both textual names and numeric codes are valid. It is distinguishable from sibling tools like get_departments_and_semesters because it focuses on searching codes by keyword rather than listing all departments.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool should be used when the agent needs to find a department code from a partial name or code keyword, but it does not explicitly state when not to use it. It also does not mention alternatives such as get_departments_and_semesters for obtaining the full department list. Usage context is present but left to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_departments_and_semestersA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Even with no annotations provided, the description clearly signals a read-only retrieval operation with the verb 'Retrieve' and the scope 'all available'. It does not add richer behavioral context such as authentication requirements, rate limits, or what happens if the source is empty, so the description carries only partial behavioral burden.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler and includes a helpful example of the semester format. The parenthetical '64' in 'Program Course Details (64)' is unexplained but does not create significant bloat.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter tool with an output schema, the description sufficiently covers what the tool returns: department codes/names and semester terms. It could be more complete by explaining the 'Program Course Details (64)' reference and when to call this versus a sibling search tool, but nothing essential to invoking it is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and the input schema is empty, so there is no parameter documentation burden on the description. The description appropriately focuses on what the response will contain rather than input details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Retrieve all available...') and a clear resource: METU academic departments and semester terms. It also distinguishes itself from search_departments by emphasizing 'all available' rather than filtered search, and it provides a concrete semester format example.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The word 'all available' implies this is the tool to use when you need the complete unfiltered enumeration of departments and semesters. However, it does not explicitly mention alternatives or state when not to use it, such as preferring search_departments for targeted lookup.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_student_course_categoriesA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden of disclosing safety and side effects. It accurately describes this as a retrieval operation ('Retrieve') scoped to the logged-in student's data, making it clear that no mutation occurs. It also names the data source (Service 178) and enumerates the content returned, which adds useful behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that states the action, the scope, the key result types, and examples, with no filler or redundancy. Every element contributes to an agent's understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is sufficiently complete for a parameterless read-only tool with an output schema present. It specifies the scope (logged-in student), the data categories, and the source service. It could slightly improve by mentioning that the returned categories are intended as inputs for category-based queries, but nothing critical is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so there is nothing to explain. The schema coverage is trivially 100% and the description correctly avoids inventing parameters. This matches the baseline for parameterless tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the verb ('Retrieve'), the resource ('logged-in student's program types and curriculum course categories'), and provides concrete examples of the values returned (MAJOR, MUST COURSE, etc.). This distinguishes it from the sibling get_student_courses_by_category, which retrieves courses rather than the categories themselves.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this is a lookup/context tool that returns the set of categories available to the logged-in student, likely to be used before filtering courses by category. However, it does not explicitly state when to use this tool versus alternatives such as get_student_courses_by_category or list_program_courses, leaving the routing decision to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.