Skip to main content
Glama

IA-QA — 152 QA & Dev Tools for AI Agents

cot_analyzer

Read-onlyIdempotent

Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation.

Input Schema

NameRequiredDescriptionDefault
reasoningYesThe CoT / reasoning trace text (e.g. from <think> tags or step-by-step output)
expected_conclusionNoExpected final answer to check against (optional)

Output Schema

NameRequiredDescriptionDefault
markersNo
step_countNo
total_charsNo
total_linesNo
has_conclusionNo
reasoning_depthNo
backtracking_signalsNo
reasoning_depth_labelNo
conclusion_matches_expectedNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • addedInput schema / examples
      Added value: +[
      +  {
      +    "expected_conclusion": "7",
      +    "reasoning": "Step 1: 12 apples minus 5 eaten.\nStep 2: 12 - 5 = 7.\nTherefore, the answer is 7."
      +  }
      +]
  2. Changed1 schema field changed
    • addedOutput schema / properties
      Added value: +{
      +  "backtracking_signals": {},
      +  "conclusion_matches_expected": {},
      +  "has_conclusion": {
      +    "type": "boolean"
      +  },
      +  "markers": {},
      +  "reasoning_depth": {},
      +  "reasoning_depth_label": {},
      +  "step_count": {
      +    "type": "number"
      +  },
      +  "total_chars": {
      +    "type": "number"
      +  },
      +  "total_lines": {
      +    "type": "number"
      +  }
      +}
  3. Changed1 schema field changed
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
      +  "additionalProperties": true,
      +  "type": "object"
      +}
  4. First observed

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true and destructiveHint=false, so the safety profile is covered. The description adds real behavioral value beyond that by listing exactly which signals the analysis emits (step count, logical flow, backtracking, depth estimate). It does not say anything about thresholds, scoring scale, or what happens with an unstructured trace, but given the annotation coverage that is a tolerable gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the core action, and each sentence earns its place: one states what is analyzed and what is detected, the second gives the target use case. No filler or restated title.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, the description needn't explain return values, and annotations cover the read-only/idempotent profile; the description fills in the analytical scope well. The remaining shortfall is that it doesn't set expectations about input quality (e.g. that traces without explicit steps are handled differently), but overall an agent has enough to call this correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and both parameters carry their own descriptions, so the baseline of 3 applies. The description never mentions 'expected_conclusion' or the optional comparison behavior it enables, adding essentially nothing about parameters beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('Analyze') and resource ('Chain-of-Thought (CoT) or reasoning trace from an LLM') and enumerates the outputs it produces: step count, logical flow, conclusion presence, backtracking, reasoning depth. It is clear what it does, but it never names a sibling it is distinct from, so the agent must infer the boundary with analyze_responses, response_quality_score, or validate_agent_trajectory.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The sentence 'Useful for o1/o3/DeepSeek-R1 evaluation' implies a context (reasoning-model traces) but not a decision rule. There is no when-not guidance, no prerequisite (e.g. trace must be a CoT rather than a final answer only), and no routing to or away from overlapping siblings such as analyze_responses or response_quality_score.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.