Server Quality Checklist

Tool Scores

• traceA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses read-only and deterministic behavior, the network/cache side effect on first use, CPU default performance, and the structured error return on failure. This is rich behavioral context far beyond the basic facts.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is lengthy but every sentence earns its place: purpose, use case, model support, constraints, failure behavior, and parameter definitions. It front-loads the primary purpose and then methodically covers context, making it efficiently structured for its complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity, the description covers all essential context: output schema elements, failure modes, network behavior, performance caveats, and supported model families. It is complete enough for an agent to invoke correctly and interpret results, especially with the detailed return signature.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has no parameter descriptions (0% coverage), so the description must compensate. It explains 'model' with accepted name examples, 'prompt' as input text, and 'top_k' with default and meaning (caps how many top heads and neurons). This adds meaning well beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Run a forward pass... and report which attention heads and MLP neurons were most responsible for its predicted next token.' It clearly answers 'why did the model predict X' for a specific prompt, distinguishing it from sibling tools like activations, patch, and circuit.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Call this to answer why did the model predict X for a specific prompt,' which is clear when-to-use guidance. It also provides when-not-to-use constraints (only TransformerLens-supported open-weight models, not closed-source APIs). However, it does not name alternative sibling tools for other interpretability tasks, so it lacks explicit alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• activationsA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, but description discloses read-only and deterministic behavior, the only side effect (HuggingFace Hub caching on first use), CPU default, output size scaling with model depth, and structured error handling on failure. This fully covers operational behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but every sentence adds value. It is front-loaded with purpose, then flows through usage guidance, constraints, side effects, failure mode, parameters, example, and return structure. No redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity and minimal schema, the description is highly complete. It explains the output schema in detail (schema_version, operation, model, prompt, n_tokens, activations list), covers failure behavior, scalability, and side effects, leaving no significant gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but description compensates fully by defining both parameters: 'model (str), any name HookedTransformer.from_pretrained accepts, e.g. gpt2' and 'prompt (str), the input text.' It also gives an explicit example call, making parameter meaning unambiguous.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Opening phrase 'Dump raw per-layer activation summary statistics' clearly states the action and target, listing specific statistics and components (residual stream, MLP, attention). It distinguishes itself from sibling trace by explicitly stating it is used when 'trace's top-k ranking isn't enough detail'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly directs when to call: 'Call this when trace's top-k ranking isn't enough detail and you need the raw scale/shape of a specific hook point before deciding what to inspect further or patch with the patch tool.' Also names alternatives (trace, patch) and notes the same model constraint as trace.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• circuitA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden and does so excellently: it discloses read-only nature, caching/network/CPU behavior, higher cost than trace, determinism, failure behavior (structured error object), and the methodological limitation that it does not capture interaction effects. This goes far beyond simple mutation/read hints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but every sentence earns its place: purpose, usage, exclusions, behavioral traits, parameter details, example, and return schema. It is front-loaded with the core mechanism, uses clear paragraph separation, and contains no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity and presence of an output schema, the description is complete: it covers what the tool does, when to use it, alternatives, limitations, operational behavior, parameter semantics, and the return JSON structure. The output schema is present but the description still enriches it by explaining each field's meaning.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description explicitly enumerates all three parameters with types and semantics: model (str), prompt (str), and top_k (int, default 10) explaining 'how many top-attributed components to test via ablation.' The example call further clarifies expected usage, fully compensating for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb+resource+method: 'ranks candidate attention heads and MLP neurons by direct logit attribution, then measures each candidate's individual causal effect via single-component zero-ablation.' It also distinguishes from siblings by explicitly referencing trace's correlational ranking and patch, making the tool's unique purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit when-to-use guidance is given: 'Call this when trace's correlational ranking isn't enough and you want a causal pass across multiple candidates without manually calling patch on each one.' It also states when NOT to use it ('This is NOT full path-patching...') and points to an alternative ('use a dedicated tool such as Anthropic's circuit-tracer instead').

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• patchA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully discloses behavioral traits: read-only in-memory effect, no persistence, determinism, failure modes returning structured errors, and dependency on trace's caching/CPU defaults. It also clarifies what 'read-only' means, which is critical for a tool named 'patch.'

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Though long, the description is dense and logically organized: purpose, usage context, behavioral guarantees, error handling, parameters, example, and return fields. Given the tool's complexity and lack of annotations, every sentence serves a purpose and the key information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, usage, safety, error behavior, parameter semantics, and output shape. It also references sibling tools appropriately. Even with an output schema present, the description adds essential context about causal intervention, determinism, and failure handling.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description defines every parameter: model as str, prompt as str, layer as zero-indexed transformer block, and component with its full enum of allowed values. An example call grounds the semantics concretely.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource: 'Zero-ablate one component ... and report how the predicted token and its logit changed.' It clearly distinguishes from siblings by stating it does not search for candidates itself, and explicitly frames it as a causal intervention after trace/circuit.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance is provided: 'use it to test whether a component trace or circuit flagged as correlated ... is actually causally responsible,' and 'Call it after trace or circuit has surfaced a candidate layer/component; it does not search for candidates itself.' This gives clear when-to-use and relationship to alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.