tokenize

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations to rely on, the description still discloses key behavior: the fallback ordering, that a forced but unavailable engine raises an actionable install hint rather than silently degrading, and the lang auto-detection default. It omits return-shape and performance details, though an output schema covers the return shape.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The core purpose and the character-offset guarantee are front-loaded in one sentence, followed by only the engine and lang semantics an agent needs. No filler or repeated information from the schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values needn't be described; for the 3-parameter surface the description covers engine selection, fallback, error behavior, and lang accepted values. The remaining gap is interaction with sibling tools and any limits on input size.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description carries the burden, and it does: it enumerates the engine values (jieba/spacy/stanza) and lang values (auto/zh/en/any stanza code) that the bare schema lacks entirely. It doesn't explicitly define what engine="auto" resolves to at call time beyond implication.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

"Segment text into tokens with exact character offsets" gives a specific verb (segment into tokens) plus the differentiating detail of exact character offsets. That distinguishes it from segment-adjacent siblings like sent_split, pos_tag, and noun_phrases without ambiguity.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description clearly establishes default engine selection per language (Chinese→jieba, English→spaCy/stanza/regex fallback) and explains how to override it with engine. It doesn't explicitly compare against sibling tokenization-adjacent tools such as sent_split or pos_tag, so it falls short of full when/when-not routing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.