Skip to main content
Glama

aa-mcp

ci

MCP server for artificialanalysis.ai — live AI model benchmarks, pricing, speed, context/parameter specs, provider endpoints and the image/video/speech/music arenas — shaped for agents that pay per token.

No API key needed. Two interchangeable data backends, same records out:

aa:rsc (default)

aa:api (optional)

source

the site's Next.js RSC payloads

GET /api/v2/language/models/free

needs

nothing

AA_API_KEY (free tier is enough)

adds

~50 metrics per model incl. 17 cross-model benchmarks, pricing, speed, per-model cost-per-task breakdown, provider endpoints, media boards

six Capability Indexes + Coding/Agentic indexes (not in the scrape)

quota

none

100 req/24 h → cached 6 h by default

If a key is present the API is tried first and the server falls back to scraping on 401/403 rather than failing. Every response header shows which backend answered.

Install

One line, no clone (uv fetches and runs it for you):

uvx --from git+https://github.com/HenryGetz/aa-mcp aa-mcp

Or install the aa-mcp executable once:

uv tool install --from git+https://github.com/HenryGetz/aa-mcp aa-mcp   # -> ~/.local/bin/aa-mcp

Local checkout / development:

git clone <this repo> && cd aa-mcp
uv sync                      # creates .venv with the mcp SDK
uv run aa-mcp                # serve on stdio

Register with an MCP client (omp, Claude Code, Codex, …):

{
  "mcpServers": {
    "aa": {
      "type": "stdio",
      "command": "/home/USER/.local/bin/aa-mcp",
      "timeout": 120000
    }
  }
}

Related MCP server: aa-mcp

Tools

tool

answers

aa_leaderboard

smartest / best / cheapest / fastest models, best value (mode=value), Pareto frontier (mode=frontier, delegates to aa_pareto), budget and capability filters

aa_model

everything AA knows about one model: indices, benchmark scores, pricing incl. cache, speed, cost-per-task breakdown, license, modalities, cutoff

aa_pareto

the trade-off frontier for any score vs any x-axis (cost per task, whole-index cost, output tokens, time, context, params, TTFT, in/out price) with the knee — no per-benchmark tool needed

aa_compare

2-12 models side by side on the metrics you care about (metrics=all/benchmarks/list, view=diff for deltas, spec= for a reproducible YAML/JSON round-trip)

aa_benchmarks

the search index over everything rankable (benchmarks, indexes, derived metrics, arenas): what it measures, its unit, live coverage and the exact How to query call; filter with query=/domain=/tag=

aa_benchmark_leaderboard

who leads a given benchmark (full leaderboard when AA publishes one, otherwise AA's chart set — stated in the header; lower-is-better metrics rank lowest-first)

aa_providers

which provider serves a model, at what price, quantization, TTFT and feature support; or the whole provider × model board

aa_media

text-to-image, image-editing, text-to-video, image-to-video, video-editing, TTS, music, speech-to-text, realtime voice (speech_to_speech == realtime_voice)

aa_agents

AA's coding-agent, chatbot and search-API leaderboards

aa_search

resolve a fuzzy model name to real names + slugs (use it before guessing)

aa_reference

metric definitions, index composition, field names, provenance

aa_raw

raw JSON escape hatch for anything the tables do not cover

Resources (15): aa://reference/metrics (full definitions), one per topic (overview, indexes, benchmarks, pricing, speed, providers, media, fields, sources), aa://benchmarks (registry JSON), aa://rankables and aa://rankables.yaml (every rankable thing, for building your own queries), aa://comparisons/spec (a aa_compare spec= template) and aa://sources. Prompts: best_model_for_task, compare_models.

Comparing models

Everything AA's own comparison pages do, without the pages:

# same models as artificialanalysis.ai/models/comparisons?compare=a,b,c,d
aa_compare '{"queries":["mimo-v2-6-pro","deepseek-v4-1-flash","gpt-6-luna-xhigh","muse-wonder-full"]}'

# every benchmark AA has for those models, with deltas vs the best
aa_compare '{"queries":[...],"metrics":"all","view":"diff"}'

# reproducible / reusable comparison (YAML or JSON); the output echoes the spec back
aa_compare '{"spec":"models: [mimo-v2-6-pro, kimi-k3]\nmetrics: [intelligence, cost_per_task, gpqa, terminalBench40]\nview: diff"}'

Trade-off questions always have one call:

aa_pareto '{}'                             # Intelligence vs $/task frontier + knee
aa_pareto '{"x":"index_total_cost"}'       # vs cost of the whole index run
aa_pareto '{"x":"out_tokens_per_task"}'    # vs verbosity
aa_pareto '{"score":"cyberIndex","x":"cost_per_task"}'

Finding the right benchmark for a domain, without knowing its name:

aa_benchmarks '{"domain":"security"}'   # -> CWE-Bench, CyberGym, Cyber Index, …
aa_benchmarks '{"query":"speech"}'      # -> STT/TTS/STS boards + the exact call to make

Token discipline

  • Tables, not JSON — rounded values, one header line, no argument echo.

  • Default 10-15 rows; every table tool accepts top and fields.

  • Long-form documentation lives in resources/reference, so it never costs context unless the agent asks for it.

  • aa_raw warns when used.

Measured live (12-call panel, one call per tool): ≈3.3k tokens total, ≈277 per call, e.g. a 5-row leaderboard is ~180 tokens, a full model digest ~470, a media board ~160, the aa_benchmarks search index ~235, aa_reference topic=overview ~915 (the one call that is deliberately long-form).

Example — aa_leaderboard {"mode":"value","top":5} (verbatim):

# 5 of 109 scored by value (700 tracked) · value = Intel per $/task, active only · stale: 23m ago · aa:rsc
| Model | Slug | Creator | Intel | $/Task | tok/s | TTFT | Ctx | Intel/$ |
|---|---|---|---|---|---|---|---|---|
| GPT-6 Luna (Low) | gpt-6-luna-low | OpenAI | 21.5 | $0.0045 | 121.9 | 2.6s | 1.0M | 4766.0 |
| GPT-6 Luna (Medium) | gpt-6-luna-medium | OpenAI | 29.9 | $0.02 | - | - | 1.0M | 1712.3 |
| GPT-6 Luna (Non-reasoning) | gpt-6-luna-non-reasoning | OpenAI | 18.5 | $0.01 | 112.3 | 0.9s | 1.0M | 1627.8 |
| Granite 4.2 3B | granite-4-2-3b | IBM | 9.1 | $0.0060 | 215.4 | 0.5s | 131k | 1513.0 |
| Claude Haiku 5.5 (Low) | claude-haiku-5-5-low | Anthropic | 29.5 | $0.02 | 170.0 | 9.2s | 1.0M | 1204.1 |
— 514 rows skipped: no measured cost per task (AA has cost per task for 186/700 variants)

Coverage and limits

Covered: 700 model variants × ~50 leaderboard metrics merged with a 700 × 94 dataset (capability indexes, AA-Briefcase/GDPval Elo, AIME, LiveCodeBench, GDP.pdf, AutomationBench, Harvey LAB, MLCR, cost breakdowns, per-eval costs, licenses, effort labels); per-model digests with prompt-type speed breakdowns; 1,201 provider endpoints across 64 hosts with prices, quantization, latency and feature flags; 9 media arenas plus speech-to-text / streaming / realtime voice; AA's coding-agent, chatbot and search-API boards; a 95-entry rankable index (67 cross-model benchmarks plus indexes, derived cost/speed metrics and arenas) with descriptions, units, live coverage and the exact tool call for each.

Deliberately out of scope (and the server says so instead of guessing):

  • Benchmarks AA does not track (MathVista, DocVQA, BFCL) — no data exists.

  • Cross-model signature leaderboards for benchmarks AA only exposes as a chart subset: the tool labels these not a full leaderboard and reports real coverage.

  • Coding/Agentic indexes need the JSON API (AA_API_KEY).

  • Performance-over-time history and raw per-provider measurements are Commercial-tier API features; not implemented.

  • Image/video/music model detail pages (per-family specs beyond arena rows) and hardware/inference-stack pages are not exposed.

Environment

variable

default

meaning

AA_API_KEY

unset

AA JSON API key; enables the aa:api backend

AA_SOURCE

auto

auto | api | rsc — backend selection

AA_CACHE_DIR

~/.cache/aa-mcp

payload cache

AA_CACHE_TTL

3600

scrape cache TTL (seconds)

AA_API_TTL

21600

API cache TTL (seconds)

AA_RICH_TTL

21600

rich-dataset (model-page overlay) cache TTL

AA_MAX_BYTES

50331648

decoded-body cap (gzip-bomb guard)

AA_HTTP_TIMEOUT

30

per-attempt HTTP timeout (seconds)

AA_HTTP_DEADLINE

45

total budget for one fetch (seconds)

AA_OFFLINE

unset

1 — serve from cache only, never touch the network

AA_BASE_URL

https://artificialanalysis.ai

override host (tests/mirrors)

AA_LOG_LEVEL

WARNING

stderr log level

Behaviour worth knowing

  • Freshness is labelled. Payloads are cached for an hour; the header shows the age and says stale when a network failure forced a cached copy. Pass refresh=true for freshly released models.

  • Coverage is labelled. AA scores a benchmark on a subset of models; tables state how many of the 700 tracked variants carry a value, and warn when a filter dropped rows that simply have no measurement.

  • Cost per task is cache-aware and must not be recomputed from $/1M rates; raw rates are separate fields.

  • Effort variants (-low … -xhigh) are separate rows — always name the effort.

  • Replies are bounded and cells are neutralized. Scraped text is sanitized (markdown metacharacters escaped, 120 chars per cell) and any reply is capped at 200 kB, so one hostile page cannot blow up your context; oversized tables say how many rows were kept.

  • Hostile arguments are rejected up front, not passed to a fetch or a cache path: control characters / lone surrogates, over-long strings (>256 chars, 2000 for fields) and non-finite numbers (NaN/Infinity) return a ValueError naming the argument, and error text never echoes more than 60 chars of your input.

  • AA does not track MathVista, DocVQA or BFCL; the server says so instead of inventing.

Troubleshooting

  • Cloudflare / bot challenge (403/503 or a tiny body): the error names the exact cache path. Render the URL in a browser, save the HTML there, re-run — or set AA_OFFLINE=1 to keep working from the last good capture.

  • Site structure change: the error says which array was missing; the parsers live in src/aa_mcp/loaders.py (flight.py finds the arrays).

  • AA redeploys often — never hardcode chunk or build-id URLs; the client always re-parses freshly fetched payloads.

  • Cache size: payloads are stored verbatim (a rich model page is ~3 MB, so the cache settles around 50-120 MB). Clear it with python -c "from aa_mcp.fetch import clear_cache; print(clear_cache())" — worst case the next call refetches.

Development

uv run pytest -q                 # offline: fixtures in tests/fixtures (real payloads)
uvx ruff check .                 # lint (CI runs this)
.venv/bin/python eval/client.py tools
.venv/bin/python eval/client.py call aa_leaderboard '{"top":5}'
.venv/bin/python eval/client.py bench                 # per-call chars/tokens report
.venv/bin/python scripts/surface_sweep.py             # every tool × every documented enum, live
.venv/bin/python scripts/gen_reference_md.py --check  # docs/REFERENCE.md in sync

scripts/surface_sweep.py is the end-to-end guard: it drives the real stdio server through ~220 calls covering every tool, board, sort key, axis, section, topic and raw kind, and fails if anything errors unexpectedly (it is how the speech-board NameError was caught).

docs/REFERENCE.md is generated from src/aa_mcp/reference.py (scripts/gen_reference_md.py); docs/INTERFACES.md documents the internal data model. eval/ holds the evaluation harness and every audit report: the independent field-by-field parser audit (eval/reports/fidelity.md), the code review (eval/reports/audit_quality.md), the adversarial attack report (eval/reports/adversarial.md) and five rounds of agent usage reports (eval/runs/round*.md).

Available Tools

12 tools
aa_agentsAgent leaderboardsB

AA agent leaderboards: coding agents, chatbots and search APIs.

Args: board: coding_agents (default), chatbots or search_api. top: rows to return (default 15). refresh: bypass the cache and re-fetch.

ParametersJSON Schema
NameRequiredDescriptionDefault
topNo
boardNocoding_agents
refreshNo

TDQS

B3.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must carry the full behavioral burden. It does disclose caching behavior ('refresh: bypass the cache and re-fetch'), which is genuinely useful, but says nothing about auth requirements, rate limits, or that this is a read-only retrieval (implied but unstated). Partial coverage of a no-annotation tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded one-line summary followed by a compact args block; each line earns its place. The args section is slightly redundant with schema defaults but no sentence is wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema and no annotations exist, so the description is the sole source of information. It covers inputs well but never describes what a returned 'row' contains or the leaderboard structure, leaving the agent unable to anticipate the response shape.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, and it does: all three parameters are documented, including the valid values for 'board' (coding_agents, chatbots, search_api) which are absent from the schema and not constrained by any enum. Defaults for top and board are restated, which is minor redundancy.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States the resource clearly (agent leaderboards) and enumerates the three content types it covers (coding agents, chatbots, search APIs). An agent understands the deliverable, but the definition never distinguishes itself from close siblings like aa_leaderboard or aa_benchmark_leaderboard, so the boundary is left to inference.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no statement of when to use this tool versus the eleven siblings, several of which (aa_leaderboard, aa_benchmark_leaderboard) sound nearly identical. The only routing clue is the word 'agent' in the description, which the agent must decode on its own.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

aa_benchmark_leaderboardBenchmark leaderboardA

Rank models on one benchmark, stating the source and coverage honestly.

Auto-selects the richest source: the full 700-row leaderboard when the field exists there, else AA's curated eval chart set, else the homepage panel; the header says which and warns when it is not a full leaderboard.

Args: benchmark: benchmark key, label or eval slug (e.g. gpqa, gpqa-diamond, "GPQA Diamond", briefcaseElo). top: rows to return (default 15). fields: comma list of columns to show instead of the default set. creator: substring match on the creator/lab. open_weights: True/False to keep only open- or closed-weights models. min_intelligence: drop models below this Intelligence Index value. include_estimates: include AA-estimated scores (default True); estimates are marked with * and footnoted. refresh: bypass the cache and re-fetch.

ParametersJSON Schema
NameRequiredDescriptionDefault
topNo
fieldsNo
creatorNo
refreshNo
benchmarkYes
open_weightsNo
min_intelligenceNo
include_estimatesNo

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so the description carries the full burden. It discloses the auto-source-selection behavior, that headers warn about incomplete data, that estimates are marked with * and footnoted, and that refresh bypasses cache. Strong behavioral disclosure, though it doesn't state auth needs or rate limits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with purpose and auto-selection logic, then args in a clear list. Every sentence earns its place, though the eval slug examples in the arg list are slightly verbose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite 8 params, no annotations, and no output schema, the description covers purpose, source-selection behavior, estimate marking, cache behavior, and all parameters. Missing only auth/rate-limit context or return-format hints, but otherwise complete for an agent to invoke correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate entirely, and it does: it documents all 8 params with meaning beyond types (benchmark accepts key/label/slug, top default 15, fields as comma list overriding defaults, creator substring match, open_weights filter, min_intelligence threshold, include_estimates default True with marking behavior, refresh bypasses cache).

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (rank) and resource (models on one benchmark) and explicitly names the honesty constraint about source/coverage. Distinct from siblings like aa_leaderboard (general) and aa_model (single model detail).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly explains the auto-selection hierarchy (700-row leaderboard > curated eval chart > homepage panel) and warns when it falls back to a non-full leaderboard. This tells the agent exactly when this tool is appropriate versus alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

aa_benchmarksBenchmark & metric search indexA

Search everything rankable: benchmarks, indexes, derived metrics, arenas.

Each hit carries its How to query hint (the exact tool call), so finding "Artificial Analysis Speech to Speech Index" immediately gives you aa_media board=speech_to_speech.

Args: query: tokenised AND-match over key/label/aliases/description/tags (e.g. "speech", "cyber", "long context"). domain: one domain name (coding, math, science, security, agentic, tool-use, long-context, instruction-following, knowledge, reasoning, safety, multimodal, speech, media, music, video, business, finance, legal, medical, engineering, economics, cost, speed, openness, ...). tag: one raw tag (a narrower filter than domain=). verbose: also show what each thing measures.

ParametersJSON Schema
NameRequiredDescriptionDefault
tagNo
queryNo
domainNo
verboseNo

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral burden. It does add real value by disclosing that every hit embeds a 'How to query' hint (the exact follow-up tool call), which is a non-obvious output behavior. It says nothing about result limits, pagination, ordering, or whether the index is read-only, so the disclosure is partial.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded one-line purpose, then the key output affordance, then a scoped Args block. Every sentence earns its place and the concrete 'Artificial Analysis Speech to Speech Index' → aa_media example is high-value rather than filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 4-parameter, annotation-free, output-schema-free search tool, the definition covers purpose, filtering dimensions, and hit contents. It leaves minor gaps around result volume, ordering, and pagination, but nothing that would cause an agent to invoke it incorrectly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate and it largely does: it defines query's matching semantics (AND over key/label/aliases/description/tags), enumerates realistic domain values, explains that tag is a narrower filter than domain, and says verbose surfaces what each thing measures. Only the lack of a tag vocabulary or worked examples keeps it short of complete.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource set: 'Search everything rankable: benchmarks, indexes, derived metrics, arenas.' An agent can tell this is a discovery/search index rather than a leaderboard tool. It does not, however, distinguish itself from the similarly-named sibling aa_search or explain how it relates to aa_benchmark_leaderboard, so sibling differentiation is only partial.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The Args block implies usage: tokenised AND-match queries, a domain filter, or a narrower tag filter, which tells the agent how to drive the tool. But there is no explicit statement of when to prefer this over aa_search, aa_benchmark_leaderboard, or aa_leaderboard, and no exclusions or prerequisites. Usage is inferable but not guided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

aa_compareCompare modelsA

Side-by-side comparison of 2-12 models on any benchmark (site parity).

Mirrors artificialanalysis.ai's comparison pages: models as columns (or a per-metric delta view), every rankable metric, and a reproducible YAML spec. Rows AA publishes only on an eval chart set / homepage panel (the cyber boards, AA-3D, …) are joined in and the source is footnoted; any cell that stays - is footnoted (a per-model missing on requested metrics: … count, the per-metric detail while the gap is small, or a whole-model "no published pricing or speed" line) so no gap is silent.

Media/speech arena models (e.g. "Grok Voice 3.0") also resolve: comparing only arena entries renders their board metrics (Elo / STS / WER, appearances, price, release, open weights) with a footer warning that per-board scores are not cross-comparable; mixing a language model with an arena entry is refused with an explanation.

Args: queries: 2-12 model names or slugs (language models, or arena entries on one board). metrics: "all" (every rankable key any selected model has), "benchmarks" (benchmark/index kinds only), or a comma list of keys/labels/aliases; empty uses the default rows (Intelligence, Intel/$, $/task, in/out $/1M, tok/s, TTFT, Context). For an arena comparison the names are arena fields (elo, sts, wer, ci, appearances, price, release, open_weights, rank, board, creator). view: "table" (models as columns) or "diff" (one row per metric, value plus the delta vs the first model, e.g. 57.6 (+4.9)). compact: default metric set only — keep the core rows instead of the full set; ignored (and stated in the footer) whenever metrics= is given. spec: a YAML or JSON document with models/metrics/view/compact/filters, merged with the explicit arguments; every reply ends with a copy-pasteable spec: block that reproduces the call (view and compact included, so the round-trip is lossless). refresh: bypass the cache and re-fetch.

ParametersJSON Schema
NameRequiredDescriptionDefault
specNo
viewNotable
compactNo
metricsNo
queriesNo
refreshNo

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and largely meets it: it discloses how missing cells are footnoted so no gap is silent, that arena scores are not cross-comparable, that mixing model types is refused, that refresh bypasses the cache, and that a reproducible spec block is always returned. It does not state auth needs or rate limits, leaving a small gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose and scope are front-loaded and the Args section is well organized, but the opening prose paragraph is long and dense with operational detail that arguably belongs in contextual notes rather than a purpose statement. Most sentences carry information, yet the description leans toward over-specification relative to a concise tool blurb.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema and no annotations, the description covers what an agent needs: input ranges, view modes and their rendered output (columns vs. per-metric deltas), the default metric set, arena behavior, refusal conditions, caching, and the guaranteed spec block. Nothing essential for a correct call is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate and it does: each of the six parameters is documented with accepted values and defaults (metrics accepts 'all'/'benchmarks'/comma lists, view is 'table'/'diff', compact is ignored when metrics is given, spec merges with explicit args, refresh bypasses cache). It adds semantics well beyond the bare schema types.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Side-by-side comparison of 2-12 models on any benchmark'), names the input domain (language models or arena entries), and mirrors a concrete external artifact (artificialanalysis.ai comparison pages). An agent can distinguish it from aa_leaderboard, aa_model, and aa_pareto without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the compare semantics and edge cases are described (arena-only comparisons render board metrics; mixing a language model with an arena entry is refused with an explanation). However, there is no explicit routing guidance telling the agent when to pick aa_compare over aa_leaderboard, aa_pareto, or aa_model, which is the core 'vs alternatives' question.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

aa_leaderboardModel leaderboardA

Rank and filter the ~700 tracked model variants — the default starting point.

Answers "which model is smartest / cheapest / fastest / best value right now". Returns a compact markdown table sorted by sort (default: intelligence).

Args: mode: "rank" (default), "value" (intelligence per $/task), or "frontier" (the Pareto intelligence-vs-$/task frontier — the same code path as aa_pareto, which also plots other x axes; the footnote gives the knee). sort: metric/alias to sort by — e.g. intelligence, cost_per_task, tps, ttft, context_tokens, price_in, price_out, params_b, release_date, or any benchmark key. Use aa_reference topic=fields for all names. fields: comma list of columns to show instead of the default set. top: rows to return (default 10; >15 drops the Creator column for brevity). creator: substring match on the creator/lab (e.g. "anthropic"). name: substring match on the model name/slug. open_weights: True/False to keep only open- or closed-weights models. reasoning: True/False to keep only reasoning or non-reasoning models. min_intelligence: drop models below this Intelligence Index value. max_cost_per_task: drop models above this $/task (cache-aware). min_speed: drop models below this median output tok/s. max_ttft: drop models above this median time-to-first-token (seconds). min_context: drop models with a smaller context window (tokens). released_after: ISO date (YYYY-MM-DD); keep models released on/after it. include_deprecated: include deprecated rows (default False). refresh: bypass the cache and re-fetch (use for "released today").

ParametersJSON Schema
NameRequiredDescriptionDefault
topNo
modeNorank
nameNo
sortNointelligence
fieldsNo
creatorNo
refreshNo
max_ttftNo
min_speedNo
reasoningNo
min_contextNo
open_weightsNo
released_afterNo
min_intelligenceNo
max_cost_per_taskNo
include_deprecatedNo

TDQS

A4.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden and adds meaningful behavior: returns a compact markdown table sorted by sort, default intelligence; top >15 drops the Creator column; refresh bypasses cache; frontier mode shares code path with aa_pareto. It stops short of stating read-only nature, auth needs, or rate limits, so not full 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads purpose and user questions before the Args list, and the Args block is tightly formatted with no redundant sentences. Length is justified by the 16 parameters.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 16 parameters, no annotations, and no output schema, the description covers output format, default sort, column behavior, caching, and all parameters. It leaves no critical gap for an agent to call it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, and it does: all 16 parameters are documented with defaults, allowed values, units, examples, and semantics (e.g., max_cost_per_task cache-aware, min_speed median output tok/s). It also points to aa_reference for the full sort alias list.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('Rank and filter the ~700 tracked model variants') and positions itself as the default starting point. Answers concrete user questions ('which model is smartest / cheapest / fastest / best value right now') and distinguishes itself from aa_pareto and aa_reference where relevant.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says this is the default starting point for leaderboard queries and names alternatives for specific needs: aa_reference for field names, aa_pareto for other x-axis plots. Mode descriptions clarify when to use rank, value, or frontier.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

aa_mediaMedia leaderboardsA

Human-preference arenas: image, video, speech, music (Elo + 95% CI).

Args: board: any image/video/TTS/music board key (text_to_image, image_editing, text_to_video, image_to_video, video_editing, text_to_speech, text_to_speech_controlled, music_instrumental, music_with_vocals) or a speech board (speech_to_text, speech_to_text_streaming, speech_to_speech — also accepted as realtime_voice / "realtime voice"). Hyphens/spaces/case are ignored (text-to-image == text_to_image). top: rows to return (default 15). fields: comma string (or list) of columns; supports ci_lower, ci_upper, ci (95% CI), open_weights, url. Elo CI is not shown by default. open_weights: keep only open- or closed-weights entries. refresh: bypass the cache and re-fetch.

ParametersJSON Schema
NameRequiredDescriptionDefault
topNo
boardYes
fieldsNo
refreshNo
open_weightsNo

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden and does disclose real behavioral traits: refresh bypasses the cache, Elo CI is hidden by default, fuzzy key normalization (text-to-image == text_to_image), and alias acceptance (realtime_voice). It stops short of stating read-only nature or rate limits, but the caching and default-output behavior are genuinely useful context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A one-line summary is front-loaded before the Args block, and the structured arg list is dense with no filler. The board enumeration is long but each entry is load-bearing; the only mild cost is length.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, yet the description conveys the key return content (Elo plus 95% CI, selectable columns) so the agent knows what comes back. It could state the return format/row shape more explicitly, but for a leaderboard lookup it is adequately complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate and it does thoroughly: it enumerates every valid board key plus aliases, explains normalization, gives the top default, lists supported field columns including ci_lower/ci_upper/ci/open_weights/url, and defines open_weights and refresh semantics. This is far more than the bare schema offers.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific resource — human-preference arenas for image, video, speech, and music with Elo + 95% CI — which is concrete and distinguishes it from generic leaderboard tools. However, it never explicitly contrasts itself with close siblings like aa_leaderboard or aa_benchmark_leaderboard, so the agent must infer the boundary.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied through the enumeration of valid board keys, which tells an agent this tool is for media arenas rather than text/benchmark leaderboards. There is no explicit when-to-use/when-not or named alternative, which is a gap given three leaderboard-family siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

aa_modelModel digestA

One model's full digest: specs, indices, benchmark scores, pricing and speed.

Use after aa_leaderboard/aa_search when you need everything about a single model. The header and footnote always carry the round-trippable slug.

Args: query: model name or slug (fuzzy). Ambiguous names return a candidate table. sections: comma string (or list) of identity/indexes/benchmarks/pricing/ speed/cost/openness/modalities, or all. Default: identity, indexes, benchmarks, pricing, speed. Alias: specs = identity. Unknown names error with the valid list. refresh: bypass the cache and re-fetch.

ParametersJSON Schema
NameRequiredDescriptionDefault
queryYes
refreshNo
sectionsNo

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and mostly succeeds: it discloses ambiguity handling ('Ambiguous names return a candidate table'), error behavior for invalid sections ('Unknown names error with the valid list'), caching semantics ('refresh: bypass the cache and re-fetch'), and the round-trippable slug in header/footnote. It does not mention auth or whether the call has side effects, which is a minor residual gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded summary sentence, then a short usage sentence, then a compact Args block that keeps each parameter to one to two lines. No filler; every clause carries information an agent needs to invoke the tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite no output schema and no annotations, the description covers the return categories, the ambiguity path, error cases for bad section names, the slug round-trip, and all inputs. An agent has everything required to call it correctly and to interpret an ambiguous result.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate fully — and it does: it documents query's fuzzy matching and ambiguity fallback, enumerates the valid section names, gives the default set, notes the 'specs = identity' alias, and explains that refresh bypasses the cache. All three parameters are meaningfully specified beyond the bare types in the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States precisely what is returned for a single resource — 'One model's full digest: specs, indices, benchmark scores, pricing and speed' — and immediately differentiates from the leaderboard/search siblings. An agent can tell that this is the deep single-model view rather than a list view without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly positions the tool in a workflow: 'Use after aa_leaderboard/aa_search when you need everything about a single model,' naming the alternatives. It stops short of stating when NOT to use it (e.g., for multiple models use aa_compare), so the exclusion side is inferred rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

aa_paretoPareto trade-off frontierA

The Pareto frontier: any score vs any cost/size/speed axis, with the knee.

Always available — no per-benchmark tool needed. Returns the models where no other model has both a lower-or-equal x and a higher-or-equal score, ordered by x ascending, plus a knee line and coverage.

The three site views this covers:

  • aa_pareto (default) = "Intelligence Index vs. Cost per Index Task"

  • aa_pareto x=out_tokens_per_task = "Intelligence Index vs. Output Tokens per Task"

  • aa_pareto x=index_total_cost = "Intelligence Index vs. Cost to Run the AA Intelligence Index" (whole suite)

Args: score: any rankable with per-model numbers (default intelligence); e.g. cyberIndex, terminalBench40, gpqa, capabilities.legal, openness. x: the trade-off axis — cost_per_task (default), index_total_cost, out_tokens_per_task, time_per_task, context_tokens, params_b, ttft, price_in, price_out. top: frontier points to return (default 15; cheapest first). creator: substring match on the creator/lab. open_weights: True/False to keep only open- or closed-weights models. min_intelligence: drop models below this Intelligence Index value. min_context: drop models with a smaller context window (tokens). include_deprecated: include deprecated rows (default False). include_all: return every frontier point instead of the top cheapest. refresh: bypass the cache and re-fetch.

ParametersJSON Schema
NameRequiredDescriptionDefault
xNocost_per_task
topNo
scoreNointelligence
creatorNo
refreshNo
include_allNo
min_contextNo
open_weightsNo
min_intelligenceNo
include_deprecatedNo

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations the description carries the full burden and does well: it defines the dominance rule, the ordering (x ascending), the default cheapest-first truncation, that deprecated rows are excluded by default, and that refresh bypasses the cache. It does not state freshness/rate-limit behavior or the exact response shape, but the cached-vs-refreshed distinction and default filtering are real disclosures.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core definition and dominance rule, then organized into site-view mapping and Args sections so an agent can scan it. The site-view block is slightly verbose for three examples and the smart-quote formatting is noisy, but no sentence is purely filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, and the description does explain what is returned (non-dominated models, x-ascending order, knee line, coverage) as well as default truncation and filtering. It stops short of describing the concrete response structure or error behavior, leaving a small gap for a 10-parameter analytical tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and all 10 parameters are undocumented in the schema, yet the Args block defines every one, including enumerated values for x (cost_per_task, index_total_cost, out_tokens_per_task, time_per_task, context_tokens, params_b, ttft, price_in, price_out) and example values for score. This is exactly the compensation the zero-coverage schema requires.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb/resource (compute the Pareto frontier of any score against any trade-off axis) and quantifies the output (frontier points plus a knee and coverage). This is clearly distinguishable from aa_leaderboard or aa_compare, which rank rather than compute a non-dominated set.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly tells the agent it is 'Always available — no per-benchmark tool needed' and maps three concrete site views onto parameter combinations, which is strong when-to-use context. It never excludes or routes against specific siblings such as aa_compare, so it stops short of full 5-level routing guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

aa_providersProvider endpointsA

Provider endpoints: per-host pricing, precision, speed and feature flags.

Pass model for one model's serving endpoints (with per-endpoint $/task and precision), or host/filters for the 1,200-row provider board. With neither, returns the cheapest endpoints globally.

Args: model: model name or slug — switches to per-model endpoint detail. host: substring match on the serving host (e.g. "groq"). top: rows to return (default 15). sort: cost_per_task (default), price_in, price_out, tps, ttft, intelligence or context_tokens. Lower-is-better keys sort ascending. fields: comma list of columns; supports precision, cache_read, discount, e2e, function_calling, json_mode, context_tokens. open_weights: keep only endpoints for open- or closed-weights models. min_context: drop endpoints with a smaller context window (tokens). refresh: bypass the cache and re-fetch.

ParametersJSON Schema
NameRequiredDescriptionDefault
topNo
hostNo
sortNocost_per_task
modelNo
fieldsNo
refreshNo
min_contextNo
open_weightsNo

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations present, the description carries the full behavioral burden and does meaningfully cover it: it discloses the default result set when no arguments are given, that a cache exists and ``refresh`` bypasses it, that lower-is-better sort keys order ascending, and the default row count. It omits auth requirements, rate limits, and any failure behavior, so it is strong but not exhaustive for an unannotated tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The mode-selection rule is front-loaded in the first paragraph before the Args list, so the most decision-relevant information comes first. The parameter block is long but each line adds distinct semantics rather than restating the schema, so little is wasted; a tighter single paragraph could have trimmed a few lines.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description must convey the return shape, and it does name the key outputs ($/task, precision, the 1,200-row provider board). Combined with the mode rules and full parameter documentation, an agent has enough to call it correctly, though a note on result shape for the board mode (columns, pagination) would close the remaining gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, yet all eight parameters are explained in the Args block with genuine meaning beyond the schema: ``sort`` enumerates six keys plus the lower-is-better ascending rule, ``fields`` lists the extra columns it unlocks (cache_read, discount, e2e, etc.), ``host`` gives an example substring, and ``min_context``/``open_weights`` state their filtering semantics. This fully compensates for the empty schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening line states the resource and the data it exposes (per-host pricing, precision, speed, feature flags), and the body distinguishes three operating modes: per-model endpoints, the host/filtered provider board, and the global cheapest-endpoint fallback. It never references the sibling tools (aa_model, aa_leaderboard) that an agent might confuse it with, so it falls short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit routing logic inside the tool: pass ``model`` for one model's serving endpoints, ``host``/filters for the provider board, and 'with neither, returns the cheapest endpoints globally.' That resolves the main ambiguity about how to invoke it. It gives no guidance on when to prefer this over siblings like aa_model or aa_leaderboard, which keeps it from a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

aa_rawRaw JSON escape hatchA

Raw JSON escape hatch (token-expensive — prefer the table tools).

Args: kind: model, leaderboard, providers, providers_board, media, agents, home, eval or catalog. query: slug/board/slug argument for the chosen kind (e.g. a model slug for "model", a board key for "media"/"agents", an eval slug for "eval"). fields: comma list of top-level keys to keep per record. top: records to return, capped at 5 (default 1). refresh: bypass the cache and re-fetch.

ParametersJSON Schema
NameRequiredDescriptionDefault
topNo
kindYes
queryNo
fieldsNo
refreshNo

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must carry the behavioral load. It discloses two important traits: token expense and cache-bypass via refresh. It doesn't say whether it's read-only, nor whether large result sets are truncated or how errors surface, leaving meaningful gaps.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

An args list with one line per parameter is appropriately structured for a five-parameter tool. The kind line enumerates nine values in one long sentence, but nothing is wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Explains inputs well but omits output shape and return semantics, and the schema offers no help (0% coverage, no output schema). For a generic escape hatch the input docs are decent, but an agent still lacks a picture of what it gets back.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate and largely does: it enumerates the valid kind values, explains query's meaning per kind with examples, defines fields for top-level projection, and notes top's cap of 5 with default 1.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Describes itself as a raw JSON escape hatch that returns unfiltered records, contrasted with a family of table tools. It's clear it's a generic low-level fetcher, though it doesn't state a single verb+resource precisely because it's a multi-kind passthrough.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly steers the agent away: 'prefer the table tools, token-expensive'. This gives a clear when-not-to-use signal. What's missing is a positive condition for when the escape hatch is actually the right choice.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

aa_referenceReference & provenanceA

Static reference text: tool routing, metric/eval definitions, provenance.

No network. Use to learn what the fields mean and which tool answers what.

Args: topic: overview (default), indexes, benchmarks, pricing, speed, providers, media, fields (alias list), sources, or all.

ParametersJSON Schema
NameRequiredDescriptionDefault
topicNooverview

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden, and it does disclose a meaningful trait: 'No network' signals static, offline, deterministic, side-effect-free retrieval. That is genuinely beyond structured data. It does not discuss output size, caching, or error behavior, keeping it from a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose and the key constraint ('Static reference text', 'No network') are front-loaded in the first two sentences, and the argument list is compact. The stray blank lines and 'Args:' block add minor formatting noise but no wasted prose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter, no-annotation, no-output-schema documentation tool, the description covers what it returns, its offline nature, and every valid topic value. The only omission is any hint about the shape or length of the returned reference text, which is minor.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the schema declares no enum for `topic`, so the description is the only source of valid values. It lists all ten accepted topics (overview, indexes, benchmarks, pricing, speed, providers, media, fields, sources, all) plus the default and the 'fields' alias caveat, fully compensating for the schema gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States exactly what the tool returns (static reference text) and enumerates its subject matter: tool routing, metric/eval definitions, provenance. The phrase 'which tool answers what' explicitly positions it as the meta/routing doc relative to the many aa_* siblings, so an agent can distinguish it without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Use to learn what the fields mean and which tool answers what' gives a clear affirmative trigger for invocation, and 'No network' implies it is the fallback for terminology questions rather than data. It stops short of naming exclusions (e.g., don't call for live numbers – use aa_leaderboard), so it is not a full 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 12 tool updatesv0.1.0
    • First observedaa_agents
    • First observedaa_benchmark_leaderboard
    • First observedaa_benchmarks
    • First observedaa_compare
    • First observedaa_leaderboard
    • First observedaa_media
    • First observedaa_model
    • First observedaa_pareto
    • First observedaa_providers
    • First observedaa_raw
    • First observedaa_reference
    • First observedaa_search

TDQS

A4/5.0

Scored across 12 tools

Disambiguation4/5

Tools mostly have distinct roles: aa_search resolves names, aa_reference explains fields, aa_raw is an escape hatch, aa_model is single-model detail. There is some overlap between aa_leaderboard, aa_benchmark_leaderboard, aa_pareto and aa_compare since all rank/compare models and share many filter args, but the descriptions clearly delineate rank-vs-frontier-vs-side-by-side-vs-single-benchmark usage.

Naming Consistency5/5

All 12 tools use the aa_ prefix with lowercase snake_case (aa_leaderboard, aa_model, aa_pareto, aa_benchmark_leaderboard, aa_reference, aa_raw). The pattern is predictable throughout, with only the shared 'aa_' namespace prefix adding any variation.

Tool Count5/5

12 tools is a well-scoped set for a model-intelligence data server, each mapping to a coherent capability (leaderboard, single model, Pareto, compare, benchmarks, providers, media, agents, search, reference, raw). Nothing feels redundant or missing at the count level.

Completeness4/5

Coverage is strong: ranking, single-model digests, Pareto frontiers, side-by-side comparison, benchmark search, provider endpoints, media/agents arenas, fuzzy name resolution, static reference, and a raw JSON escape hatch. Minor gaps remain (no explicit write/refresh-scheduling or per-benchmark-specific tools), but aa_benchmark_leaderboard and aa_raw largely cover the workarounds.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    D
    maintenance
    Provides access to real-time LLM pricing, speed metrics, and performance benchmarks for over 300 models from Artificial Analysis. It enables users to list, filter, and compare models based on costs, tokens per second, and intelligence indices.
    2
    30 npm
    9
    MIT
  • A
    license
    A
    quality
    D
    maintenance
    Enables AI agents to query LLM and multimodal model benchmarks, pricing, speed, and track model updates via structured diffs using the Artificial Analysis public API.
    7
    27 PyPI
    2
    MIT
  • F
    license
    Not graded
    quality
    B
    maintenance
    Provides protocol-neutral market intelligence for the AI agent economy, with read-only tools to search agents, retrieve details and histories, compare agents, list categories, view category rankings, and access methodology.
    -