aa-mcp
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@aa-mcpshow me the Pareto frontier for intelligence vs cost per task"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
aa-mcp
MCP server for artificialanalysis.ai — live AI model benchmarks, pricing, speed, context/parameter specs, provider endpoints and the image/video/speech/music arenas — shaped for agents that pay per token.
No API key needed. Two interchangeable data backends, same records out:
|
| |
source | the site's Next.js RSC payloads |
|
needs | nothing |
|
adds | ~50 metrics per model incl. 17 cross-model benchmarks, pricing, speed, per-model cost-per-task breakdown, provider endpoints, media boards | six Capability Indexes + Coding/Agentic indexes (not in the scrape) |
quota | none | 100 req/24 h → cached 6 h by default |
If a key is present the API is tried first and the server falls back to scraping on 401/403 rather than failing. Every response header shows which backend answered.
Install
One line, no clone (uv fetches and runs it for you):
uvx --from git+https://github.com/HenryGetz/aa-mcp aa-mcpOr install the aa-mcp executable once:
uv tool install --from git+https://github.com/HenryGetz/aa-mcp aa-mcp # -> ~/.local/bin/aa-mcpLocal checkout / development:
git clone <this repo> && cd aa-mcp
uv sync # creates .venv with the mcp SDK
uv run aa-mcp # serve on stdioRegister with an MCP client (omp, Claude Code, Codex, …):
{
"mcpServers": {
"aa": {
"type": "stdio",
"command": "/home/USER/.local/bin/aa-mcp",
"timeout": 120000
}
}
}Related MCP server: aa-mcp
Tools
tool | answers |
| smartest / best / cheapest / fastest models, best value ( |
| everything AA knows about one model: indices, benchmark scores, pricing incl. cache, speed, cost-per-task breakdown, license, modalities, cutoff |
| the trade-off frontier for any score vs any x-axis (cost per task, whole-index cost, output tokens, time, context, params, TTFT, in/out price) with the knee — no per-benchmark tool needed |
| 2-12 models side by side on the metrics you care about ( |
| the search index over everything rankable (benchmarks, indexes, derived metrics, arenas): what it measures, its unit, live coverage and the exact |
| who leads a given benchmark (full leaderboard when AA publishes one, otherwise AA's chart set — stated in the header; lower-is-better metrics rank lowest-first) |
| which provider serves a model, at what price, quantization, TTFT and feature support; or the whole provider × model board |
| text-to-image, image-editing, text-to-video, image-to-video, video-editing, TTS, music, speech-to-text, realtime voice ( |
| AA's coding-agent, chatbot and search-API leaderboards |
| resolve a fuzzy model name to real names + slugs (use it before guessing) |
| metric definitions, index composition, field names, provenance |
| raw JSON escape hatch for anything the tables do not cover |
Resources (15): aa://reference/metrics (full definitions), one per topic
(overview, indexes, benchmarks, pricing, speed, providers, media,
fields, sources), aa://benchmarks (registry JSON), aa://rankables and
aa://rankables.yaml (every rankable thing, for building your own queries),
aa://comparisons/spec (a aa_compare spec= template) and aa://sources.
Prompts: best_model_for_task, compare_models.
Comparing models
Everything AA's own comparison pages do, without the pages:
# same models as artificialanalysis.ai/models/comparisons?compare=a,b,c,d
aa_compare '{"queries":["mimo-v2-6-pro","deepseek-v4-1-flash","gpt-6-luna-xhigh","muse-wonder-full"]}'
# every benchmark AA has for those models, with deltas vs the best
aa_compare '{"queries":[...],"metrics":"all","view":"diff"}'
# reproducible / reusable comparison (YAML or JSON); the output echoes the spec back
aa_compare '{"spec":"models: [mimo-v2-6-pro, kimi-k3]\nmetrics: [intelligence, cost_per_task, gpqa, terminalBench40]\nview: diff"}'Trade-off questions always have one call:
aa_pareto '{}' # Intelligence vs $/task frontier + knee
aa_pareto '{"x":"index_total_cost"}' # vs cost of the whole index run
aa_pareto '{"x":"out_tokens_per_task"}' # vs verbosity
aa_pareto '{"score":"cyberIndex","x":"cost_per_task"}'Finding the right benchmark for a domain, without knowing its name:
aa_benchmarks '{"domain":"security"}' # -> CWE-Bench, CyberGym, Cyber Index, …
aa_benchmarks '{"query":"speech"}' # -> STT/TTS/STS boards + the exact call to makeToken discipline
Tables, not JSON — rounded values, one header line, no argument echo.
Default 10-15 rows; every table tool accepts
topandfields.Long-form documentation lives in resources/reference, so it never costs context unless the agent asks for it.
aa_rawwarns when used.
Measured live (12-call panel, one call per tool): ≈3.3k tokens total, ≈277 per
call, e.g. a 5-row leaderboard is ~180 tokens, a full model digest ~470, a media
board ~160, the aa_benchmarks search index ~235, aa_reference topic=overview
~915 (the one call that is deliberately long-form).
Example — aa_leaderboard {"mode":"value","top":5} (verbatim):
# 5 of 109 scored by value (700 tracked) · value = Intel per $/task, active only · stale: 23m ago · aa:rsc
| Model | Slug | Creator | Intel | $/Task | tok/s | TTFT | Ctx | Intel/$ |
|---|---|---|---|---|---|---|---|---|
| GPT-6 Luna (Low) | gpt-6-luna-low | OpenAI | 21.5 | $0.0045 | 121.9 | 2.6s | 1.0M | 4766.0 |
| GPT-6 Luna (Medium) | gpt-6-luna-medium | OpenAI | 29.9 | $0.02 | - | - | 1.0M | 1712.3 |
| GPT-6 Luna (Non-reasoning) | gpt-6-luna-non-reasoning | OpenAI | 18.5 | $0.01 | 112.3 | 0.9s | 1.0M | 1627.8 |
| Granite 4.2 3B | granite-4-2-3b | IBM | 9.1 | $0.0060 | 215.4 | 0.5s | 131k | 1513.0 |
| Claude Haiku 5.5 (Low) | claude-haiku-5-5-low | Anthropic | 29.5 | $0.02 | 170.0 | 9.2s | 1.0M | 1204.1 |
— 514 rows skipped: no measured cost per task (AA has cost per task for 186/700 variants)Coverage and limits
Covered: 700 model variants × ~50 leaderboard metrics merged with a 700 × 94 dataset (capability indexes, AA-Briefcase/GDPval Elo, AIME, LiveCodeBench, GDP.pdf, AutomationBench, Harvey LAB, MLCR, cost breakdowns, per-eval costs, licenses, effort labels); per-model digests with prompt-type speed breakdowns; 1,201 provider endpoints across 64 hosts with prices, quantization, latency and feature flags; 9 media arenas plus speech-to-text / streaming / realtime voice; AA's coding-agent, chatbot and search-API boards; a 95-entry rankable index (67 cross-model benchmarks plus indexes, derived cost/speed metrics and arenas) with descriptions, units, live coverage and the exact tool call for each.
Deliberately out of scope (and the server says so instead of guessing):
Benchmarks AA does not track (MathVista, DocVQA, BFCL) — no data exists.
Cross-model signature leaderboards for benchmarks AA only exposes as a chart subset: the tool labels these
not a full leaderboardand reports real coverage.Coding/Agentic indexes need the JSON API (
AA_API_KEY).Performance-over-time history and raw per-provider measurements are Commercial-tier API features; not implemented.
Image/video/music model detail pages (per-family specs beyond arena rows) and hardware/inference-stack pages are not exposed.
Environment
variable | default | meaning |
| unset | AA JSON API key; enables the |
|
|
|
|
| payload cache |
|
| scrape cache TTL (seconds) |
|
| API cache TTL (seconds) |
|
| rich-dataset (model-page overlay) cache TTL |
|
| decoded-body cap (gzip-bomb guard) |
|
| per-attempt HTTP timeout (seconds) |
|
| total budget for one fetch (seconds) |
| unset |
|
|
| override host (tests/mirrors) |
|
| stderr log level |
Behaviour worth knowing
Freshness is labelled. Payloads are cached for an hour; the header shows the age and says
stalewhen a network failure forced a cached copy. Passrefresh=truefor freshly released models.Coverage is labelled. AA scores a benchmark on a subset of models; tables state how many of the 700 tracked variants carry a value, and warn when a filter dropped rows that simply have no measurement.
Cost per task is cache-aware and must not be recomputed from $/1M rates; raw rates are separate fields.
Effort variants (
-low…-xhigh) are separate rows — always name the effort.Replies are bounded and cells are neutralized. Scraped text is sanitized (markdown metacharacters escaped, 120 chars per cell) and any reply is capped at 200 kB, so one hostile page cannot blow up your context; oversized tables say how many rows were kept.
Hostile arguments are rejected up front, not passed to a fetch or a cache path: control characters / lone surrogates, over-long strings (>256 chars, 2000 for
fields) and non-finite numbers (NaN/Infinity) return aValueErrornaming the argument, and error text never echoes more than 60 chars of your input.AA does not track MathVista, DocVQA or BFCL; the server says so instead of inventing.
Troubleshooting
Cloudflare / bot challenge (403/503 or a tiny body): the error names the exact cache path. Render the URL in a browser, save the HTML there, re-run — or set
AA_OFFLINE=1to keep working from the last good capture.Site structure change: the error says which array was missing; the parsers live in
src/aa_mcp/loaders.py(flight.pyfinds the arrays).AA redeploys often — never hardcode chunk or build-id URLs; the client always re-parses freshly fetched payloads.
Cache size: payloads are stored verbatim (a rich model page is ~3 MB, so the cache settles around 50-120 MB). Clear it with
python -c "from aa_mcp.fetch import clear_cache; print(clear_cache())"— worst case the next call refetches.
Development
uv run pytest -q # offline: fixtures in tests/fixtures (real payloads)
uvx ruff check . # lint (CI runs this)
.venv/bin/python eval/client.py tools
.venv/bin/python eval/client.py call aa_leaderboard '{"top":5}'
.venv/bin/python eval/client.py bench # per-call chars/tokens report
.venv/bin/python scripts/surface_sweep.py # every tool × every documented enum, live
.venv/bin/python scripts/gen_reference_md.py --check # docs/REFERENCE.md in syncscripts/surface_sweep.py is the end-to-end guard: it drives the real stdio server
through ~220 calls covering every tool, board, sort key, axis, section, topic and raw
kind, and fails if anything errors unexpectedly (it is how the speech-board NameError
was caught).
docs/REFERENCE.md is generated from src/aa_mcp/reference.py
(scripts/gen_reference_md.py); docs/INTERFACES.md documents the internal data
model. eval/ holds the evaluation harness and every audit report: the independent
field-by-field parser audit (eval/reports/fidelity.md), the code review
(eval/reports/audit_quality.md), the adversarial attack report
(eval/reports/adversarial.md) and five rounds of agent usage reports
(eval/runs/round*.md).
Available Tools
12 toolsaa_agentsAgent leaderboardsB
AA agent leaderboards: coding agents, chatbots and search APIs.
Args: board: coding_agents (default), chatbots or search_api. top: rows to return (default 15). refresh: bypass the cache and re-fetch.
| Name | Required | Description | Default |
|---|---|---|---|
| top | No | ||
| board | No | coding_agents | |
| refresh | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must carry the full behavioral burden. It does disclose caching behavior ('refresh: bypass the cache and re-fetch'), which is genuinely useful, but says nothing about auth requirements, rate limits, or that this is a read-only retrieval (implied but unstated). Partial coverage of a no-annotation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded one-line summary followed by a compact args block; each line earns its place. The args section is slightly redundant with schema defaults but no sentence is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema and no annotations exist, so the description is the sole source of information. It covers inputs well but never describes what a returned 'row' contains or the leaderboard structure, leaving the agent unable to anticipate the response shape.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it does: all three parameters are documented, including the valid values for 'board' (coding_agents, chatbots, search_api) which are absent from the schema and not constrained by any enum. Defaults for top and board are restated, which is minor redundancy.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States the resource clearly (agent leaderboards) and enumerates the three content types it covers (coding agents, chatbots, search APIs). An agent understands the deliverable, but the definition never distinguishes itself from close siblings like aa_leaderboard or aa_benchmark_leaderboard, so the boundary is left to inference.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no statement of when to use this tool versus the eleven siblings, several of which (aa_leaderboard, aa_benchmark_leaderboard) sound nearly identical. The only routing clue is the word 'agent' in the description, which the agent must decode on its own.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
aa_benchmark_leaderboardBenchmark leaderboardA
Rank models on one benchmark, stating the source and coverage honestly.
Auto-selects the richest source: the full 700-row leaderboard when the field exists there, else AA's curated eval chart set, else the homepage panel; the header says which and warns when it is not a full leaderboard.
Args: benchmark: benchmark key, label or eval slug (e.g. gpqa, gpqa-diamond, "GPQA Diamond", briefcaseElo). top: rows to return (default 15). fields: comma list of columns to show instead of the default set. creator: substring match on the creator/lab. open_weights: True/False to keep only open- or closed-weights models. min_intelligence: drop models below this Intelligence Index value. include_estimates: include AA-estimated scores (default True); estimates are marked with * and footnoted. refresh: bypass the cache and re-fetch.
| Name | Required | Description | Default |
|---|---|---|---|
| top | No | ||
| fields | No | ||
| creator | No | ||
| refresh | No | ||
| benchmark | Yes | ||
| open_weights | No | ||
| min_intelligence | No | ||
| include_estimates | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so the description carries the full burden. It discloses the auto-source-selection behavior, that headers warn about incomplete data, that estimates are marked with * and footnoted, and that refresh bypasses cache. Strong behavioral disclosure, though it doesn't state auth needs or rate limits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with purpose and auto-selection logic, then args in a clear list. Every sentence earns its place, though the eval slug examples in the arg list are slightly verbose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite 8 params, no annotations, and no output schema, the description covers purpose, source-selection behavior, estimate marking, cache behavior, and all parameters. Missing only auth/rate-limit context or return-format hints, but otherwise complete for an agent to invoke correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate entirely, and it does: it documents all 8 params with meaning beyond types (benchmark accepts key/label/slug, top default 15, fields as comma list overriding defaults, creator substring match, open_weights filter, min_intelligence threshold, include_estimates default True with marking behavior, refresh bypasses cache).
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (rank) and resource (models on one benchmark) and explicitly names the honesty constraint about source/coverage. Distinct from siblings like aa_leaderboard (general) and aa_model (single model detail).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly explains the auto-selection hierarchy (700-row leaderboard > curated eval chart > homepage panel) and warns when it falls back to a non-full leaderboard. This tells the agent exactly when this tool is appropriate versus alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
aa_benchmarksBenchmark & metric search indexA
Search everything rankable: benchmarks, indexes, derived metrics, arenas.
Each hit carries its How to query hint (the exact tool call), so finding
"Artificial Analysis Speech to Speech Index" immediately gives you
aa_media board=speech_to_speech.
Args: query: tokenised AND-match over key/label/aliases/description/tags (e.g. "speech", "cyber", "long context"). domain: one domain name (coding, math, science, security, agentic, tool-use, long-context, instruction-following, knowledge, reasoning, safety, multimodal, speech, media, music, video, business, finance, legal, medical, engineering, economics, cost, speed, openness, ...). tag: one raw tag (a narrower filter than domain=). verbose: also show what each thing measures.
| Name | Required | Description | Default |
|---|---|---|---|
| tag | No | ||
| query | No | ||
| domain | No | ||
| verbose | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden. It does add real value by disclosing that every hit embeds a 'How to query' hint (the exact follow-up tool call), which is a non-obvious output behavior. It says nothing about result limits, pagination, ordering, or whether the index is read-only, so the disclosure is partial.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded one-line purpose, then the key output affordance, then a scoped Args block. Every sentence earns its place and the concrete 'Artificial Analysis Speech to Speech Index' → aa_media example is high-value rather than filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 4-parameter, annotation-free, output-schema-free search tool, the definition covers purpose, filtering dimensions, and hit contents. It leaves minor gaps around result volume, ordering, and pagination, but nothing that would cause an agent to invoke it incorrectly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate and it largely does: it defines query's matching semantics (AND over key/label/aliases/description/tags), enumerates realistic domain values, explains that tag is a narrower filter than domain, and says verbose surfaces what each thing measures. Only the lack of a tag vocabulary or worked examples keeps it short of complete.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource set: 'Search everything rankable: benchmarks, indexes, derived metrics, arenas.' An agent can tell this is a discovery/search index rather than a leaderboard tool. It does not, however, distinguish itself from the similarly-named sibling aa_search or explain how it relates to aa_benchmark_leaderboard, so sibling differentiation is only partial.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The Args block implies usage: tokenised AND-match queries, a domain filter, or a narrower tag filter, which tells the agent how to drive the tool. But there is no explicit statement of when to prefer this over aa_search, aa_benchmark_leaderboard, or aa_leaderboard, and no exclusions or prerequisites. Usage is inferable but not guided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
aa_compareCompare modelsA
Side-by-side comparison of 2-12 models on any benchmark (site parity).
Mirrors artificialanalysis.ai's comparison pages: models as columns (or a
per-metric delta view), every rankable metric, and a reproducible YAML spec.
Rows AA publishes only on an eval chart set / homepage panel (the cyber
boards, AA-3D, …) are joined in and the source is footnoted; any cell that
stays - is footnoted (a per-model missing on requested metrics: …
count, the per-metric detail while the gap is small, or a whole-model
"no published pricing or speed" line) so no gap is silent.
Media/speech arena models (e.g. "Grok Voice 3.0") also resolve: comparing only arena entries renders their board metrics (Elo / STS / WER, appearances, price, release, open weights) with a footer warning that per-board scores are not cross-comparable; mixing a language model with an arena entry is refused with an explanation.
Args:
queries: 2-12 model names or slugs (language models, or arena entries on
one board).
metrics: "all" (every rankable key any selected model has), "benchmarks"
(benchmark/index kinds only), or a comma list of keys/labels/aliases;
empty uses the default rows (Intelligence, Intel/$, $/task, in/out
$/1M, tok/s, TTFT, Context). For an arena comparison the names are
arena fields (elo, sts, wer, ci, appearances, price, release,
open_weights, rank, board, creator).
view: "table" (models as columns) or "diff" (one row per metric, value
plus the delta vs the first model, e.g. 57.6 (+4.9)).
compact: default metric set only — keep the core rows instead of the full
set; ignored (and stated in the footer) whenever metrics= is given.
spec: a YAML or JSON document with models/metrics/view/compact/filters,
merged with the explicit arguments; every reply ends with a
copy-pasteable spec: block that reproduces the call (view and
compact included, so the round-trip is lossless).
refresh: bypass the cache and re-fetch.
| Name | Required | Description | Default |
|---|---|---|---|
| spec | No | ||
| view | No | table | |
| compact | No | ||
| metrics | No | ||
| queries | No | ||
| refresh | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and largely meets it: it discloses how missing cells are footnoted so no gap is silent, that arena scores are not cross-comparable, that mixing model types is refused, that refresh bypasses the cache, and that a reproducible spec block is always returned. It does not state auth needs or rate limits, leaving a small gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The purpose and scope are front-loaded and the Args section is well organized, but the opening prose paragraph is long and dense with operational detail that arguably belongs in contextual notes rather than a purpose statement. Most sentences carry information, yet the description leans toward over-specification relative to a concise tool blurb.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema and no annotations, the description covers what an agent needs: input ranges, view modes and their rendered output (columns vs. per-metric deltas), the default metric set, arena behavior, refusal conditions, caching, and the guaranteed spec block. Nothing essential for a correct call is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate and it does: each of the six parameters is documented with accepted values and defaults (metrics accepts 'all'/'benchmarks'/comma lists, view is 'table'/'diff', compact is ignored when metrics is given, spec merges with explicit args, refresh bypasses cache). It adds semantics well beyond the bare schema types.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Side-by-side comparison of 2-12 models on any benchmark'), names the input domain (language models or arena entries), and mirrors a concrete external artifact (artificialanalysis.ai comparison pages). An agent can distinguish it from aa_leaderboard, aa_model, and aa_pareto without opening a schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied by the compare semantics and edge cases are described (arena-only comparisons render board metrics; mixing a language model with an arena entry is refused with an explanation). However, there is no explicit routing guidance telling the agent when to pick aa_compare over aa_leaderboard, aa_pareto, or aa_model, which is the core 'vs alternatives' question.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
aa_leaderboardModel leaderboardA
Rank and filter the ~700 tracked model variants — the default starting point.
Answers "which model is smartest / cheapest / fastest / best value right now".
Returns a compact markdown table sorted by sort (default: intelligence).
Args:
mode: "rank" (default), "value" (intelligence per $/task), or "frontier"
(the Pareto intelligence-vs-$/task frontier — the same code path as
aa_pareto, which also plots other x axes; the footnote gives the
knee).
sort: metric/alias to sort by — e.g. intelligence, cost_per_task, tps,
ttft, context_tokens, price_in, price_out, params_b, release_date, or
any benchmark key. Use aa_reference topic=fields for all names.
fields: comma list of columns to show instead of the default set.
top: rows to return (default 10; >15 drops the Creator column for brevity).
creator: substring match on the creator/lab (e.g. "anthropic").
name: substring match on the model name/slug.
open_weights: True/False to keep only open- or closed-weights models.
reasoning: True/False to keep only reasoning or non-reasoning models.
min_intelligence: drop models below this Intelligence Index value.
max_cost_per_task: drop models above this $/task (cache-aware).
min_speed: drop models below this median output tok/s.
max_ttft: drop models above this median time-to-first-token (seconds).
min_context: drop models with a smaller context window (tokens).
released_after: ISO date (YYYY-MM-DD); keep models released on/after it.
include_deprecated: include deprecated rows (default False).
refresh: bypass the cache and re-fetch (use for "released today").
| Name | Required | Description | Default |
|---|---|---|---|
| top | No | ||
| mode | No | rank | |
| name | No | ||
| sort | No | intelligence | |
| fields | No | ||
| creator | No | ||
| refresh | No | ||
| max_ttft | No | ||
| min_speed | No | ||
| reasoning | No | ||
| min_context | No | ||
| open_weights | No | ||
| released_after | No | ||
| min_intelligence | No | ||
| max_cost_per_task | No | ||
| include_deprecated | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden and adds meaningful behavior: returns a compact markdown table sorted by sort, default intelligence; top >15 drops the Creator column; refresh bypasses cache; frontier mode shares code path with aa_pareto. It stops short of stating read-only nature, auth needs, or rate limits, so not full 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads purpose and user questions before the Args list, and the Args block is tightly formatted with no redundant sentences. Length is justified by the 16 parameters.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 16 parameters, no annotations, and no output schema, the description covers output format, default sort, column behavior, caching, and all parameters. It leaves no critical gap for an agent to call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it does: all 16 parameters are documented with defaults, allowed values, units, examples, and semantics (e.g., max_cost_per_task cache-aware, min_speed median output tok/s). It also points to aa_reference for the full sort alias list.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource ('Rank and filter the ~700 tracked model variants') and positions itself as the default starting point. Answers concrete user questions ('which model is smartest / cheapest / fastest / best value right now') and distinguishes itself from aa_pareto and aa_reference where relevant.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says this is the default starting point for leaderboard queries and names alternatives for specific needs: aa_reference for field names, aa_pareto for other x-axis plots. Mode descriptions clarify when to use rank, value, or frontier.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
aa_mediaMedia leaderboardsA
Human-preference arenas: image, video, speech, music (Elo + 95% CI).
Args: board: any image/video/TTS/music board key (text_to_image, image_editing, text_to_video, image_to_video, video_editing, text_to_speech, text_to_speech_controlled, music_instrumental, music_with_vocals) or a speech board (speech_to_text, speech_to_text_streaming, speech_to_speech — also accepted as realtime_voice / "realtime voice"). Hyphens/spaces/case are ignored (text-to-image == text_to_image). top: rows to return (default 15). fields: comma string (or list) of columns; supports ci_lower, ci_upper, ci (95% CI), open_weights, url. Elo CI is not shown by default. open_weights: keep only open- or closed-weights entries. refresh: bypass the cache and re-fetch.
| Name | Required | Description | Default |
|---|---|---|---|
| top | No | ||
| board | Yes | ||
| fields | No | ||
| refresh | No | ||
| open_weights | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden and does disclose real behavioral traits: refresh bypasses the cache, Elo CI is hidden by default, fuzzy key normalization (text-to-image == text_to_image), and alias acceptance (realtime_voice). It stops short of stating read-only nature or rate limits, but the caching and default-output behavior are genuinely useful context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A one-line summary is front-loaded before the Args block, and the structured arg list is dense with no filler. The board enumeration is long but each entry is load-bearing; the only mild cost is length.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, yet the description conveys the key return content (Elo plus 95% CI, selectable columns) so the agent knows what comes back. It could state the return format/row shape more explicitly, but for a leaderboard lookup it is adequately complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate and it does thoroughly: it enumerates every valid board key plus aliases, explains normalization, gives the top default, lists supported field columns including ci_lower/ci_upper/ci/open_weights/url, and defines open_weights and refresh semantics. This is far more than the bare schema offers.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific resource — human-preference arenas for image, video, speech, and music with Elo + 95% CI — which is concrete and distinguishes it from generic leaderboard tools. However, it never explicitly contrasts itself with close siblings like aa_leaderboard or aa_benchmark_leaderboard, so the agent must infer the boundary.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied through the enumeration of valid board keys, which tells an agent this tool is for media arenas rather than text/benchmark leaderboards. There is no explicit when-to-use/when-not or named alternative, which is a gap given three leaderboard-family siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
aa_modelModel digestA
One model's full digest: specs, indices, benchmark scores, pricing and speed.
Use after aa_leaderboard/aa_search when you need everything about a single model. The header and footnote always carry the round-trippable slug.
Args: query: model name or slug (fuzzy). Ambiguous names return a candidate table. sections: comma string (or list) of identity/indexes/benchmarks/pricing/ speed/cost/openness/modalities, or all. Default: identity, indexes, benchmarks, pricing, speed. Alias: specs = identity. Unknown names error with the valid list. refresh: bypass the cache and re-fetch.
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | ||
| refresh | No | ||
| sections | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and mostly succeeds: it discloses ambiguity handling ('Ambiguous names return a candidate table'), error behavior for invalid sections ('Unknown names error with the valid list'), caching semantics ('refresh: bypass the cache and re-fetch'), and the round-trippable slug in header/footnote. It does not mention auth or whether the call has side effects, which is a minor residual gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded summary sentence, then a short usage sentence, then a compact Args block that keeps each parameter to one to two lines. No filler; every clause carries information an agent needs to invoke the tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite no output schema and no annotations, the description covers the return categories, the ambiguity path, error cases for bad section names, the slug round-trip, and all inputs. An agent has everything required to call it correctly and to interpret an ambiguous result.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate fully — and it does: it documents query's fuzzy matching and ambiguity fallback, enumerates the valid section names, gives the default set, notes the 'specs = identity' alias, and explains that refresh bypasses the cache. All three parameters are meaningfully specified beyond the bare types in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States precisely what is returned for a single resource — 'One model's full digest: specs, indices, benchmark scores, pricing and speed' — and immediately differentiates from the leaderboard/search siblings. An agent can tell that this is the deep single-model view rather than a list view without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly positions the tool in a workflow: 'Use after aa_leaderboard/aa_search when you need everything about a single model,' naming the alternatives. It stops short of stating when NOT to use it (e.g., for multiple models use aa_compare), so the exclusion side is inferred rather than stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
aa_paretoPareto trade-off frontierA
The Pareto frontier: any score vs any cost/size/speed axis, with the knee.
Always available — no per-benchmark tool needed. Returns the models where no
other model has both a lower-or-equal x and a higher-or-equal score,
ordered by x ascending, plus a knee line and coverage.
The three site views this covers:
aa_pareto(default) = "Intelligence Index vs. Cost per Index Task"aa_pareto x=out_tokens_per_task= "Intelligence Index vs. Output Tokens per Task"aa_pareto x=index_total_cost= "Intelligence Index vs. Cost to Run the AA Intelligence Index" (whole suite)
Args:
score: any rankable with per-model numbers (default intelligence); e.g.
cyberIndex, terminalBench40, gpqa, capabilities.legal, openness.
x: the trade-off axis — cost_per_task (default), index_total_cost,
out_tokens_per_task, time_per_task, context_tokens, params_b, ttft,
price_in, price_out.
top: frontier points to return (default 15; cheapest first).
creator: substring match on the creator/lab.
open_weights: True/False to keep only open- or closed-weights models.
min_intelligence: drop models below this Intelligence Index value.
min_context: drop models with a smaller context window (tokens).
include_deprecated: include deprecated rows (default False).
include_all: return every frontier point instead of the top cheapest.
refresh: bypass the cache and re-fetch.
| Name | Required | Description | Default |
|---|---|---|---|
| x | No | cost_per_task | |
| top | No | ||
| score | No | intelligence | |
| creator | No | ||
| refresh | No | ||
| include_all | No | ||
| min_context | No | ||
| open_weights | No | ||
| min_intelligence | No | ||
| include_deprecated | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations the description carries the full burden and does well: it defines the dominance rule, the ordering (x ascending), the default cheapest-first truncation, that deprecated rows are excluded by default, and that refresh bypasses the cache. It does not state freshness/rate-limit behavior or the exact response shape, but the cached-vs-refreshed distinction and default filtering are real disclosures.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core definition and dominance rule, then organized into site-view mapping and Args sections so an agent can scan it. The site-view block is slightly verbose for three examples and the smart-quote formatting is noisy, but no sentence is purely filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, and the description does explain what is returned (non-dominated models, x-ascending order, knee line, coverage) as well as default truncation and filtering. It stops short of describing the concrete response structure or error behavior, leaving a small gap for a 10-parameter analytical tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% and all 10 parameters are undocumented in the schema, yet the Args block defines every one, including enumerated values for x (cost_per_task, index_total_cost, out_tokens_per_task, time_per_task, context_tokens, params_b, ttft, price_in, price_out) and example values for score. This is exactly the compensation the zero-coverage schema requires.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb/resource (compute the Pareto frontier of any score against any trade-off axis) and quantifies the output (frontier points plus a knee and coverage). This is clearly distinguishable from aa_leaderboard or aa_compare, which rank rather than compute a non-dominated set.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly tells the agent it is 'Always available — no per-benchmark tool needed' and maps three concrete site views onto parameter combinations, which is strong when-to-use context. It never excludes or routes against specific siblings such as aa_compare, so it stops short of full 5-level routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
aa_providersProvider endpointsA
Provider endpoints: per-host pricing, precision, speed and feature flags.
Pass model for one model's serving endpoints (with per-endpoint $/task and
precision), or host/filters for the 1,200-row provider board. With neither,
returns the cheapest endpoints globally.
Args: model: model name or slug — switches to per-model endpoint detail. host: substring match on the serving host (e.g. "groq"). top: rows to return (default 15). sort: cost_per_task (default), price_in, price_out, tps, ttft, intelligence or context_tokens. Lower-is-better keys sort ascending. fields: comma list of columns; supports precision, cache_read, discount, e2e, function_calling, json_mode, context_tokens. open_weights: keep only endpoints for open- or closed-weights models. min_context: drop endpoints with a smaller context window (tokens). refresh: bypass the cache and re-fetch.
| Name | Required | Description | Default |
|---|---|---|---|
| top | No | ||
| host | No | ||
| sort | No | cost_per_task | |
| model | No | ||
| fields | No | ||
| refresh | No | ||
| min_context | No | ||
| open_weights | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations present, the description carries the full behavioral burden and does meaningfully cover it: it discloses the default result set when no arguments are given, that a cache exists and ``refresh`` bypasses it, that lower-is-better sort keys order ascending, and the default row count. It omits auth requirements, rate limits, and any failure behavior, so it is strong but not exhaustive for an unannotated tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The mode-selection rule is front-loaded in the first paragraph before the Args list, so the most decision-relevant information comes first. The parameter block is long but each line adds distinct semantics rather than restating the schema, so little is wasted; a tighter single paragraph could have trimmed a few lines.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description must convey the return shape, and it does name the key outputs ($/task, precision, the 1,200-row provider board). Combined with the mode rules and full parameter documentation, an agent has enough to call it correctly, though a note on result shape for the board mode (columns, pagination) would close the remaining gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, yet all eight parameters are explained in the Args block with genuine meaning beyond the schema: ``sort`` enumerates six keys plus the lower-is-better ascending rule, ``fields`` lists the extra columns it unlocks (cache_read, discount, e2e, etc.), ``host`` gives an example substring, and ``min_context``/``open_weights`` state their filtering semantics. This fully compensates for the empty schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The opening line states the resource and the data it exposes (per-host pricing, precision, speed, feature flags), and the body distinguishes three operating modes: per-model endpoints, the host/filtered provider board, and the global cheapest-endpoint fallback. It never references the sibling tools (aa_model, aa_leaderboard) that an agent might confuse it with, so it falls short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit routing logic inside the tool: pass ``model`` for one model's serving endpoints, ``host``/filters for the provider board, and 'with neither, returns the cheapest endpoints globally.' That resolves the main ambiguity about how to invoke it. It gives no guidance on when to prefer this over siblings like aa_model or aa_leaderboard, which keeps it from a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
aa_rawRaw JSON escape hatchA
Raw JSON escape hatch (token-expensive — prefer the table tools).
Args: kind: model, leaderboard, providers, providers_board, media, agents, home, eval or catalog. query: slug/board/slug argument for the chosen kind (e.g. a model slug for "model", a board key for "media"/"agents", an eval slug for "eval"). fields: comma list of top-level keys to keep per record. top: records to return, capped at 5 (default 1). refresh: bypass the cache and re-fetch.
| Name | Required | Description | Default |
|---|---|---|---|
| top | No | ||
| kind | Yes | ||
| query | No | ||
| fields | No | ||
| refresh | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must carry the behavioral load. It discloses two important traits: token expense and cache-bypass via refresh. It doesn't say whether it's read-only, nor whether large result sets are truncated or how errors surface, leaving meaningful gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
An args list with one line per parameter is appropriately structured for a five-parameter tool. The kind line enumerates nine values in one long sentence, but nothing is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Explains inputs well but omits output shape and return semantics, and the schema offers no help (0% coverage, no output schema). For a generic escape hatch the input docs are decent, but an agent still lacks a picture of what it gets back.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate and largely does: it enumerates the valid kind values, explains query's meaning per kind with examples, defines fields for top-level projection, and notes top's cap of 5 with default 1.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Describes itself as a raw JSON escape hatch that returns unfiltered records, contrasted with a family of table tools. It's clear it's a generic low-level fetcher, though it doesn't state a single verb+resource precisely because it's a multi-kind passthrough.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly steers the agent away: 'prefer the table tools, token-expensive'. This gives a clear when-not-to-use signal. What's missing is a positive condition for when the escape hatch is actually the right choice.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
aa_referenceReference & provenanceA
Static reference text: tool routing, metric/eval definitions, provenance.
No network. Use to learn what the fields mean and which tool answers what.
Args: topic: overview (default), indexes, benchmarks, pricing, speed, providers, media, fields (alias list), sources, or all.
| Name | Required | Description | Default |
|---|---|---|---|
| topic | No | overview |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden, and it does disclose a meaningful trait: 'No network' signals static, offline, deterministic, side-effect-free retrieval. That is genuinely beyond structured data. It does not discuss output size, caching, or error behavior, keeping it from a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Purpose and the key constraint ('Static reference text', 'No network') are front-loaded in the first two sentences, and the argument list is compact. The stray blank lines and 'Args:' block add minor formatting noise but no wasted prose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter, no-annotation, no-output-schema documentation tool, the description covers what it returns, its offline nature, and every valid topic value. The only omission is any hint about the shape or length of the returned reference text, which is minor.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% and the schema declares no enum for `topic`, so the description is the only source of valid values. It lists all ten accepted topics (overview, indexes, benchmarks, pricing, speed, providers, media, fields, sources, all) plus the default and the 'fields' alias caveat, fully compensating for the schema gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States exactly what the tool returns (static reference text) and enumerates its subject matter: tool routing, metric/eval definitions, provenance. The phrase 'which tool answers what' explicitly positions it as the meta/routing doc relative to the many aa_* siblings, so an agent can distinguish it without opening a schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Use to learn what the fields mean and which tool answers what' gives a clear affirmative trigger for invocation, and 'No network' implies it is the fallback for terminology questions rather than data. It stops short of naming exclusions (e.g., don't call for live numbers – use aa_leaderboard), so it is not a full 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
aa_searchFind a model slugA
Cheap name -> slug resolver; call this first whenever a name is uncertain.
Always returns the slug so the next tool call can be exact. Media/speech arena models (image/video/speech/music boards) are searched too and returned with their board, so a name like "Grok Voice 3.0" resolves instead of coming back empty.
Args: query: fuzzy model name or partial slug (e.g. "claude opus"). creator: optional creator/lab substring filter. open_weights: True/False to keep only open- or closed-weights models. top: rows to return (default 15). refresh: bypass the cache and re-fetch.
| Name | Required | Description | Default |
|---|---|---|---|
| top | No | ||
| query | No | ||
| creator | No | ||
| refresh | No | ||
| open_weights | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden and does add real behavioral context: it is "cheap," results are cached and `refresh` bypasses the cache, a slug is always returned, and media/speech boards are included with their board label. It never mentions auth requirements, rate limits, or error/empty-result behavior, so it is not fully transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core purpose and routing instruction in the first sentence, and the Args block is compact. The media/speech sentence is slightly defensive prose, but it justifies a real edge case rather than being filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a five-parameter, no-annotation, no-output-schema tool, the description covers all parameters, the cache model, and the guarantee of a slug return (plus board labels for media models). It stops short of describing the shape of returned rows or how many results `top` can safely request.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and the Args block documents all five parameters with meaning beyond their names: query is a fuzzy name or partial slug (with an example), creator is a substring filter, open_weights keeps only one weight class, top is a row count with default, refresh bypasses cache. It omits acceptable query syntax limits and slug format conventions, keeping it from a 5.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource+scope: a name-to-slug resolver, with the explicit routing claim that it should be called first when a name is uncertain. It is clearly distinguishable from detail-oriented siblings like aa_model or aa_compare without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
"Call this first whenever a name is uncertain" gives a concrete trigger condition and an ordering rule relative to the next tool call. It lacks explicit when-not guidance (e.g., when a name is already known to be an exact slug), so it falls just short of the top band.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
12 tool updates
v0.1.0- First observed
aa_agents - First observed
aa_benchmark_leaderboard - First observed
aa_benchmarks - First observed
aa_compare - First observed
aa_leaderboard - First observed
aa_media - First observed
aa_model - First observed
aa_pareto - First observed
aa_providers - First observed
aa_raw - First observed
aa_reference - First observed
aa_search
TDQS
Scored across 12 tools
Tools mostly have distinct roles: aa_search resolves names, aa_reference explains fields, aa_raw is an escape hatch, aa_model is single-model detail. There is some overlap between aa_leaderboard, aa_benchmark_leaderboard, aa_pareto and aa_compare since all rank/compare models and share many filter args, but the descriptions clearly delineate rank-vs-frontier-vs-side-by-side-vs-single-benchmark usage.
All 12 tools use the aa_ prefix with lowercase snake_case (aa_leaderboard, aa_model, aa_pareto, aa_benchmark_leaderboard, aa_reference, aa_raw). The pattern is predictable throughout, with only the shared 'aa_' namespace prefix adding any variation.
12 tools is a well-scoped set for a model-intelligence data server, each mapping to a coherent capability (leaderboard, single model, Pareto, compare, benchmarks, providers, media, agents, search, reference, raw). Nothing feels redundant or missing at the count level.
Coverage is strong: ranking, single-model digests, Pareto frontiers, side-by-side comparison, benchmark search, provider endpoints, media/agents arenas, fuzzy name resolution, static reference, and a raw JSON escape hatch. Minor gaps remain (no explicit write/refresh-scheduling or per-benchmark-specific tools), but aa_benchmark_leaderboard and aa_raw largely cover the workarounds.
Maintenance
Related MCP Connectors
Live data on AI agents, APIs and models: prices, free tiers, limits, outages, real-world usage.
Market intelligence for the AI agent economy: rankings, trust signals, liveness. 13 tools.
Public web tools for agents: product extraction, claim checks, webpage QA and ranked audits.
Live web access for agents: scrape, SERP search, crawl/map, 100+ collectors, datasets, proxies.
Related MCP Servers
- AlicenseAqualityDmaintenanceProvides access to real-time LLM pricing, speed metrics, and performance benchmarks for over 300 models from Artificial Analysis. It enables users to list, filter, and compare models based on costs, tokens per second, and intelligence indices.230 npm9MIT
- AlicenseAqualityDmaintenanceEnables AI agents to query LLM and multimodal model benchmarks, pricing, speed, and track model updates via structured diffs using the Artificial Analysis public API.727 PyPI2MIT
- AlicenseNot gradedqualityCmaintenanceProvides AI agents with honest benchmark rankings (Agentic Memory Index and Agentic Search Index) for AI tools, plus graded checks and telemetry for x402 endpoints.6 npmMIT
- FlicenseNot gradedqualityBmaintenanceProvides protocol-neutral market intelligence for the AI agent economy, with read-only tools to search agents, retrieve details and histories, compare agents, list categories, view category rankings, and access methodology.-