The Aggregate — LLM benchmark aggregate
Server Details
Fused LLM rankings: one IRT/Elo scale across ~5,000 public benchmark leaderboards, updated daily.
- Status
- Healthy
- Last Tested
- Transport
- Streamable HTTP
- URL
Glama MCP Gateway
Connect through Glama MCP Gateway for full control over tool access and complete visibility into every call.
Full call logging
Every tool call is logged with complete inputs and outputs, so you can debug issues and audit what your agents are doing.
Tool access control
Enable or disable individual tools per connector, so you decide what your agents can and cannot do.
Managed credentials
Glama handles OAuth flows, token storage, and automatic rotation, so credentials never expire on your clients.
Usage analytics
See which tools your agents call, how often, and when, so you can understand usage patterns and catch anomalies.
Tool Definition Quality
Average 3.8/5 across 7 of 8 tools scored.
Each tool has a distinct purpose: about, compare, get single benchmark/leaderboard/model, prediction duel, search benchmarks/models. No overlapping functionality, clear boundaries.
All names follow a consistent verb_noun pattern in snake_case (e.g., compare_models, get_benchmark, search_benchmarks). Only 'about_the_aggregate' deviates slightly but remains descriptive and consistent in style.
8 tools is well-scoped for an LLM benchmark aggregator. Covers core actions: about, search, retrieve details, compare, leaderboard, and a unique prediction duel feature. No excess or deficiency.
The tool surface covers all obvious use cases: learning about the aggregate, viewing leaderboards, getting details on benchmarks/models, comparing models, searching, and the prediction duel. Minor gaps include no explicit 'list all benchmarks' tool, but search effectively fills that role.
Available Tools
8 toolsabout_the_aggregateAbout The AggregateAInspect
What this data is: how the IRT fusion works, current coverage counts, update cadence, and how to cite it.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Tool Definition Quality
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses what information is provided (IRT fusion, coverage, cadence, citation) but does not mention any behavioral traits such as read-only nature, rate limits, or return format. With no parameters, the transparency is adequate but not enriched.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that efficiently lists the covered topics. It is front-loaded with the purpose. No superfluous information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has no parameters and no output schema, the description adequately covers the purpose and the specific aspects of the aggregate it will explain. It provides sufficient context for an agent to understand what this tool offers.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has zero parameters, and schema description coverage is 100% (no parameters to describe). Per guidelines, baseline is 4. The description does not need to add parameter meaning.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool provides information about the aggregate: IRT fusion, coverage counts, update cadence, and citation. This distinguishes it from sibling tools like get_model or get_leaderboard which focus on specific models or benchmarks.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool should be used when one wants details about the aggregate, but it does not explicitly state when to use it versus alternatives or provide exclusion criteria. No sibling references are made.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
compare_modelsCompare modelsAInspect
Head-to-head between 2-4 models: aggregate ranks, Elo gap with a significance note based on the standard errors, and notable benchmarks they share.
| Name | Required | Description | Default |
|---|---|---|---|
| models | Yes | Two to four model names or slugs. |
Tool Definition Quality
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided. Description mentions outputs but fails to disclose behavioral traits such as error handling, prerequisites, or side effects. Meets basic expectations but lacks depth.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, front-loaded with core purpose, no wasted words. Each segment adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given only one parameter and no output schema, the description covers the basic operation well. Missing edge-case info, but adequate for a straightforward tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with clear parameter description. The tool description restates the range but adds no new semantic details beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description specifies exact verb+resource ('head-to-head between 2-4 models') and lists outputs, clearly differentiating from sibling tools like get_model or get_leaderboard.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for comparing 2-4 models, but provides no explicit guidance on when to use this tool versus alternatives like get_model or search_models.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_benchmarkBenchmark detailAInspect
One benchmark in depth: what it measures, the original source leaderboard URL, IRT stats (difficulty, noise, model coverage), skill weights, and the current top models on it.
| Name | Required | Description | Default |
|---|---|---|---|
| top | No | How many top models to list (1-50, default 10). | |
| benchmark | Yes | Benchmark name or slug, e.g. "Aider polyglot". |
Tool Definition Quality
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. It details return content but does not mention non-destructive nature (though implied), error handling, auth needs, or rate limits. Adequate for a read operation but not fully transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single, well-structured sentence that efficiently communicates the tool's purpose and return value. No verbose or redundant language.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, but description compensates by enumerating return fields (IRT stats, skill weights, top models). Parameter descriptions are complete. Lacks some behavioral details but overall complete for a retrieval tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so parameters are already described. The description adds no new constraints or formatting details beyond schema. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states what the tool returns: measurements, source URL, IRT stats, skill weights, and top models. Distinguishes from siblings like get_leaderboard (list) and search_benchmarks (search) by offering in-depth detail on a single benchmark.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implies usage for deep-diving into a specific benchmark, but no explicit when-to-use or when-not-to-use guidance. With multiple sibling tools, clearer differentiation would help avoid confusion (e.g., vs get_leaderboard or search_benchmarks).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_leaderboardAggregate leaderboardAInspect
Top of the cross-benchmark aggregate ranking: every model placed on one Elo scale by an IRT model fit over ~5,000 public benchmark leaderboards. Supports paging via limit/offset.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Rows to return (1-100, default 25). | |
| offset | No | Rows to skip from the top (default 0). |
Tool Definition Quality
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description bears full responsibility. It only states the tool returns ranking data and supports paging, without disclosing behavioral traits like read-only nature, authentication requirements, or potential side effects. This is minimal for a tool without annotation support.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences packing all necessary information: what the tool returns and paging capability. No filler words, front-loaded with the primary purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the tool's purpose and paging, but without an output schema, it omits details about return format (e.g., model names, Elo scores, ranking order). Adequate but leaves some ambiguity for an agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with both 'limit' and 'offset' parameters described. The description reinforces paging ('Supports paging via limit/offset') but does not add substantive meaning beyond what the schema already provides. Baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it returns the top of the cross-benchmark aggregate ranking, using an IRT model over ~5,000 public leaderboards. This specific verb-resource combination distinguishes it from sibling tools like 'about_the_aggregate' (which explains methodology) and 'get_benchmark' (which focuses on individual benchmarks).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly indicates this tool is for retrieving the aggregate leaderboard with paging support. It does not explicitly list when not to use it or mention alternatives, but the context is sufficiently clear given the sibling tool names.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_modelModel profileAInspect
One model in depth: aggregate rank, Elo with standard error, provider, what it is, cost per task where known, and its most notable benchmark results (with percentiles).
| Name | Required | Description | Default |
|---|---|---|---|
| model | Yes | Model name or slug, e.g. "Claude Opus 4.5" or "gpt-5-5". |
Tool Definition Quality
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It details the returned data (rank, Elo, etc.) but does not disclose side effects, authentication needs, or rate limits. For a read-only tool, this is adequate but not comprehensive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence that efficiently conveys the tool's purpose and output. Every element earns its place with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given only one parameter and no output schema, the description provides a solid overview of the tool's return value. It is complete enough for a simple lookup, though it could mention pagination or result limits.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The only parameter is 'model', which has 100% schema description coverage. The description does not add new information about the parameter, only implying that the model name is needed. Baseline score of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states this tool retrieves a single model's in-depth profile, listing specific attributes (rank, Elo, provider, cost, benchmarks). This distinguishes it from siblings like compare_models (comparison) and get_leaderboard (list).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for detailed info on one model ("One model in depth"). However, it does not explicitly state when not to use it (e.g., for comparisons) or mention alternatives among sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_prediction_duelPrediction duel standingsAInspect
Guesswork — the public prediction duel: every day frontier LLMs and The Aggregate's own IRT model predict newly scraped benchmark scores before seeing them, and the errors are scored. Returns the current monthly standings, wins and losses included.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Tool Definition Quality
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden. It discloses the return (monthly standings, wins/losses) and implies a read-only operation without side effects. It could mention freshness or if it's cached, but it's adequate for a simple tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise: two sentences with no wasted words. It front-loads the engaging 'Guesswork' context, then immediately states the purpose and return value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no parameters and no output schema, the description adequately explains what is returned (monthly standings, wins/losses). It could specify the output format or data structure, but it is sufficient for the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
No parameters exist, and schema coverage is 100% (empty). The description adds meaning beyond the schema by providing context about the prediction duel, which is the baseline of 4 for zero-parameter tools.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool returns monthly standings of a prediction duel, explaining the context (frontier LLMs and IRT model predicting benchmark scores) and distinguishes it from sibling tools like get_leaderboard or get_model. It uses specific verbs and resource.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use when wanting prediction duel standings but does not provide explicit when-to-use or when-not-to-use guidance, nor does it compare with alternatives. Usage is implied but not clarified.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_benchmarksSearch benchmarksAInspect
Find benchmarks in the aggregate by (partial) name. Returns model coverage, difficulty on the Elo scale, and the benchmark page URL.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Max results (1-25, default 10). | |
| query | Yes | Benchmark name fragment, e.g. "swe-bench" or "arena". |
Tool Definition Quality
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It discloses that the tool returns specific fields (coverage, difficulty, URL) but does not mention any behavioral traits such as read-only nature, rate limits, or side effects. For a search tool, more behavioral context would be beneficial.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence that efficiently conveys purpose and output. It is concise but could be slightly expanded to include usage hints without loss of brevity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description compensates by listing three return items (model coverage, difficulty, URL). It is fairly complete for a simple search tool, though it omits details like default sorting or pagination behavior beyond the limit parameter.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with descriptions for both parameters ('query' and 'limit'). The description adds 'by (partial) name' which aligns with the schema's query description, but does not provide additional meaning beyond what the schema already conveys.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Find' and resource 'benchmarks', specifies the search method 'by (partial) name', and explicitly lists the returned data (model coverage, difficulty on Elo scale, and benchmark page URL). This distinguishes the tool from siblings like 'get_benchmark' (single benchmark) and 'search_models' (different resource).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description indicates usage for finding benchmarks by partial name, but does not provide explicit when-to-use or when-not-to-use guidance, nor does it mention alternatives among siblings. The context is clear but lacks explicit exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_modelsSearch modelsAInspect
Find ranked models by (partial) name or provider. Returns rank, Elo and the model page URL.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Max results (1-25, default 10). | |
| query | Yes | Model or provider name fragment, e.g. "opus" or "deepseek". |
Tool Definition Quality
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description bears full burden. It states the return values but does not disclose request limits, pagination, or safety characteristics. The behavior is implied to be read-only but not explicitly stated.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence with no superfluous words. It efficiently conveys purpose and return information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Without an output schema, the description covers return values adequately. However, it lacks details on sorting, error handling, and case sensitivity, which are useful for a search tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already documents both parameters. The description adds context about return fields but does not provide additional syntax or format details for the parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it finds ranked models by partial name or provider, and specifies the return fields. It distinguishes from sibling tools like compare_models and get_model by focusing on search, but does not explicitly differentiate.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for partial name or provider searches, but provides no guidance on when not to use it or alternatives. Sibling tools are listed but not compared.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Claim this connector by publishing a /.well-known/glama.json file on your server's domain with the following structure:
{
"$schema": "https://glama.ai/mcp/schemas/connector.json",
"maintainers": [{ "email": "your-email@example.com" }]
}The email address must match the email associated with your Glama account. Once published, Glama will automatically detect and verify the file within a few minutes.
Control your server's listing on Glama, including description and metadata
Access analytics and receive server usage reports
Get monitoring and health status updates for your server
Feature your server to boost visibility and reach more users
For users:
Full audit trail – every tool call is logged with inputs and outputs for compliance and debugging
Granular tool control – enable or disable individual tools per connector to limit what your AI agents can do
Centralized credential management – store and rotate API keys and OAuth tokens in one place
Change alerts – get notified when a connector changes its schema, adds or removes tools, or updates tool definitions, so nothing breaks silently
For server owners:
Proven adoption – public usage metrics on your listing show real-world traction and build trust with prospective users
Tool-level analytics – see which tools are being used most, helping you prioritize development and documentation
Direct user feedback – users can report issues and suggest improvements through the listing, giving you a channel you would not have otherwise
The connector status is unhealthy when Glama is unable to successfully connect to the server. This can happen for several reasons:
The server is experiencing an outage
The URL of the server is wrong
Credentials required to access the server are missing or invalid
If you are the owner of this MCP connector and would like to make modifications to the listing, including providing test credentials for accessing the server, please contact support@glama.ai.
Discussions
No comments yet. Be the first to start the discussion!