Skip to main content
Glama

rerank_evaluate

Read-only

Evaluate RAG retrieval: rank passages against a query, compute Precision@k / Recall@k and a PASS/FAIL verdict from your relevance labels (INCONCLUSIVE without them). Modes: (1) give every passage the score YOUR reranker produced — deterministic, offline, the mode to gate CI on; (2) no scores, no key — a local BM25 keyword baseline, never a judgement of a neural reranker; (3) api_key for an NVIDIA account that still has reranking (NVIDIA retired the hosted endpoints on 2026-05-18).

Example tool call

This call uses example arguments supplied in the tool’s input schema.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "query": "capital of France",
      "api_key": "<your-api-key>",
      "passages": [
        {
          "text": "Paris is the capital of France.",
          "relevant": true
        },
        {
          "text": "Bananas are rich in potassium.",
          "relevant": false
        }
      ]
    },
    "name": "rerank_evaluate"
  }
}

Input Schema

NameRequiredDescriptionDefault
queryYes
top_kNoDefault 3.
api_keyNoNVIDIA key, used only when no passage has a score; never stored.
passagesYes2 to 20 passages.
thresholdNoMinimum Precision@k to PASS (default 0.5).

Output Schema

NameRequiredDescriptionDefault
kNo
modeNo
modelNo
rankedNo
verdictNo
thresholdNo
latency_msNo
recall_at_kNo
precision_at_kNo
total_passagesNo
has_ground_truthNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed8 schema fields changed
    • addedInput schema / examples
      Added value: +[
      +  {
      +    "api_key": "<your-api-key>",
      +    "passages": [
      +      {
      +        "relevant": true,
      +        "text": "Paris is the capital of France."
      +      },
      +      {
      +        "relevant": false,
      +        "text": "Bananas are rich in potassium."
      +      }
      +    ],
      +    "query": "capital of France"
      +  }
      +]
    • changedInput schema / properties / api_key / description
      Previous value: -"Your NVIDIA API key (BYOK), used only when no passage carries a score. Transits RAM for the single call, never stored."New value: +"NVIDIA key, used only when no passage has a score; never stored."
    • changedInput schema / properties / passages / description
      Previous value: -"Array of passage objects to rank (min 2, max 20)"New value: +"2 to 20 passages."
    • changedInput schema / properties / passages / items / properties / relevant / description
      Previous value: -"Ground truth: is this passage relevant?"New value: +"Ground truth."
    • changedInput schema / properties / passages / items / properties / score / description
      Previous value: -"Relevance score from YOUR reranker. Present on every passage → ranking is done from these (offline, no key). Higher = more relevant. Score all passages or none."New value: +"Your reranker's score (higher = more relevant). On all passages or none."
    • removedInput schema / properties / query / description
      Removed value: -"The search query or question to rank against"
    • changedInput schema / properties / threshold / description
      Previous value: -"Minimum Precision@k to PASS (0-1, default 0.5)"New value: +"Minimum Precision@k to PASS (default 0.5)."
    • changedInput schema / properties / top_k / description
      Previous value: -"k for Precision@k evaluation (default 3)"New value: +"Default 3."
  2. First observed

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations, the description discloses that the api_key path is contingent on NVIDIA's retirement of hosted endpoints on 2026-05-18, that the scored mode is deterministic and offline, and that the key is never stored. That is exactly the kind of auth/deprecation context annotations cannot carry. It does not explain why idempotentHint is false when mode 1 is deterministic, leaving a small unexplained inconsistency in the behavioral profile.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One dense paragraph, front-loaded with the output (metrics and verdict) before the mode enumeration, and every clause carries information. The parenthetical about NVIDIA's retirement is long but is the kind of fact that prevents a wasted call, so it earns its space. Slightly heavy sentence structure keeps it below top marks.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a five-parameter evaluation tool with an output schema and a detailed annotations profile, the description covers modes, prerequisites, verdict semantics, and failure cases (INCONCLUSIVE) without needing to describe return values. The only remaining gap is the exact definition of the precision/recall computation and how top_k bounds it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is already 80%, so the baseline is 3; the description earns an extra point by explaining the semantics of the score field as 'YOUR reranker produced' and tying api_key to the no-score condition, which frames how parameters interact rather than restating them. It still does not clarify how top_k affects the reported Precision@k/Recall@k or how passages beyond top_k are treated.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: rank passages against a query and compute Precision@k / Recall@k plus a PASS/FAIL verdict. It goes further by enumerating three execution modes, which makes the tool's behavior unambiguous. It stops short of naming a sibling tool it should be chosen over (e.g. rag_relevance_rank or bm25_score), so the differentiation from the catalog is implied rather than explicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Clear conditions are attached to each mode: mode 1 is 'the mode to gate CI on', mode 2 is explicitly 'never a judgement of a neural reranker', and mode 3 requires an NVIDIA account with surviving reranking access. The INCONCLUSIVE-without-labels rule tells the agent what happens if it omits ground truth. What is missing is routing guidance against the sibling tools that overlap with each mode.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.