IA-QA — 152 QA & Dev Tools for AI Agents

rerank_evaluate

Read-only

Evaluate RAG retrieval: rank passages against a query, compute Precision@k / Recall@k and a PASS/FAIL verdict from your relevance labels. Modes: (1) give every passage the score YOUR reranker produced — deterministic, offline, the mode to gate CI on; (2) no scores, no key — a local BM25 keyword baseline, never a judgement of a neural reranker; (3) api_key for an NVIDIA account that still has reranking (NVIDIA retired the hosted endpoints on 2026-05-18).

Input Schema

Name	Required	Description
`query`	Yes
`top_k`	No	Default 3.
`api_key`	No	NVIDIA key, used only when no passage has a score; never stored.
`passages`	Yes	2 to 20 passages.
`threshold`	No	Minimum Precision@k to PASS (default 0.5).

Output Schema

Name	Required	Description	Default
`k`	No
`mode`	No
`model`	No
`ranked`	No
`verdict`	No
`threshold`	No
`latency_ms`	No
`recall_at_k`	No
`precision_at_k`	No
`total_passages`	No
`has_ground_truth`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed8 schema fields changedOct 31, 2026

addedInput schema / examples

Added value: +[
+  {
+    "api_key": "<your-api-key>",
+    "passages": [
+      {
+        "relevant": true,
+        "text": "Paris is the capital of France."
+      },
+      {
+        "relevant": false,
+        "text": "Bananas are rich in potassium."
+      }
+    ],
+    "query": "capital of France"
+  }
+]

changedInput schema / properties / api_key / description

Previous value: -"Your NVIDIA API key (BYOK), used only when no passage carries a score. Transits RAM for the single call, never stored."New value: +"NVIDIA key, used only when no passage has a score; never stored."

changedInput schema / properties / passages / description

Previous value: -"Array of passage objects to rank (min 2, max 20)"New value: +"2 to 20 passages."

changedInput schema / properties / passages / items / properties / relevant / description

Previous value: -"Ground truth: is this passage relevant?"New value: +"Ground truth."

changedInput schema / properties / passages / items / properties / score / description

Previous value: -"Relevance score from YOUR reranker. Present on every passage → ranking is done from these (offline, no key). Higher = more relevant. Score all passages or none."New value: +"Your reranker's score (higher = more relevant). On all passages or none."

removedInput schema / properties / query / description

Removed value: -"The search query or question to rank against"

changedInput schema / properties / threshold / description

Previous value: -"Minimum Precision@k to PASS (0-1, default 0.5)"New value: +"Minimum Precision@k to PASS (default 0.5)."

changedInput schema / properties / top_k / description

Previous value: -"k for Precision@k evaluation (default 3)"New value: +"Default 3."

Changed15 schema fields changedAug 20, 2026

addedInput schema / properties / api_key

Added value: +{
+  "description": "Your NVIDIA API key (BYOK), used only when no passage carries a score. Transits RAM for the single call, never stored.",
+  "type": "string"
+}

addedInput schema / properties / passages / items / properties / score

Added value: +{
+  "description": "Relevance score from YOUR reranker. Present on every passage → ranking is done from these (offline, no key). Higher = more relevant. Score all passages or none.",
+  "type": "number"
+}

addedOutput schema / properties / has_ground_truth
```
Added value: +{
+  "type": "boolean"
+}
```
addedOutput schema / properties / k
```
Added value: +{
+  "type": "number"
+}
```
addedOutput schema / properties / latency_ms
```
Added value: +{
+  "type": "number"
+}
```
addedOutput schema / properties / mode
```
Added value: +{
+  "type": "string"
+}
```
addedOutput schema / properties / precision_at_k
```
Added value: +{
+  "type": "number"
+}
```

removedOutput schema / properties / query

Removed value: -{
-  "type": "string"
-}

addedOutput schema / properties / ranked

Added value: +{
+  "items": {
+    "type": "object"
+  },
+  "type": "array"
+}

addedOutput schema / properties / recall_at_k
```
Added value: +{
+  "type": "number"
+}
```

removedOutput schema / properties / results

Removed value: -{
-  "items": {
-    "type": "object"
-  },
-  "type": "array"
-}

addedOutput schema / properties / threshold
```
Added value: +{
+  "type": "number"
+}
```

removedOutput schema / properties / top_n

Removed value: -{
-  "type": "number"
-}

addedOutput schema / properties / total_passages
```
Added value: +{
+  "type": "number"
+}
```
addedOutput schema / properties / verdict
```
Added value: +{
+  "type": "string"
+}
```

Changed1 schema field changedJun 21, 2026

addedOutput schema / properties

Added value: +{
+  "model": {
+    "type": "string"
+  },
+  "query": {
+    "type": "string"
+  },
+  "results": {
+    "items": {
+      "type": "object"
+    },
+    "type": "array"
+  },
+  "top_n": {
+    "type": "number"
+  }
+}

Changed1 schema field changedJun 10, 2026

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "additionalProperties": true,
+  "type": "object"
+}

First observedMay 26, 2026

TDQS

A4.1/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare the operation as read-only and non-destructive, and the description adds useful traits: mode 1 is deterministic and offline and suitable for CI gating, while mode 2 is a keyword baseline that cannot judge a neural reranker. The retirement date for hosted NVIDIA endpoints is meaningful context beyond the annotations. It does not describe pagination or output shape, but an output schema exists.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with the core action and metrics, then enumerates modes compactly; every clause carries information. The mode list is dense and slightly crowded into one paragraph, but no sentence is wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Since an output schema exists, the description need not explain return values; what remains is exactly what an agent needs: what the tool does, which mode it is in based on the inputs, and which mode is safe for CI. The combination of modes, caveats, and example-annotated schema makes the definition self-sufficient.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 80% schema coverage, the baseline is 3, and the description adds meaningful meaning: the `score` value must be the caller's reranker output and applies to all passages or none, while `api_key` is needed only when no passage has a score. That's real mode-selection semantics beyond the schema's field text.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

It names a specific verb and resource: rank passages against a query and compute Precision@k / Recall@k with a PASS/FAIL verdict. It also distinguishes its BM25 fallback mode from a neural reranker, which implicitly separates it from sibling tools like rag_relevance_rank and bm25_score. It does not name those siblings explicitly, so an agent must infer the boundary.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The three modes are enumerated with conditions: supply your own scores for deterministic offline CI gating, omit scores for a keyword baseline, or supply api_key. The description explicitly warns that mode 2 is not a judgment of a neural reranker and that NVIDIA retired hosted endpoints. It never explicitly says when to choose rag_relevance_rank instead.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All IA-QA — 152 QA & Dev Tools for AI Agents tools Inspect

Try in Browser

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.