evaluate_retrieval
Tests retrieval quality by running query cases and checking whether expected documents appear in top-5 results, then reports MRR@5 and Recall@5 for auditing.
Instructions
Evaluate search quality by testing whether search_knowledge() retrieves expected documents.
Read-only. Runs multiple search queries internally. No side effects on the index.
Args: test_cases: JSON string array of test cases. Each item requires "query" (search string) and "expected_filepath" (path of the document that should appear in top-5 results). Example: [{"query": "suid exploit", "expected_filepath": "security/suid.md"}]
Returns: JSON string with MRR@5 (Mean Reciprocal Rank), Recall@5, and per-query hit/miss breakdown. MRR@5 above 0.7 indicates good retrieval quality.
Usage: Use to audit search quality after bulk document ingestion, a model change or a reindex. Each case runs as one top-5 search with the defaults of search_knowledge() (hybrid_alpha 0.3, search_method auto, no category or folder filter), so the figures describe what a client gets when it passes no options; this tool takes no search options. Use get_index_stats() for system health checks instead. Use search_knowledge() for actual document retrieval — this tool is for quality measurement only.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"test_cases": "<test_cases>"
},
"name": "evaluate_retrieval"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| test_cases | Yes |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |