run_semantic_tests
Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.
Example tool call
This call uses example arguments supplied in the tool’s input schema.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"cases": [
{
"actual": "Paris is the capital of France",
"expected": "The capital of France is Paris"
}
]
},
"name": "run_semantic_tests"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | tfidf (default): fast, free, lexical. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key. | |
| cases | Yes | Array of (actual, expected) pairs to evaluate. | |
| api_key | No | OpenAI API key — required only when mode is embeddings. | |
| thresholds | No | Pass/fail thresholds (defaults: cosine 0.75, rouge_l 0.5). | |
| require_all | No | If true (default), all cases must pass for overall PASS. If false, at least one case passing returns PASS. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | ||
| cases | No | ||
| summary | No | ||
| verdict | No | ||
| thresholds | No |