consistency_check
Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. True semantic consistency: run_semantic_tests in embedding mode. Essential for determinism testing.
Example tool call
This call uses example arguments supplied in the tool’s input schema.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"responses": [
"The capital of France is Paris.",
"Paris is the capital of France."
]
},
"name": "consistency_check"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| responses | Yes | Array of 2+ LLM responses to compare (same prompt, different runs) | |
| check_facts | No | Check for contradictory numbers/facts across responses (default: true) |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| verdict | No | ||
| fact_drift | No | ||
| avg_similarity | No | ||
| response_count | No | ||
| pairwise_scores | No | ||
| fact_contradiction | No | ||
| length_variance_percent | No |