consistency_check
Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. For true semantic consistency, use run_semantic_tests with embedding mode. Essential for determinism testing.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| responses | Yes | Array of 2+ LLM responses to compare (same prompt, different runs) | |
| check_facts | No | Check for contradictory numbers/facts across responses (default: true) |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| verdict | No | ||
| fact_drift | No | ||
| avg_similarity | No | ||
| response_count | No | ||
| pairwise_scores | No | ||
| fact_contradiction | No | ||
| length_variance_percent | No |