eval_schema_compliance
Check that LLM-generated JSON conforms to a provided schema, reporting per-field validation errors for targeted fixes.
Instructions
Validate that an LLM output conforms to a JSON Schema.
Wraps multivon-eval's SchemaEvaluator. Parses the LLM output
as JSON (tolerantly strips markdown code fences), then validates
the parsed structure against the provided JSON Schema dict. Reports
per-field validation errors — not just "valid/invalid".
For Pydantic-model validation or more advanced setups (custom validators, recursive schemas), use the multivon-eval SDK directly.
Args:
output: The LLM-generated text expected to contain JSON.
schema: A JSON Schema dict (Draft 7). Example:
{"type": "object", "required": ["title", "score"], "properties": {"title": {"type": "string"}, "score": {"type": "number"}}}.
strict: If True, additional fields not in the schema are
treated as failures.
Returns:
{"score": 0.0-1.0, "passed": bool, "reason": str, "threshold": float, "evaluator": "schema_compliance"}.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| output | Yes | ||
| schema | Yes | ||
| strict | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||