response_quality_score
Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with the reason when none is given, because relevance and correctness cannot be read off word overlap (a correct one-word answer shares no words with its question). Meaning against a reference answer: run_semantic_tests. Also returns unscored signals: question-term overlap, average sentence length, markdown.
Example tool call
This call uses example arguments supplied in the tool’s input schema.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"question": "q",
"response": "anything"
},
"name": "response_quality_score"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| question | Yes | The original question/prompt | |
| response | Yes | The LLM response to score | |
| max_length | No | Maximum character length. Scored as a proportional penalty beyond it. | |
| expected_keywords | No | Keywords a correct answer must contain, as whole words (case-insensitive; a plural in -s/-es counts). Scored as coverage. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| grade | No | ||
| stats | No | ||
| reason | No | ||
| signals | No | ||
| breakdown | No | ||
| max_score | No | ||
| total_score | No | ||
| not_measured | No |