Grounding Benchmark
faf_benchMeasure how much project.faf context improves AI answers on a repo: generate questions, grade cold vs. with-faf responses, and compute the context lift.
Instructions
Prove the .faf earns its place — measure how much the context is worth, on THIS repo, falsifiably. Questions derive from the project.faf's own populated slots (the .faf is the answer key), so grading is mechanical — no judge, no rubric. action=questions returns the answer-key-safe question set; action=grade takes your answers WITHOUT the .faf (cold) and WITH it (faf), grades both, and returns the cold→with-faf lift with a receipt hash. The delta is the product; the cold number belongs to the absence of context, never to FAF. Reads only.
Tool call template
This tool declares no required parameters. Add optional parameters from the input schema as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {},
"name": "faf_bench"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| faf | No | action=grade: answers produced WITH the project.faf in context. Map of question number → answer string. | |
| cold | No | action=grade: answers produced WITHOUT the .faf (general repo knowledge only). Map of question number → answer string. | |
| path | No | Project path (optional — the active project if omitted). | |
| model | No | action=grade (optional): the model that produced the answers. | |
| action | No | questions = get the answer-key-safe question set to answer; grade = submit cold + with-faf answers to score the delta. Default: questions. | |
| fafTokens | No | action=grade (optional): tokens spent answering with the .faf. | |
| coldTokens | No | action=grade (optional): tokens spent answering cold. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| faf | No | action=grade — score WITH the .faf. | |
| cold | No | action=grade — score WITHOUT context (absence baseline). | |
| delta | No | with-faf minus cold — the product. | |
| total | No | Number of questions in the set. | |
| action | No | ||
| receipt | No | faf-cli's bench receipt — sha256 over the canonical projection; third-party verifiable. | |
| version | No | ||
| protocol | No | in-session — answers are self-reported by the agent under test. | |
| qsetHash | No | Hash of the question set — rides the receipt; same .faf reproduces it. | |
| questions | No | action=questions only — NEVER includes the answer key. |