faf_bench
Read-only
Measure how much the project.faf context improves answer accuracy on your repo. Generates questions from the .faf answer key, grades answers without and with the .faf, and returns the lift with a receipt.
Instructions
Prove the .faf earns its place — measure how much the context is worth, on THIS repo, falsifiably. Questions derive from the project.faf's own populated slots (the .faf is the answer key), so grading is mechanical — no judge, no rubric. action=questions returns the answer-key-safe question set; action=grade takes your answers WITHOUT the .faf (cold) and WITH it (faf), grades both, and returns the cold→with-faf lift with a ✪ receipt. The delta is the product; the cold number belongs to the absence of context, never to FAF.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| faf | No | action=grade: answers produced WITH the project.faf in context. Map of question number → answer string. | |
| cold | No | action=grade: answers produced WITHOUT the .faf (general repo knowledge only). Map of question number → answer string. | |
| path | No | Project path (optional — current directory if omitted). | |
| model | No | action=grade (optional): the model that produced the answers. | |
| action | No | questions = get the answer-key-safe question set to answer; grade = submit cold + with-faf answers to score the delta. Default: questions. | |
| fafTokens | No | action=grade (optional): tokens spent answering with the .faf. | |
| coldTokens | No | action=grade (optional): tokens spent answering cold. |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| faf | No | action=grade — score WITH the .faf. | |
| cold | No | action=grade — score WITHOUT context (absence baseline). | |
| delta | No | with-faf minus cold — the product. | |
| total | No | Number of questions in the set. | |
| action | No | ||
| receipt | No | ✪ receipt — sha256 over the canonical projection; third-party verifiable. | |
| version | No | ||
| protocol | No | in-session — answers are self-reported by the agent under test. | |
| qsetHash | No | Hash of the question set — rides the receipt; same .faf reproduces it. | |
| questions | No | action=questions only — NEVER includes the answer key. |