faf_bench
Read-only
Grade answers against a project's .faf context to measure the lift in accuracy. Submit cold and with-context responses; receive a mechanized delta and receipt.
Instructions
Prove the .faf earns its place — measure how much the context is worth, on THIS repo, falsifiably. Questions derive from the project.faf's own populated slots (the .faf is the answer key), so grading is mechanical — no judge, no rubric. action=questions returns the answer-key-safe question set; action=grade takes your answers WITHOUT the .faf (cold) and WITH it (faf), grades both, and returns the cold→with-faf lift with a ✪ receipt. The delta is the product; the cold number belongs to the absence of context, never to FAF.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| faf | No | action=grade: answers produced WITH the project.faf in context. Map of question number → answer string. | |
| cold | No | action=grade: answers produced WITHOUT the .faf (general repo knowledge only). Map of question number → answer string. | |
| path | No | Project path (optional — current directory if omitted). | |
| model | No | action=grade (optional): the model that produced the answers. | |
| action | No | questions = get the answer-key-safe question set to answer; grade = submit cold + with-faf answers to score the delta. Default: questions. | |
| fafTokens | No | action=grade (optional): tokens spent answering with the .faf. | |
| coldTokens | No | action=grade (optional): tokens spent answering cold. |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| faf | No | action=grade — score WITH the .faf. | |
| cold | No | action=grade — score WITHOUT context (absence baseline). | |
| delta | No | with-faf minus cold — the product. | |
| total | No | Number of questions in the set. | |
| action | No | ||
| receipt | No | ✪ receipt — sha256 over the canonical projection; third-party verifiable. | |
| version | No | ||
| protocol | No | in-session — answers are self-reported by the agent under test. | |
| qsetHash | No | Hash of the question set — rides the receipt; same .faf reproduces it. | |
| questions | No | action=questions only — NEVER includes the answer key. |