claude-faf-mcp

Overview Schema Source Related Servers Discussions Admin

Grounding Benchmark

faf_bench

Read-onlyIdempotent

Measure how much project.faf context improves AI answers on a repo: generate questions, grade cold vs. with-faf responses, and compute the context lift.

Instructions

Prove the .faf earns its place — measure how much the context is worth, on THIS repo, falsifiably. Questions derive from the project.faf's own populated slots (the .faf is the answer key), so grading is mechanical — no judge, no rubric. action=questions returns the answer-key-safe question set; action=grade takes your answers WITHOUT the .faf (cold) and WITH it (faf), grades both, and returns the cold→with-faf lift with a receipt hash. The delta is the product; the cold number belongs to the absence of context, never to FAF. Reads only.

Input Schema

TableJSON Schema

Name	Required	Description
`faf`	No	action=grade: answers produced WITH the project.faf in context. Map of question number → answer string.
`cold`	No	action=grade: answers produced WITHOUT the .faf (general repo knowledge only). Map of question number → answer string.
`path`	No	Project path (optional — the active project if omitted).
`model`	No	action=grade (optional): the model that produced the answers.
`action`	No	questions = get the answer-key-safe question set to answer; grade = submit cold + with-faf answers to score the delta. Default: questions.
`fafTokens`	No	action=grade (optional): tokens spent answering with the .faf.
`coldTokens`	No	action=grade (optional): tokens spent answering cold.

Output Schema

TableJSON Schema

Name	Required	Description
`faf`	No	action=grade — score WITH the .faf.
`cold`	No	action=grade — score WITHOUT context (absence baseline).
`delta`	No	with-faf minus cold — the product.
`total`	No	Number of questions in the set.
`action`	No
`receipt`	No	faf-cli's bench receipt — sha256 over the canonical projection; third-party verifiable.
`version`	No
`protocol`	No	in-session — answers are self-reported by the agent under test.
`qsetHash`	No	Hash of the question set — rides the receipt; same .faf reproduces it.
`questions`	No	action=questions only — NEVER includes the answer key.

Schema Changelog

Changes observed during successful MCP inspections.

Changed3 schema fields changedv6.0.0Sep 13, 2026

changedInput schema / properties / path / description

Previous value: -"Project path (optional — current directory if omitted)."New value: +"Project path (optional — the active project if omitted)."

changedOutput schema / description

Previous value: -"Question set (action=questions) or the cold→with-faf grading + ✪ receipt (action=grade)."New value: +"Question set (action=questions) or the cold→with-faf grading and its receipt (action=grade)."

changedOutput schema / properties / receipt / description

Previous value: -"✪ receipt — sha256 over the canonical projection; third-party verifiable."New value: +"faf-cli's bench receipt — sha256 over the canonical projection; third-party verifiable."

Addedv5.12.0Jun 22, 2026

TDQS

A3.9/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, and non-destructive; the description adds real context beyond them - the questions are 'answer-key-safe', grading is mechanical with no judge or rubric, and the result is a cold→with-faf lift with a receipt hash. It also confirms the read-only nature ('Reads only'), consistent with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Content is dense and front-loaded: the purpose, the mechanism, and the two-action flow appear in order with no filler. Sentences are long and somewhat jargon-heavy, but each carries information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, the description needn't explain return values, and it still covers the two actions, their sequencing, and the meaning of the cold/faf split. An agent has enough to invoke it correctly, with only minor gaps like token-field usage not being explained narratively.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, including per-parameter descriptions for faf, cold, action, and token fields, so the schema does the heavy lifting. The description reinforces the cold-vs-faf distinction and the default action but adds little syntactic detail beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a concrete purpose - measuring how much the project.faf context is worth on the current repo via mechanically graded questions - and enumerates both actions (questions, grade). It clearly differs from sibling tools like faf_score or faf_about, though the metaphorical opening ('earns its place') adds flavor before the operational statement.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives a clear workflow: call action=questions to get the answer-key-safe set, then action=grade with cold and faf answers to obtain the delta. It does not explicitly state when to prefer this over siblings such as faf_score, but the internal usage guidance is concrete.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server