Deployment-readiness harnesses for AI deployment patterns.
Redaction · evals from your real failures · a CI gate that fails fabrication
Website · Getting started · Examples · Roadmap · Contributing · Changelog
You have a prototype: your documents, a model, useful output. What you don't have is the apparatus that makes it responsible to ship — a privacy pre-step, an evaluation built from your own failures, and a gate that stops a quiet prompt tweak from deploying a system that fabricates. That apparatus usually requires an ML engineer. Groundwork scaffolds it instead.
Groundwork's units are deployment patterns, not products — think Terraform modules for trustworthy AI deployments. groundwork init <archetype> scaffolds the whole harness for a pattern:
Archetype | The pattern | Status |
| documents → retrieval → grounded answers → human review | ✅ shipped |
| documents → structured fields → schema validation → human review | ✅ shipped |
research synthesis | questions → evidence → synthesis → human review | planned — see the roadmap |
Every check is deterministic: same inputs, same result, every time — no LLM judge, and no API key for the checks themselves.
git clone https://github.com/nickjlamb/groundwork.git && cd groundwork
npm install && npm run build
npm run demo # document QA: the whole loop passes, fully offline
npm run demo:break # the system starts fabricating — watch the gate go red✗ GROUNDING demo-savings: ungrounded number "14" — not in the provided context
✗ GROUNDING demo-unanswerable-appeals: unanswerable question — the answer did not abstainThe same loop for structured extraction:
npm run demo:extraction # documents → fields, every check green
npm run demo:extraction:break # a guessed date of birth, a dropped field — red, by name✗ EXTRACTION demo-referral: SCHEMA: /referral_date must be string
✗ EXTRACTION demo-referral: FABRICATED field "date_of_birth": document does not state it (got "12 April 1988")That red build is the product: a fabricated figure, a guessed field, or a failed abstention becomes a CI failure, not a user complaint.
Ready for your own system?
npm install -D @pharmatools/groundwork
npx groundwork init # document QA
npx groundwork init extraction # structured extractionRelated MCP server: PubCrawl
You edit | Groundwork runs | The gate enforces |
| Redaction pre-step (Redacta) — identifiers become labelled tokens locally, before anything is sent |
|
| The archetype's eval (OpenGATE) — document QA: anchored facts · numbers trace to context · abstention. Extraction: schema validity · field accuracy vs gold · no fabricated fields |
|
| Cost profiling — measured token usage, savings in leverage order (caching → batching → trimming → routing) | A human still reviews high-stakes outputs — the gate is a floor, not a certification |
For extraction, one schema choice does most of the safety work: nullability is the abstention contract. A field the system must always find is non-nullable; a field the document may not state is nullable, and the system returns null for it — unknown → null, never guessed. A guessed value in a null-gold field fails by name: FABRICATED field "date_of_birth": document does not state it.
Example | What it shows | Run |
Document QA: the full loop on a tiny local system — and the gate catching fabrication by name. Zero network. |
| |
Structured extraction: schema validation, field accuracy, and the gate naming a guessed date of birth and a dropped required field. Zero network. |
| |
A real Claude-backed system: answer-from-document-only prompting, prompt caching, key-gated evals, measured usage feeding |
| |
Claude filling a schema from fictional referral letters: unknown → null in the prompt contract, the schema as the cached prefix, and the gate catching a guessed date of birth — plus a mock API for offline CI. |
|
groundwork init [archetype] scaffold the harness for a pattern (document-qa, extraction)
groundwork check redaction self-test + the archetype's eval
groundwork check --baseline freeze this run as the regression floor
groundwork check --ci exit non-zero on failure or regression vs baseline
groundwork cost measured token usage + savings, in leverage orderTwo ways to learn it: docs/GETTING-STARTED.md is the reference walkthrough (about half an hour), and the course covers the same ground as checkpoint-driven lessons — including the sabotage test, where you deliberately break your own system to prove the gate catches it, and a lesson on the extraction archetype. The scaffolded playbook (GROUNDWORK.md) lives inside your repo, where your team will actually read it.
The same checks ship as an MCP server (groundwork-mcp, stdio) so Claude Code, Cowork, or Claude Desktop can run them conversationally — check_readiness on a repo, check_answer_grounding on a single answer, check_extraction on a single record (no repo needed), scaffold_harness, cost_summary:
{ "mcpServers": { "groundwork": { "command": "npx", "args": ["-y", "-p", "@pharmatools/groundwork", "groundwork-mcp"] } } }There's also an Agent Skill (skills/groundwork-readiness/) that teaches an agent to run the gate and report results honestly — including refusing to present a green check as a safety certification.
Groundwork is a strong floor, not a guarantee. Deterministic checks catch the failures that can be caught deterministically; they cannot certify an AI system safe. For high-stakes outputs — anything touching health, money, legal standing, or safety — a human must review before the output reaches the person it affects. Extraction raises the stakes quietly: the record feeds decisions, so a wrong field is a wrong decision. The scaffolded playbook says this too, on purpose.
Status — two archetypes shipped (document QA, structured extraction) of a planned three. Depth before breadth: each pattern gets finished properly before the next starts. See the roadmap.
Contributing — bug reports, gold-case patterns, and examples are especially welcome: CONTRIBUTING.md.
Releases — tagged on GitHub, versions on npm, history in CHANGELOG.md.
Built on — OpenGATE (evaluation) and Redacta (privacy), both open source.
MIT. Everything init scaffolds into your repo is MIT-0 — yours, no attribution needed.
Deterministically check a single answer against the context it should be grounded in: required facts (anchors) present, every number traceable to the context, abstention when the question is unanswerable. Same logic as the CI gate — no model, no network. Returns pass/fail with named issues.
| Name | Required | Description | Default |
|---|---|---|---|
| answer | Yes | The answer to check | |
| context | Yes | The retrieved passage(s) the answer must be grounded in | |
| question | Yes | The question that was asked | |
| answerable | No | false → the context cannot answer this; the only correct behaviour is abstention | |
| answer_anchors | No | Facts a correct answer MUST contain (with acceptable rephrasings) | |
| abstain_markers | No | Phrases that count as a valid refusal (defaults cover common forms) | |
| allowed_new_numbers | No | Numbers legitimately absent from the context (e.g. computed totals) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly states that the tool is deterministic, requires no model or network, and returns pass/fail with named issues. It also reveals key validation behaviors around anchors, numbers, and abstention, which goes well beyond a bare description.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single dense sentence with no filler. It front-loads the tool's core purpose, then lists the specific checks and the deterministic/offline nature, with return behavior stated at the end. Every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers what the tool validates, that it is deterministic and offline, and what it returns (pass/fail with named issues). With no output schema present, this is sufficient for an agent to understand the tool's behavior. Minor gaps are the absence of explicit usage examples or edge-case notes, but the overall context is adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description references anchors, traceable numbers, and abstention, which loosely map to answer_anchors, allowed_new_numbers, and answerable/abstain_markers, but it does not add much parameter-specific meaning beyond what the schema already documents.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb and resource ('check a single answer against the context') and enumerates precise checks: anchors present, numbers traceable, abstention when unanswerable. It clearly identifies the tool's function, though it does not explicitly differentiate it from sibling check tools like check_extraction or check_readiness.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives useful context ('Same logic as the CI gate — no model, no network'), implying use when a deterministic, offline check is needed. However, it does not explicitly state when to prefer this tool over alternatives or when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Deterministically check a structured extraction against its hand-labelled gold record: schema validity (nullable fields are the abstention contract), field accuracy with per-field normalisers (dates → ISO, money → minor units) and aliases, and fabrication — a non-null value in a field whose gold is null (a guessed date of birth the document never stated). Same logic as the CI gate — no model, no network. Returns pass/fail with named issues and a per-field verdict.
| Name | Required | Description | Default |
|---|---|---|---|
| gold | Yes | The hand-labelled gold record; null means THE DOCUMENT DOES NOT STATE IT — gold's keys define which fields are compared | |
| record | Yes | The record the system extracted ({ field: value | null }) | |
| schema | No | The JSON Schema the record must validate against (type a must-find field non-nullable, a may-be-absent field nullable) | |
| aliases | No | Per-field acceptable alternative gold values (matched after normalisation) | |
| normalize | No | Per-field normaliser applied to both sides before comparing |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses that it is deterministic, performs no network/model call, and explains the fabrication logic (null gold vs non-null value). It also states the return format ('pass/fail with named issues and a per-field verdict'). It does not mention error conditions or performance, but for a stateless deterministic check this is sufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, with the first sentence packing the purpose and main checks, and the second noting determinism and output. It is concise and front-loaded, though the first sentence is dense and technically heavy, which slightly impacts readability. Overall it earns its place with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has 5 parameters, nested objects, and no output schema, so the description must compensate. It covers the validation logic (schema, accuracy, fabrication), the normalisation and alias handling, and the return structure. It omits potential error cases or size limits, but for a deterministic check these are minor. It gives the agent everything needed to call it and interpret results.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Although the schema provides 100% coverage with descriptions for each parameter, the tool description adds meaningful semantics beyond the schema: it explains that gold's null means 'the document does not state it' and that gold's keys define which fields are compared, clarifies that aliases are matched after normalisation, and elaborates on the normaliser types. This enriches the parameter understanding well past the schema baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('check') and a precise resource ('a structured extraction against its hand-labelled gold record'), then enumerates the exact checks performed: schema validity, field accuracy with normalisers/aliases, and fabrication. This distinguishes it clearly from sibling tools like check_answer_grounding or check_readiness, which address different evaluation concerns.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Same logic as the CI gate — no model, no network', which communicates the deterministic, offline nature and implies it is the authoritative extraction validator. It does not explicitly list when NOT to use it or name an alternative, but the CI gate reference provides strong contextual guidance. A borderline 5, but the lack of explicit 'use this instead of X' keeps it at 4.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run groundwork check in a repo that has a scaffolded harness: redaction self-test, then the archetype's eval (grounding for document-qa, extraction for extraction) against the repo's own gold set. Returns the exit code, the main scorer's metrics, and any named failures — exactly what CI would report.
| Name | Required | Description | Default |
|---|---|---|---|
| ci | No | Also compare against the committed baseline and fail on regression | |
| dir | Yes | Absolute path to the repo root (the directory containing groundwork/) | |
| baseline | No | Freeze this run as the regression floor |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses the operations (redaction self-test, eval against gold set), the return payload (exit code, metrics, failures), and frames it as 'exactly what CI would report,' implying a non-mutating status check. It does not explicitly state non-destructiveness or side effects, but the tone and purpose strongly imply it.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single dense sentence, front-loaded with the command, no wasted words. Every element (prerequisite, steps, output) earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and no annotations, the description provides the essential context: prerequisite, process, and return value. It does not cover error cases or side effects, but for a check tool this is sufficiently complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The description adds no extra meaning beyond the schema for parameters; it mentions the run and output but does not explain how ci or baseline affect behavior, so no additional contribution.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Run groundwork check in a repo'), describes the exact sequence (redaction self-test then archetype eval), and specifies the output (exit code, metrics, failures). This clearly distinguishes it from sibling tools by mentioning the redaction self-test and the full gate.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Clearly states the prerequisite ('a repo that has a scaffolded harness') and what the tool does, making when to use it obvious. It does not name alternatives or explicitly state when not to use it, but the condition is specific enough to guide selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run groundwork cost in a repo: measured token usage from eval runs, with savings recommendations in leverage order (prompt caching, batching, context trimming, model routing last).
| Name | Required | Description | Default |
|---|---|---|---|
| dir | Yes | Absolute path to the repo root |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden; it discloses that it executes a command and describes the output shape, including the leverage-order for recommendations. It does not state whether the command has side effects, requires network access, or has prerequisites, which is a gap for an execution-type tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The entire description is one efficient sentence that leads with the concrete command and packs in the key outputs and their ordering. There is no filler or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter tool with no output schema, the description covers the core behavior and the nature of the returned recommendations. It omits prerequisites or failure behavior, but these are not critical for a straightforward cost-summary invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the only parameter (dir) is already documented as 'Absolute path to the repo root'. The description reinforces the repo context but adds no new parameter semantics beyond the schema, so the baseline score applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb/resource pair ('Run groundwork cost') and states the exact output: measured token usage from eval runs and savings recommendations. This clearly distinguishes it from the sibling correctness/readiness/scaffolding tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides clear context that the tool operates in a repo and produces cost information, so an agent can infer when to select it. It does not explicitly name alternatives or when-not-to-use conditions, but with unrelated siblings this is a minor gap.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run groundwork init [archetype] in a repo: writes the adapter boundary, redaction config, gold-set templates, GitHub Action, and playbook for a deployment pattern — document-qa (documents → grounded answers) or extraction (documents → structured fields). Never overwrites existing files. After scaffolding, the human wires groundwork/adapter.mjs and replaces the example gold cases with real ones.
| Name | Required | Description | Default |
|---|---|---|---|
| dir | Yes | Absolute path to the repo root to scaffold into | |
| archetype | No | Deployment pattern to scaffold (default: document-qa) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses the key non-destructive behavior ('Never overwrites existing files') and lists what is written. However, it does not explain what happens if files already exist (fail, skip, or warn), whether permissions are needed, or how errors are handled. This is adequate but not thorough for a mutation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single compact block, front-loading the command and its purpose, then listing outputs and archetypes. It also includes a critical safety note and a follow-up step without unnecessary filler. Slightly dense but well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a scaffolding tool with two parameters and no output schema, the description covers what it does, what it writes, the archetype options, the non-overwrite guarantee, and the human follow-up. It lacks details on error handling or behavior when files already exist, which are minor gaps. Overall, it is complete enough for an agent to call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for both parameters. The description adds meaningful semantics for the archetype enum by explaining what each pattern does ('documents → grounded answers' vs 'documents → structured fields'), which is not in the schema. For dir, it does not go beyond the schema's 'absolute path' description, but the added archetype context raises the score above baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb ('Run groundwork init') and the specific resources written (adapter boundary, redaction config, gold-set templates, GitHub Action, playbook). It also names the two deployment patterns, distinguishing it from sibling tools that focus on checking or costing rather than scaffolding.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool is used to scaffold a repo for a deployment pattern, and it notes a follow-up human action. However, it does not explicitly state when to use this tool versus alternatives or when not to use it. No direct sibling performs scaffolding, so some guidance would help, but the context is implied.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.4.0check_answer_groundingcheck_extractioncheck_readinesscost_summaryscaffold_harnessScored across 5 tools
Each tool has a unique role: scaffolding, deterministic per-example checks, full CI-readiness evaluation, and cost reporting. The two check tools are cleanly separated by archetype (answer grounding vs extraction), and check_readiness is a higher-level composite rather than a duplicate.
Four tools follow a clear verb_noun snake_case pattern (check_*, scaffold_*), but cost_summary breaks the pattern by being noun_noun. The naming is still readable and predictable overall, with no mixed casing or chaotic style.
Five tools is a well-scoped set for the Groundwork CLI: scaffold, validate readiness, run archetype-specific checks, and analyze cost. Each tool earns its place with no redundancy or bloat.
The tool surface covers the full core workflow: scaffold_harness sets up a project, check_readiness runs the CI-equivalent eval, check_answer_grounding and check_extraction handle both supported archetypes, and cost_summary closes the loop on optimization. There are no obvious dead ends or missing lifecycle steps.
Enterprise AI Control Plane: governance, guardrails, spend tracking, compliance & smart routing.
AI routing, memory, guardrails, and governance. Routes across Claude, GPT, Gemini.
Extract PDFs to Markdown, RAG chunks and cited tables; publish tracked Doc Links with read stats.
Deterministic runtime safety for AI agents: scan PII, gate tool actions, verify LLM output.
