engineering-audit
You can perform comprehensive engineering audits of code repositories. The server lets you list and retrieve rule domains; start and monitor audit runs with metadata; configure audits interactively via a local web page or headlessly via environment variables; record per-rule verdicts, findings, and coverage with mandatory completeness; view run progress; preview and (with explicit confirmation) file GitHub issues; submit feedback to the tool author; and render final self-contained HTML and JSON reports. All operations guard against accidental overwrites and require user consent for outward-facing actions.
Integrates with OpenAI Codex CLI to run standalone audits against engineering rules and load rules inline via AGENTS.md fragments.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@engineering-audit Audit the current repo against the engineering rules and generate a report"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
engineering-audit
TL;DR: engineering-audit turns the AI coding assistant you already use (Claude Code, Codex CLI, Gemini CLI) into an engineering-practice auditor. It has two audit modes:
Full repository audit: your assistant sweeps a whole repository against a pack of sourced engineering rules and produces a self-contained HTML report. Every finding says what is wrong, why it matters and how to fix it, with the citation behind the claim attached, and can be filed as a GitHub issue or copied out for pasting anywhere.
Inline decision-time checks: one-line triggers in your assistant's context make it load the relevant rules at the moment you are making a matching decision (designing a schema, cutting a branch, shaping an API). No report, just the right rules at the right moment.
Starting a new project? The optional Engineering Grill uses the same rules before you build. Your assistant interviews you, works out which domains matter, and records the decisions and checks the project will need, without expecting you to understand the framework first.
Three complete rule domains ship in this repository, ready to run: data modelling, testing strategy and presenting data, 54 rules with their full source citations, in examples/taster-rules/. You can run a real audit right now with no sign-up; the full sixteen-domain, 260-rule pack is available on request (Rules access).
Need a walk through? There is a free interactive walkthrough → that takes you from a bare Windows, Mac or Linux PC to a finished audit of your own repository. It rehearses every step below in a simulated terminal, with the wrong turns coached, before you do them for real. No terminal experience assumed.
Save me the chit chat: show me how to install it →
How to use
From nothing to a first audit in five steps. The taster rules ship in this repository, so none of this needs access to the full pack.
Never touched a terminal? Rehearse these five steps first in the free interactive walkthrough linked at the top: it takes a non-technical founder from a bare Windows, Mac or Linux PC to a finished audit, in a simulated terminal with the wrong turns coached.
Step 1: pick your assistant
The tool works through the assistant you already drive: Claude Code or OpenAI Codex CLI (both proven end to end; standalone audits recorded 2026-08-09 and 2026-08-10, with a further Codex run on Linux 2026-08-19), or Gemini CLI (documented, not yet exercised end to end; see the support matrix). GitHub Copilot is not supported.
Step 2: dependencies
Dependency | Why you need it | Check it |
runs the MCP server via |
| |
git | clones this repository for the taster rules, and any other rules pack |
|
your assistant's CLI | drives the audit and hosts the MCP registration |
|
GitHub CLI | only if the assistant should file findings as GitHub issues; filing from the report page instead needs only a PAT in your browser |
|
There is no pip install and no npm anywhere: uvx fetches and runs the tagged release
directly.
Step 3: get the rules onto disk
Clone this repository. The taster pack (the three complete domains named in the TL;DR) is
a working rules directory at examples/taster-rules/:
git clone https://github.com/rodlunt/engineering-audit
cd engineering-auditHave the full pack instead? Registration below is identical, just point --rules-dir at
that clone's domains/ directory. See Rules access for how to ask.
Step 4: register the tool with your assistant
Every command below is pinned to the current release tag (@v0.15.0) rather than the
moving main branch: an unpinned git dependency resolves to whatever main holds at
install time and silently moves on later cache refreshes. Find the latest tag on the
Releases page. Updating to a newer
tag is a deliberate act with its own command, below, not something that happens on a
git pull: the pin lives in your assistant's MCP registration, not in this repository.
Paths beginning with /path/to/ are placeholders. Replace them with the real absolute path
on your computer; run pwd inside a folder when you need to see its absolute path.
Claude Code
Register the server (swap in the taster path from Step 3, or your full-pack path):
claude mcp add engineering-audit --scope user -- uvx --from git+https://github.com/rodlunt/engineering-audit@v0.15.0 \
engineering-audit-mcp --rules-dir /path/to/engineering-audit/examples/taster-rules--scope user registers it for every repository. Without it claude mcp add defaults to
local scope, which registers the server for the current directory alone and leaves it
unavailable everywhere else. That failure is silent and lands later (issue #245): a skill
run in any other project reports list_domains unavailable with nothing pointing at scope
as the cause. If that happens, run claude mcp list in the project where it failed; if
engineering-audit is missing there but present in the directory you installed from, it
was registered without --scope user. Fix it with claude mcp remove engineering-audit
followed by the add command above.
To update to a newer tag, remove first and re-add. claude mcp add refuses to
overwrite an existing name (MCP server engineering-audit already exists in user config),
so changing the tag on its own is not enough:
claude mcp remove engineering-audit
claude mcp add engineering-audit --scope user -- uvx --from git+https://github.com/rodlunt/engineering-audit@v0.15.0 \
engineering-audit-mcp --rules-dir /path/to/engineering-audit/examples/taster-rulesThe change only takes effect in a new session. The one you are in keeps the server it started with, so verify after restarting rather than before:
claude mcp list | grep engineering-audit # must show the tag you just setbegin_run's response states tool_version too. If it names the old version, the session
predates the re-registration and the run is not exercising the build you think it is.
Install the audit skill (gives you a natural-language entry point: "audit this repo"):
cd /path/to/engineering-audit
scripts/install-skills.sh auditIt copies rather than symlinks, so the installed skill does not change under you when this
checkout switches branch. scripts/install-skills.sh --check reports when a copy is stale.
Full details: integrations/claude-code/.
OpenAI Codex CLI
Register the server (verified against codex-cli 0.114.0):
codex mcp add engineering-audit \
--env ENGINEERING_AUDIT_RULES_DIR=/path/to/engineering-audit/examples/taster-rules \
-- uvx --from git+https://github.com/rodlunt/engineering-audit@v0.15.0 engineering-audit-mcpInline mode: generate the trigger fragment and append it to your repo's AGENTS.md (or
~/.codex/AGENTS.md for all repos):
uvx --from git+https://github.com/rodlunt/engineering-audit@v0.15.0 engineering-audit-fragments \
--rules-dir /path/to/engineering-audit/examples/taster-rules --out-dir .
cat AGENTS-fragment.md >> AGENTS.mdStandalone audit: in a codex session, ask it to read AUDIT.md from this repository and
run the audit. Headless notes and caveats: integrations/codex/.
Gemini CLI
There is no packaged extension. Gemini CLI resolves an extension only from a repository
root, which would put a manifest, a GEMINI.md and a commands/ directory in the root of a
tool that also serves Claude Code and Codex, and none of it was ever exercised against a real
Gemini CLI. Registering the server by hand is one paste and has no such cost.
Add the server to ~/.gemini/settings.json (or a project-level .gemini/settings.json),
swapping in the taster path from Step 3:
{
"mcpServers": {
"engineering-audit": {
"command": "uvx",
"args": [
"--from",
"git+https://github.com/rodlunt/engineering-audit@v0.15.0",
"engineering-audit-mcp"
],
"env": {
"ENGINEERING_AUDIT_RULES_DIR": "/path/to/engineering-audit/examples/taster-rules"
}
}
}
}Then start gemini in the repository you want audited and paste:
Audit this repository against the engineering rules. Read AUDIT.md from the
engineering-audit repository and follow it, driving the engineering-audit MCP tools
through to a rendered report.Inline triggers work too, and need no extension: generate the fragment and merge it into
whichever GEMINI.md tier you want it to apply to.
uvx --from git+https://github.com/rodlunt/engineering-audit@v0.15.0 engineering-audit-fragments \
--rules-dir /path/to/engineering-audit/examples/taster-rules --out-dir .
cat GEMINI-fragment.md >> GEMINI.mdGemini support is documented, untested: Gemini CLI was not available to exercise any of
it. Check gemini --help before an unattended run. Details and caveats:
integrations/gemini/.
Optional: plan before you build
Install Engineering Grill after registering the MCP server if you want a guided, plain-English planning conversation before code is written. It reads the domains from the connected rules pack, so the taster pack gives a three-domain grill and the full rules pack gives the complete framework.
Headless / CI
Skip the interactive configuration page by pointing ENGINEERING_AUDIT_CONFIG at a saved
configuration JSON (shape documented in AUDIT.md); get_config then returns
immediately. Example driver, Claude Code:
claude -p "Read AUDIT.md at <path> and audit this repository via the engineering-audit \
MCP tools." --mcp-config mcp.json --allowedTools "mcp__engineering-audit__*,Read,Glob,Grep"Every run also checks this repository's tags for a newer release, on by default (see
Security for what that discloses). On an air-gapped machine, or anywhere the
network cost or the call itself is unwanted, pass --no-update-check to engineering-audit-mcp
or set the ENGINEERING_AUDIT_NO_UPDATE_CHECK environment variable.
What keeps the staleness checks working
Two checks tell you whether what you are running is current: one for the tool, one for the rules
pack. Neither ever guesses. A check that could not run reports could-not-check, which is a
distinct state from current, so a stale build is never reported as fine. What that honesty does
not do is tell you when a check has gone blind, and both go blind on install shapes that look
perfectly ordinary.
Check | Attached when | Blind when | What you lose |
Tool build | installed from a git URL, which is what every command in Step 4 does | installed from a downloaded archive or a plain wheel, or run from a local or editable checkout | nothing warns you that a pin left on an old tag, or a stale |
Rules pack | the rules directory is a git clone with an | the rules arrived as a downloaded zip, were vendored into another repository, have no remote, or have uncommitted changes | nothing warns you that the run is judging your repository against superseded rules |
Following Step 3 and Step 4 as written keeps both attached: git clone for the rules, and
uvx --from git+...@<tag> for the tool. Any other shape still runs and still audits correctly.
It just cannot tell you it is out of date.
The provenance rows in the report header carry the answer for the run in front of you. Both
reading could-not-check at once is the combination to watch for, because that is a build of
unknown age judging your repository against rules of unknown age, with nothing able to detect
either.
Step 5: run your first audit
Ask your assistant to audit the repository you have open ("audit this repo against the
engineering rules"), tick the domains on the configuration page that opens, wait for the
sweep, then open audit-output/report.html. What to expect while it runs is in
What a run looks like; what it costs in tokens is in
What a full run costs.
Related MCP server: code-review-mcp
Engineering Grill, before the code exists (BETA)
Status: beta, and the label is meant literally. The interview shipped in v0.13.0 as a separate
interrogate skill and was folded into Engineering Grill by issue #239: one pre-build skill, not
two. Nobody has yet completed a full interactive session with it. Expect it to change.
An audit sweeps the rules over a repository that already exists. Engineering Grill runs them the other way, as questions about work that has not started. It classifies every domain the pack returns, derives the full question set from the rules of those whose triggers genuinely fire, then puts the highest-consequence questions first, one at a time, in the Hot Seat. The rest are held, not discarded: it names the total, offers a deep dive through everything remaining, and records what was answered, what was deferred and what was never asked.
Triage is global on purpose. One domain's third-best question routinely matters more than another's first, and a per-domain ranking cannot see that. Questions carry a reversibility grade and a blast radius so they can be compared across domains at all.
The rule that shapes the whole design is that no full domain document enters the conversation with the user. That is stated as an invariant rather than an architecture: sub-agents satisfy it where the host has them, serial read-and-discard satisfies it where the host does not, which is what keeps one skill shippable on more than one assistant.
Ten of the sixteen domains are design-time by their own Load this when: statements, so most of
the pack was already pointed at the moment before the code, with nothing to deliver it there.
It never starts a run. It calls list_domains and get_domain only, both of which work with no
run in progress, so it works on a directory that is not a repository yet or on nothing but a
description. Setup is in integrations/engineering-grill/, and
the optional hook that offers it when plan mode starts is in
integrations/claude-code/README.md.
What has been exercised: question derivation on three domains (d02, d01, d15) against one brief, the no-run guarantee, and the hook's failure paths.
What has never been run even once: the cross-domain triage. It is the newest part and the part everything else now depends on, so treat its output with more suspicion than the rest.
What has not: the other thirteen domains, and the interactive loop itself with a real person answering. Nobody has yet finished a full interrogation. Until they have, treat the shape of the session as unproven and the question quality as sampled rather than measured.
This is a beta of a Claude Code integration specifically. There is no Codex or Gemini equivalent,
and unlike audit there is no assistant-neutral protocol document behind it: the skill file is
the whole specification. That is deliberate for now and is the first thing to change if it earns
its place.
Scope
This documentation states functional behaviour (what the tool does) and the security and
privacy properties documented inline (how the access token is held, what telemetry you opt
into, what leaves your machine and when). The two human-facing surfaces, the configuration page
and the report page, target WCAG 2.2 Level AA. Applying the
same proven versus documented, untested distinction used below for the
assistant integrations: one criterion is proven, the rest are documented, untested. Contrast is
machine-checked on every test run, tests/test_report_stylesheet.py computes contrast ratios
from the CSS custom properties and asserts 4.5:1 in both the light and dark palettes. No
keyboard-only pass and no screen reader pass has been recorded against either surface, and
neither template currently sets an explicit minimum target size on its interactive controls, so
those criteria (and WCAG 2.2's target size, success criterion 2.5.8, specifically) have not been
checked by anyone. The target is real; only the contrast ratio has a recorded check behind it.
Performance is explicitly out of scope for now: no throughput or latency target is stated or
tested anywhere in this repository.
How it works
The tool is a local MCP server (Python, stdio). It points at a local directory of rule documents and serves them to the agent driving the audit. The agent supplies the judgement; the server supplies everything that must not depend on an LLM's memory: schema-validated finding capture (a rule the agent did not check can never be recorded as a pass), the configuration page, deterministic report rendering, source citations attached from the rules pack itself, and GitHub issue filing with an explicit confirmation step.
Two audit modes, plus an optional project-start skill:
Engineering Grill: before code is written, the assistant sorts the loaded domains by relevance, asks framework-backed questions and records the resulting plan. It uses only the read-only
list_domainsandget_domaintools and does not start an audit run.Standalone audit: tick the domains to audit on a local configuration page (or supply a saved config for headless runs), the agent sweeps the repository, and you get
report.htmlplus optional GitHub issues.Inline: one-line triggers merged into your assistant's instruction context tell it to call
get_domain(...)at decision moments (designing a schema, cutting a branch, shaping an API), so the rules arrive exactly when they are useful.
Support matrix
Assistant | Inline mode | Standalone audit |
Claude Code | proven (in daily use via skills) | proven (recorded run 2026-08-09) |
OpenAI Codex CLI | documented, untested | proven (recorded run 2026-08-10) |
Gemini CLI | documented, untested | documented, untested |
GitHub Copilot | unsupported | unsupported |
"Proven" means a recorded end-to-end run exists. "Documented, untested" means the integration follows the assistant's official documentation, with individually verified pieces labelled in the integration README, but no full audit has been exercised on it yet. Copilot is deliberately unsupported rather than silently absent.
What a run looks like
A standalone audit is a conversation plus one browser page. From your seat:
Ask for the audit ("audit this repo against the engineering rules"). The assistant gathers run metadata and starts the run.
A configuration page opens in your browser (
http://127.0.0.1:<port>/). Opening it is best-effort: in a remote or display-less session no tab can appear, so the assistant also prints the URL; open it yourself if nothing popped up. Tick the domains to audit, choose report-only or GitHub issue filing, and submit. Nothing proceeds until you submit: the tool never falls back to a domain selection you did not make.The assistant sweeps the repository domain by domain. This is the slow part: minutes for a small repository and a few domains, longer for a big selection. You can ask for progress; it can report which domains are recorded and which remain.
Everything lands in
audit-output/inside the audited repository:report.html(the deliverable, openable in any browser; the assistant offers to open it when the run finishes) andrun-state.json(the raw machine-readable results, which can re-render the same report later viaengineering-audit-render). If you chose GitHub filing, the assistant previews the issues and asks before filing anything.
audit-output/ belongs to the audited repository, not to this tool. Commit it, ignore it
or delete it as that repository's own conventions dictate.
What a full run costs
A full sweep is token-hungry, and the configuration page's domain tick boxes are the cost control: cost scales close to linearly with the domains you tick. Budget from these recorded runs rather than guessing, one row per host that has completed one:
Host | Tool version | Scope | Active time | Findings | Tokens |
Claude Code, Fable 5 orchestrating Sonnet subagents (this repository, 2026-08-09) | 0.4.0 (established from tag history, not stamped into a retained report; see docs/example-audit-cost.md) | all 16 domains, 260 rules | 47 minutes end to end, sweeps running four at a time | 33 (every one filed as a GitHub issue) | 2,010,691 subagent tokens, roughly 100k to 170k per domain (excludes the orchestrator) |
Codex CLI 0.147.0, gpt-5.6-sol at high reasoning effort (external React SPA, roughly 344 files, 2026-08-10) | 0.5.1 | all 16 domains of the standard pack | 19 minutes 21 seconds | 32 (122 rules could not be evaluated) | 6,172,397 input plus output, of which 96% is cached input; 269,293 non-cached input plus output |
Both rows predate v0.9.0 and v0.9.1, which added required per-finding and per-domain
output that neither run had to produce; see the comparability note in
docs/example-audit-cost.md before budgeting a current run from
either figure.
The two token columns are not the same measurement and must not be subtracted or averaged. Codex does not fan out to one subagent per domain the way the Claude Code skill does, and that single difference drives everything: fanning out gives each subagent a small fresh context and bills mostly uncached input, while staying in one long context re-reads a large accumulated context every turn. Hence Codex's 6.17M being 96 per cent cache reads. The Claude Code figure counts per-subagent totals and excludes the orchestrator conversation entirely. Use each row within itself, for the shape of run it describes. Full per-host detail, including what each run did and did not measure, is in docs/example-audit-cost.md.
The report
Self-contained HTML, generated locally; nothing leaves your machine unless you choose to send or file it. It is a written report meant to be read top to bottom, in this order:
A computed headline, first thing on the page: one sentence naming what needs attention first, built from the run's own counts, with a second line saying what was set aside or could not be evaluated so a partial sweep cannot read as a clean bill of health.
Findings, sorted worst first rather than in the order they were recorded, each in three parts (the issue and location, why it matters, suggested fix) with the rule's citation appended automatically from the rules pack. The tool refuses to publish a finding whose rule carries no citation. Each finding also carries its domain's self-assessed confidence and whether that domain's rule text was fetched this run, so a finding from a shaky domain does not look identical to one from a solid one. The confidence never appears on its own: it ships with how many of that domain's rules could not be evaluated, out of how many, so a domain claiming high confidence over half a domain it could not check says so in the same breath. The four severity levels are defined on the page, and stated as assigned by the assistant named in the header rather than measured.
Issues: tick boxes to select findings, then file them to GitHub directly from the report (fine-grained PAT, used in memory only, sent only to api.github.com), or copy the selected set for pasting into an LLM or editor, or copy them one at a time. Only critical and high findings are ticked on load; nothing is hidden, and an already-filed finding shows unticked with a link to it.
A tool performance summary about the audit run itself, below the findings rather than above them: one table with a row per domain carrying rule verdicts, findings by severity, files inspected and skipped, confidence and fetch status, with the could-not-evaluate and not-applicable reasons in full, and an evidence boundary block naming what each domain did not read. Every number ships with its base, a domain set aside in full cannot read as a domain swept clean, an unchecked rule is never presented as a pass, and a value the tool was told rather than measured says so. Longer sections sit behind summaries that each carry their own numbers, so the detail is one click away rather than occupying the top of the page.
Feedback to the author: freeform text, two optional questions asking what you concluded from the report and what you would fix first, and tick-box consent over which run statistics accompany any of it. Everything here is off by default and never prefilled. Finding text never leaves your machine through this channel.
It prints sensibly too: the issue-filing section is dropped from print rather than clipped, with a line saying where to find it, and collapsed sections are expanded on paper.
A live example: docs/demo/report.html (download and open locally;
GitHub does not render raw HTML in the browser). Generated from the invented demo rules pack
in tests/fixture_pack, not a real audit against a real repository.
The rules
The author's rules pack covers sixteen decision domains, 260 rules in all, each rule carrying a cited source, a volatility tier and a verification date, and each domain proven against a real system before it is trusted:
# | Domain | Rules | Fires when you are... |
d01 | Designing a Data Model | 15 | modelling entities, choosing keys, constraints, normalising, writing DDL or migrations |
d02 | Eliciting and Specifying Requirements | 16 | deciding what to build, writing requirements or user stories, checking the right problem is being solved |
d03 | Modelling Structure and Behaviour Before Building | 15 | deciding what to diagram before coding, drawing or reviewing FMC/UML/SysML models |
d04 | Structuring Code and Applying Design Patterns | 14 | designing classes or modules, weighing a design pattern, choosing data structures or error handling |
d05 | Choosing What to Test and How Much | 18 | choosing test levels and coverage, weighing testing against risk, planning load or soak tests and CI gates |
d06 | Structuring a Repo, Branches and CI/CD | 15 | structuring a repository, writing CI/CD workflows, handling automation credentials, cutting releases |
d07 | Handling Untrusted Input and Secure Coding | 16 | writing code untrusted input can reach: forms, auth flows, credentials, sessions |
d08 | Threat Modelling and Security Risk Decision-Making | 15 | running a risk assessment, threat-modelling a system, prioritising vulnerabilities, justifying a control |
d09 | Responding When Something Breaks in Production | 16 | writing incident response plans, defining recovery objectives, running post-incident reviews |
d10 | Designing APIs and Service Contracts | 14 | creating or extending an HTTP API, choosing verbs and status codes, versioning or deprecating an interface |
d11 | Choosing Architecture and Deployment Topology | 16 | picking an application architecture, deciding VM/container/serverless topology, planning scaling and rollout |
d12 | Making an Ethical or Professional Judgement Call | 17 | facing pressure to cut a corner, decisions affecting users or the public, handling personal data |
d13 | Estimating and Pricing Work | 16 | scoping work before quoting, choosing estimation methods, setting contingency, defending an estimate |
d14 | Fault Diagnosis of a Running System | 19 | investigating an outage, a slow or wrong-answering service, or an intermittent bug |
d15 | Interface Design and Prototyping | 17 | laying out a screen, designing a form, writing button and error copy, deciding confirmation vs undo |
d16 | Presenting Data for Decisions | 21 | putting a number, chart or table in front of somebody who has to decide something |
Try it right now: the taster pack
Three complete domains (d01, d05, d16, 54 rules with their full source citations) are published in examples/taster-rules/ as point-in-time exports from the maintained pack. They are a working rules directory, and they are what the How to use steps register by default: run a real audit before asking for anything.
Rules access
The full pack lives in a private repository with access granted per user (the maintained originals, their revision history and proving records). Ask via the rules pack access request form.
Once granted, the pack arrives the same way the taster did: a repository you clone.
git clone https://github.com/rodlunt/engineering-frameworkThen swap the rules path in your Step 4 registration for the clone's domains/ directory
(claude mcp remove then re-add for Claude Code; codex mcp add overwrites in place).
Updating the rules afterwards is git pull in that clone. Keep it a real clone rather
than a downloaded archive: the clone keeps its origin remote, so the run's rules-pack
staleness check stays attached, exactly as described under "What keeps the staleness
checks working".
The
tooling works with any rules directory in the expected format (**Trigger:** header,
### N. Title rules, Rule id: footers with Source: fragments), so you can also write
your own pack.
Development
uv sync
uv run pytest -qCI runs the same suite on every push and pull request. Tests use an invented fixture rules pack; no private rule content exists in this repository. The renderer and configuration page are deterministic and fully testable with no LLM involved.
This is a solo-maintainer repository, and its merge gate is deliberately CI-only: every change
lands via a pull request that must pass the check status; no human review requirement is
configured.
Tracking issues and PRs
Up to now, the PR description has been the deliberate change record for this project: each PR
body explains what changed and why, and that has been treated as sufficient in place of a
separate issue-linked history. From now on, every PR that has a tracking issue links it with a
Closes #N line (or Fixes #N), so the issue tracker and the merge history stay in step
instead of relying on the PR description alone. Release PRs (chore(release): X.Y.Z) and
housekeeping PRs with no issue behind them are the standing exception: there is nothing for
either to close, so no keyword is expected on them. Nothing currently checks for the keyword on
a PR that does carry a tracking issue; a CI check failing a PR with no closing keyword and no
opt-out label is an option the maintainer can pick up separately, not something this policy
statement adds on its own.
Eval harness
evals/ holds a deterministic scorer for audit quality: a small fictional golden repository with
known planted findings and controls, and engineering-audit-eval to check a run-state.json
against them. The scorer is CI-safe and has its own tests; the audit run that feeds it calls a
real LLM and is run and checked by hand. See evals/README.md.
Roadmap
Thin CLI wrapper driving an agent CLI headlessly end to end (a manual, scripted version of this now lives in evals/README.md; a first-class wrapper command is still open).
Remotely served rules with revocable access.
Codex and Gemini support-matrix rows moving to proven once live runs are recorded.
Licence
The tooling in this repository (the MCP server, the deterministic report renderer, the configuration page, and every supporting script) is licensed under Apache-2.0. Rules packs are licensed separately and are not covered by this repository's licence; see Rules access.
Built by Rodney Lunt. If this saved you some time, you can buy me a coffee.
Available Tools
10 toolsbegin_runA
Start a fresh audit run and create its output directory, or resume an interrupted one.
assistant/model/repo_name/repo_commit/started are supplied by the
calling agent; tool_version defaults to the installed package version
if omitted. repo_dir is
the path to the repository being audited, on disk; it is optional,
but file_issues needs it to detect the GitHub repository to file
against, unless a repo is given explicitly on that call instead.
Calling this twice without finishing the first run (via
render_report) is an error, since it would silently discard whatever
domain results have already been recorded; pass replace=True to
explicitly discard the in-progress run and start over.
A run's progress is saved to a crash-recovery file in output_dir as it
goes, so a server that stops mid-run (host restart, dropped
connection, machine asleep) loses at most the domain in flight. When
this call finds such a file for an unfinished run in output_dir it
starts nothing and returns a description of it plus an instruction:
run_started is False, "meta" is absent, and "resumable" says whether
it can be continued at all. Call begin_run again with resume=True to
continue that run (its recorded domains are kept, and the response
lists which domains are still missing), or resume=False to discard it
and start fresh. resume=False is the only way to overwrite saved
results, and replace=True counts as the same explicit decision.
Resuming a run for a DIFFERENT repository is refused outright, as is
resuming one whose saved state cannot be read; either way, nothing is
started and nothing is deleted until told.
environment records the host facts the report header cannot carry, and
its keys are a closed set: 'os' (e.g. "macOS 15.2", "Ubuntu 24.04"),
'host_cli' (the CLI application driving this audit, e.g. "codex",
"claude-code") and 'host_cli_version' (that CLI's version string).
Collect them from the machine you are running on rather than guessing,
and omit any key you cannot determine: an omitted fact and a guessed
one are not the same thing. Any other key is refused outright, because
this metadata is included in feedback issues filed publicly on the
tool's own repository. Do not name the assistant, the model or the tool
version here; all three are already fixed rows in the report header.
The recorded metadata also stamps two provenance SHAs, best-effort:
tool_commit (the git commit the installed tool build was made from,
via its PEP 610 install record) and rules_pack_commit (the loaded
rules pack directory's git HEAD, '-dirty' suffixed if it has
uncommitted changes). Either is None when it could not be
determined, which the report renders as "unknown" rather than
guessing: a report must be traceable to the exact tool build and
rules version that produced it, not just a package version number
that can lag behind either.
started is the caller's own claim about when the run began, taken on
trust like everything else the calling agent asserts. This call also
stamps meta.server_started from the server's own clock at the moment
it runs, independent of that claim; render_report does the same for
meta.server_finished. Neither figure is treated as more authoritative
than the other in the rendered report: a resumed run genuinely spans a
wall-clock gap that is not audit work, so the server's elapsed time is
not automatically the truer duration, but an assistant-supplied
duration that was never checked against anything is worse. The report
states both and flags it when they diverge by more than expected,
rather than presenting an unmeasured number as fact.
The run also performs a best-effort tool update check, comparing
tool_commit against the tool's latest tagged release on GitHub, and
the rules pack against its own remote the same way. Each result
lands in the returned meta (update_check and pack_update_check
respectively), prefixed "current", "stale", "could-not-check" or
"not-checked" (see engineering_audit.update_check for the exact
strings). The calling agent MUST tell the user when either reports
stale or could-not-check, rather than silently proceeding as if the
installed build were confirmed current: this tool is installed via a
pinned uvx reference, and a stale pin or cache would otherwise serve
an old build forever with nothing to say so. This check runs
automatically and discloses only the caller's IP address and the
fact that this repository's tags were queried, no repository
content, findings or paths; it can be turned off with
--no-update-check or the ENGINEERING_AUDIT_NO_UPDATE_CHECK
environment variable, in which case both fields read "not-checked",
which is not something to warn the user about, since turning it off
was their own choice.
Inputs: model (required), started (required), assistant (required), repo_name (required), +7 more. Returns: object
file_issuesA
Preview or file GitHub issues for every recorded finding, via the
user's own gh CLI.
Requires config.issue_mode == "github": if the user chose in-report
delivery instead, this raises rather than filing issues nobody asked
for. Requires at least one recorded domain result.
confirm=False (the default) NEVER files anything and never invokes
gh at all: it returns a preview {repo, count, titles, instruction}
so the calling agent can show the user exactly what is about to be
filed on their repository, and get explicit agreement, before a
single issue goes out. Filing on someone's repo is outward-facing;
this confirmation step is mandatory, not decorative.
confirm=True files one issue per finding that has not already been
filed, so retrying after a partial failure does not double-file the
ones that succeeded. Filed issues are tracked, and returned, per
finding under a key of the form "<rule id>#<n>" (n counting that
rule's findings in recording order), not per rule id: a domain result
may carry two findings for the same rule, and both of their issue
urls have to survive. The target repository is `repo` if given,
otherwise detected from the audited repository directory recorded
by begin_run's repo_dir. If any issue fails to file, filing stops
immediately and the error lists exactly which findings were filed
(with their URLs) and which were not, so a retry knows where to
resume.
Each filed issue carries the "engineering-audit" label. The label is
checked once per call and created on the target repository if it is
missing; the response's label field reports which of present,
created or unavailable happened. Unavailable (creation failed) files
the issues unlabelled and says so once, in warnings, rather than
once per issue.
Inputs: repo, confirm. Returns: object
get_configA
Fetch the resolved audit configuration, or report that the user has not submitted the configuration page yet.
Requires start_config to have been called first. Every response carries
a "status" field, and it is the only field worth branching on:
- "configured": the configuration is resolved and is in the response's
"config" and "selected_domain_ids". Stop calling this tool.
- "waiting": the interactive page is up and nobody has submitted it
yet. This is NOT a failure and NOT a configuration. Tell the user the
audit is waiting on them at the "url" in the response, then CALL THIS
TOOL AGAIN. Keep calling it while the status says "waiting".
- a raised error: the run's overall deadline (timeout_s) elapsed with
no submission. Tell the user the audit is not proceeding. Never fall
back to a domain selection nobody chose.
In preset mode the configuration is already known and comes back as
"configured" on the first call.
This tool deliberately blocks for at most a short interval per call
(about 25 seconds) and then returns "waiting", rather than holding one
call open for the whole of timeout_s. Hosts impose their own per-tool
timeouts, independent of timeout_s (Codex has
mcp_servers.<name>.tool_timeout_sec), and a call held open past one of
those is cancelled by the host, which can take the whole MCP process
and this run's configuration page down with it (issue #85). timeout_s
remains
the run's overall waiting budget and is enforced here, cumulatively,
across however many calls it takes: it is measured from the moment the
page opened, so polling more often does not buy the user more time, and
polling less often does not cost them any. To keep waiting past the
deadline, call again with a larger timeout_s; that is an explicit
decision to extend, not a silent one.
Inputs: timeout_s. Returns: object
get_domainA
Return the full document text for one domain, given its id (e.g. 'd01').
This tool serves the full rule text: it is meant for the local agent
driving the audit, which needs the rules to apply them. Nothing else
in this package returns rule body text.
Because of that, this call is recorded against the run in progress: it
is the one observable event that could have supplied the rules a
verdict is meant to rest on. record_domain_result says so when verdicts
arrive for a domain this was never called for, and the report names
that domain. The claim either way is only ever that the text was
fetched, never that it was read.
A fetch made when no run is in progress belongs to no run and is not
recorded: call begin_run first, then fetch each domain as you come to
it.
Inputs: domain_id (required). Returns: result
list_domainsA
List every domain loaded from the rules pack, and report any files in the pack directory that were skipped because they had no Trigger line.
Inputs: no named parameters. Returns: object
record_domain_resultA
Record the audit result for one domain.
The payload itself is pydantic-validated by DomainResult (finding and
verdict consistency, could-not-run reason, could-not-evaluate and
not-applicable notes, both of which are the verdict's stated reason,
every finding's precondition, the completed domain's
uninspected_evidence, and that every consulted_sources entry has a
non-blank url, title and why). On top of that: the domain must be one of the domains
selected for this run, a completed result must carry a verdict for
every rule the domain defines, and every consulted_sources rule_id
must be one of this domain's own rules; a completed result missing a
verdict raises IncompleteResultError listing exactly which rule ids
are missing, and an unattributable consulted source raises
UnknownRuleIdError, so the agent can fix and resubmit rather than a
skipped rule silently passing or a citation silently pointing at
nothing. Re-recording an already-recorded domain requires
replace=True, to guard against an accidental overwrite.
Verdicts for a domain get_domain was never called for during this run
are recorded, not refused, and the response says "rules_fetched": false
and carries a warning naming what that means. The report names the
domain too. Recording rather than refusing is deliberate: refusing
would be trivially satisfied by fetching the text and ignoring it,
which destroys the signal, while the verdicts and the fact that they
were unsupported both survive this way. Tell the user when you see it.
Two fields are refused outright rather than recorded with a warning,
because unlike an unfetched domain there is no signal to preserve by
letting them through: a finding without a `precondition` (issue #178)
and a completed domain without `uninspected_evidence` (issue #179).
Both are one sentence the auditor already knows the answer to, and in
both cases being unable to write it is the finding. A finding whose
precondition cannot be named belongs at not-applicable, and a domain
that cannot say what it did not read has not established what its
absence claims are worth. See AUDIT.md step 3 and step 4.
Inputs: result (required), replace. Returns: object
render_reportA
Finish the run and render its report.
Requires a resolved configuration. Sets meta.finished to the given
ISO timestamp, renders the deterministic HTML report (which itself
refuses to render an incomplete run: a selected domain with no
recorded result, or a completed result missing a rule verdict, raises
rather than producing a report that looks clean over a gap), and
writes both report.html and run-state.json to the run's deliverables
directory: config.deliverables_dir if the configuration page (or a
preset AuditConfig) named one, otherwise the run's own output_dir,
unchanged from how every run before that choice existed behaved.
output_dir itself is never affected by this choice; it stays the
run's working directory for the crash-recovery progress file
regardless of where the finished deliverables land (issue #109).
Any issue URLs filed this run via file_issues, and any
feedback issue filed via submit_feedback, are carried on the
RunState itself, so the written run-state.json is self-sufficient:
it (and its schema_version) can be handed to
engineering-audit-render later to re-render the same report without
this server, this run tracker, or either URL, still in memory.
This call also stamps meta.server_finished from the server's own
clock, alongside the caller-supplied finished. See begin_run's
server_started for why the report keeps both this figure and the
caller's rather than trusting either one alone.
The finished run stays reachable for one last submit_feedback (the
order AUDIT.md documents), which rewrites both files to carry the
feedback issue's link. It stops being reachable at the next
begin_run.
Both files are written atomically, and the run's crash-recovery file
is removed once they are on disk: from here the run-state.json is the
record, and a later begin_run on this output directory starts clean
rather than offering to resume a run that is already finished.
The response also carries "rules_fetched": which domains had their
rule text fetched this run, which recorded verdicts without it, and
which were carried in from a saved run that never recorded it. Any
domain in the second list is named in the report and must be named to
the user as well: it says the verdicts for that domain were reached
without the rules they are verdicts on.
Inputs: finished (required). Returns: object
run_statusA
Report progress for the current run: which selected domains have recorded results, which are still missing, and the findings count so far. Read-only over the run itself; it also carries any queued crash-recovery warning that no earlier response has reported yet.
Inputs: no named parameters. Returns: object
start_configA
Begin configuring the audit run.
If the ENGINEERING_AUDIT_CONFIG environment variable names a path to
a valid AuditConfig JSON file, it is loaded immediately (the
documented headless/CI path); an invalid or unreadable file is a
loud error, never a silently-applied default. Otherwise this starts
the interactive localhost configuration page, opens it in the user's
browser when one is available (best-effort; the response's
opened_in_browser field says whether a tab actually opened), and
returns its URL for the agent to show the user as the fallback.
Inputs: no named parameters. Returns: object
submit_feedbackA
Send optional run feedback to the tool author.
Requires a resolved configuration. There is nothing to send unless
config.feedback_text was set on the configuration page, or the
calling agent supplies extra_text; if neither is present this
raises rather than filing an empty, pointless issue.
The feedback body always carries the free text plus a run-metadata
section (tool version, rules pack, assistant, model, repository,
timestamps), and then each telemetry section the user consented to
on the configuration page (coverage totals, findings rollup by
severity/domain id, self-assessment, environment, consulted sources
by rule id/url/why, rule verdict distribution by domain and in
total, run duration and the divergence verdict between its two
measurements, which domains had their rule text fetched via
get_domain, and the reader's own conclusions after reading the
report); an unconsented section is left out entirely. Finding text
itself is never included, only counts.
report_conclusion and report_fix_first (issue #135) are the
reader's own answers, in their own words, to the two questions the
finished report's own feedback form asks: in one sentence, what did
this report tell them about their repository, and what would they
fix first. Pass these only if the human using this session actually
read the finished report and dictated an answer back; never guess
or paraphrase one on their behalf. Both are ignored unless the
reader_conclusions section was consented to on the configuration
page, same as every other telemetry section here.
Files a labelled issue on the tool author's feedback repository via
gh. If gh is unavailable or filing fails for any reason, the
feedback is never lost: this returns a mailto fallback instead,
with the same body, so the agent can offer to open the user's mail
client or hand over the text to paste in manually.
May be called either before or after render_report. Called after,
it sends feedback for the run just finished and rewrites that run's
report.html and run-state.json so both carry the feedback issue's
link; the response's report_updated field says whether that rewrite
succeeded, and a failed rewrite is reported as a warning rather than
an error, because the issue is already filed by then and raising
would invite a retry that double-files it.
Inputs: extra_text, report_fix_first, report_conclusion. Returns: object
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 10 tools
Each tool targets a distinct stage of the audit workflow: configuration, run lifecycle, domain metadata, rule text, result recording, progress reporting, report rendering, issue filing, and feedback. Even closely related pairs like list_domains/get_domain and start_config/get_config are cleanly separated by action and state. No two tools plausibly do the same job.
Nine of ten tools follow a clear verb_noun snake_case pattern: begin_run, list_domains, get_domain, record_domain_result, render_report, file_issues, submit_feedback, start_config, get_config. The outlier is run_status, which reads as a noun phrase rather than an imperative verb+object form. It is still stylistically consistent and readable, so this is only a minor deviation.
Ten tools map cleanly onto the full audit lifecycle without redundancy or bloat. Each tool earns its place, covering configuration, run start/resume/status, domain and rule inspection, result recording, report rendering, issue filing, and feedback submission. The scope is well matched to the server's purpose.
The tool surface covers the core audit lifecycle well: configuration, run management, domain discovery, rule text retrieval, result recording, rendering, issue filing, and feedback. The main gap is that previously recorded findings are only summarized by run_status or surfaced indirectly via file_issues and render_report; there is no direct tool to reread detailed recorded results after resuming a run. This is a workable omission rather than a dead end.
Maintenance
Related MCP Connectors
One MCP server for all your MCP servers: code mode, per-tool permissions, approvals and audit.
AI-native git hosting — repos, PRs, issues, CI gates, and AI code review over MCP (60 tools).
Audit your MCP servers for poisoned tools and leaked secrets, and check AI-generated code for flaws.
MCP server that lets your AI assistant build and deploy apps, hosting each one at a live URL.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceA production-ready MCP server that provides AI assistants with comprehensive GitHub developer tooling including PR analysis, code review, changelog generation, dependency auditing, commit summarization, and refactoring suggestions.MIT
- AlicenseAqualityBmaintenanceAn MCP server that provides local code quality analysis for AI coding assistants, supporting file analysis, git diff review, and full project scanning with quality scoring.53MIT
- AlicenseNot gradedqualityCmaintenanceAn MCP server that provides code quality checks for AI coding assistants, including file size limits, ESLint, architecture compliance, anti-pattern scanning, and comment compliance.6 npmMIT
- AlicenseNot gradedqualityDmaintenanceMCP server for automated code review using AI agents. It analyzes code diffs or file paths for bugs, security issues, and style violations.MIT