| list_domainsA | List every domain loaded from the rules pack, and report any files
in the pack directory that were skipped because they had no Trigger
line. |
| get_domainA | Return the full document text for one domain, given its id (e.g. 'd01'). This tool serves the full rule text: it is meant for the local agent
driving the audit, which needs the rules to apply them. Nothing else
in this package returns rule body text.
Because of that, this call is recorded against the run in progress: it
is the one observable event that could have supplied the rules a
verdict is meant to rest on. record_domain_result says so when verdicts
arrive for a domain this was never called for, and the report names
that domain. The claim either way is only ever that the text was
fetched, never that it was read.
A fetch made when no run is in progress belongs to no run and is not
recorded: call begin_run first, then fetch each domain as you come to
it.
|
| begin_runA | Start a fresh audit run and create its output directory, or resume
an interrupted one. assistant/model/repo_name/repo_commit/started are supplied by the
calling agent; tool_version defaults to the installed package version
if omitted. repo_dir is
the path to the repository being audited, on disk; it is optional,
but file_issues needs it to detect the GitHub repository to file
against, unless a repo is given explicitly on that call instead.
Calling this twice without finishing the first run (via
render_report) is an error, since it would silently discard whatever
domain results have already been recorded; pass replace=True to
explicitly discard the in-progress run and start over.
A run's progress is saved to a crash-recovery file in output_dir as it
goes, so a server that stops mid-run (host restart, dropped
connection, machine asleep) loses at most the domain in flight. When
this call finds such a file for an unfinished run in output_dir it
starts nothing and returns a description of it plus an instruction:
run_started is False, "meta" is absent, and "resumable" says whether
it can be continued at all. Call begin_run again with resume=True to
continue that run (its recorded domains are kept, and the response
lists which domains are still missing), or resume=False to discard it
and start fresh. resume=False is the only way to overwrite saved
results, and replace=True counts as the same explicit decision.
Resuming a run for a DIFFERENT repository is refused outright, as is
resuming one whose saved state cannot be read; either way, nothing is
started and nothing is deleted until told.
environment records the host facts the report header cannot carry, and
its keys are a closed set: 'os' (e.g. "macOS 15.2", "Ubuntu 24.04"),
'host_cli' (the CLI application driving this audit, e.g. "codex",
"claude-code") and 'host_cli_version' (that CLI's version string).
Collect them from the machine you are running on rather than guessing,
and omit any key you cannot determine: an omitted fact and a guessed
one are not the same thing. Any other key is refused outright, because
this metadata is included in feedback issues filed publicly on the
tool's own repository. Do not name the assistant, the model or the tool
version here; all three are already fixed rows in the report header.
The recorded metadata also stamps two provenance SHAs, best-effort:
tool_commit (the git commit the installed tool build was made from,
via its PEP 610 install record) and rules_pack_commit (the loaded
rules pack directory's git HEAD, '-dirty' suffixed if it has
uncommitted changes). Either is None when it could not be
determined, which the report renders as "unknown" rather than
guessing: a report must be traceable to the exact tool build and
rules version that produced it, not just a package version number
that can lag behind either.
started is the caller's own claim about when the run began, taken on
trust like everything else the calling agent asserts. This call also
stamps meta.server_started from the server's own clock at the moment
it runs, independent of that claim; render_report does the same for
meta.server_finished. Neither figure is treated as more authoritative
than the other in the rendered report: a resumed run genuinely spans a
wall-clock gap that is not audit work, so the server's elapsed time is
not automatically the truer duration, but an assistant-supplied
duration that was never checked against anything is worse. The report
states both and flags it when they diverge by more than expected,
rather than presenting an unmeasured number as fact.
The run also performs a best-effort tool update check, comparing
tool_commit against the tool's latest tagged release on GitHub, and
the rules pack against its own remote the same way. Each result
lands in the returned meta (update_check and pack_update_check
respectively), prefixed "current", "stale", "could-not-check" or
"not-checked" (see engineering_audit.update_check for the exact
strings). The calling agent MUST tell the user when either reports
stale or could-not-check, rather than silently proceeding as if the
installed build were confirmed current: this tool is installed via a
pinned uvx reference, and a stale pin or cache would otherwise serve
an old build forever with nothing to say so. This check runs
automatically and discloses only the caller's IP address and the
fact that this repository's tags were queried, no repository
content, findings or paths; it can be turned off with
--no-update-check or the ENGINEERING_AUDIT_NO_UPDATE_CHECK
environment variable, in which case both fields read "not-checked",
which is not something to warn the user about, since turning it off
was their own choice.
|
| start_configA | Begin configuring the audit run. If the ENGINEERING_AUDIT_CONFIG environment variable names a path to
a valid AuditConfig JSON file, it is loaded immediately (the
documented headless/CI path); an invalid or unreadable file is a
loud error, never a silently-applied default. Otherwise this starts
the interactive localhost configuration page, opens it in the user's
browser when one is available (best-effort; the response's
opened_in_browser field says whether a tab actually opened), and
returns its URL for the agent to show the user as the fallback.
|
| get_configA | Fetch the resolved audit configuration, or report that the user has
not submitted the configuration page yet. Requires start_config to have been called first. Every response carries
a "status" field, and it is the only field worth branching on:
- "configured": the configuration is resolved and is in the response's
"config" and "selected_domain_ids". Stop calling this tool.
- "waiting": the interactive page is up and nobody has submitted it
yet. This is NOT a failure and NOT a configuration. Tell the user the
audit is waiting on them at the "url" in the response, then CALL THIS
TOOL AGAIN. Keep calling it while the status says "waiting".
- a raised error: the run's overall deadline (timeout_s) elapsed with
no submission. Tell the user the audit is not proceeding. Never fall
back to a domain selection nobody chose.
In preset mode the configuration is already known and comes back as
"configured" on the first call.
This tool deliberately blocks for at most a short interval per call
(about 25 seconds) and then returns "waiting", rather than holding one
call open for the whole of timeout_s. Hosts impose their own per-tool
timeouts, independent of timeout_s (Codex has
mcp_servers.<name>.tool_timeout_sec), and a call held open past one of
those is cancelled by the host, which can take the whole MCP process
and this run's configuration page down with it (issue #85). timeout_s
remains
the run's overall waiting budget and is enforced here, cumulatively,
across however many calls it takes: it is measured from the moment the
page opened, so polling more often does not buy the user more time, and
polling less often does not cost them any. To keep waiting past the
deadline, call again with a larger timeout_s; that is an explicit
decision to extend, not a silent one.
|
| record_domain_resultA | Record the audit result for one domain. The payload itself is pydantic-validated by DomainResult (finding and
verdict consistency, could-not-run reason, could-not-evaluate and
not-applicable notes, both of which are the verdict's stated reason,
every finding's precondition, the completed domain's
uninspected_evidence, and that every consulted_sources entry has a
non-blank url, title and why). On top of that: the domain must be one of the domains
selected for this run, a completed result must carry a verdict for
every rule the domain defines, and every consulted_sources rule_id
must be one of this domain's own rules; a completed result missing a
verdict raises IncompleteResultError listing exactly which rule ids
are missing, and an unattributable consulted source raises
UnknownRuleIdError, so the agent can fix and resubmit rather than a
skipped rule silently passing or a citation silently pointing at
nothing. Re-recording an already-recorded domain requires
replace=True, to guard against an accidental overwrite.
Verdicts for a domain get_domain was never called for during this run
are recorded, not refused, and the response says "rules_fetched": false
and carries a warning naming what that means. The report names the
domain too. Recording rather than refusing is deliberate: refusing
would be trivially satisfied by fetching the text and ignoring it,
which destroys the signal, while the verdicts and the fact that they
were unsupported both survive this way. Tell the user when you see it.
Two fields are refused outright rather than recorded with a warning,
because unlike an unfetched domain there is no signal to preserve by
letting them through: a finding without a `precondition` (issue #178)
and a completed domain without `uninspected_evidence` (issue #179).
Both are one sentence the auditor already knows the answer to, and in
both cases being unable to write it is the finding. A finding whose
precondition cannot be named belongs at not-applicable, and a domain
that cannot say what it did not read has not established what its
absence claims are worth. See AUDIT.md step 3 and step 4.
|
| run_statusA | Report progress for the current run: which selected domains have
recorded results, which are still missing, and the findings count so
far. Read-only over the run itself; it also carries any queued
crash-recovery warning that no earlier response has reported yet. |
| file_issuesA | Preview or file GitHub issues for every recorded finding, via the
user's own gh CLI. Requires config.issue_mode == "github": if the user chose in-report
delivery instead, this raises rather than filing issues nobody asked
for. Requires at least one recorded domain result.
confirm=False (the default) NEVER files anything and never invokes
gh at all: it returns a preview {repo, count, titles, instruction}
so the calling agent can show the user exactly what is about to be
filed on their repository, and get explicit agreement, before a
single issue goes out. Filing on someone's repo is outward-facing;
this confirmation step is mandatory, not decorative.
confirm=True files one issue per finding that has not already been
filed, so retrying after a partial failure does not double-file the
ones that succeeded. Filed issues are tracked, and returned, per
finding under a key of the form "<rule id>#<n>" (n counting that
rule's findings in recording order), not per rule id: a domain result
may carry two findings for the same rule, and both of their issue
urls have to survive. The target repository is `repo` if given,
otherwise detected from the audited repository directory recorded
by begin_run's repo_dir. If any issue fails to file, filing stops
immediately and the error lists exactly which findings were filed
(with their URLs) and which were not, so a retry knows where to
resume.
Each filed issue carries the "engineering-audit" label. The label is
checked once per call and created on the target repository if it is
missing; the response's label field reports which of present,
created or unavailable happened. Unavailable (creation failed) files
the issues unlabelled and says so once, in warnings, rather than
once per issue.
|
| submit_feedbackA | Send optional run feedback to the tool author. Requires a resolved configuration. There is nothing to send unless
config.feedback_text was set on the configuration page, or the
calling agent supplies extra_text; if neither is present this
raises rather than filing an empty, pointless issue.
The feedback body always carries the free text plus a run-metadata
section (tool version, rules pack, assistant, model, repository,
timestamps), and then each telemetry section the user consented to
on the configuration page (coverage totals, findings rollup by
severity/domain id, self-assessment, environment, consulted sources
by rule id/url/why, rule verdict distribution by domain and in
total, run duration and the divergence verdict between its two
measurements, which domains had their rule text fetched via
get_domain, and the reader's own conclusions after reading the
report); an unconsented section is left out entirely. Finding text
itself is never included, only counts.
report_conclusion and report_fix_first (issue #135) are the
reader's own answers, in their own words, to the two questions the
finished report's own feedback form asks: in one sentence, what did
this report tell them about their repository, and what would they
fix first. Pass these only if the human using this session actually
read the finished report and dictated an answer back; never guess
or paraphrase one on their behalf. Both are ignored unless the
reader_conclusions section was consented to on the configuration
page, same as every other telemetry section here.
Files a labelled issue on the tool author's feedback repository via
gh. If gh is unavailable or filing fails for any reason, the
feedback is never lost: this returns a mailto fallback instead,
with the same body, so the agent can offer to open the user's mail
client or hand over the text to paste in manually.
May be called either before or after render_report. Called after,
it sends feedback for the run just finished and rewrites that run's
report.html and run-state.json so both carry the feedback issue's
link; the response's report_updated field says whether that rewrite
succeeded, and a failed rewrite is reported as a warning rather than
an error, because the issue is already filed by then and raising
would invite a retry that double-files it.
|
| render_reportA | Finish the run and render its report. Requires a resolved configuration. Sets meta.finished to the given
ISO timestamp, renders the deterministic HTML report (which itself
refuses to render an incomplete run: a selected domain with no
recorded result, or a completed result missing a rule verdict, raises
rather than producing a report that looks clean over a gap), and
writes both report.html and run-state.json to the run's deliverables
directory: config.deliverables_dir if the configuration page (or a
preset AuditConfig) named one, otherwise the run's own output_dir,
unchanged from how every run before that choice existed behaved.
output_dir itself is never affected by this choice; it stays the
run's working directory for the crash-recovery progress file
regardless of where the finished deliverables land (issue #109).
Any issue URLs filed this run via file_issues, and any
feedback issue filed via submit_feedback, are carried on the
RunState itself, so the written run-state.json is self-sufficient:
it (and its schema_version) can be handed to
engineering-audit-render later to re-render the same report without
this server, this run tracker, or either URL, still in memory.
This call also stamps meta.server_finished from the server's own
clock, alongside the caller-supplied finished. See begin_run's
server_started for why the report keeps both this figure and the
caller's rather than trusting either one alone.
The finished run stays reachable for one last submit_feedback (the
order AUDIT.md documents), which rewrites both files to carry the
feedback issue's link. It stops being reachable at the next
begin_run.
Both files are written atomically, and the run's crash-recovery file
is removed once they are on disk: from here the run-state.json is the
record, and a later begin_run on this output directory starts clean
rather than offering to resume a run that is already finished.
The response also carries "rules_fetched": which domains had their
rule text fetched this run, which recorded verdicts without it, and
which were carried in from a saved run that never recorded it. Any
domain in the second list is named in the report and must be named to
the user as well: it says the verdicts for that domain were reached
without the rules they are verdicts on.
|