Prowl Research Agent

Runbook-driven research agent. It follows structured research runbooks (SaaS competitor teardowns, ads & creative research, subscription-app audits, market sizing, equity research, SEO, AI visibility and more), calls the Prowl MCP tool bank (440+ data tools: SEO, ads libraries, reviews, scraping, app intelligence, LLM cross-checks), stores extracted claims in an evidence ledger, and produces reports with source citations and verification disclosures.

It is a CLI for humans and has two MCP surfaces for agents:

• the Fabric service (prowl-research-service, HTTP MCP at http://127.0.0.1:18764/mcp, the connection fabric-agent.json declares) — 7 tools: research.run, research.tools.call, research.tutorial, research.list_runbooks, research.artifact.get, fabric.job.get, fabric.job.cancel (see Fabric service);

• the legacy local stdio MCP server (prowl-research mcp) — 4 tools: research.run, research.list_runbooks, research.get_status, research.get_report.

Quick start for a new teammate

Install

git clone https://github.com/PROWL-AI/research-agent.git
cd research-agent
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

Configure

Two keys, both via environment (key names only — values never in the repo):

• PROWL_API_KEY — your Prowl API key (prowl_…), billed per tool call. Get one at https://prowl.chat.

• OPENROUTER_API_KEY — the agent's own LLM (planning, synthesis, writing). Any OpenAI-compatible endpoint works: set RESEARCH_LLM_BASE_URL / RESEARCH_LLM_API_KEY / RESEARCH_LLM_MODEL to override.

Other environment knobs (all optional):

• PROWL_MCP_URL — Prowl MCP endpoint (default https://prowl.chat/mcp).

• RESEARCH_LLM_MODEL_STRONG — the strong-tier model (planning, writing, repair; default xiaomi/mimo-v2.6-pro); RESEARCH_LLM_MODEL covers the cheap tier (claim extraction, transforms, citation checks; default deepseek/deepseek-v4.1-flash); RESEARCH_JUDGE_MODEL the eval judge (default anthropic/claude-haiku-5.5 — another vendor than the default writer, so a report is not graded by the family that wrote it; keep that true if you change the writer). Defaults apply to new installs and the environment: an installed service keeps the models saved in its settings. Judge scores before and after the judge default changed (2026-10-17, from the strong model) are not comparable. Prices for the cost estimate are in research_agent/pricing.py (OpenRouter catalog, read 2026-10-17; scripts/check_prices.py re-reads it).

• RESEARCH_MAX_LLM_USD — the run's LLM spend ceiling (see "How it works"). Claim extraction and citation checks send reasoning: {enabled: false} to OpenRouter: on the cheap tier reasoning could spend the whole 2000-token output budget before any JSON, and an answer cut that way is not retried, because the same request is cut the same way (B-050, measured 2026-10-18/20).

• RESEARCH_RUNS_DIR — where runs are written, for the stdio MCP server and every CLI command alike (run, status, report, rewrite, export, prune). Default ./runs relative to the process; set it explicitly in MCP client configs.

• RESEARCH_NO_SUBAGENTS=1 — force sequential execution (no worker fan-out).

• RESEARCH_PROWL_CALL_TIMEOUT_S — read timeout in seconds for each Prowl MCP call (default 180; a hung server fails the call instead of parking it).

CLI subcommands: run <runbook> [--run-id <id>] [--max-llm-usd <usd>] [--key value …], list-runbooks, status <run_id>, report <run_id>, rewrite <run_id>, export <run_id> [--format html|md], prune [--older-than 30d] [--keep-last N] [--yes], validate [--online], mcp.

Exit codes: run exits 1 when the run ends partial (budget or failure — the report says why); rewrite exits 1 when lint issues remain; validate exits 0 when all runbooks are valid, 1 when a runbook fails validation, 2 on usage or network/config errors (CI can tell "runbook invalid" apart from "catalog unreachable"); other input and usage errors exit 2. list-runbooks, status, report, export, prune exit 0 on success; mcp serves on stdio until stopped.

runs/ grows forever unless you prune it: prune lists runs older than --older-than (by checkpoint created_at, default 30d) without deleting anything; pass --yes to actually delete. --keep-last N always spares the N newest runs, and any run whose .lock is held by a live process is skipped.

MCP

Register the agent as an MCP server (stdio) in any MCP client:

{
  "mcpServers": {
    "research-agent": {
      "command": "prowl-research",
      "args": ["mcp"],
      "env": { "PROWL_API_KEY": "prowl_…", "OPENROUTER_API_KEY": "…" }
    }
  }
}

The stdio server exposes four tools: research.run, research.list_runbooks, research.get_status, research.get_report. It is the legacy local surface; the Fabric bundle points at the HTTP service, which exposes seven (listed at the top of this file).

Proving tool call (verifies install and registration; needs no keys and makes no billed calls):

research.list_runbooks()

Expected: the runbook list (saas-competitor-teardown, …) with inputs and budgets. research.get_status and research.get_report are key-free too — only research.run requires the keys above (it fails fast, naming the missing env var, before spending anything).

Runs take minutes, so for long runs prefer the job pattern: call research.run with wait=false (the default) to get {run_id, status: "running"} back immediately, then poll research.get_status(run_id) until the checkpoint shows complete, partial, failed, or interrupted. failed/interrupted require inspecting the checkpoint; a report may not exist. For complete/partial, fetch the result with research.get_report(run_id). Use wait=true only when the caller can block for the whole run.

Next to status, every terminal state carries a terminal_code saying why the run stopped (null while it is live). The set is closed:

Run state has exactly one writer: the process holding the run's lock (runs/<id>/.lock). The orchestrator records cancelled/error itself before releasing the lock; a second process on the same run is refused (run … is locked by another process) and writes nothing — on stdio MCP the refusal is returned synchronously, also for wait=false. get_status reports active (some process holds the run lock) and orphaned (running, no live owner anywhere). interrupted is written by reconciliation — at stdio server start, at CLI run/status, and at Fabric service start — and only for a running checkpoint whose lock is free. A run whose version-1 checkpoint cannot be migrated without guessing is reported with status: needs_attention and the reason in stop_reason: its report stays readable and exportable, it can be pruned, and research.run refuses to resume it. get_status also reports has_checkpoint: false while a run accepted by this server is still starting, or when it failed or was cancelled before it took the run lock (the status then comes from the in-process job and disappears with it).

First real run (makes billed Prowl calls; see budget limits below):

prowl-research run saas-competitor-teardown --competitors example.com,rival.com

Expected: a markdown report under runs/<id>/report.md plus an HTML export, with a source log and confidence ladder.

Develop

pytest                                  # unit tests + runbook validation (offline)
python scripts/validate_runbooks.py --online  # frontmatter + tool names vs live Prowl catalog (needs PROWL_API_KEY)
python evals/judge.py --run runs/<id>   # 5-dimension quality rubric for a finished run (exit 0 pass, 1 fail, 2 judge error)
python evals/calibrate.py               # the judge against planted-defect fixtures (offline; --live spends)

The judge's verdict is vetoed by defects the code proves — a citation of no claim; a quotation no cited claim backs; a sentence whose metric figure none of its cited claims carries (a sentence the writer marked "derived" excepted); a conflicting claim cited on a line without the upper-case CONFLICT tag and outside a conflict register (rows under a "Source log" heading are provenance, not findings); a citation tagged VERIFIED whose claim the ledger does not hold as verified; a partial run that does not say "partial" before its first section — so a lenient model cannot pass them. An uncited figure lowers the citation score instead. Each judge LLM call is checked by estimate against --max-usd (default 0.50) before it is sent and recorded in <run>/judge-ledger.json, never in the run's checkpoint. The judge needs only an LLM key.

Authoring a new runbook: docs/runbook-authoring.md. Architecture decisions and inherited traps: docs/knowledge-pack.md.

Related MCP server: Advanced Web Search MCP Server

How it works

runbook (SKILL.md)  →  brief  →  plan  →  parallel research sub-agents
                     (data segments; transforms stay on the lead)
                     →  evidence ledger  →  writer + citation-repair loop
                     →  citation verification pass  →  markdown + HTML

• Runbooks ask for sourced numbers and triangulation. The writer consumes the evidence ledger and runs citation repair/verification, but this is not a guarantee that every fact is true or has two independent sources. Inspect the report's verification disclosures and source evidence before relying on it.

• The repair pass fixes what the judge would veto. Besides the citation lint, the writer's repair pass gets the defects the eval judge proves by code (research_agent/agent/report_checks.py): a VERIFIED tag on a claim the ledger does not hold as verified, a conflict cited without its [CONFLICT] tag, a quotation no cited claim holds, a figure no cited claim carries. lint_issues_before/lint_issues_after count both (B-057). The repair pass sends only the lines at fault and patches the lines it gets back (L<n>: …), so a long report is not rewritten whole (B-059). The citation lint also flags drafting residue inside a citation (a question mark, a note or a nested bracket, such as [C186? no, [C237]]) as malformed_citation, which the repair pass rewrites to plain claim ids (B-068).

• One refused step does not end the run. A tool that is refused (no results, not found, a provider, scope or argument error) marks that step failed (or skipped_on_error when the plan marked it optional) with the server's reason, and the run goes on; with a refused required step it ends partial / step_failed, and the writer is told which steps were refused and why, so the report says what is missing. What still stops the run: an empty Prowl wallet (insufficient_funds), the service's own call limit or an unavailable call journal, a call whose outcome is unknown, five refusals of required steps in a row across the run (optional steps neither count nor reset it), and every budget. A budget stop keeps its own terminal_code; its stop_reason also names the refused steps. A resume asks every failed step again (a new attempt, which the provider may bill again).

• A value only an earlier step returns is a placeholder. The plan is written before any tool runs, so an argument such as an app's numeric store id is planned as {{headspace_apple_id}}, with a transform before the step that names it. That transform ends its note with a fenced bindings block; a value is kept only when a raw artifact of the run states it verbatim (checkpoint.bindings), and the step sends the filled arguments (bound_arguments; arguments keeps the plan as written). A placeholder nobody resolved is never sent: the step is skipped_on_error (or failed, when required) with not sent: no earlier step resolved {{name}}, and it does not count toward the refusals in a row (B-065).

• Runbooks declare their tool allowlist and budget (max_tool_calls / max_usd / max_minutes). On budget exhaustion the agent emits a partial report that marks what is missing — it never fails silently. The budget is read from the run's attempt ledger: every dispatched call is recorded before it is sent (so in-flight calls count against max_tool_calls), and its cost is stored when it returns or fails — a billed failure costs money too. Spend = server-reported actual costs + estimates for calls without one (the catalog price, else the runbook's per-call share of max_usd, never zero); stats.cost_source says server, mixed, estimate, or null before any tool attempt. The runbook budget caps Prowl tool calls; the agent's own LLM spend (planning, claim extraction, transforms, writing, repair, citation checks) is capped separately by max_llm_usd: --max-llm-usd / MCP max_llm_usd, else RESEARCH_MAX_LLM_USD, else the cap a resumed run recorded, else the runbook's budget.max_llm_usd, else USD 5. It is a check before every LLM HTTP attempt (retries included), against an estimate of that call: the last call it admits can overshoot by its actual cost minus its estimate, so stats.llm_budget.spend_usd can end slightly above the cap. Each attempt is recorded in the attempt ledger with the provider's usage.cost (OpenRouter is asked for it), else a price-table estimate (research_agent/pricing.py; an unlisted model is priced at USD 15 / 75 per 1M tokens, above every listed model — never 0); an HTTP error response (429, 5xx) is recorded as unbilled. Research stages stop at the cap minus a writer reserve (25%, at most USD 1) with terminal_code=budget_llm, and once a research call is refused no further tool is dispatched; the writer, repair and citation passes may spend the reserve — a refusal there also ends the run partial / budget_llm — and if even that is gone the report is the evidence ledger as gathered. A resume reads the spend and cap already recorded (also after a re-plan) and keeps an existing report when it gathered nothing new and cannot afford a writer. rewrite --max-llm-usd spends from the same ledger. stats.llm_budget reports cap, reserve, spend, cost source and refused stages; stats.llm_usage the tokens.

• Evidence statuses (ledger schema 2). A claim's status is recomputed over its whole subject group on every change and on load, so it never depends on arrival order. verified needs either:

  • span — the claim's quote is found in the step's whole stored raw (its values or its JSON, sha256-checked on resume; a bare number is not a quote) and the value sits inside the quote on number boundaries (1200 is not in 11,200). A span-verified claim is one source saying it; or

  • corroboration — two agreeing claims from two different steps with distinct known sources: different publishers (registrable domain of a URL the raw or the request actually names — an extractor-supplied URL that the raw does not contain is dropped) or different upstream data providers (delivery wrappers removed: google_trends and serpapi_google_trends are one Google Trends source).

  Never a source: a transform note and an LLM-answer tool (Perplexity, Gemini, Google AI mode/overview, *_responses) — their own text never span-verifies and they add no independence; a web search/scrape claim without a URL; a claim whose quote was not found. Two values that disagree make every valued claim of the group conflict (a missing unit is compatible with any unit; $ and USD are one unit); a claim without a value never confirms or resolves one, and a chart over a conflicting claim is rejected. The extractor's own word never verifies; a v1 ledger's verbatim claim without a quote loads as assumed with the reason in verification.detail. Reports written before schema 2 keep their old tags — regenerate them with rewrite_report, which also re-saves the ledger; judge status shares are not comparable across this change. Events: claim.verified, claim.conflict, claim.quote_not_found (with run_id; under the service they go to ~/Library/Logs/prowl-research/events.log, rotated at 5 × 5 MB).

• Charts are rendered deterministically from ledger data, never described by the LLM.

Current verification and resume limits

The deep agent audit adds seven defects: lifecycle ownership (KA-06 — wave 1), hidden artifact/extraction failures and budget accounting across mixed receipts/resume (KA-10…KA-12 — wave 2), and false evidence verification (KA-07…KA-09 — wave 3, ledger schema 2). The ported contracts live in tests/test_ka_contracts.py. The historical probe file stays as the audit's cut: 11 of its 12 cases pass; its success control asserts a cwd-relative raw_ref, which wave 2 replaced with a run-dir-relative one. The 2026-10-10 handoff links the remediation plan and records the audited source revision, checks and next repair tasks. The offline suite is .venv/bin/python -m pytest tests/ -q (network-free); its pass count is a fact about one revision, so it is recorded with that revision in the handoff rather than here. A green run does not establish live provider quality or Fabric host acceptance.

Resume works from the checkpoint's step records (schema version 2, wave 2): every step has an immutable step_id assigned when the plan is made, so a tool leaving the catalog excludes that step by id and shifts nothing. A step's state is pending, dispatching, outcome_unknown, raw_saved, completed, failed, skipped_on_error, skipped_budget or excluded. A resume runs pending and failed steps, finishes raw_saved steps from their saved raw without a new paid call, and re-sends an outcome_unknown call with the same idempotency key (<run_id>:<step_id>:<attempt>) so the server can replay it instead of billing twice. A client read timeout counts as an unknown outcome, never as a refusal, and a budget stop leaves such a step outcome_unknown so its key survives. The Prowl server deduplicates a re-sent key only within its retry window (15 minutes by default): a resume after that may be charged again, and the re-send is still recorded as one attempt — the budget then undercounts by that call. A step migrated from version 1 whose dispatch may have reached the server is first re-sent under its v1 key (<run_id>:<index>:<tool>). Version-1 checkpoints are migrated on load; a v1 state that cannot be mapped without guessing is reported needs_attention (above) and never resumed. The upgrade is one-way for a run directory: the first save of a migrated run keeps its v1 file as checkpoint.v1.json, but a pre-v2 build reading the new checkpoint.json re-plans (and re-bills) the run. Restart stdio sessions and reinstall the Fabric service before resuming runs a v2 build has touched; a build refuses a checkpoint whose schema_version is newer than it understands.

Status and checkpoint shape (schema 2). research.get_status returns completed_steps as step ids (not indexes), planned_steps without excluded steps, step_counts per state, skipped_steps entries carrying step_id, index, step, tool, reason and state (while a run is live, steps still in flight are not listed), and counters with data_calls, attempted_calls, cost_usd, cost_estimate_usd, spend_usd and cost_unknown_attempts (the same keys appear in a run's stats). stats.cost_source is server, mixed, estimate or null (no tool attempt yet). checkpoint.json (also what research.artifact.get serves) holds plan_raw, steps and attempts; it no longer persists plan, completed_steps, skipped_steps or counters. Crash injection at every durable write is covered by tests/test_durability.py.

Finance/subscription live acceptance remains unverified. With several workers, the USD check can still overshoot by the in-flight calls' unknown cost (each counts at its estimate); max_usd is not a cap on the separate LLM provider spend.

Fabric service and operator dashboard

The separate prowl-research-service entry point adds durable jobs, observed tool/LLM receipts, human decisions and a service-owned dashboard for Fabric Dashboards. It reuses the existing research core. See Fabric setup and architecture, dashboard design, and current verification/handoff. Legacy CLI/stdio and their existing run directories remain separate entry points. Other agents call the service directly with a delegation token from the Fabric hub (DEC-0039; the service's own agent token from loopback needs none), and an ended research job is continued with research.run + resumeJobId without paying again for what was already paid (FABRIC.md).

License

MIT — see LICENSE.