# it-desk

A self-healing monitor and ticket desk for a fleet of local HTTP services on one Windows machine.
It probes each service, restarts the ones that fail, verifies that exactly one process is serving
the port afterwards, records every step as a ticket with evidence, asks a model for a diagnosis when
scripted restarts stop working, and notifies a human. Coding agents reach it through an MCP server
so they can check health, file bugs, request restarts and work a shared ticket queue instead of
killing processes themselves.

It is a single FastAPI process (`run.py`) with a SQLite store, a static dashboard, and a stdio MCP
adapter (`mcp_server.py`).

## Architecture

```
probe (HTTP, bounded timeout)
  -> state machine per service (GREEN / RED / BREAKER, startup grace, consecutive-red gate)
  -> diagnose (process dead, orphan listener, zombie duplicates, hang)
  -> restart: drain_url (optional) -> exclusive restart lease (optional broker)
              -> kill by command-line regex (denylist enforced) -> launch from registry
              -> verify listener + probe
  -> ticket with timeline events (probe_fail, diagnosis, kill, launch, verify_ok/verify_fail, ...)
  -> recovered: ticket auto_resolved
  -> scripts give up (verify fails, breaker trips, check_only service down): escalate
       -> AI triage (structured JSON first, prose fallback; diagnosis only, never executed)
       -> notification (Discord webhook and/or signed off-site heartbeat)
  -> humans or agents claim / release / resolve tickets through the dashboard or MCP
```

Main modules:

| Module | Role |
|---|---|
| `app/monitor.py` | Poll loop, per-service state machine, restart policy, breaker, escalation |
| `app/probes.py` | HTTP health probes |
| `app/diagnose.py` | Tier-1 classification before anything is touched |
| `app/remediate.py` | The only action surface: drain, lease, kill-by-match, launch, verify |
| `app/procs.py` | Process and listener helpers; the never-kill denylist lives here in code |
| `app/tickets.py` | SQLite (WAL) store: tickets, events, probe history, restart log, playbooks |
| `app/triage.py` | Tier-2 model diagnosis; output is attached to the ticket only |
| `app/actions.py` | Catalog of verbs a model may select; model-selected actions are recorded, only the monitor executes restarts |
| `app/storm.py` | Groups 3+ services failing within 120 s into one fleet event |
| `app/heartbeat.py` | Optional signed heartbeat to an off-site watchdog (dead-man alerting) |
| `app/explain.py` | Optional plain-language explanation feed over open tickets |
| `app/prime.py`, `app/prime_routes.py` | Optional persistent Claude Code session in a ConPTY, shared over a WebSocket |
| `app/scrub.py` | Secret scrubber applied to every off-machine or untrusted-text sink |
| `app/routes.py` | Dashboard and JSON API |
| `mcp_server.py` | stdio MCP server proxying to the desk |

More detail: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).

### The state-0 listener problem

Restart verification asks "is exactly one live process serving this port?" (`procs.verify_listener`).
On some Windows builds a healthy uvicorn listener is reported with TCP state 0 (`psutil` status
`NONE`): `netstat` prints no `LISTENING` row and `Get-NetTCPConnection` shows a blank state. A
filter on `LISTEN` therefore sees no owner for a working service, every restart is recorded as
failed, and the failures trip the circuit breaker.

`procs.port_owners` counts any socket bound to a loopback or wildcard address on the port with no
remote end and an owning PID that is alive, whatever state it reports. Connected sockets
(`ESTABLISHED`, `TIME_WAIT`) and binds on other addresses (for example a VPN proxy re-listening on
the same port) are ignored. `verify_listener` then returns:

- `pid-bound`: exactly one owner and the probe passed;
- `http`: no socket row visible but the probe passed;
- failure when the probe failed or two or more processes own the port (zombie duplicates).

The `netstat` fallback (used only when `psutil` cannot read the table) is still blind to state-0
rows. Tests: `tests/test_listener_probe.py`.

## MCP tools

Register: `claude mcp add -s user it-department -- py -3.12 <path-to-checkout>/mcp_server.py`

| Tool | Purpose |
|---|---|
| `it_status` | Fleet health plus open ticket counts by kind |
| `it_report` | File a bug the agent could not fix (merged with an open ticket of the same normalised title, reopened if closed within 7 days) |
| `it_restart` | Request a coordinated restart (drain, lease, kill, relaunch, verify, ticketed) |
| `it_investigate` | Open an investigation ticket with evidence |
| `it_queue` | List tickets ready to work |
| `it_claim` / `it_release` | Take or hand back a ticket |
| `it_resolve` | Close a claimed ticket with a resolution |

Writes carry a session id (`CLAUDE_CODE_SESSION_ID` or a random one). Inside the prime terminal
(`IT_PRIME=1`) writes are stamped as the prime, and the desk refuses restarts from that source.

## Running

Requirements: Windows, Python 3.12.

```
py -3.12 -m pip install --require-hashes -r requirements.lock
copy config\services.example.yaml config\services.yaml   (then edit)
mkdir %USERPROFILE%\.it-desk
py -3.12 -c "import secrets; print(secrets.token_urlsafe(32))" > %USERPROFILE%\.it-desk\token
py -3.12 run.py
```

The dashboard is at http://127.0.0.1:8107. The API and dashboard accept loopback Host headers only,
and every write needs `Authorization: Bearer <token>`. With no token configured, writes fail closed.

## Configuration

### Environment

| Variable | Default | Meaning |
|---|---|---|
| `IT_DESK_SERVICES` | `config/services.yaml`, else `config/services.example.yaml` | Service registry path |
| `IT_DESK_SETTINGS` | `config/settings.yaml` | Settings path |
| `IT_DESK_DATA_DIR` | `data/` | SQLite DB (`itdept.db`), caches, prime state |
| `IT_DESK_LOG_DIR` | `logs/` | `itdept.log` (self-trimming) |
| `IT_DESK_TOKEN_FILE` | `~/.it-desk/token` | Bearer token file (read by the desk and the MCP server) |
| `IT_DESK_TOKEN` | unset | Token fallback when the file is absent |
| `IT_DESK_URL` | `http://127.0.0.1:8107` | Desk URL used by `mcp_server.py` and `scripts/migrate_bugs.py` |
| `IT_DESK_LEASE_BROKER_URL` | `http://127.0.0.1:8106` | Optional lease broker for exclusive restarts; unreachable = leaseless |
| `IT_DESK_BEAT_SECRET_FILE` | `~/.it-desk/beat_secret` | DPAPI-encrypted HMAC key for the heartbeat |
| `IT_DESK_BUGS_DIR` | `~/.it-desk/bugs` | Markdown mirror of bug tickets (`BUGS.md`) and the offline `bugs.jsonl` inbox |
| `IT_DESK_REAPER_LOG` | unset | Optional log of an external orphan-MCP reaper; feeds the MCP-zero alert and `/api/logs/reaper`. Unset = off |
| `IT_DESK_CLAUDE_PROJECTS` | `~/.claude/projects` | Where Claude Code writes session transcripts; the prime terminal reads its own transcript there for context usage |
| `IT_DESK_AUDIT_LOG` | unset | Optional JSONL audit log for the consultation-storm detector; unset = off |
| `IT_DESK_PROVIDER_CHAIN` | unset | Optional `module:function` called as `fn(prompt, system)` returning `{"text", "provider", "errors"}`; tried before the built-in providers |
| `DEEPSEEK_API_KEY` | unset | Key for the built-in DeepSeek provider |
| `CONDUCTOR_URL` | unset | Optional agent-run conductor whose approval gates appear in the explain feed |
| `IT_DESK_DB` | unset | DB path for `scripts/stats.py` |
| `IT_PRIME` | unset | Set to `1` by the prime terminal for its MCP child |

### `config/services.yaml`

A list under `services:`. See `config/services.example.yaml`. Validation is in
`app/config.py:_validate_services` and runs on every save through the API.

| Key | Meaning |
|---|---|
| `id` | Unique id |
| `name` | Display name |
| `type` | `managed` (may be restarted) or `check_only` (alert only) |
| `port` | Port verified after a restart. Omit on `check_only` rows |
| `probe.url` | Health URL; must start with `http://127.0.0.1` |
| `probe.expect_status`, `probe.timeout_s` | Expected HTTP status and timeout |
| `launch.command` | argv list; `[0]` must be `py`, `python.exe`, or an absolute path ending in `python.exe` (bare `python` is refused) |
| `launch.cwd` | Absolute working directory |
| `launch.env` | Optional `UPPER_SNAKE: "string"` map layered over the desk's environment |
| `kill_match` | Regex over the process command line; at least 6 characters, must contain a backslash or `.py`, no catch-alls |
| `drain_url` | Optional; POSTed before a kill so the service can finish in-flight work (15 s deadline) |
| `startup_grace_s` | Probe failures within this window after a launch are ignored |
| `depends_on` | Parent id; a child failing while its parent is down is grouped under the parent |
| `enabled` | `false` skips the service |

### `config/settings.yaml`

| Key | Default | Meaning |
|---|---|---|
| `poll_interval_s` | 15 | Probe interval (minimum 3) |
| `fail_threshold` | 2 | Consecutive probe failures before a service goes RED |
| `restart_verify_wait_s` | 8 | Wait before verifying a relaunch |
| `probe_history_days` | 7 | Probe history retention |
| `stale_incident_min` | 30 | An incident open this long, given up on by the monitor, appears in the work queue |
| `bug_autoclose_green_min` | 30 | A bug linked to a service closes after the service has been green this long |
| `circuit_breaker.max_restarts_per_hour` | 4 | Restart budget per service; exceeding it opens the breaker and escalates |
| `circuit_breaker.cooldown_min` | 30 | Breaker open time |
| `restart_policy.min_gap_s` / `max_gap_s` | 60 / 900 | Backoff between restarts of one service |
| `restart_policy.reset_after_green_s` | 600 | Green time that resets the backoff |
| `restart_policy.consecutive_red` | `{}` | Per-service RED ticks required before the first restart |
| `triage.provider` | `claude_cli` | `claude_cli`, `deepseek` or `off`; the other built-in provider is the fallback. DeepSeek is only usable when `DEEPSEEK_API_KEY` is set; otherwise it fails fast with that message |
| `triage.claude_cmd` | `["claude", "-p", "--tools", "", "--strict-mcp-config"]` | Command for the `claude_cli` provider (prompt on stdin). The default disables all built-in tools and all MCP servers, so injected ticket text cannot make the CLI act. Keep those flags if you override it |
| `triage.timeout_s` | 180 | Provider timeout |
| `triage.max_runs_per_ticket` | 1 | Triage runs per ticket |
| `discord.webhook_url` | `''` | Notification webhook; blank = no-op |
| `discord.notify` | `[escalated, recovered_after_escalation]` | Events that notify |
| `dashboard.host` / `port` | `127.0.0.1` / 8107 | Bind address; host must stay loopback |
| `heartbeat.enabled`, `url`, `interval_s`, `timeout_s`, `sleep_hours`, `critical_services` | off | Signed off-site heartbeat; see `app/heartbeat.py` |
| `paused` | `false` | Monitor loop paused (no probes, no restarts) |
| `consult_storm.enabled`, `window_min`, `threshold`, `every_s` | on, 10, 5, 300 | Consultation-storm detector (needs `IT_DESK_AUDIT_LOG`) |
| `cli_version_check.enabled`, `every_s` | on, 3600 | Notes Claude Code CLI version changes on matching tickets |

## Incident statistics

```
py -3.12 scripts/stats.py [--db PATH] [--days N]
```

Prints incident tickets opened in the window, recovered without escalation, escalated, went green after escalation,
restart budget exhausted, closed with a written resolution, still open, median and p90 time from
open to recovery without escalation, and a per-service breakdown. Bugs and investigations filed by agents are not
counted. The database is opened read-only.

### Measured figures

Figures measured on the author's own fleet over 114 days (2026-07-05 to 2026-10-26)
with `scripts/stats.py`; the database is not published.

| Measure | Value |
|---|---|
| Incidents opened | 1543 |
| Recovered without escalation (green again before any escalation; includes restarts and transient blips) | 1123 (72.78%) |
| Escalated (an `escalation_reason` note exists) | 407 (26.38%) |
| Escalated, then went green again (automatic or manual) | 405 (26.25%) |
| Restart budget exhausted (breaker opened) | 4 (0.26%) |
| Closed with a written resolution by a person or agent | 11 (0.71%) |
| Open to recovered without escalation, median / p90 | 16 s / 58 s |

Most escalations come from `check_only` services, which are never restarted and escalate on first
failure; one such service accounts for 338 of the 407.

## Threat model

The desk assumes a single trusted local user.

- Read routes (`GET /api/status`, `/api/tickets`, `/api/bugs`, `/api/settings` and similar) are
  unauthenticated local reads. They expose ticket text, error output and settings (the Discord
  webhook URL is redacted). Anything that can reach `127.0.0.1:8107` with a loopback Host header can
  read them. `GET /api/registry` and the prime routes require the token.
- A bearer token holder can change `config/services.yaml` (`launch.command`, `launch.cwd`) and
  `triage.claude_cmd` through the API. That is authenticated local code execution, by design: the
  desk exists to launch processes.
- The token is a file in the user profile (`IT_DESK_TOKEN_FILE`). Any process running as that user
  can read it.
- The prime-source check (`X-IT-Source: prime` header or `source`/`who` body fields) is a
  convenience guard that stops the prime session from requesting restarts. It is not a security
  boundary; any token holder can omit it.
- Ticket text is treated as untrusted: it is fenced and scrubbed before it reaches a model, and the
  default triage CLI runs with no tools.

## Tests

```
py -3.12 -m pip install -r requirements-dev.txt
py -3.12 -m pytest -q
```

`tests/conftest.py` redirects every data, log and config path to a temporary directory and fails
the session if anything writes under the repo's `data/`, `logs/` or `config/`. Cross-checks that
need the optional `winspawn` spawn auditor are skipped when it is not installed. CI runs on
`windows-latest`.

## Limitations

- Windows only: process handling, ConPTY (`pywinpty`), DPAPI secret storage, `CREATE_NO_WINDOW`
  spawns and the listener semantics above are Windows-specific. `requirements.lock` pins
  `pywinpty`, so it does not install on Linux.
- Managed services must be Python processes launched from the registry; `kill_match` selects
  processes by command line.
- One machine. Probes must target `127.0.0.1`; the dashboard binds loopback only.
- AI triage is advisory. Model-selected actions are recorded; only the monitor executes restarts.
- The prime terminal needs the Claude Code CLI on PATH, `~/.it-desk/prime_mcp.json` and
  `~/.it-desk/prime_settings.json`.
- The bug mirror writes `BUGS.md` into `IT_DESK_BUGS_DIR` on every bug write.

## License

MIT. See [LICENSE](LICENSE).