Havlio
by kanduit-lab
README.md
# Havlio
**Self-hosted PaaS with a write-capable MCP control plane.**
Havlio is an on-premise deploy hub for Docker services: Vercel-style deploy
workflows with runtime, secrets, and data staying on infrastructure you own.
> ## ⚠️ Proprietary — source-visible, not open source
>
> **You may read this code. You may not run it.**
>
> No license to use, deploy, self-host, copy, modify, or redistribute is
> granted — including for personal, internal, educational, or evaluation
> purposes. A public repository is not a grant of rights.
>
> See [LICENSE](LICENSE) and [NOTICE](NOTICE). The installation instructions
> below apply only to holders of a signed commercial license.
>
> **한국어:** 소스는 공개되어 있으나 오픈소스가 아닙니다. 열람만 허용되며 실행,
> 배포, 복제, 수정, 상업적·사내 이용은 모두 금지됩니다. 이용을 원하시면 사전에
> 서면 라이선스를 취득하십시오.
---
## What makes it different
Agents do not just read Havlio — they operate it, and they cannot outrank the
human who authorized them.
- **102 MCP tools, 40 of them mutating.** Deploy, roll back, manage domains,
sidecars, networks, users, and audit — all callable by an agent.
Comparable self-hosted control planes expose read-only MCP or none at all.
- **Agent permissions can never exceed human RBAC**, and CI proves it. Every
MCP tool is mapped to a hub permission in `MCP_TOOL_PERMISSIONS`
(`apps/api/core/mcp-tools-registry.ts`); the regression checks in
`apps/api/scripts/checks/` fail the build if a tool ships without one.
- **Agent actions are separable in the audit ledger.** `security_audit_events`
records `source_channel` (`mcp_oauth` / `mcp_api_token` / `api`) and
`mcp_tool`, so what an AI did and what a person did are distinct rows — not
a merged "system" actor.
---
**Frameworks:** Next.js · React SPA · Node.js · Spring Boot (auto-detected)
**Operating modes:** headless CD scripts only · full console/API control plane
**Core surfaces:** dashboard · services · deploy history · activity · builds ·
containers · domains · sidecars · networks · cleanup · email delivery · notices ·
users · roles · OAuth · MCP · audit log · settings
**Platform posture:** Havlio is moving from a deployment console into an
installed PaaS control plane. The product owns local identity, RBAC, MCP/API
tokens, audit history, gateway/domain operations, Docker runtime controls,
sidecars, deploy safety, and operator-visible failure contracts. External OAuth
providers and CI systems are adapters; they do not own Havlio authorization
or audit semantics.
**Security posture:** privileged actions must be permission-gated, step-up
guarded where sensitive, recorded in the audit ledger with HTTP status and
stable application error codes, and safe to review without leaking secrets,
tokens, raw logs, or terminal output. Audit read/export, owner/admin controls,
blocked-permission states, and deploy lifecycle errors all
use the same operator-facing status/error contract.
**Documentation:** feature explanations live in [docs/features/](docs/features/),
operator/user docs live in [docs/](docs/), and internal engineering docs live in
[docs/internal/](docs/internal/). The docs ownership map is
[docs/README.md](docs/README.md).
**Contributions:** this project does not accept external pull requests. See
[LICENSE](LICENSE).
---
## Architecture
```
Service Repo Hub (This Repo)
┌───────────────────────┐ ┌──────────────────────────────────┐
│ .deploy.yml │ │ Reusable Workflows │
│ .github/workflows/ │ │ ci-reusable.yml (build/push) │
│ deploy.yml ────────┼──── @v2 ───►│ cd-reusable.yml (deploy) │
│ │ │ │
│ dockerfiles/ │ │ Deploy Scripts │
│ .dockerignore │ │ lib/cd/ (blue-green logic) │
└───────────────────────┘ │ lib/config/ (yml parsing) │
│ │
Secrets: │ Management Console (optional) │
ENV_FILE_BASE64 │ apps/api (Fastify API) │
DEPLOY_REPO_TOKEN (private hub) │ apps/web (React SPA) │
NOTIFICATION_WEBHOOK_URL (opt) │ infra/ (nginx + gateway) │
│ Gateway (nginx container) │
│ per-app network isolation │
└──────────────────────────────────┘
```
**CI** runs on GitHub-hosted or self-hosted AMD64/ARM64 runners:
- `build-amd64` → native on AMD64 runner
- `build-arm64` → native on ARM64 runner (QEMU fallback if unavailable)
- `merge` → combines into a multi-arch manifest
**CD** runs on your self-hosted deploy server.
---
## Quick Start
### 1. Server Setup
```bash
# Public hub repo
HAVLIO_REPO=kanduit-lab/havlio
HAVLIO_TAG=$(curl -fsSL "https://api.github.com/repos/${HAVLIO_REPO}/releases/latest" \
| sed -n 's/.*"tag_name": "\(v[^"]*\)".*/\1/p' | head -1)
curl -fsSL "https://github.com/${HAVLIO_REPO}/releases/download/${HAVLIO_TAG}/setup-${HAVLIO_TAG}.sh" -o setup.sh
bash setup.sh
# Private hub repo (gh CLI required)
gh auth login
gh auth refresh -s read:packages
gh auth token | docker login ghcr.io -u $(gh api user -q .login) --password-stdin
gh release download -R kanduit-lab/havlio -p "setup-*.sh" --clobber
bash setup-*.sh
```
The interactive installer defaults to `/srv/havlio` for new installs,
creating it with sudo and handing ownership to the installing user when needed.
If an existing legacy install is found at `~/havlio`, the installer reuses
that path instead of silently creating a second hub. Choose the final install
directory at setup time with `--dir`; moving an existing runtime directory is
not recommended because SQLite state and app data may include Docker-created
ownership. When both default locations exist, pass `--dir` explicitly. It
configures gateway ports, SSL, update channel, container names, and optionally
the management UI. No manual YAML editing is required for normal installs.
### 2. Self-Hosted Runners
Runners handle CD (deploy) and optionally CI (native builds). Register runners on each **service repo** (Settings → Actions → Runners → New self-hosted runner).
#### CD Runner (deploy server)
```bash
mkdir -p ~/actions-runner && cd ~/actions-runner
# Follow GitHub's download + install instructions for your OS
./config.sh --url https://github.com/OWNER/SERVICE-REPO --token YOUR_TOKEN --labels self-hosted,production
sudo ./svc.sh install && sudo ./svc.sh start
```
The default `runner_labels` in `.deploy.yml` is `["self-hosted"]`. Add extra labels like `production`, `staging`, or machine-specific tags as needed.
#### CI Runners (native builds, optional)
Native builds avoid QEMU and are significantly faster, especially for ARM64.
**AMD64 runner:**
```bash
./config.sh --url https://github.com/OWNER/SERVICE-REPO --token YOUR_TOKEN \
--labels self-hosted,X64,build
```
**ARM64 runner:**
```bash
./config.sh --url https://github.com/OWNER/SERVICE-REPO --token YOUR_TOKEN \
--labels self-hosted,ARM64,build
```
Then configure in `.deploy.yml`:
```yaml
ci:
runner_labels_amd64: '["self-hosted", "X64", "build"]'
runner_labels_arm64: '["self-hosted", "ARM64", "build"]'
```
> **If a runner with the configured labels is not available**, the build job will wait up to 60 minutes then fail. If you don't have native runners, leave these unset to use GitHub-hosted runners with QEMU.
### 3. Hub Repo Settings (once per org)
Settings → Actions → General:
- Actions permissions: **Allow all actions and reusable workflows**
- If private: Access → **Accessible from repositories in the organization**
### 4. Service Repo Onboarding
```bash
# Download deploy-manager (public hub)
HAVLIO_REPO=kanduit-lab/havlio
HAVLIO_TAG=$(curl -fsSL "https://api.github.com/repos/${HAVLIO_REPO}/releases/latest" \
| sed -n 's/.*"tag_name": "\(v[^"]*\)".*/\1/p' | head -1)
curl -fsSL "https://github.com/${HAVLIO_REPO}/releases/download/${HAVLIO_TAG}/deploy-manager-${HAVLIO_TAG}.sh" -o deploy-manager.sh
# Or private hub
gh release download -R kanduit-lab/havlio -p "deploy-manager-*.sh" --clobber
mv deploy-manager-*.sh deploy-manager.sh
# Install
bash deploy-manager.sh install
```
The install command:
1. Detects your framework
2. Generates `.deploy.yml`
3. Copies workflow templates, Dockerfiles, health check files
4. Optionally configures GitHub secrets via `gh` CLI
**Secrets — two approaches (choose one):**
A) **GitHub Environment secrets (recommended)**
Settings → Environments → create `production` (and `staging` if needed):
| Environment | Secret | Value |
| ------------- | -------- | ------- |
| `production` | `ENV_FILE_BASE64` | `base64 -w 0 < .env.prod` |
| `staging` | `ENV_FILE_BASE64` | `base64 -w 0 < .env.stg` |
Workflows use `secrets: inherit` — each environment gets its own .env automatically.
B) **Repo-level secrets (legacy)**
| Secret | When | Value |
| -------- | ------ | ------- |
| `ENV_FILE_BASE64` | Single env | `base64 -w 0 < .env` |
**Common secrets** (Settings → Secrets → Actions):
| Secret | When | Value |
| -------- | ------ | ------- |
| `DEPLOY_REPO_TOKEN` | Hub repo is private | Fine-grained PAT: `contents:read` on hub repo |
| `NOTIFICATION_WEBHOOK_URL` | Optional | Slack/Discord webhook URL |
### 5. Deploy
```bash
# Production (stable v* tag → build + blue-green deploy + GitHub Release)
git tag v1.0.0 && git push origin v1.0.0
# Staging (develop branch push or beta tag → staging deploy)
git push origin develop
git tag v1.1.0-beta.1 && git push origin v1.1.0-beta.1
# Preview (PR open/update → ephemeral preview deploy, cleanup on PR close)
# Automatic — just open a PR
# Manual trigger: GitHub → Actions → Deploy → Run workflow
```
### 6. Server Management
Use the console for service status, rollback, restart, stop, cleanup, logs, and
domain operations. The server scripts are reserved for installation, updates,
and break-glass recovery when the console is unavailable.
```bash
HAVLIO_DIR=/srv/havlio # use ~/havlio for older installs not yet migrated
bash "$HAVLIO_DIR/bin/havlio.sh" # operator menu
bash "$HAVLIO_DIR/bin/havlio.sh" version # version info + update check
bash "$HAVLIO_DIR/bin/havlio.sh" update # update to latest
bash "$HAVLIO_DIR/bin/havlio.sh" update --tag v2.5.0 # update to specific version
bash "$HAVLIO_DIR/bin/havlio.sh" restart # emergency gateway + console restart
bash "$HAVLIO_DIR/bin/havlio.sh" reset --dry-run # preview emergency soft reset
bash "$HAVLIO_DIR/bin/havlio.sh" uninstall --backup # full removal with config metadata backup
```
```bash
# Service repo: keep deploy-manager and workflows up to date
bash deploy-manager.sh update # update to latest hub version
bash deploy-manager.sh update --check # check only, no changes
bash deploy-manager.sh update --tag v2.6.0 # pin to specific version
```
---
## `.deploy.yml` Reference
```yaml
# ── Required ──────────────────────────────────────────────────────────────────
app_name: my-app # container name prefix (my-app-blue / my-app-green)
domain: app.example.com # public domain for this service
framework: nextjs # nextjs | react | nodejs | spring | auto
# ── CI ────────────────────────────────────────────────────────────────────────
ci:
trivy_scan: true # vulnerability scan after build (default: true)
# Native build runners (optional — defaults to GitHub-hosted ubuntu-latest + QEMU)
# Labels must match runners registered on this repo.
# If set but no matching runner is online, build waits up to 60 min then fails.
runner_labels_amd64: '["ubuntu-latest"]' # AMD64 build runner
runner_labels_arm64: '["self-hosted", "ARM64"]' # ARM64 build runner (native = no QEMU)
# ── CD ────────────────────────────────────────────────────────────────────────
cd:
runner_labels: '["self-hosted"]' # deploy server runner labels
deploy_env: production # used in container naming + concurrency group
health_path: "" # override default health path for this app
network: auto # docker network name (auto = {app_name}-net)
network_strict: false # true = fail if network doesn't already exist
memory_limit: "" # e.g. 512m, 2g (empty = no limit)
cpu_limit: "" # e.g. 0.5, 2.0 (empty = no limit)
drain_timeout: 5 # seconds to wait before stopping old container
verify_deployment: true # curl health check from runner after deploy
notifications: true # send Slack/Discord on success/failure
# ── Preview deploys (PR-based) ────────────────────────────────────────────────
preview:
enabled: true
max: 5 # max concurrent previews (oldest removed first)
public: true # verify deployment via public URL
# ── Staging (multi-environment) ──────────────────────────────────────────────
# staging:
# enabled: true
# domain: "app-{channel}.example.com" # {channel} = branch slug, {hash} = sha7
# branches: [develop] # branches that trigger staging (default: develop)
# beta: true # deploy beta/rc tags to staging (default: true)
# ── Sidecar services ──────────────────────────────────────────────────────────
# services:
# postgresql:
# enabled: true
# version: "16"
# redis:
# enabled: true
# version: "7-alpine"
```
---
## `gateway.yml` Reference
Located at `/srv/havlio/gateway/gateway.yml` by default, or under the
explicit `--dir` install path. Managed by `havlio-install.sh`.
```yaml
# ── Gateway listener ─────────────────────────────────────────────────────────
port: 80 # HTTP port; use 8080 when behind a local proxy
# ── SSL — choose one mode ─────────────────────────────────────────────────────
# Mode 1: External proxy owns TLS
# Omit ssl_* keys when Cloudflare, nginx, Caddy, or a load balancer terminates TLS.
# Then set public_url — the gateway only sees plain HTTP and cannot infer it.
# public_url: https://hub.example.com
# Mode 2: Auto cert via Let's Encrypt
# ssl: auto
# ssl_email: admin@example.com
# ssl_staging: true # use Let's Encrypt staging CA for test certificates
# ssl_port: 443
# Mode 3: Manual cert (HTTP + HTTPS, both ports exposed)
# ssl_cert: /path/fullchain.pem
# ssl_key: /path/privkey.pem
# ssl_port: 443 # HTTPS port (port above becomes HTTP)
# https_redirect: true # HTTP→HTTPS redirect
# ── Update channel ───────────────────────────────────────────────────────────
update_channel: stable # stable | beta | auto | manual | notify
# ── Management UI/API (optional) ─────────────────────────────────────────────
# Leave omitted/commented for headless CD mode.
# dashboard: true
# dashboard_domain: hub.example.com
# ── Infrastructure naming (optional, defaults shown) ─────────────────────────
# network_name: havlio-net
# gateway_container: havlio-gateway
# web_container: havlio-web
# api_container: havlio-api
```
---
## Features
| Category | Feature |
| ---------- | --------- |
| **CI** | Auto framework detection |
| **CI** | Parallel native AMD64 + ARM64 builds (QEMU fallback) |
| **CI** | Configurable build runners per architecture |
| **CI** | Registry-based Docker layer cache |
| **CI** | Trivy vulnerability scan (CRITICAL/HIGH, toggleable) |
| **CD** | Blue-green zero-downtime deploy |
| **CD** | Health check with retry (10 attempts, 3s interval) |
| **CD** | Graceful drain before stopping old container |
| **CD** | Per-app Docker network isolation |
| **CD** | Deploy notifications (Slack/Discord/Mattermost) |
| **CD** | Multi-environment: production, staging, preview |
| **CD** | GitHub Environment secrets with `secrets: inherit` |
| **CD** | Instant rollback (no rebuild) |
| **CD** | Version compatibility check (client vs server major) |
| **CD** | Auto-injected `HAVLIO_*` env (app name, deploy env, image tag, sha, branch, actor, run URL, typed sidecar URLs, ...) + `NODE_ENV`, `PORT`, canonical sidecar URLs |
| **CD** | Service deletion cleans up GitHub Deployments/Environments |
| **CD** | Headless CD mode: gateway + scripts only; console UI/API optional |
| **Staging** | Auto-deploy on configured branches (default: `develop`); per-channel GH Environment for isolated secrets |
| **Staging** | Dynamic domain: `app-{channel}.example.com` |
| **Staging** | Own sidecar containers, production DB copy on first deploy |
| **Preview** | PR-based auto-deploy, cleanup on PR close |
| **Preview** | Pattern: `{subdomain}-{branch}-{sha7}.{root}` |
| **Preview** | Shared sidecar with isolated sandbox DB per preview |
| **Preview** | Auto-cleanup (keep max N, remove oldest) |
| **Preview** | PR comment with preview URL, GitHub Deployment card |
| **Console** | Web dashboard (session-based auth) |
| **Console** | Browser push notifications for automatic deploy/runtime events with per-user preferences, plus admin-selected notice delivery |
| **Console** | SMTP email delivery for email verification, user invitations, password reset links, account security, admin notices, and selected operational event email |
| **Console** | Branded HTML/plain-text email template previews in Settings -> Templates, with placeholder identity rows and reusable notification variants |
| **Console** | Installable PWA shell with service-worker background notification handling |
| **Console** | Per-env scope: pill selector at top scopes Overview/Logs/Terminal/Env Vars/Deployments to the selected environment |
| **Console** | Service management (blue/green switch, rollback, restart, stop) per environment |
| **Console** | Env Vars KV editor with hub-injected variables grouped by category (read-only) |
| **Console** | Manual deploy from image or git URL with live log stream |
| **Console** | Domain management (nginx CRUD + zero-downtime reload + DNS guide using `public_ip`) |
| **Console** | MCP and DNS integration guides with client presets, permission presets, explicit risk opt-in, service scoping, and copyable setup commands |
| **Console** | Container, network, sidecar list pages with search / filter / app-grouping |
| **Console** | Sidecar operations with mobile-friendly credentials, dump, restart, stop, and log actions |
| **Console** | Resource monitor (CPU/mem charts, host memory, load average) |
| **Console** | Docker cleanup preview + selective cleanup for images, exited containers, orphan networks, and volumes |
| **Console** | Cleanup schedules with presets plus explicit category selection |
| **Console** | Users table with server-side search, role/status/email filters, paging, lifecycle actions, and email verification actions |
| **Console** | Audit Log with review tabs, status/error-code filters, actor/target/resource filters, safe event details, and CSV/JSON export |
| **Console** | Permission-blocked page and embedded states with status code, reason code, policy/resource context, required permissions, request id when available, and next action |
| **Console** | Live Docker event feed (SQLite-backed, replays last N on connect) |
| **Console** | Build history page separates CI/BuildKit noise from runtime activity |
| **Console** | Deployment history (SQLite-backed, populated by hub-pipeline endpoint) |
| **Console** | Log viewer (streaming, container selector scoped to selected env) |
| **Console** | Web terminal (docker exec via xterm.js) |
| **Console** | Pretendard-based typography, dark mode, mobile layouts, global search |
| **Auth** | Multi-user RBAC with fine-grained permissions, preset roles + custom roles |
| **Auth** | Per-service and global role bindings (e.g. `deployer@web-store`, `admin@global`) |
| **Auth** | Session impersonation — test-as-user or test-as-role |
| **Auth** | TOTP (RFC 6238) 2FA with QR-code enrollment + 2-step login |
| **Auth** | Passkey / WebAuthn (FIDO2) — passwordless sign-in, platform or roaming |
| **Auth** | External OAuth providers — GitHub / Google / GitLab / Microsoft / Discord presets + any OIDC |
| **Auth** | Setup wizard captures admin account, hub contact, gateway/SSL intent, and external identity plan |
| **Auth** | Email ownership verification; OIDC `email_verified` is only true after SMTP verification or provider-verified email |
| **Auth** | Expiring user invitations with resend/revoke/audit flow instead of temporary password handoff |
| **Auth** | Activate/deactivate lifecycle for local users; inactive users are blocked across login, sessions, MCP tokens, and impersonation |
| **Auth** | Profile avatars via direct URL, opt-in Gravatar-compatible email fallback, or local upload history with crop/select/delete controls |
| **Auth** | Stable service IDs (`havlio_svc_*`) — rename-safe references |
| **Security** | Append-only security audit ledger with stable audit identities, authorized profile joins, redacted metadata, status codes, stable error codes, request/correlation ids, and owner-critical alert fanout |
| **Security** | Owner/admin policy gates for sensitive account, role, session, and privileged mutation paths, with remaining high-risk policy gaps documented rather than claimed complete |
| **Security** | Central API error response contract for auth, RBAC, validation, conflict, rate-limit, upstream, Docker/gateway, audit, and internal failures |
| **MCP** | `/api/mcp` JSON-RPC endpoint with tools for services, envs, deploys, logs, sidecars, domains, gateway, and events |
| **MCP** | OAuth 2.1 authorization server — PKCE S256 + RFC 7591 Dynamic Client Registration |
| **MCP** | Least-privilege discovery scope (`service.read activity.read`) with per-tool `insufficient_scope` step-up |
| **Console/MCP** | Secret-read boundary: env reads mask values by default; raw `.env` values require explicit reveal plus `service.env.read.secrets` |
| **MCP** | Multi-service scope via CSV (`?service=a,b,c`) — restrict one endpoint to a subset |
| **MCP** | Generated server key follows the saved brand slug as `<brand-slug>-havlio`, with render-only per-connection prefix/base/suffix edits |
| **MCP** | Supported clients: Claude Desktop / Claude Code / Codex / Cline / Cursor / any MCP client |
| **MCP** | DCR dedup by client name + loopback host — no duplicate entries on port rotation |
| **SSL** | Let's Encrypt auto-issuance + renewal (certbot sidecar) |
| **SSL** | Manual cert, auto cert, or HTTP-only mode |
| **SSL** | HTTP + HTTPS with configurable redirect |
| **SSL** | Wildcard cert detection (skips per-domain issuance) |
| **Sidecar** | Postgres, Redis, MySQL, MariaDB, MongoDB |
| **Sidecar** | Auto-generated credentials, persistent data via bind mount |
| **Infra** | Tarball-based distribution (no git on server) |
| **Infra** | Floating major version tags (`@v2` auto-receives updates) |
| **Infra** | Update channels: stable, beta, auto (patch only), manual |
| **Infra** | Configurable container/network names via `gateway.yml` |
| **Infra** | Concurrent deploy safety (flock on registry + nginx) |
| **Infra** | GHCR auto-login via `gh auth token` |
---
## Headless CD Mode
The console is useful, but not required for basic deployments. In headless CD mode the server runs the gateway, registry/state files, Docker networks, and deploy scripts only. Service repos can still deploy through GitHub Actions because `cd-reusable.yml` runs `lib/cd/deploy.sh` directly on the self-hosted runner.
What still works without `apps/api` / `apps/web`:
- blue-green deploys, rollback slot preservation, nginx reloads, sidecars, previews, preview cleanup, `havlio-restart.sh`, and `havlio-update.sh`
- registry updates in `/srv/havlio/registry/apps.json`
- external proxy or manual/wildcard certificate setups
What needs the console/API:
- dashboard, RBAC/users, OAuth, MCP, audit log/export,
permission-blocked UI, web terminal/logs, activity feed, cleanup UI,
settings UI, manual deploy API, and SQLite deployment history
- `/api/observability/pipeline` reporting from workflows; it is best-effort and skipped when `havlio_api_token` or the local API is absent
For direct HTTPS without an external proxy, prefer a manual/wildcard certificate in headless mode until wildcard DNS-01 issuance is implemented.
---
## Management Console
Enable the management console when you want the UI/API control plane, RBAC,
OAuth/MCP, audit review, activity, cleanup, settings, and
operator workflows:
```yaml
dashboard: true
dashboard_domain: hub.example.com
```
First access opens the setup wizard to create the admin account and apply safe
initial settings. When the installer creates a first setup key, the CLI prints it
after installation and the setup screen requires it before creating the admin.
The key is stored under the registry directory and removed after setup succeeds.
Appearance can be pre-configured in this first setup flow, SMTP can be configured
or skipped, and Gateway/SSL plus OAuth provider setup remain planned work behind
safer authenticated controls. The web/API images are pulled from GHCR on
install/update; no source checkout or local build is required on the server.
**Primary surfaces:** Dashboard · Services · Domains · Containers · Networks ·
Sidecars · Resources · Activity · Builds · Email Delivery · Notices · Users ·
Roles · Audit Log · Settings · MCP/DNS Integrations · Profile ·
About · Web Terminal
Security and audit behavior is intentionally operator-facing. Audit Log,
permission-blocked states, owner/admin policy gates, settings/security changes,
and deploy lifecycle errors follow the same stable
status/error-code contract where coded. Run
`pnpm -C apps/api check:manual-deploy-smoke` in a Docker-enabled environment
to verify the current state of live deploy smoke. See
[docs/features/securitySurfaceOverview.md](<docs/features/securitySurfaceOverview.md>)
for the durable beta security-surface guide.
Browser push notifications are configured per user from Profile → Notifications. Automatic deploy/runtime events honor the user's event preferences; admin notices use the channels selected by the sender and require an active browser subscription for push. The API auto-generates VAPID keys in `REGISTRY_DIR/web-push-keys.json`; set `WEB_PUSH_PUBLIC_KEY`, `WEB_PUSH_PRIVATE_KEY`, and optionally `WEB_PUSH_SUBJECT` when you want externally managed keys.
SMTP can be pre-configured during first setup or configured later from Settings -> SMTP. The hub stores host, port, username, password, sender address, TLS mode, and email delivery policy in its SQLite registry database, then uses that account for email verification, user invitations, password reset links, account security messages, admin notices, and selected operational event email. Admins can enable or disable transactional email and notification email separately, choose the default operational email opt-in, and tune email-change limits, outbound volume caps, rate limits, and link expiration for verification, password reset, invitation, SMTP test, and notification email flows. Settings -> Templates previews the standard branded HTML/plain-text templates for account, notification, and announcement email. Add SPF records for the SMTP provider, publish that provider's DKIM records, and set a DMARC policy for the sender domain; Havlio does not hold DKIM private keys.
Local avatars are stored under `REGISTRY_DIR/avatars`, replace the previous local avatar on upload, accept PNG/JPEG/WebP/GIF only, and are capped at 512KB. Removing a local avatar deletes the stored file and falls back to the normal URL avatar mode.
---
## Preview Deploys
Triggered automatically on PR open/update. Cleaned up on PR close.
```yaml
preview:
enabled: true
max: 5
```
Preview URL pattern: `{subdomain}-{branch}-{sha7}.{root-domain}`
Example: `app-feature-abc1234.example.com`
Requires wildcard DNS (`*.example.com`).
- Each preview gets its own container and nginx config
- Sidecars are shared with production — isolated via sandbox DB (`{app}_preview_{sha7}`)
- PR comment auto-updated with preview URL
- Old previews beyond `max` are auto-removed (oldest first)
- PR close triggers full cleanup (container, nginx config, sandbox DB)
---
## SSL
**Behind external proxy** (Cloudflare, nginx, Caddy handles SSL):
```yaml
port: 8080 # proxy forwards plain HTTP to this port
public_url: https://hub.example.com # the origin browsers actually use
```
`public_url` is required here, not optional. The gateway sees only plain HTTP, so
without it the hub publishes an `http://` OAuth issuer and rejects every browser
write with `BROWSER_SESSION_ORIGIN_REJECTED`. Auto and manual cert modes derive
the origin themselves and ignore the key.
**Direct exposure:**
```yaml
# Let's Encrypt (auto)
ssl: auto
ssl_email: admin@example.com
# Manual cert
ssl_cert: /etc/ssl/fullchain.pem
ssl_key: /etc/ssl/privkey.pem
# HTTP + HTTPS dual
ssl_cert: /etc/ssl/fullchain.pem
ssl_key: /etc/ssl/privkey.pem
ssl_port: 443 # HTTP stays on `port`, HTTPS on `ssl_port`
```
---
## Sidecar Services
```yaml
services:
postgresql:
enabled: true
version: "16"
redis:
enabled: true
version: "7-alpine"
```
Supported: `postgresql` · `redis` · `mysql` · `mariadb` · `mongodb`
- Accessible by name inside the app network: `myapp-postgresql:5432`
- Credentials auto-generated on first deploy → `/srv/havlio/data/{app}/{service}/.credentials`
- Data persisted to `/srv/havlio/data/{app}/{service}/data/`
---
## Framework Notes
| Framework | Health Path | Notes |
| ----------- | ------------- | ------- |
| `nextjs` | `/api/health` | Requires `output: 'standalone'` in `next.config` |
| `react` | `/` | Static SPA, Vite or CRA |
| `nodejs` | `/health` | Register route manually |
| `spring` | `/actuator/health` | Enable Actuator or add custom controller |
---
## Versioning
Service repos reference `@v2` (floating tag). The hub auto-updates the floating tag on each release:
```
hub releases v2.5.3
→ v2 floating tag moves to v2.5.3
→ all service repos on @v2 receive the update automatically on next run
```
Breaking changes (removed/renamed inputs) bump the major version (`v3`). The CD workflow rejects deploys where `client major > server major`.
Durable workflow and documentation hygiene rules live in
[docs/internal/01-project-rules.md](docs/internal/01-project-rules.md). `TODO.md`
is only a product backlog; release state belongs in GitHub Releases and tags.
---
## Project Structure
```
.github/workflows/
├── ci-reusable.yml # CI: parallel native builds + multi-arch manifest
├── cd-reusable.yml # CD: blue-green deploy via gateway
└── release.yml # Release: hub images + tarballs + floating tag
apps/
├── api/ # Fastify API, auth/RBAC, Docker/nginx control
├── web/ # React SPA (Vite + Tailwind)
└── ...
bin/ # Server commands
├── havlio.sh # Thin operator menu and command router
├── havlio-install.sh # Idempotent server provisioning
├── havlio-update.sh # Self-update from GitHub releases
├── havlio-restart.sh # Gateway + console restart with network recovery
├── havlio-version.sh # Version info + update check
├── havlio-reset.sh # Soft reset (containers only, config preserved)
└── havlio-uninstall.sh # Full teardown
lib/
├── cd/ # Blue-green deploy internals
│ ├── deploy.sh
│ ├── common.sh
│ ├── nginx.sh
│ ├── sidecar.sh
│ └── frameworks/
├── config/
│ └── parse-deploy-yml.sh # .deploy.yml parser (used by deploy.yml config job)
└── setup/
├── gateway-config.sh
├── interactive-setup.sh
└── compose-generator.sh
templates/
├── workflows/ # deploy.yml template ({{HAVLIO_MAJOR}} placeholder)
├── nginx/ # nginx.conf + app conf templates
├── dockerfiles/ # Framework Dockerfiles
├── health/ # Health check route templates
└── config/ # .dockerignore, .deploy.yml.example
docs/
├── README.md # Documentation ownership map
├── features/ # Product-facing feature explanations
├── internal/ # Maintainer-only engineering docs
│ ├── 01-project-rules.md # Durable project rules; no release/session state
│ ├── 02-console-architecture.md # Console/runtime architecture and API surface
│ ├── 03-test-implementation-plan.md # Verification layers and check:* promotion plan
│ └── 11-mcp-coverage.md # MCP dashboard/API coverage matrix
├── 10-mcp-connection-guide.md # MCP client connection runbook
├── 20-runner-tags.md # GitHub Actions runner label reference
└── 21-build-and-tls-operations.md # Build and TLS operator reference
dist/
├── setup.sh # Server bootstrap (released as asset)
├── deploy-manager.sh # Service repo CLI — Linux/Mac
└── deploy-manager.ps1 # Service repo CLI — Windows
infra/
├── Dockerfile # Runtime image for nginx + built web assets
├── nginx.conf # Gateway nginx template
└── nginx-web.conf # Console web serving config
```
## Local Development Shortcuts
Use `make` as the short local entrypoint. These targets are developer-only
wrappers; production installs still use release setup assets or
`bin/havlio-install.sh` directly. Durable rules for these shortcuts live in
[docs/internal/01-project-rules.md](docs/internal/01-project-rules.md).
```bash
make dev # dev gateway + setup key when needed + API :5000 + Vite :5174
make dev-down # stop only the background dev gateway
make demo-up # demo containers + seeded service metadata, without running pnpm dev
make dev-clean # reset dev API state to first setup and print a dev setup key
make demo-down # stop demo containers
make install # current source images + real interactive installer at ~/workspace/havlio
make reset # wrapper around ~/workspace/havlio/bin/havlio-reset.sh
make uninstall # wrapper around ~/workspace/havlio/bin/havlio-uninstall.sh
```
Forward reset/uninstall flags with `ARGS` instead of adding more make targets:
```bash
make reset ARGS="--dry-run"
make uninstall ARGS="--backup"
make uninstall ARGS="--yes-i-understand --no-backup"
make uninstall ARGS="--purge-certs" # remove preserved Let's Encrypt state too
```
`make reset` and `make uninstall` run the installed scripts through `bash`, so
the copied files do not need executable bits. If an installed script is missing
from `~/workspace/havlio/bin`, the make wrapper falls back to the source
`bin/` script with `--dir ~/workspace/havlio`. `make uninstall` is also
idempotent for local cleanup: when the local hub directory is already gone it
skips hub teardown and only removes the developer helper registry.
`make dev` starts the background dev gateway, then runs the foreground API and
Vite dev servers. It does not start or recreate demo containers, so repeated
foreground restarts do not disturb demo fixture state. If first setup is not
complete, it prints the current dev setup key or generates one before starting
the foreground dev servers. Use `make dev-down` when you need to stop the
background dev gateway.
Use `make demo-up` when an existing `make dev` or `make install` console needs
sample service containers and seeded service metadata. It starts only the demo
Docker fixture, seeds `dev/data/registry/apps.json` for plain local `pnpm dev`,
and when `~/workspace/havlio/registry` exists it also merges the same demo
service entries into the local install registry so a `make install` console can
show the same Services list. It does not start the foreground dev servers or
own the dev gateway lifecycle. Disable that install-registry sync with
`make demo-up DEV_SYNC_INSTALL=false`.
Use `make dev-clean` to replay the public first-time setup wizard without
running the full installer rehearsal: it removes the local dev registry database
and seeded service metadata under `dev/data/registry`, creates a new
`dev/data/registry/first-setup-key`, and prints that key in the terminal. Restart
`pnpm dev` after `make dev-clean` if the API was already running, because the API
keeps SQLite open while the process is alive.
`make demo-up` also recovers the common Docker stale-network case for demo
fixtures: if an old demo container points at a removed Docker network, the target
recreates only the local demo fixture and then reseeds metadata.
The `havlio-demo` Compose project is a local UI fixture, not an installed hub.
It starts sample service containers, app networks, stopped-service state, and
postgresql/redis sidecars so the console can render services, sidecars, networks,
deployments, builds, and activity without deploying real applications. Keep it in
`make demo-up`; do not make `make install` depend on demo data.
`make install` is the release-candidate installer rehearsal. It builds the
current web/API images, pushes them to a local registry, then runs the real
`bin/havlio-install.sh` flow against `~/workspace/havlio`. The installer still
prompts for gateway boundary, domain, ports, update channel, and container names; it
does not pre-generate `gateway.yml`. The resulting URL depends on the wizard
answers, with `http://localhost:8080` as the common local external-proxy choice.
The local registry is a developer helper container named
`havlio-local-registry` on `localhost:5001`. It is outside the installed hub
Compose project so the rehearsal still pulls images like a production install
would. `make uninstall` removes this helper after running the installed
uninstall script.
`LOCAL_PUBLIC_APP_URL` is a web-build metadata value. It fills `index.html`
canonical, Open Graph, and Twitter URLs; it does not control runtime routing,
API calls, login redirects, service-worker scope, or gateway behavior. For a
DDNS rehearsal, pass it so the built static metadata matches the public console
URL:
```bash
make install LOCAL_PUBLIC_APP_URL="https://dev.kanduit.site/"
```
Runtime links are owned by the API and gateway request path, not by the static
web bundle. Auth and OAuth links use the incoming forwarded host/proto headers.
Background notification and notice email links use `HAVLIO_URL`,
`HAVLIO_PUBLIC_URL`, or `PUBLIC_BASE_URL` when those API environment variables are
set; otherwise they fall back to the local development URL. If email links show
`localhost`, verify the gateway/proxy headers and API public URL environment
before treating it as a web-build issue.
`make install` remembers the last successful `LOCAL_PUBLIC_APP_URL` in
`~/workspace/havlio-local.mk`, next to the local install directory and the
preserved certbot state. Future `make install` runs reuse that value unless you
override it on the command line. Delete the file to return to the default
`http://localhost:8080/` metadata URL.
If `~/workspace/havlio/gateway/gateway.yml` already exists, the installer
reuses it. To replay the first-time wizard, run `make uninstall` and then
`make install`; the make target calls the installed hub uninstall script instead
of inventing a separate removal path. Override the path with
`make install LOCAL_HAVLIO_DIR="$HOME/workspace/havlio-alt"` when you need
another local install directory.
When management UI is enabled during a fresh first setup, the installer prints a
highlighted `FIRST SETUP KEY - COPY THIS` block after startup. Paste that value
into the dedicated Setup key screen to create the first admin and apply initial
appearance/SMTP settings. Re-running setup against an existing install checks
the install's update channel before downloading, and update/refresh paths do not
reprint the key. The key is stored at
`~/workspace/havlio/registry/first-setup-key` during setup and is removed
after the first admin is created.
Uninstall preserves Let's Encrypt certbot state by default outside the hub
directory at `~/workspace/havlio-certbot-conf`, and the next install restores
it automatically when the new `gateway/certbot/conf` directory is empty. This
keeps local release rehearsals from burning certificate rate limits. When the
restored certificate for the management domain is still valid, install reuses it,
rewrites nginx to the restored `/etc/letsencrypt/live/<domain>` paths, and skips
the initial certbot issuance attempt. Use
`make uninstall ARGS="--purge-certs"` only when you intentionally want the
certificate material removed. Root-owned certbot files are handled without
interactive sudo prompts; if they cannot be preserved or removed safely, uninstall
stops before deleting the hub directory and prints the required recovery action.
The rehearsal intentionally keeps the installer's production defaults visible in
the Advanced prompts (`havlio-net`, `havlio-gateway`,
`havlio-web`, `havlio-api`). Change them only when another hub is
already using those names on the same Docker host. Installer runs remove Compose
orphans in the hub project so old rehearsal containers from renamed local
configs do not stay attached to the active install.
## Development Verification
For local development, use type/build checks as the default fast gate:
```bash
pnpm -C apps/api typecheck
pnpm -C apps/web build
```
The API package also exposes focused `check:*` audit scripts for contracts such
as MCP coverage, permission catalog drift, email verification, invitations,
notices, and SMTP policy. These checks are intentionally selected by changed
surface instead of run as one global gate. The verification policy and future
test-suite promotion plan are owned by
[docs/internal/03-test-implementation-plan.md](docs/internal/03-test-implementation-plan.md).
---
## Troubleshooting
| Problem | Fix |
| --------- | ----- |
| Reusable workflow 404 | Check floating tag exists (`v2`). Havlio: Settings → Actions → allow org repos |
| GHCR `unauthorized` on CD | Package Settings → Manage Actions access → add repo |
| Cross-repo checkout 403 | Set `DEPLOY_REPO_TOKEN` (fine-grained PAT: `contents:read` on hub) |
| Web/API image pull fails | `gh auth refresh -s read:packages` then `bash bin/havlio-update.sh` |
| Build job stuck | No runner with matching labels online. Check `runner_labels_amd64/arm64` in `.deploy.yml`. Timeout is 60min. |
| Health check fails | `docker logs {app}-blue` |
| Next.js blank page | Ensure `output: 'standalone'` in `next.config`. Check for `app/` vs `src/app/` conflict. |
| Preview not triggering | Ensure `preview.enabled: true` in `.deploy.yml` and PR targets main/master |
| Gateway 404 for all domains | Check `gateway/conf.d/` has app configs. Run `bash bin/havlio-restart.sh` |
| Version mismatch in CD | Update server first: `bash bin/havlio-update.sh` |
| Containers not in console | Check service naming, labels, and Docker network membership; then refresh the console. |