domin8

# PROJECT DIRECTORY (OCD Librarian Edition) 📚🔬 **Purpose:** Define a perfectly deterministic, auditable, machine-enforceable project layout and the programmatic rules for how files, artifacts, and agent-created data are stored, named, and versioned. --- ## Table of Contents 1. Philosophy & Goals ✅ 2. Top-level tree (recommended) 🌲 3. Directory & file responsibilities 🔧 4. The agent artifact mirror (`~/.domin8/agent_data/`) — canonical layout 🗂️ 5. Metadata & schema conventions (JSON/Pydantic) 📄 6. Programmatic workflows & commands (CLI hooks) ⚙️ 7. Commit message & git rules (tamper-evident) 🧾 8. Naming, formatting, and style rules ✍️ 9. CI, pre-commit hooks, and enforcement 🔒 10. Retention, archival & verification policies 🕵️‍♂️ 11. Search, audit, and introspection tools 🔍 12. Example artifact lifecycle (create → approve → commit) 🔁 --- ## 1) Philosophy & Goals (one sentence) Be deterministic, auditable, and hostile-by-default: every destructive or intrusive operation must be structured, validated, human-approved, logged, hashed, and stored in a single canonical artifact directory outside the repository. --- ## 2) Top-level tree (recommended) ```text / (repo root) ├── src/ # Python source (package: domin8) │ └── domin8/ │ ├── __init__.py │ ├── server.py │ ├── tools/ # MCP tool implementations & pipelines │ ├── schemas/ # Pydantic schemas, discriminated unions │ └── storage/ # thin API layer to write to ~/.domin8/agent_data ├── tests/ # unit/integration/e2e tests ├── docs/ # markdown docs, incl. PROJECT_DIRECTORY.md ├── scripts/ # helper scripts and maintenance utilities ├── config/ # non-sensitive config templates (yaml/json) ├── .domin8-ignore # list of repo paths to ignore from mirroring ├── .github/ci/ # CI workflows and policy definitions ├── .gitignore ├── pyproject.toml ├── README.md └── main.py ~/.domin8/agent_data/ # EXTERNAL canonical mirror (never committed) └── <relative-path>/ # e.g. README.md/ └── artifacts/ └── <UUID>/ ├── meta.json ├── intent.json ├── approval.json ├── diff.patch ├── commit.md └── verification.sha256 ``` > Note: `~/.domin8/agent_data/` MUST be excluded from the repo and backed up separately as secure artifacts. --- ## 3) Directory & File Responsibilities - `src/domin8/` — Application logic. Small modules, single responsibility per file. - `src/domin8/tools/` — Each MCP tool is its own module and contains explicit Pydantic input/output types and the deterministic execution pipeline. - `src/domin8/schemas/` — All Pydantic models live here. Use discriminated unions and strict validators to force meaningful agent responses. - `src/domin8/storage/` — Exposes a deterministic API for artifact creation, hashing, atomic writes, and access control. - `tests/` — Tests are grouped by unit, integration, and e2e. Fixtures replicate deterministic system times & seeds. - `docs/` — Canonical documentation; `docs/PROJECT_DIRECTORY.md` is the authority for layout & conventions. - `scripts/` — Utilities to migrate, verify, and inspect artifact trees. - `config/` — Human-readable templates; sensitive values must be pulled from secrets manager and never committed. --- ## 4) Canonical Agent Artifact Mirror: `~/.domin8/agent_data/` 🗄️ This is the single source of truth for every artifact created by the pipeline. Layout rules: - Root: `~/.domin8/agent_data/` (per-machine) - For each repo-relative file `path/to/file.ext`, create a directory: `~/.domin8/agent_data/path/to/file.ext/` (note: file names included in directory names) - Under each file directory: an `artifacts/` directory containing one directory per artifact, named by a generated UUIDv7 (or UUIDv4 if required): `artifacts/<UUID>/` Each artifact directory contains strictly-named files: - `meta.json` — canonical metadata (see next section) - `intent.json` — the agent-supplied structured payload as validated by Pydantic - `approval.json` — human approval record (approver id, timestamp, comments, signature) - `diff.patch` — the textual patch to be applied (unified diff) - `commit.md` — synthesized commit message and references - `verification.sha256` — cryptographic verification of concatenated artifact parts Access controls: - Write via `storage` API only (atomic create + chmod 0o600) - Owner-only read/write by default on disk Tamper-evidence: - Each artifact must compute `verification.sha256` that is the SHA-256 over `meta.json|intent.json|diff.patch|approval.json` normalized with canonical JSON and unix newlines. - Optionally sign `verification.sha256` with a host key to support non-repudiation. --- ## 5) Metadata & Schema Conventions (JSON + Pydantic) 📐 Required `meta.json` fields (strict; no nulls): - `uuid` (string, UUIDv7 or UUIDv4) - `repo_relative_path` (string) - `action` (enum: `create|edit|delete|move|rename`) - `actor` (object: `{id: str, type: 'agent' | 'human' | 'system'}`) - `created_at` (ISO8601 with timezone) - `pre_state_sha` (git blob sha or `null` if create) - `post_state_sha` (expected git blob sha after commit) -- `deterministic_flags` (object; e.g., {schema_version: int, tool_version: "1.2.3"}) -- `short_summary` (string, <= 140 chars) - `status` (enum: `pending|approved|rejected|committed|rolled_back`) - `signature` (optional string; host-signed HMAC or signature) Schema rules: - All schemas live in `src/domin8/schemas/` and are imported by tools. - Use strict type validation; require non-empty strings for justifications. - Include `explainability` field where agent must provide at least 3 bullet points describing why change is required. Validators must enforce meaning (non-boilerplate length-checking + semantic checks where possible). --- ## 6) Programmatic workflows & interactions (chat & API hooks) 🔁 Core operations are exposed via a deterministic CLI and the MCP server. Example high-level steps for a single artifact: 1. `domin8 request-change --file README.md --payload ./payload.json` 2. Server validates using Pydantic (fail fast). On success: create `artifacts/<UUID>/intent.json` and `meta.json` with `status: pending`. 3. Emit human-review notification and wait for `approval.json` to be written with `status: approved`. 4. On approval: run pre-commit checks (linters, tests, diff sanity). 5. Generate `diff.patch`, apply in isolated temporary repo, compute `post_state_sha`, and write to artifact dir atomically. 6. Commit with standardized commit message and update `meta.json` to `committed`. 7. Compute and write `verification.sha256` and, optionally, host signature. Atomicity rules: - All writes to artifact directories must be atomic: write-to-temp + fsync + rename. - If anything fails, record `status: rolled_back` and persist diagnostics. Sample interactions & API verbs: - Create artifact via MCP tool call (`request_change`) - Approve via Continue (MCP elicitation) or via programmatic call to the library's approval API - Commit the artifact programmatically via the server's commit pipeline (often triggered after approvalften triggered after approval) --- ## 7) Commit Message & Git Rules 🧾 Commit message generation — rules and template (automatically generated from artifact data): - The **git commit message MUST be generated programmatically** by the commit pipeline using data copied from `intent.json`, `meta.json`, `approval.json`, and a machine-computed diff summary from `diff.patch`. - **Human-edited commit messages are disallowed** for MCP-managed changes; the generator is the single source of truth for commit content. Template (subject + body) — filled by the generator using artifact fields: ```text domin8: <UUID> <repo_relative_path> — <ACTION> — <short_summary> * artifact: ~/.domin8/agent_data/<repo_relative_path>/artifacts/<UUID>/ * actor: <actor.id> (<actor.type>) * pre_sha: <pre_state_sha> * post_sha: <post_state_sha> * created_at: <created_at> * explainability: - <bullet 1 from intent.json.explainability> - <bullet 2> - <bullet 3> Diff summary (automatically generated from diff.patch): - files changed: <n> - insertions: <n> - deletions: <n> Artifact reference: ~/.domin8/agent_data/<repo_relative_path>/artifacts/<UUID>/ ``` Operational details: - The pipeline synthesizes the exact finalized commit message into `commit.md` inside the artifact directory before performing the `git commit`. - Commits MUST be performed by the pipeline with `git commit --file commit.md` (or equivalent git API call) so the repository history contains the canonical message. - The commit generator must copy the relevant fields from `intent.json` (including `short_summary` and `explainability`) and any deterministic metadata from `meta.json` into the message. The human `approval.json` record (approver id, timestamp, comments) MUST be included in the commit body when present. Enforcement: - Install a `pre-commit` hook that rejects commits to MCP-governed files unless the commit message in the git object **exactly matches** the content of a `commit.md` from a `status: committed` artifact that references the same UUID. - CI checks must verify that the commit message was generated from and matches the artifacts (`intent.json`, `meta.json`, `approval.json`, `diff.patch`, and `verification.sha256`). Any mismatch causes the build to fail and the PR to be rejected. - Any manual attempt to edit the commit message will be flagged and rejected by the enforcement hooks. (See enforcement section below for additional CI and hook policies.) --- ## 8) Naming, formatting, & style rules ✍️ - Files: `snake_case.py`, modules lowercase. - Tests: `tests/unit/test_*.py`, `tests/integration/test_*.py`. - Schema files: `schemas/<resource>_schema.py`. - UUIDs: Use RFC-compliant UUIDs; prefer time-ordered option (UUIDv7) for easier indexing. - Timestamps: ISO8601 with timezone (e.g., 2026-01-12T16:34:00+01:00). --- ## 9) CI, pre-commit hooks, and enforcement 🔐 Minimum set of checks: - `python -m pytest --maxfail=1 -q` (deterministic tests only) - `ruff`/`flake8` + `black` for formatting - `domin8 verify-artifact --uuid <UUID>` run as required Policy enforcement: - Local CI must run artifact verification and block merges if verification fails or artifact lineage is missing. --- ## 10) Retention, archival & verification policies 🕰️ - Artifacts are immutable once `status: committed`. - Retention default: keep all artifacts indefinitely (until processed intelligently) - Verification: periodic cron to re-hash and verify `verification.sha256`. --- ## 11) Search, audit & introspection tooling 🔍 - Provide an API / MCP listing to list artifacts with statuses and short summaries (used by Continue sidebar). - Provide an audit endpoint or library function to retrieve full artifact details and produce exports (e.g., `audit_report_<UUID>.json`). - Use a small SQLite index in `~/.domin8/index.sqlite` that stores artifact rows for fast search, updated atomically when artifacts are created. --- ## 12) Example artifact lifecycle (end-to-end) 🔁 1. Agent calls tool with payload → Pydantic validates → `artifacts/<UUID>/intent.json` + `meta.json` (status=pending) 2. Human reviews → writes `approval.json` (status=approved) 3. Tool runs pre-commit checks, computes `diff.patch`, applies in isolated workspace, computes `post_state_sha` and commits with message containing UUID 4. Tool writes `commit.md` and `verification.sha256`, sets `meta.json.status=committed` 5. Audit can re-run `domin8 verify --uuid <UUID>` anytime --- ## Minimal Implementation Checklist (for maintainers) ✅ - [ ] Create `src/domin8/storage/__init__.py` with atomic write primitives - [ ] Implement `src/domin8/schemas/` with strict validators - [ ] Expose programmatic APIs and MCP tools for creating artifacts, reviewing, approving, committing, and auditing (no CLI). - [ ] Add pre-commit hook and CI verification job - [ ] Implement indexing service for `~/.domin8/index.sqlite` - [ ] Add periodic verification cron script --- ## Closing Notes 💡 This file is the single-source-of-truth for how the repo organizes and manages programmatically-created artifacts and agent actions. Keep it obsessive, keep it explicit, and keep the mirror directory canonical and off-repo.