Mnehmos Synch

# Orchestrator Prompt: The Archivist Implementation

**Objective**: Implement The Archivist companion webapp as a standalone project, following the validated design in [`archivist-companion-app.md`](./archivist-companion-app.md).

**Target Repository**: `archivist-companion` (new standalone repo)

---

## Project Initialization

```
TASK: Initialize archivist-companion as a new standalone project

Create the following structure:
- Python backend (FastAPI + archivist core)
- React frontend (Vite + TailwindCSS)  
- Shared config and Docker support

Reference: docs/design/archivist-companion-app.md - "Appendix: File Structure"
```

---

## Task Map

### Phase 1: Core Infrastructure (MVP)

| task_id | name | mode | deps | workspace_path | file_patterns | acceptance_criteria |
|---------|------|------|------|----------------|---------------|---------------------|
| `P1-01` | Initialize Python project structure | code | - | `archivist/` | `*.py`, `requirements.txt`, `setup.py` | Python package structure with `__init__.py`, entry point in `main.py`, dependencies in `requirements.txt` |
| `P1-02` | Implement configuration management | code | P1-01 | `archivist/` | `config.py` | Pydantic-based config loading from JSON/env vars, supports all settings from design doc |
| `P1-03` | Implement file system watcher | code | P1-01 | `archivist/` | `watcher.py` | Uses watchdog, monitors `~/.agent-synch/logs/inbox`, emits events on new files |
| `P1-04` | Implement SQLite queue manager | code | P1-01 | `archivist/` | `queue.py` | CRUD for processing_queue table, status transitions, retry logic |
| `P1-05` | Implement MCP client wrapper | code | P1-02 | `archivist/` | `mcp_client.py` | Connects to mnehmos.synch.mcp via stdio, exposes all required tools as async methods |
| `P1-06` | Write tests for Phase 1 | red-phase | P1-03, P1-04, P1-05 | `tests/` | `test_*.py` | Tests for watcher, queue, MCP client - must fail initially |
| `P1-07` | Make Phase 1 tests pass | green-phase | P1-06 | `archivist/`, `tests/` | `*.py` | All red phase tests turn green |

### Phase 2: Processing Pipeline

| task_id | name | mode | deps | workspace_path | file_patterns | acceptance_criteria |
|---------|------|------|------|----------------|---------------|---------------------|
| `P2-01` | Implement LLM extractor (Ollama) | code | P1-07 | `archivist/` | `extractor.py` | Connects to Ollama API, sends extraction prompt, parses JSON response |
| `P2-02` | Implement main processor loop | code | P2-01, P1-05 | `archivist/` | `processor.py` | Async loop: dequeue → extract → call MCP tools → emit checkpoint → mark complete |
| `P2-03` | Add lock management to processor | code | P2-02 | `archivist/` | `processor.py` | acquire_lock before writes, release_lock in finally block, handle timeout |
| `P2-04` | Implement traffic log parser | code | P1-01 | `archivist/` | `parser.py` | Parse JSONL traffic files, extract metadata, handle malformed entries |
| `P2-05` | Write tests for Phase 2 | red-phase | P2-03, P2-04 | `tests/` | `test_processor.py`, `test_extractor.py`, `test_parser.py` | Tests with mocked Ollama and MCP - must fail initially |
| `P2-06` | Make Phase 2 tests pass | green-phase | P2-05 | `archivist/`, `tests/` | `*.py` | All red phase tests turn green |
| `P2-07` | Integration test: end-to-end | code | P2-06 | `tests/` | `test_integration.py` | Full flow: write traffic file → watcher detects → processor extracts → MCP tools called |

### Phase 3: Webapp Backend

| task_id | name | mode | deps | workspace_path | file_patterns | acceptance_criteria |
|---------|------|------|------|----------------|---------------|---------------------|
| `P3-01` | Initialize FastAPI app structure | code | P1-07 | `archivist/api/` | `__init__.py`, `routes.py`, `schemas.py` | FastAPI app with CORS, health endpoint returns 200 |
| `P3-02` | Implement queue status endpoints | code | P3-01, P1-04 | `archivist/api/` | `routes.py` | GET /api/queue/status, GET /api/queue/items work |
| `P3-03` | Implement logs endpoints | code | P3-01 | `archivist/api/` | `routes.py` | GET /api/logs/inbox, GET /api/logs/{project_id} list files |
| `P3-04` | Implement memory proxy endpoints | code | P3-01, P1-05 | `archivist/api/` | `routes.py` | Proxy to MCP: search_memory, list_projects, get_active_context, get_context_events |
| `P3-05` | Implement manual trigger endpoints | code | P3-01, P2-02 | `archivist/api/` | `routes.py` | POST /api/process/trigger queues all inbox items |
| `P3-06` | Write API tests | red-phase | P3-05 | `tests/` | `test_api.py` | FastAPI TestClient tests for all endpoints |
| `P3-07` | Make API tests pass | green-phase | P3-06 | `archivist/api/`, `tests/` | `*.py` | All API tests green |

### Phase 4: Webapp Frontend

| task_id | name | mode | deps | workspace_path | file_patterns | acceptance_criteria |
|---------|------|------|------|----------------|---------------|---------------------|
| `P4-01` | Initialize React + Vite + Tailwind | code | - | `webapp/` | `package.json`, `vite.config.ts`, `tailwind.config.js`, `src/App.tsx` | `npm run dev` starts dev server, Tailwind works |
| `P4-02` | Create Dashboard page | code | P4-01 | `webapp/src/` | `pages/Dashboard.tsx`, `components/QueueStatus.tsx` | Shows queue metrics, Archivist health status |
| `P4-03` | Create Event Log page | code | P4-01 | `webapp/src/` | `pages/EventLog.tsx`, `components/TrafficViewer.tsx` | Lists traffic logs, expandable entries, filter by project |
| `P4-04` | Create Memory Browser page | code | P4-01, P3-04 | `webapp/src/` | `pages/MemoryBrowser.tsx`, `components/SearchResults.tsx` | Search input, results display, navigate to file details |
| `P4-05` | Create Settings page | code | P4-01 | `webapp/src/` | `pages/Settings.tsx` | Configure project mappings, LLM settings |
| `P4-06` | Add routing and navigation | code | P4-02, P4-03, P4-04, P4-05 | `webapp/src/` | `App.tsx`, `components/Nav.tsx` | React Router, sidebar navigation, all pages accessible |
| `P4-07` | Connect frontend to backend API | code | P4-06, P3-07 | `webapp/src/` | `api/*.ts`, `hooks/*.ts` | API client, React Query hooks, data flows correctly |

### Phase 5: Polish & Integration

| task_id | name | mode | deps | workspace_path | file_patterns | acceptance_criteria |
|---------|------|------|------|----------------|---------------|---------------------|
| `P5-01` | Implement Chat interface | code | P4-07 | `webapp/src/` | `pages/Chat.tsx` | Chat UI, sends to backend, backend calls LLM with MCP tools available |
| `P5-02` | Add WebSocket for real-time updates | code | P5-01 | `archivist/api/`, `webapp/src/` | `ws.py`, `hooks/useWebSocket.ts` | Dashboard updates in real-time when queue changes |
| `P5-03` | Create Dockerfile and docker-compose | code | P3-07, P4-07 | `.` | `Dockerfile`, `docker-compose.yml` | `docker-compose up` runs full stack |
| `P5-04` | Create PM2 ecosystem config | code | P3-07 | `.` | `ecosystem.config.js` | `pm2 start ecosystem.config.js` works |
| `P5-05` | Write README and documentation | memory | P5-04 | `.` | `README.md`, `docs/*.md` | Installation, configuration, usage documented |
| `P5-06` | Refactor for code quality | blue-phase | P5-05 | `archivist/`, `webapp/src/` | `*.py`, `*.ts`, `*.tsx` | Clean code, consistent style, no TODO comments |
| `P5-07` | Final integration test | debug | P5-06 | `tests/` | `test_e2e.py` | Full end-to-end: traffic logger → Archivist → MCP → webapp displays |

---

## Parallel Execution Groups

The following tasks can be executed in parallel as they have non-overlapping file patterns:

**Group A** (can run together):
- `P1-01` through `P1-05` (sequential within group)
- `P4-01` (frontend init - separate workspace)

**Group B** (after P1-07):
- `P2-01` through `P2-04` (can parallelize extractor, parser)
- `P3-01` (API init - different file patterns)

**Group C** (after P3-01):
- `P3-02`, `P3-03` (different route handlers, can merge)
- `P4-02` through `P4-05` (different pages, can parallelize)

---

## Key Contracts

### MCP Tool Usage Contract

The Archivist MUST use the following mnehmos.synch.mcp tools:

| Tool | When Used | Required Params |
|------|-----------|-----------------|
| `acquire_lock` | Before ANY write operation | resource_id, agent_id="archivist", operation="write" |
| `release_lock` | In finally block after writes | resource_id, agent_id="archivist" |
| `file_to_cabinet` | When indexing decisions/files | project_id, file_path, summary |
| `set_active_context` | When updating working state | project_id, summary, focus (optional) |
| `emit_context_event` | After processing session | project_id, agent_id="archivist", event_type="checkpoint", summary |
| `log_bug` | When extracting bugs from traffic | project_id, title, description, severity |
| `search_memory` | Webapp memory browser | query, project_id (optional) |
| `get_active_context` | Webapp context display | project_id |
| `get_context_events` | Webapp event log | project_id, limit |

### Storage Path Contract

All paths MUST use `~/.agent-synch/` as base:
- Logs: `~/.agent-synch/logs/inbox/` and `~/.agent-synch/logs/{project_id}/`
- Queue DB: `~/.agent-synch/archivist.db`
- Config: `~/.agent-synch/config/archivist.json`

### Error Handling Contract

All processing MUST:
1. Log errors to queue with `error_message`
2. Increment `attempts` counter
3. Retry up to 3 times with exponential backoff
4. After max retries, mark as `failed` and emit error event

---

## Acceptance Criteria (Project-Level)

1. **Traffic Logger**: A new traffic JSONL file in inbox/ triggers processing within 5 seconds
2. **Extraction**: LLM correctly identifies decisions, schema changes, bugs from sample traffic
3. **MCP Integration**: All tool calls succeed and data appears in mnehmos.synch.mcp
4. **Webapp**: All pages render correctly, API calls work, real-time updates function
5. **Reliability**: System recovers from crashes, queue persists across restarts
6. **Documentation**: README sufficient for new user to install and run

---

## How to Start

Copy this prompt to Orchestrator mode:

```
Initialize new project "archivist-companion" in F:\Github\archivist-companion

Follow the task map in docs/design/archivist-orchestrator-prompt.md

Start with Phase 1 tasks, executing in order of dependencies.
Delegate to appropriate worker modes based on task assignments.
Ensure parallel safety by respecting workspace_path and file_patterns.

Reference design document: F:\Github\mnehmos.synch.mcp\docs\design\archivist-companion-app.md
```

---

*Task Map generated by Architect Mode | 2025-12-31*