# Implementation Plan: Pathway 1 - Conscious Memory Architecture
**Memori Integration with Multi-Provider LLM Cascade**
**Status:** 🟢 Complete
**Timeline:** 4-6 weeks (Standard)
**Last Updated:** November 20, 2025
______________________________________________________________________
## Progress Tracker
### Overall Progress: 100% Complete
| Phase | Status | Progress | Target Completion |
|-------|--------|----------|-------------------|
| **Phase 0: Preparation** | 🟢 Complete | 4/4 tasks | Week 0 |
| **Phase 1: Enhanced Schema** | 🟢 Complete | 5/5 tasks | Week 1 |
| **Phase 2: Multi-Provider Extraction** | 🟢 Complete | 6/6 tasks | Week 2 |
| **Phase 3: Conscious Agent** | 🟢 Complete | 5/5 tasks | Week 3-4 |
| **Phase 4: Filesystem Integration** | 🟢 Complete | 4/4 tasks | Week 5 |
| **Phase 5: Testing & Rollout** | 🟢 Complete | 6/6 tasks | Week 6 |
**Legend:** 🔵 Not Started | 🟡 In Progress | 🟢 Complete | 🔴 Blocked
______________________________________________________________________
## Overview
Implement native Memori-inspired patterns (Conscious Agent, multi-provider entity extraction, memory categorization) within session-buddy while preserving superior ONNX vector search and development workflow tools.
### Key Features
- ✅ **Multi-provider LLM cascade:** OpenAI → Claude → Pattern-based → Ollama (future)
- ✅ **Smart threshold extraction:** Only extract from significant files (relevance > 0.7)
- ✅ **Graceful degradation:** All features optional with fallback
- ✅ **Zero breaking changes:** Backward compatible with existing installations
- ✅ **Auto-migration:** Intelligent detection and migration of existing data
______________________________________________________________________
## Design Decisions
### 1. Multi-Provider LLM Cascade
**Extraction attempts in order:**
```
1. OpenAI (gpt-4o-mini) [$0.15/$0.60 per 1M tokens]
↓ (if API key missing or fails)
2. Anthropic Claude (haiku/sonnet) [$0.25/$1.25 per 1M tokens]
↓ (if API key missing or fails)
3. Pattern-Based [Free, 60% accuracy]
↓ (future)
4. Ollama (local LLM) [Zero cost, privacy-first]
```
**Rationale:**
- **OpenAI primary:** Cheapest, fastest, structured outputs
- **Claude fallback:** Same provider as main app, high quality
- **Pattern-based:** Always available, no dependencies
- **Ollama future:** Privacy-first, zero cost option
**Monthly cost estimate (5,000 extractions):** ~$5-6/month
### 2. Smart Threshold Auto-Extraction
**Filesystem monitoring triggers extraction only when:**
- ✅ `relevance_score > 0.7` (significant files)
- ✅ Active development (modified in last 5 minutes)
- ✅ Critical file patterns (auth, database, config, API, security)
**Rationale:**
- Balances automation with API cost control
- Focuses on meaningful changes
- Avoids noise from temporary/build files
### 3. Graceful Degradation
**All LLM features are optional:**
- ✅ System works without any API keys
- ✅ Pattern-based extraction always available
- ✅ Feature flags for gradual rollout
- ✅ User chooses extraction method via config
______________________________________________________________________
## Phase 0: Preparation (Week 0)
**Status:** 🟢 Complete
**Duration:** 1 week
**Blocking:** None
### Tasks
- [ ] **0.1** Set up feature flag system
- Create `config.feature_flags` module
- Add toggle mechanism for each feature
- Default all flags to `False` (safe rollout)
- [ ] **0.2** Design database migration strategy
- Document v1 → v2 schema changes
- Plan parallel schema operation (v1 + v2 simultaneously)
- Create rollback plan
- [ ] **0.3** Create test plan and fixtures
- Define test coverage requirements (>85%)
- Create test data for entity extraction
- Set up performance benchmarks
- [ ] **0.4** Secure API keys (optional)
- Document API key setup instructions
- Create `.env.example` template
- Test API key validation
### Deliverables
- `session_buddy/config/feature_flags.py`
- `docs/MIGRATION_STRATEGY.md`
- `docs/API_KEY_SETUP.md`
- Test plan document
______________________________________________________________________
## Phase 1: Enhanced Memory Schema (Week 1)
**Status:** 🟢 Complete
**Duration:** 1 week
**Depends on:** Phase 0
### Tasks
- [ ] **1.1** Create schema_v2.py with enhanced DuckDB tables
- `conversations_v2` with Memori categorization
- `memory_entities` for entity storage
- `memory_relationships` for entity connections
- `memory_promotions` for tier tracking
- `memory_access_log` for Conscious Agent
- [ ] **1.2** Implement migration script (v1 → v2)
- Best-effort categorization of existing data
- Preserve existing ONNX embeddings
- Gradual migration with rollback capability
- [ ] **1.3** Add database version tracking
- Schema version table
- Migration history log
- Health check for schema compatibility
- [ ] **1.4** Create indexes for performance
- Category + namespace indexes
- Memory tier + importance indexes
- Timestamp + access pattern indexes
- [ ] **1.5** Unit tests for schema
- Table creation tests
- Migration tests (v1 → v2)
- Index performance tests
- Rollback tests
### Schema Changes
**New Tables:**
```sql
-- Enhanced conversations with Memori categorization
CREATE TABLE conversations_v2 (
id TEXT PRIMARY KEY,
content TEXT NOT NULL,
embedding FLOAT[384], -- Keep ONNX vectors
-- NEW: Memori categorization
category TEXT NOT NULL, -- facts, preferences, skills, rules, context
subcategory TEXT,
importance_score REAL DEFAULT 0.5,
-- NEW: Memory tier management
memory_tier TEXT DEFAULT 'long_term',
access_count INTEGER DEFAULT 0,
last_accessed TIMESTAMP,
-- NEW: Extraction metadata
extracted_by TEXT, -- openai, claude, pattern, ollama
extraction_confidence REAL,
-- Existing fields
project TEXT,
namespace TEXT DEFAULT 'default',
timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
searchable_content TEXT,
reasoning TEXT
);
CREATE TABLE memory_entities (
id TEXT PRIMARY KEY,
memory_id TEXT NOT NULL,
entity_type TEXT NOT NULL,
entity_value TEXT NOT NULL,
confidence REAL DEFAULT 1.0,
extracted_by TEXT,
FOREIGN KEY (memory_id) REFERENCES conversations_v2(id) ON DELETE CASCADE
);
CREATE TABLE memory_relationships (
id TEXT PRIMARY KEY,
from_entity_id TEXT NOT NULL,
to_entity_id TEXT NOT NULL,
relationship_type TEXT NOT NULL,
strength REAL DEFAULT 1.0,
FOREIGN KEY (from_entity_id) REFERENCES memory_entities(id) ON DELETE CASCADE,
FOREIGN KEY (to_entity_id) REFERENCES memory_entities(id) ON DELETE CASCADE
);
CREATE TABLE memory_promotions (
id TEXT PRIMARY KEY,
memory_id TEXT NOT NULL,
from_tier TEXT NOT NULL,
to_tier TEXT NOT NULL,
reason TEXT,
priority_score REAL,
timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (memory_id) REFERENCES conversations_v2(id) ON DELETE CASCADE
);
CREATE TABLE memory_access_log (
id TEXT PRIMARY KEY,
memory_id TEXT NOT NULL,
access_type TEXT,
timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (memory_id) REFERENCES conversations_v2(id) ON DELETE CASCADE
);
```
### Migration Strategy
**Parallel Schema Operation:**
```python
# Run v1 and v2 schemas simultaneously during transition
# Feature flag controls which schema is used
if config.use_schema_v2:
# Use new schema with enhanced features
db = ReflectionDatabaseV2()
else:
# Use legacy schema (backward compatible)
db = ReflectionDatabase()
# Migration runs in background
# Once complete, flip feature flag
```
### Deliverables
- `session_buddy/memory/schema_v2.py`
- `session_buddy/memory/migration.py`
- `tests/unit/test_schema_v2.py`
- `tests/unit/test_migration.py`
______________________________________________________________________
## Phase 2: Multi-Provider Entity Extraction (Week 2)
**Status:** 🟢 Complete
**Duration:** 1 week
**Depends on:** Phase 1
### Tasks
- [ ] **2.1** Create entity extraction engine with cascade logic
- `EntityExtractionEngine` class
- Provider cascade with fallback
- Timeout and retry handling
- [ ] **2.2** Implement OpenAI provider (primary)
- Structured outputs with JSON schema
- ProcessedMemory Pydantic model
- Error handling and logging
- [ ] **2.3** Implement Anthropic provider (fallback)
- Tool use for structured extraction
- Same ProcessedMemory schema
- Failover logic
- [ ] **2.4** Implement pattern-based extractor (final fallback)
- Current regex/keyword approach
- Always available (no dependencies)
- Confidence scoring
- [ ] **2.5** Add Ollama provider stub (future)
- Interface defined, not implemented
- Configuration placeholder
- Documentation for future addition
- [ ] **2.6** Unit tests for all providers
- Test each provider independently
- Test cascade logic (OpenAI → Claude → Pattern)
- Test failover scenarios
- Mock API responses
### Multi-Provider Architecture
```python
class EntityExtractionEngine:
"""Multi-provider extraction with cascade fallback."""
def __init__(self, config: Config):
self.providers = self._init_providers(config)
self.fallback_extractor = PatternBasedExtractor()
async def extract_entities(
self, user_input: str, ai_output: str
) -> EntityExtractionResult:
"""Try each provider in order until one succeeds."""
for provider in self.providers:
try:
return await provider.extract_entities(user_input, ai_output)
except Exception as e:
logger.warning(f"{provider.name} failed: {e}")
continue
# Final fallback
return await self.fallback_extractor.extract_entities(user_input, ai_output)
```
### Provider Implementations
**1. OpenAI Provider:**
```python
class OpenAIProvider(LLMProvider):
async def extract_entities(...) -> ProcessedMemory:
response = await self.client.chat.completions.create(
model="gpt-4o-mini",
response_format={
"type": "json_schema",
"json_schema": ProcessedMemory.model_json_schema()
}
)
return ProcessedMemory.model_validate_json(response)
```
**2. Anthropic Provider:**
```python
class AnthropicProvider(LLMProvider):
async def extract_entities(...) -> ProcessedMemory:
response = await self.client.messages.create(
model="claude-3-5-haiku-20241022",
tools=[extraction_tool]
)
return ProcessedMemory.model_validate(tool_result)
```
**3. Pattern-Based:**
```python
class PatternBasedExtractor:
async def extract_entities(...) -> ProcessedMemory:
# Current regex/keyword approach (60% accuracy)
entities = self._extract_with_patterns(text)
return ProcessedMemory(
category=self._categorize(text),
entities=entities,
extracted_by="pattern"
)
```
### Configuration
```python
class Config:
# API keys (all optional)
openai_api_key: str | None = None
anthropic_api_key: str | None = None
ollama_base_url: str | None = None # Future
# Feature flags
enable_llm_entity_extraction: bool = True
enable_ollama: bool = False
# Cascade settings
llm_extraction_timeout: int = 10
llm_extraction_retries: int = 1
```
### Deliverables
- `session_buddy/memory/entity_extractor.py`
- `session_buddy/memory/llm_providers.py`
- `session_buddy/memory/processed_memory.py` (Pydantic models)
- `tests/unit/test_entity_extraction.py`
- `tests/unit/test_providers.py`
______________________________________________________________________
## Phase 3: Conscious Agent (Week 3-4)
**Status:** 🟢 Complete
**Duration:** 2 weeks
**Depends on:** Phase 2
### Tasks
- [ ] **3.1** Implement Conscious Agent background loop
- 6-hour analysis cycle
- Async task management
- Start/stop controls
- [ ] **3.2** Create priority scoring algorithm
- Frequency score (40%)
- Recency score (30%)
- Semantic importance (20%)
- Category weight (10%)
- Threshold: 0.75 for promotion
- [ ] **3.3** Implement memory promotion logic
- Long-term → Short-term → Working tier flow
- Promotion tracking in memory_promotions table
- Reasoning/justification for each promotion
- [ ] **3.4** Implement memory demotion logic
- Stale memory detection (7+ days no access)
- Short-term → Long-term demotion
- Cleanup of working tier
- [ ] **3.5** Integration tests
- Test promotion/demotion workflows
- Test concurrent access (race conditions)
- Test scalability (10k+ memories)
- Test background loop reliability
### Conscious Agent Implementation
```python
class ConsciousAgent:
"""Background memory optimization (Memori pattern)."""
async def _analyze_and_optimize(self) -> dict:
# 1. Analyze access patterns
patterns = await self._analyze_access_patterns()
# 2. Calculate priority scores
candidates = await self._calculate_promotion_priorities(patterns)
# 3. Promote high-priority (score >= 0.75)
promoted = await self._promote_memories(candidates)
# 4. Demote stale (7+ days no access)
demoted = await self._demote_stale_memories()
return {"promoted_count": len(promoted), "demoted_count": len(demoted)}
```
### Priority Scoring Formula
```python
priority_score = (
frequency_score * 0.4 # Access count normalized
+ recency_score * 0.3 # Exponential decay
+ semantic_score * 0.2 # Importance from extraction
+ category_score * 0.1 # Category weight
)
# Category weights:
# preferences: 1.0 (highest)
# skills: 0.9
# rules: 0.8
# facts: 0.7
# context: 0.6
```
### Deliverables
- `session_buddy/memory/conscious_agent.py`
- `session_buddy/memory/priority_scoring.py`
- `tests/unit/test_conscious_agent.py`
- `tests/integration/test_memory_tiers.py`
______________________________________________________________________
## Phase 4: Filesystem Integration (Week 5)
**Status:** 🟢 Complete
**Duration:** 1 week
**Depends on:** Phase 3
### Tasks
- [ ] **4.1** Enhance ProjectActivityMonitor with extraction hooks
- Smart threshold filtering (relevance > 0.7)
- Critical file pattern detection
- Duplicate prevention (recently processed files)
- [ ] **4.2** Implement file context builder
- Extract file metadata (path, type, size)
- Read file snippet (first 50 lines)
- Determine project context
- [ ] **4.3** Create activity-based importance scoring
- File activity score influences memory importance
- Recent edits boost priority
- Frequent changes indicate significance
- [ ] **4.4** Integration tests
- Test file change → extraction workflow
- Test smart threshold filtering
- Test critical file detection
- Test activity scoring
### Smart Threshold Logic
```python
async def process_file_change(
self, activity_event: ActivityEvent
) -> EntityExtractionResult | None:
"""Extract entities only for significant files."""
# Check threshold
if activity_event.relevance_score < 0.7:
return None # Skip low-relevance files
# Check critical patterns
if not is_critical_file(activity_event.details["file_path"]):
if activity_event.relevance_score < 0.9:
return None # Non-critical files need higher threshold
# Extract entities
return await self.entity_extractor.extract_entities(...)
```
### Critical File Patterns
```python
CRITICAL_FILE_PATTERNS = {
"auth": ["auth", "login", "session", "jwt", "oauth"],
"database": ["db", "database", "migration", "schema"],
"config": ["config", "settings", "env"],
"api": ["api", "endpoint", "route", "controller"],
"security": ["security", "encrypt", "hash", "crypto"],
}
```
### Deliverables
- `session_buddy/app_monitor.py` (enhanced)
- `session_buddy/memory/file_context.py`
- `tests/integration/test_filesystem_extraction.py`
______________________________________________________________________
## Phase 5: Testing & Rollout (Week 6)
**Status:** 🟢 Complete
**Duration:** 1 week
**Depends on:** Phase 4
### Tasks
- [ ] **5.1** Comprehensive test suite
- Unit tests (target: >85% coverage)
- Integration tests (end-to-end workflows)
- Performance tests (latency, scalability)
- [ ] **5.2** Feature flag deployment
- Defaults now ON; use env to disable for staged rollout
- Staged enablement plan
- Rollback procedures
- [ ] **5.3** Gradual rollout (10% → 50% → 100%)
- Day 1-2: Enable schema_v2 (parallel with v1)
- Day 3-4: Enable LLM extraction (10% of conversations)
- Day 5-6: Enable Conscious Agent
- Day 7: Enable filesystem extraction
- [ ] **5.4** Monitoring and metrics
- Track extraction accuracy
- Monitor API costs
- Measure latency
- Collect user feedback
- [ ] **5.5** Documentation updates
- User guide (API key setup, configuration)
- Developer guide (architecture, testing)
- Migration guide (v1 → v2)
- Changelog
- [ ] **5.6** Full production deployment
- Enable all features for 100% of users
- Announce new capabilities
- Monitor for issues
### Test Coverage Requirements
**Unit Tests (>85% coverage):**
- ✅ Multi-provider cascade
- ✅ Entity extraction (all providers)
- ✅ Conscious Agent logic
- ✅ Priority scoring
- ✅ Schema migration
- ✅ Smart threshold filtering
**Integration Tests:**
- ✅ End-to-end extraction workflow
- ✅ Filesystem → extraction → storage
- ✅ Memory tier promotion/demotion
- ✅ Provider failover scenarios
- ✅ Concurrent access handling
**Performance Tests:**
- ✅ Extraction latency (\<500ms per provider)
- ✅ Cascade fallback speed
- ✅ Conscious Agent scalability (10k+ memories)
- ✅ Database query performance
### Rollout Schedule
| Day | Action | Percentage | Risk |
|-----|--------|------------|------|
| 1-2 | Enable schema_v2 | 100% (parallel) | Low |
| 3-4 | Enable LLM extraction | 10% | Medium |
| 5-6 | Enable Conscious Agent | 50% | Low |
| 7 | Enable filesystem extraction | 100% | Low |
### Success Metrics
| Metric | Baseline | Target |
|--------|----------|--------|
| **Entity Extraction Accuracy** | 60% | **85%+** |
| **Memory Categorization** | N/A | **90%+** |
| **Search Relevance** | 70% | **85%+** |
| **Retrieval Latency** | 50ms | **20ms** |
| **API Cost** | $0 | **$5-8/month** |
| **Provider Cascade Success** | N/A | **\<5% fallback to patterns** |
### Deliverables
- Complete test suite with >85% coverage
- Rollout documentation
- User guides and API documentation
- Monitoring dashboard
- Post-deployment report
______________________________________________________________________
## Migration Strategy (Existing Projects)
### Auto-Detection & Auto-Migration
**The system will automatically:**
1. **Detect schema version on startup**
1. **Run migration if needed**
1. **Preserve all existing data**
1. **Enable new features gradually**
### Migration Workflow
```mermaid
graph TD
A[Session Startup] --> B{Check Schema Version}
B -->|v1 detected| C[Run Auto-Migration]
B -->|v2 exists| D[Use Enhanced Features]
C --> E[Migrate Data to v2]
E --> F[Validate Migration]
F -->|Success| D
F -->|Failure| G[Rollback + Alert]
D --> H[Enable Feature Flags]
```
### Migration Implementation
**Automatic Detection:**
```python
async def check_and_migrate():
"""Auto-detect and migrate on startup."""
# 1. Check schema version
current_version = await get_schema_version()
if current_version == "v1":
logger.info("Detected v1 schema. Starting auto-migration...")
# 2. Create backup
await create_backup(f"backup_v1_{datetime.now()}.db")
# 3. Run migration
migration_result = await migrate_v1_to_v2()
if migration_result.success:
logger.info(f"Migration complete: {migration_result.stats}")
await update_schema_version("v2")
else:
logger.error(f"Migration failed: {migration_result.error}")
await rollback_to_backup()
raise MigrationError(migration_result.error)
elif current_version == "v2":
logger.info("Schema v2 detected. Using enhanced features.")
else:
logger.warning(f"Unknown schema version: {current_version}")
```
**Migration Script:**
```python
async def migrate_v1_to_v2() -> MigrationResult:
"""Migrate v1 data to v2 schema."""
stats = {"migrated": 0, "failed": 0, "skipped": 0}
# 1. Create v2 tables (if not exist)
await create_v2_schema()
# 2. Migrate conversations
async for conv in get_v1_conversations():
try:
# Best-effort categorization
category = categorize_pattern_based(conv.content)
await insert_conversations_v2(
id=conv.id,
content=conv.content,
embedding=conv.embedding, # Preserve ONNX vectors
category=category,
memory_tier="long_term",
extracted_by="pattern", # Legacy data
extraction_confidence=0.6,
**conv.metadata,
)
stats["migrated"] += 1
except Exception as e:
logger.error(f"Failed to migrate {conv.id}: {e}")
stats["failed"] += 1
# 3. Validate migration
v1_count = await count_v1_conversations()
v2_count = await count_v2_conversations()
if v2_count >= v1_count:
return MigrationResult(success=True, stats=stats)
else:
return MigrationResult(
success=False,
error=f"Missing data: v1={v1_count}, v2={v2_count}",
stats=stats,
)
```
### User-Facing Migration Tools
**1. Migration Status Command:**
```python
@mcp.tool()
async def migration_status() -> dict:
"""Check migration status and progress."""
return {
"current_version": await get_schema_version(),
"migration_needed": await needs_migration(),
"backup_exists": await has_backup(),
"migration_history": await get_migration_history(),
"data_stats": {
"v1_conversations": await count_v1_conversations(),
"v2_conversations": await count_v2_conversations(),
"entities": await count_entities(),
"relationships": await count_relationships(),
},
}
```
**2. Manual Migration Trigger:**
```python
@mcp.tool()
async def trigger_migration(create_backup: bool = True, dry_run: bool = False) -> dict:
"""Manually trigger migration (with preview)."""
if dry_run:
# Preview what would be migrated
preview = await preview_migration()
return {
"dry_run": True,
"would_migrate": preview.conversation_count,
"estimated_time": preview.estimated_seconds,
"warnings": preview.warnings,
}
# Run actual migration
if create_backup:
await create_backup()
result = await migrate_v1_to_v2()
return {
"success": result.success,
"migrated": result.stats["migrated"],
"failed": result.stats["failed"],
"duration_seconds": result.duration,
}
```
**3. Rollback Tool:**
```python
@mcp.tool()
async def rollback_migration(backup_id: str) -> dict:
"""Rollback to previous backup if migration fails."""
# Restore from backup
await restore_backup(backup_id)
# Verify restoration
verification = await verify_database_integrity()
return {
"success": verification.success,
"restored_version": await get_schema_version(),
"conversation_count": await count_conversations(),
"issues": verification.issues,
}
```
### Migration Documentation
**User Guide Section:**
````markdown
## Migrating to Enhanced Memory Features
### Automatic Migration
session-buddy will **automatically detect and migrate** your existing
data when you upgrade to the new version. No manual intervention required.
**What happens during migration:**
1. ✅ Automatic backup created (safety first)
2. ✅ New schema tables created alongside existing ones
3. ✅ All conversations migrated with best-effort categorization
4. ✅ Existing ONNX embeddings preserved (no re-computation)
5. ✅ New features enabled gradually via feature flags
**Migration is safe:**
- Original data preserved (runs in parallel)
- Automatic rollback on failure
- No downtime required
### Manual Migration (Optional)
If you prefer to control the migration:
```bash
# Check if migration is needed
mcp migration-status
# Preview migration (dry run)
mcp trigger-migration --dry-run
# Run migration with backup
mcp trigger-migration --create-backup
# Rollback if needed
mcp rollback-migration --backup-id=<id>
````
### After Migration
Once migrated, you'll have access to:
- 🎯 **LLM-powered entity extraction** (85%+ accuracy)
- 🎯 **Conscious Agent** (background memory optimization)
- 🎯 **Memory categorization** (facts, preferences, skills, rules, context)
- 🎯 **Smart tiered search** (working → short-term → long-term)
- 🎯 **Filesystem integration** (automatic extraction from code changes)
````
### Zero-Downtime Migration
**The migration runs while the system is operational:**
```python
# During migration period:
# - v1 schema: READ-ONLY (queries work)
# - v2 schema: WRITE-ENABLED (new data)
# - Gradual cutover (no service interruption)
class DualSchemaDatabase:
"""Operates on both v1 and v2 during migration."""
async def store_conversation(self, content: str) -> str:
# Write to both schemas during transition
v1_id = await self.v1_db.store(content)
v2_id = await self.v2_db.store_enhanced(content)
return v2_id if config.use_schema_v2 else v1_id
async def search(self, query: str) -> list:
# Read from active schema
if config.use_schema_v2:
return await self.v2_db.search_tiered(query)
else:
return await self.v1_db.search(query)
````
______________________________________________________________________
## Cost Analysis
### Monthly API Costs (5,000 extractions)
| Provider | Usage | Cost per Extraction | Monthly Cost |
|----------|-------|---------------------|--------------|
| **OpenAI (90%)** | 4,500 | $0.001 | **$4.50** |
| **Claude (8%)** | 400 | $0.002 | **$0.80** |
| **Pattern (2%)** | 100 | $0 | **$0** |
| **Total** | 5,000 | - | **~$5-6/month** |
**Optional Ollama (Future):**
- Local LLM: $0/month (zero API costs)
- Requires: Local compute resources
- Trade-off: Potentially lower accuracy
______________________________________________________________________
## Risk Mitigation
| Risk | Impact | Probability | Mitigation |
|------|--------|-------------|------------|
| **LLM API downtime** | Medium | Low | Multi-provider cascade + pattern fallback |
| **High API costs** | Medium | Medium | Smart threshold + usage monitoring |
| **Migration failures** | High | Low | Parallel schemas + automatic rollback |
| **Performance degradation** | Medium | Low | Feature flags + gradual rollout |
| **Entity extraction errors** | Low | Medium | Confidence scoring + manual review |
| **User confusion** | Low | Low | Documentation + auto-migration |
______________________________________________________________________
## Dependencies
### Required (No Changes)
- `duckdb>=0.9` ✅ (existing)
- `pydantic>=2.0` ✅ (existing)
- `watchdog>=3.0` ✅ (existing)
### Optional (New - LLM Providers)
- `openai>=1.0.0` (if using OpenAI)
- `anthropic>=0.25.0` (if using Claude - likely already installed)
- `ollama` (future - local LLM)
**No breaking changes** - all new dependencies are optional.
______________________________________________________________________
## Documentation Updates
### To Be Created/Updated
1. **User Guide:**
- [ ] Multi-provider setup instructions
- [ ] API key configuration
- [ ] Cost estimation calculator
- [ ] Migration guide
- [ ] Troubleshooting
1. **Developer Guide:**
- [ ] Entity extraction architecture
- [ ] Conscious Agent internals
- [ ] Adding new LLM providers
- [ ] Testing strategies
1. **API Reference:**
- [ ] New MCP tools
- [ ] Configuration options
- [ ] Migration commands
______________________________________________________________________
## Post-Implementation
### Ongoing Monitoring
- **Weekly:** API cost reviews
- **Monthly:** Accuracy audits (sample 100 extractions)
- **Quarterly:** User feedback survey
- **Continuous:** Provider performance comparison
### Future Enhancements (Phase 6+)
- [ ] Ollama local LLM support
- [ ] Custom entity types (user-defined)
- [ ] User-defined extraction rules
- [ ] Advanced semantic search (embeddings + tiers + relationships)
- [ ] Entity visualization (knowledge graph UI)
- [ ] Cross-project entity linking
______________________________________________________________________
## Success Criteria
**Phase 1-5 Complete When:**
- ✅ All tasks marked complete (30 total tasks)
- ✅ Test coverage >85%
- ✅ Migration tested with real data
- ✅ Documentation published
- ✅ Feature flags enabled for 100% of users
- ✅ Success metrics achieved
**Metrics Must Show:**
- ✅ Entity extraction accuracy ≥85%
- ✅ Memory categorization accuracy ≥90%
- ✅ Search relevance improvement ≥15% over baseline
- ✅ Retrieval latency ≤20ms (tiered search)
- ✅ API costs ≤$10/month
- ✅ Provider cascade \<5% fallback to patterns
______________________________________________________________________
## Notes
**Status Legend:**
- 🔵 Not Started
- 🟡 In Progress
- 🟢 Complete
- 🔴 Blocked
**Update This Document:**
- Mark tasks complete as you finish them
- Update progress percentages
- Add notes/learnings in each phase
- Track blockers and resolutions
______________________________________________________________________
**Last Updated:** January 19, 2025
**Next Review:** Start of implementation
