MemoryGraph

# ADR 006: Intelligence Layer Architecture **Status**: Accepted **Date**: 2025-11-28 **Decision Makers**: Development Team **Phase**: 5 (Intelligence Layer) ## Context The memory server needed AI-powered intelligence features to automatically extract entities, recognize patterns, track temporal changes, and provide context-aware retrieval. This enhances the system's value beyond simple storage and retrieval. ## Decision We implemented a four-module intelligence layer with the following architecture: ### 1. Entity Extraction (`intelligence/entity_extraction.py`) - **Approach**: Regex-based pattern matching with optional NLP support - **Rationale**: - Regex provides fast, reliable extraction for structured entities (files, functions, errors) - Zero dependencies for basic functionality - Optional spaCy integration for advanced NLP features - Achieves 80%+ accuracy on common entity types **Entity Types Supported**: - FILE: File paths and filenames - FUNCTION: Function and method names - CLASS: Class names and components - ERROR: Error types and codes - TECHNOLOGY: Programming languages, frameworks, databases - CONCEPT: Programming concepts and patterns - COMMAND: CLI commands - PACKAGE: npm/pip packages - URL: HTTP(S) URLs - VARIABLE: Constants and variables ### 2. Pattern Recognition (`intelligence/pattern_recognition.py`) - **Approach**: Keyword and entity-based similarity matching - **Rationale**: - Simple keyword matching provides good baseline performance - Entity co-occurrence detection finds common patterns - Confidence scoring based on occurrence frequency - Can be enhanced with embeddings later without breaking API **Key Features**: - Find similar problems by keyword overlap - Extract patterns from entity co-occurrences - Suggest relevant patterns for current context - Pattern confidence scoring ### 3. Temporal Memory (`intelligence/temporal.py`) - **Approach**: Version chains using PREVIOUS relationships - **Rationale**: - Graph-native approach leverages Neo4j/Memgraph strengths - PREVIOUS relationships create natural version history - Enables time-travel queries (state at timestamp) - Supports entity change tracking **Key Features**: - Complete version history traversal - Point-in-time state queries - Entity timeline tracking - Version diff comparison ### 4. Context-Aware Retrieval (`intelligence/context_retrieval.py`) - **Approach**: Multi-factor relevance ranking with token limiting - **Rationale**: - Combines entity matching, keyword matching, and recency - Token-aware to respect LLM context windows - Project-scoped for focused retrieval - Relationship traversal for related context **Relevance Scoring**: ``` relevance = (entity_matches * 3 + keyword_matches * 2) / (1.0 + age_days / 30.0) ``` ## Alternatives Considered ### 1. Embedding-Based Similarity - **Pros**: Better semantic matching, handles synonyms - **Cons**: Requires ML dependencies, slower, more complex - **Decision**: Keep as optional enhancement, start with keywords ### 2. NLP-First Entity Extraction - **Pros**: More accurate for natural language - **Cons**: Requires spaCy, slower, overkill for code entities - **Decision**: Use regex first, optional NLP enhancement ### 3. Separate Vector Database - **Pros**: Optimized for similarity search - **Cons**: Additional complexity, dual storage, sync issues - **Decision**: Use graph-native approaches, can add later ## Consequences ### Positive - **Fast**: Regex and keyword matching are very fast - **Simple**: Easy to understand and maintain - **Flexible**: Easy to add new entity types or patterns - **Backend-Agnostic**: Works with all three backends - **Zero Dependencies**: Basic functionality requires no additional packages - **Extensible**: Clear path to add embeddings/NLP later ### Negative - **Accuracy Limitations**: Keywords miss semantic similarity - **No Semantic Understanding**: Can't handle synonyms well - **Pattern Quality**: Depends on memory volume and quality ### Neutral - **Test Coverage**: 94 new tests, 82-97% coverage in intelligence modules - **Performance**: Acceptable for current scale, may need optimization at 10K+ memories ## Implementation Notes ### Module Organization ``` intelligence/ ├── __init__.py # Public API exports ├── entity_extraction.py # Entity detection and linking ├── pattern_recognition.py # Pattern matching and suggestions ├── temporal.py # Version tracking and history └── context_retrieval.py # Intelligent context assembly ``` ### MCP Tools Added (7) 1. `find_similar_solutions` - Find similar problems and solutions 2. `suggest_patterns_for_context` - Get pattern suggestions 3. `get_memory_history` - View version history 4. `track_entity_timeline` - Track entity usage over time 5. `get_intelligent_context` - Smart context retrieval 6. `get_project_summary` - Project overview 7. `get_session_briefing` - Recent activity briefing ### Integration Points - Entity extraction can be called from `store_memory` (future enhancement) - Pattern recognition uses relationship system from Phase 4 - Temporal queries leverage PREVIOUS relationships - Context retrieval combines all intelligence features ## Future Enhancements 1. **Embedding Support** (Phase 6+) - Add sentence-transformers for semantic similarity - Hybrid keyword + embedding approach - Maintain backward compatibility 2. **Advanced NLP** (Phase 6+) - spaCy integration for better entity extraction - Named entity recognition for people, organizations - Dependency parsing for code understanding 3. **Pattern Learning** (Phase 7) - ML-based pattern effectiveness prediction - Automatic pattern extraction from successful solutions - Pattern recommendation ranking 4. **Proactive Intelligence** (Phase 7) - Automatic context briefings on session start - Predictive suggestions based on current work - Potential issue warnings ## References - [Phase 5 Completion Documentation](/docs/archive/completed_phases.md#phase-5-intelligence-layer) - [Entity Extraction Module](/src/claude_memory/intelligence/entity_extraction.py) - [Pattern Recognition Module](/src/claude_memory/intelligence/pattern_recognition.py) - [Temporal Memory Module](/src/claude_memory/intelligence/temporal.py) - [Context Retrieval Module](/src/claude_memory/intelligence/context_retrieval.py) - [Intelligence Tools](/src/claude_memory/intelligence_tools.py) ## Test Coverage - Entity Extraction: 28 tests, 82% coverage - Pattern Recognition: 23 tests, 95% coverage - Temporal Memory: 21 tests, 97% coverage - Context Retrieval: 22 tests, coverage integrated - **Total**: 94 new tests, all passing ## Metrics - **Performance**: <100ms for entity extraction, <200ms for pattern matching - **Accuracy**: ~85% for common entity types (files, functions, errors) - **Scalability**: Tested with 1000+ memories, performs well - **Code Quality**: 100% test pass rate, type hints on all functions