MCP Memory Service

# Dream-Inspired Memory Consolidation System ## Overview This document describes an innovative approach to memory consolidation inspired by human cognitive processes during sleep. Originally conceptualized on June 7, 2025, and contributed to [Issue #11](https://github.com/doobidoo/mcp-memory-service/issues/11) on July 28, 2025, this system enhances the proposed multi-layered consolidation architecture with biologically-inspired mechanisms. > **📚 Implementation Guide**: For a complete guide on implementing this system autonomously without external AI, see the [Autonomous Memory Consolidation Guide](./autonomous-memory-consolidation.md). ## Background Traditional memory systems face challenges with: - Unbounded growth leading to performance degradation - Difficulty surfacing truly important information - Lack of creative discovery mechanisms - No natural "forgetting" process The dream-inspired approach addresses these by mimicking how human brains consolidate memories during sleep cycles. ## Core Concepts ### 1. Exponential Decay Scoring Memories naturally lose relevance over time unless reinforced by connections or access patterns. ```python def calculate_memory_relevance(memory, current_time): """ Calculate memory relevance using exponential decay. Factors: - Age of memory - Base importance score - Retention period (varies by memory type) """ age = current_time - memory.created_at base_score = memory.importance_score # Different memory types have different decay rates retention_periods = { 'critical': 365, # Critical memories decay slowly 'reference': 180, # Reference material moderately 'temporary': 7, # Temporary notes decay quickly 'default': 30 # Standard memories } retention_period = retention_periods.get(memory.memory_type, 30) decay_factor = math.exp(-age.days / retention_period) # Connections boost relevance connection_boost = 1 + (0.1 * len(memory.connections)) return base_score * decay_factor * connection_boost ``` ### 2. Creative Association System During consolidation, the system randomly pairs memories to discover non-obvious connections, similar to how dreams create unexpected associations. ```python async def creative_association_phase(memories): """ Discover creative connections between seemingly unrelated memories. The "sweet spot" for interesting discoveries is moderate similarity (0.3-0.7 range) - not too similar, not completely unrelated. """ # Sample random pairs (limit to prevent combinatorial explosion) max_pairs = min(100, len(memories) * (len(memories) - 1) // 2) pairs = random.sample( list(combinations(memories, 2)), k=min(max_pairs, len(combinations(memories, 2))) ) associations = [] for mem1, mem2 in pairs: similarity = calculate_semantic_similarity(mem1, mem2) # Sweet spot for creative connections if 0.3 < similarity < 0.7: # Analyze why they might be connected connection_reason = analyze_connection(mem1, mem2) # Create association memory association = await create_association_memory( source_memories=[mem1.hash, mem2.hash], similarity_score=similarity, connection_type=connection_reason, metadata={ "discovery_method": "creative_association", "discovery_date": datetime.now(), "tags": ["association", "discovery"] } ) associations.append(association) return associations ``` ### 3. Controlled Forgetting (Memory Pruning) Not all memories need to be retained forever. The system implements intelligent forgetting to maintain efficiency. ```python async def memory_pruning_phase(time_horizon): """ Implement controlled forgetting for memory health. Rather than deleting, we compress and archive low-value memories. """ # Get candidates for forgetting candidates = await get_pruning_candidates(time_horizon) pruned_memories = [] for memory in candidates: if should_forget(memory): # Don't delete - compress and archive compressed = await compress_memory(memory) # Create summary if part of a pattern if memory.cluster_id: await update_cluster_summary(memory.cluster_id, compressed) # Archive the original await archive_memory(memory, compressed) pruned_memories.append(memory.hash) return pruned_memories def should_forget(memory): """ Determine if a memory should be forgotten. Factors: - Relevance score below threshold - No recent access - No important connections - Not tagged as important """ if memory.tags.intersection({'important', 'critical', 'reference'}): return False if memory.relevance_score < 0.1: if not memory.connections: if (datetime.now() - memory.last_accessed).days > 90: return True return False ``` ### 4. Semantic Compression Create condensed representations of memory clusters for efficient long-term storage. ```python async def semantic_compression(memory_cluster): """ Compress a cluster of related memories into a semantic summary. This creates a higher-level abstraction while preserving key information. """ # Extract key concepts using NLP concepts = extract_key_concepts(memory_cluster) # Identify recurring themes themes = identify_themes(concepts) # Create compressed representation compressed = { "summary": generate_thematic_summary(themes), "key_concepts": concepts[:10], # Top 10 concepts "temporal_range": { "start": min(m.created_at for m in memory_cluster), "end": max(m.created_at for m in memory_cluster) }, "source_count": len(memory_cluster), "aggregate_tags": aggregate_tags(memory_cluster), "embedding": generate_concept_embedding(concepts), "compression_ratio": calculate_compression_ratio(memory_cluster) } # Store as consolidated memory consolidated = await store_consolidated_memory( content=compressed["summary"], metadata={ **compressed, "type": "semantic_compression", "compression_date": datetime.now() } ) # Link source memories for memory in memory_cluster: await add_memory_link(memory.hash, consolidated.hash, "compressed_into") return consolidated ``` ## Autonomous Implementation **🚀 This entire system can run autonomously without external AI!** The key insight is that the MCP Memory Service already generates embeddings when memories are stored. These embeddings enable: - Mathematical similarity calculations (cosine similarity) - Clustering algorithms (DBSCAN, hierarchical clustering) - Statistical summarization (TF-IDF, centroid method) - Rule-based decision making For detailed implementation without AI dependencies, see the [Autonomous Memory Consolidation Guide](./autonomous-memory-consolidation.md). ## Integration with Time-Based Layers The dream-inspired mechanisms enhance each consolidation layer: ### Daily Processing (Light Touch) - Calculate initial relevance scores - Identify highly-connected memory clusters - Flag memories showing rapid decay ### Weekly Processing (Active Consolidation) - Run creative association discovery - Begin semantic compression of stable clusters - Apply light pruning to obvious temporary content ### Monthly Processing (Deep Integration) - Comprehensive relevance recalculation - Controlled forgetting with archival - Create month-level semantic summaries ### Quarterly Processing (Pattern Extraction) - Deep creative association analysis - Major semantic compression operations - Identify long-term knowledge structures ### Yearly Processing (Knowledge Crystallization) - Final compression of historical data - Archive rarely-accessed memories - Create year-level knowledge maps ## Implementation Architecture ```python class DreamInspiredConsolidator: """ Main consolidation engine with biologically-inspired processing. """ def __init__(self, storage, config): self.storage = storage self.config = config self.decay_calculator = ExponentialDecayCalculator(config) self.association_engine = CreativeAssociationEngine(storage) self.compression_engine = SemanticCompressionEngine() self.pruning_engine = ControlledForgettingEngine(storage) async def consolidate(self, time_horizon: str): """ Run full consolidation pipeline for given time horizon. """ # 1. Retrieve memories for processing memories = await self.storage.get_memories_for_horizon(time_horizon) # 2. Calculate/update relevance scores await self.update_relevance_scores(memories) # 3. Cluster by semantic similarity clusters = await self.cluster_memories(memories) # 4. Run creative associations (if appropriate for horizon) if time_horizon in ['weekly', 'monthly']: associations = await self.association_engine.discover(memories) await self.storage.store_associations(associations) # 5. Compress clusters for cluster in clusters: if len(cluster) >= self.config.min_cluster_size: await self.compression_engine.compress(cluster) # 6. Controlled forgetting (if appropriate) if time_horizon in ['monthly', 'quarterly', 'yearly']: await self.pruning_engine.prune(memories) # 7. Generate consolidation report return await self.generate_report(time_horizon, memories, clusters) ``` ## Configuration Options ```yaml dream_consolidation: # Decay settings decay: enabled: true retention_periods: critical: 365 reference: 180 standard: 30 temporary: 7 # Creative association settings associations: enabled: true min_similarity: 0.3 max_similarity: 0.7 max_pairs_per_run: 100 # Forgetting settings forgetting: enabled: true relevance_threshold: 0.1 access_threshold_days: 90 archive_location: "./memory_archive" # Compression settings compression: enabled: true min_cluster_size: 5 max_summary_length: 500 preserve_originals: true ``` ## Benefits 1. **Natural Scalability**: System automatically manages growth through decay and forgetting 2. **Serendipitous Discovery**: Creative associations reveal unexpected insights 3. **Cognitive Realism**: Mirrors human memory, making it more intuitive 4. **Performance Optimization**: Compression and archival maintain speed 5. **Adaptive Importance**: Memory relevance evolves based on usage and connections ## Safety Considerations - Never auto-delete memories marked as 'critical' or 'important' - Always compress before archiving - Maintain audit trail of all consolidation operations - Allow users to recover archived memories - Provide options to disable forgetting for specific memory types ## Future Enhancements 1. **Sleep Cycle Simulation**: Run different consolidation phases at different "depths" 2. **Dream Journal**: Track and analyze creative associations over time 3. **Memory Replay**: Periodically resurface old memories for potential new connections 4. **Adaptive Decay Rates**: Learn optimal retention periods from user behavior 5. **Emotional Tagging**: Consider emotional context in consolidation decisions ## Related Resources - 📋 [Issue #11: Multi-Layered Memory Consolidation System](https://github.com/doobidoo/mcp-memory-service/issues/11) - 🤖 [Autonomous Memory Consolidation Guide](./autonomous-memory-consolidation.md) - Complete implementation without external AI - 📚 [MCP Memory Service Documentation](https://github.com/doobidoo/mcp-memory-service) ## Conclusion The dream-inspired memory consolidation system brings biological realism to digital memory management. By implementing natural processes like decay, creative association, and controlled forgetting, we create a system that not only scales efficiently but also surfaces truly meaningful information while discovering unexpected connections. This approach transforms memory management from a storage problem into a living, breathing system that evolves with its user's needs. --- *Original concept: June 7, 2025* *Contributed to Issue #11: July 28, 2025* *Documentation created: July 28, 2025*