Dev Memory MCP Server

# Dev Memory System - Implementation Handoff ## Current Status We are attempting to implement a file-based memory system for storing development context, but encountering persistent issues with file storage and JSON response handling. ### Core Components Implemented 1. **Base Types** (`src/core/types.ts`) - BaseMemory interface for core memory structure - MemoryType enum for different context types - EnhancedMemory interface for extended functionality - DevContext interface for development-specific context 2. **File Store** (`src/core/store.ts`) - Implementation of Store interface - File-based storage with JSON serialization - Similarity matching for context retrieval 3. **Context Manager** (`src/core/context.ts`) - Management of development contexts - Context persistence and retrieval - Code context tracking 4. **MCP Server** (`src/index.ts`) - Implementation of MCP protocol - Tools for storing and retrieving context - Memory optimization capabilities ### Current Issues 1. **File Storage Problems** - Files not being written to .dev-memory/interactions/ - Possible permission issues with directory creation - Attempted fixes: * Using explicit permissions (chmod 777) * Using synchronous file operations * Simplified file paths * Added error logging 2. **JSON Response Issues** - Bad control character errors in JSON responses - Error occurs at position 193 (line 3 column 167) - Attempted fixes: * Base64 encoding content * Sanitizing special characters * Simplified response format * Plain text responses 3. **Directory Structure** - Inconsistent directory creation - Permission issues with nested directories - Attempted fixes: * Using recursive directory creation * Explicit permission setting * Simplified directory structure ### Implementation Details 1. **File Storage Strategy** ```typescript async store(memory: T): Promise<void> { // Create base directory if needed if (!fs.existsSync(this.baseDir)) { fs.mkdirSync(this.baseDir, { recursive: true }); } // Simple filename format const filename = `memory-${Date.now()}.json`; const filePath = path.join(this.baseDir, filename); // Write data const data = JSON.stringify({ id: memory.id, content: memory.content, metadata: memory.metadata }, null, 2); fs.writeFileSync(filePath, data, 'utf8'); } ``` 2. **Context Retrieval** ```typescript async findSimilar(content: string, limit: number = 5): Promise<T[]> { const files = await fs.promises.readdir(this.baseDir); const memories: T[] = []; for (const file of files) { try { const content = await fs.promises.readFile( path.join(this.baseDir, file), 'utf-8' ); const parsed = JSON.parse(content); memories.push(parsed); } catch (error) { console.error(`Error reading file ${file}:`, error); } } return memories .sort((a, b) => this.similarity(content, a.content) - this.similarity(content, b.content)) .slice(0, limit); } ``` ### Error Logs 1. **File Write Errors** ``` Error executing MCP tool: {"code":-32603,"name":"Error","message":"MCP error -32603: Bad control character in string literal in JSON at position 193 (line 3 column 167)"} ``` 2. **Directory Structure** ``` Storage directory: /Users/chetpaslawski/Documents/VS Code Projects/dev-memory-server/.dev-memory Interactions directory: .dev-memory/interactions/ ``` ### Next Steps 1. **File Storage** - [ ] Implement robust error handling for file operations - [ ] Add file locking mechanism for concurrent writes - [ ] Implement file rotation/cleanup strategy - [ ] Add checksums for file integrity 2. **JSON Handling** - [ ] Implement strict JSON sanitization - [ ] Add JSON schema validation - [ ] Implement proper error handling for JSON parsing - [ ] Add compression for large contexts 3. **Testing** - [ ] Add unit tests for file operations - [ ] Add integration tests for context storage - [ ] Add stress tests for concurrent operations - [ ] Add validation tests for JSON handling ### References 1. **Documentation** - [DESIGN.md](./DESIGN.md) - System architecture and design decisions - [WORKING_NOTES.md](./WORKING_NOTES.md) - Implementation details and notes - [README.md](../README.md) - Project overview and setup 2. **Source Files** - [src/core/store.ts](../src/core/store.ts) - File storage implementation - [src/core/types.ts](../src/core/types.ts) - Type definitions - [src/index.ts](../src/index.ts) - MCP server implementation 3. **Scripts** - [scripts/setup.sh](../scripts/setup.sh) - System setup and initialization ### Environment Details - Operating System: macOS - Node.js Version: Latest LTS - Project Location: /Users/chetpaslawski/Documents/VS Code Projects/dev-memory-server - Storage Location: .dev-memory/ ### Required Permissions 1. **Directories** ```bash .dev-memory/ ├── interactions/ (777) ├── metadata/ (755) └── logs/ (755) ``` 2. **Files** ```bash .dev-memory/ ├── interactions/*.json (644) ├── metadata/*.json (644) └── logs/*.log (644) ``` ### Dependencies ```json { "dependencies": {}, "devDependencies": { "@types/node": "^20.0.0", "typescript": "^5.0.0", "ts-node": "^10.9.0" } } ``` ### Known Workarounds 1. **File Storage** - Using synchronous file operations for better error handling - Simplified directory structure to minimize permission issues - Direct file writes instead of temporary files 2. **JSON Handling** - Sanitizing content before JSON serialization - Using plain text responses for error cases - Simplified response structure ### Critical Considerations 1. **File System** - Ensure proper permissions are set - Handle concurrent file access - Implement proper cleanup 2. **Error Handling** - Improve JSON validation - Add better error reporting - Implement retry mechanisms 3. **Performance** - Implement caching - Add file indexing - Optimize similarity matching ### Executive Summary #### Key Issues 1. **File System Access** - Root cause appears to be Node.js process permissions when running as MCP server - File writes fail silently despite directory permissions being set - Possible solution: Use a different storage strategy (SQLite, LevelDB) instead of raw files 2. **JSON Response Handling** - MCP protocol seems sensitive to certain JSON structures - Control characters in responses causing protocol errors - Possible solution: Implement a custom serialization format or use base64 encoding consistently 3. **Directory Management** - Inconsistent directory creation between Node.js process and shell commands - Permission inheritance issues with nested directories - Possible solution: Move to a flat storage structure or use a database #### Recommended Path Forward 1. **Short Term** - Switch to SQLite for storage (removes file system complexity) - Implement strict JSON sanitization - Add comprehensive logging 2. **Medium Term** - Add proper error recovery mechanisms - Implement data migration tools - Add monitoring and health checks 3. **Long Term** - Consider moving to a proper database - Implement proper backup strategies - Add clustering support ### MCP Protocol Considerations #### Protocol Limitations 1. **Response Format** - MCP responses must be valid JSON without control characters - Responses seem to have a size limit or structure requirement - Error handling is limited to basic error codes 2. **Communication Issues** - Protocol is sensitive to JSON structure - No built-in support for binary data - Limited feedback on communication errors 3. **Server Constraints** - Server runs in a restricted environment - File system access is limited - No interactive capabilities (can't prompt user) #### Protocol Workarounds 1. **Data Encoding** - Base64 encode binary data - Sanitize all text content - Use simple response structures 2. **Error Handling** - Implement custom error codes - Add detailed error logging - Use fallback response formats 3. **File System Access** - Use database instead of files - Implement retry mechanisms - Add robust error recovery ### Planned Architecture #### Package Structure ``` @dev-memory/core/ # Core functionality ├── src/ │ ├── store/ # Storage implementations │ ├── context/ # Context management │ ├── optimization/ # Memory optimization │ ├── monitoring/ # Health metrics │ └── types/ # Shared types @dev-memory/react/ # React integration @dev-memory/node/ # Node.js integration @dev-memory/git/ # Git hooks and utilities @dev-memory/cli/ # CLI tools ``` #### Integration Plans 1. **React Integration** - Memory hooks for React components - Context providers and consumers - UI components for memory visualization 2. **Node.js Integration** - Server-side memory management - API endpoints for memory access - Background processing utilities 3. **Git Integration** - Automatic context capture from commits - Branch and merge tracking - Development history analysis 4. **CLI Tools** - Memory management commands - System health monitoring - Backup and restore utilities ### Development Timeline #### Phase 1: Robustness (January 2025) - [ ] Automated server deployment - [ ] Cross-platform testing - [ ] Error recovery improvements - [ ] Current Status: Blocked on file system issues #### Phase 2: Intelligence (February 2025) - [ ] Context relevance scoring - [ ] Smart context pruning - [ ] Pattern recognition - [ ] Current Status: Not started #### Phase 3: Integration (March 2025) - [ ] CI/CD pipeline integration - [ ] Team collaboration features - [ ] External tool connectors - [ ] Current Status: Not started ### Contact Information For questions or clarification: - Project Repository: dev-memory-server - Location: /Users/chetpaslawski/Documents/VS Code Projects/dev-memory-server - Status: Blocked on file system issues - Priority: High - blocking development context capture - Next Steps: Consider switching to SQLite storage - Architecture Lead: Pending assignment - Timeline: Phase 1 (January 2025)