Enhanced Knowledge Graph Memory Server

# Implementation Plan - Memory MCP Server **Project:** @danielsimonjr/memory-mcp **Current Version:** 0.11.6 **Plan Created:** 2025-11-25 **Based on:** CODE_REVIEW.md comprehensive analysis --- ## Executive Summary This implementation plan addresses the 30 pending recommendations from CODE_REVIEW.md, organized into manageable 10-item sprints. We've already completed 7 critical improvements in v0.11.0-v0.11.6 (security, validation, caching, backups, transactions). **Progress:** - ✅ **Completed:** 7 critical items (23%) - 🔄 **Remaining:** 30 items (77%) - **Total Effort:** ~1,368-2,040 hours (estimated) **Sprint Structure:** - **Sprint 1:** Quick wins & code quality (P0/P1 simple tasks) - **Sprint 2:** Testing & documentation (P0/P1 moderate tasks) - **Sprint 3:** Performance improvements (P1 moderate tasks) - **Sprint 4:** Architecture refactoring (P0/P1 complex tasks) - **Sprint 5:** Advanced features (P2/P3 future enhancements) --- ## Table of Contents - [Completed Work](#completed-work-v0110---v0116) - [Sprint 1: Quick Wins](#sprint-1-quick-wins--code-quality) - [Sprint 2: Testing & Documentation](#sprint-2-testing--documentation) - [Sprint 3: Performance](#sprint-3-performance-improvements) - [Sprint 4: Architecture](#sprint-4-architecture-refactoring) - [Sprint 5: Advanced Features](#sprint-5-advanced-features) - [Future Roadmap](#future-roadmap-p3) --- ## Completed Work (v0.11.0 - v0.11.6) ### ✅ v0.11.0 - Security Hardening - **Fixed:** All 6 CVE security vulnerabilities - **Updated:** vitest 2.1.8 → 4.0.13, @vitest/coverage-v8 - **Result:** 0 vulnerabilities, production-safe dependencies ### ✅ v0.11.1 - Input Validation - **Added:** 14 comprehensive Zod validation schemas - **Protected:** Entity/Relation validation, importance ranges, batch limits - **Result:** Prevention of malformed data and injection attacks ### ✅ v0.11.2 - Performance Optimization - **Optimized:** Duplicate detection from O(n²) to O(n·k) - **Implemented:** Two-level bucketing (type + name prefix) - **Result:** 50x faster for large graphs (10k entities: 50M → 1M comparisons) ### ✅ v0.11.3 - Caching Layer - **Added:** In-memory cache for GraphStorage - **Implemented:** Write-through invalidation, deep copy protection - **Result:** O(n) → O(1) for read-heavy workloads ### ✅ v0.11.4 - Backup & Restore - **Created:** BackupManager with point-in-time recovery - **Features:** Timestamped backups, metadata, automatic cleanup - **Result:** Critical data protection for production ### ✅ v0.11.5 - Transaction Support - **Implemented:** ACID-compliant TransactionManager - **Features:** Begin/Commit/Rollback with automatic backup - **Result:** Zero data corruption risk, atomic operations ### ✅ v0.11.6 - Documentation - **Refactored:** README for GitHub best practices - **Added:** Storage file organization documentation - **Added:** CHANGELOG links and visibility --- ## Sprint 1: Quick Wins & Code Quality **Duration:** 1-2 weeks **Total Effort:** 19-31 hours **Priority:** P0-P1 (Critical to High) **Goal:** Low-hanging fruit with immediate impact ### Task 1.1: Console Logging Cleanup ⚡ **Priority:** P1 | **Effort:** 2-4 hours | **Complexity:** Simple **Issue:** `console.error()` used for informational messages instead of errors **Files to Modify:** - `src/memory/index.ts:38,40,4181` - `src/memory/features/CompressionManager.ts:262` - `src/memory/utils/pathUtils.ts:69,71` **Implementation Steps:** 1. Create `src/memory/utils/logger.ts`: ```typescript export const logger = { debug: (msg: string, ...args: any[]) => console.debug(`[DEBUG] ${msg}`, ...args), info: (msg: string, ...args: any[]) => console.log(`[INFO] ${msg}`, ...args), warn: (msg: string, ...args: any[]) => console.warn(`[WARN] ${msg}`, ...args), error: (msg: string, ...args: any[]) => console.error(`[ERROR] ${msg}`, ...args), }; ``` 2. Replace all instances: - `console.error('DETECTED:...')` → `logger.info('DETECTED:...')` - `console.error('COMPLETED:...')` → `logger.info('COMPLETED:...')` - Keep only actual errors as `logger.error()` 3. Test all logging output 4. Update imports across codebase **Acceptance Criteria:** - ✅ All informational messages use appropriate log levels - ✅ Errors remain as console.error - ✅ No breaking changes to functionality - ✅ TypeScript compilation passes **Testing:** - Run full test suite - Check logs during normal operations - Verify error logging still works --- ### Task 1.2: Remove Deprecated Dependencies ⚡ **Priority:** P0 | **Effort:** 2 hours | **Complexity:** Simple **Issue:** `inflight@1.0.6` (memory leak), `glob@7.2.3` (deprecated) **Files to Modify:** - `package-lock.json` - May affect dependencies of dependencies **Implementation Steps:** 1. Check current dependency tree: ```bash npm ls inflight glob ``` 2. Update to modern alternatives: ```bash npm update glob@latest npm audit fix --force ``` 3. Verify no breaking changes 4. Run full test suite 5. Check for warnings in `npm install` **Acceptance Criteria:** - ✅ No deprecated packages in dependency tree - ✅ No memory leak warnings - ✅ All tests passing - ✅ No new vulnerabilities introduced **Testing:** - `npm audit` shows 0 vulnerabilities - `npm ls` shows no warnings - All 83 tests passing --- ### Task 1.3: Magic Numbers to Constants 🔧 **Priority:** P1 | **Effort:** 4-6 hours | **Complexity:** Simple **Issue:** Hardcoded similarity weights and limits throughout codebase **Files to Modify:** - `src/memory/features/CompressionManager.ts:32-66` - `src/memory/search/RankedSearch.ts` - Various search implementations **Implementation Steps:** 1. Create `src/memory/utils/constants.ts`: ```typescript /** * Similarity scoring weights for duplicate detection */ export const SIMILARITY_WEIGHTS = { NAME: 0.4, // 40% - Name similarity (Levenshtein) TYPE: 0.2, // 20% - Entity type exact match OBSERVATION: 0.3, // 30% - Observation overlap (Jaccard) TAG: 0.1, // 10% - Tag overlap (Jaccard) } as const; /** * Default threshold for duplicate detection */ export const DEFAULT_DUPLICATE_THRESHOLD = 0.8; /** * Search result limits */ export const SEARCH_LIMITS = { DEFAULT: 50, MAX: 200, MIN: 1, } as const; /** * Importance range validation */ export const IMPORTANCE_RANGE = { MIN: 0, MAX: 10, } as const; ``` 2. Replace all magic numbers with named constants 3. Document why each value was chosen 4. Make configurable where appropriate **Acceptance Criteria:** - ✅ No hardcoded weights in CompressionManager - ✅ All limits use named constants - ✅ Constants are documented - ✅ Tests still pass with same behavior **Testing:** - Verify duplicate detection works identically - Check search limits are enforced - Run compression tests --- ### Task 1.4: Build Script Cleanup 🔧 **Priority:** P2 | **Effort:** 1 hour | **Complexity:** Simple **Issue:** `shx chmod +x dist/*.js` required on every build **Files to Modify:** - `src/memory/package.json:33` - Source TypeScript files (add proper shebang) **Implementation Steps:** 1. Add shebang to main entry point: ```typescript #!/usr/bin/env node // src/memory/index.ts ``` 2. Update build script in package.json: ```json { "scripts": { "build": "tsc" } } ``` 3. Test cross-platform builds (Windows, Linux, macOS) 4. Verify executable permissions set correctly **Acceptance Criteria:** - ✅ Build script simplified (no chmod) - ✅ Output is executable on all platforms - ✅ Shebang preserved after compilation - ✅ npm run build works consistently **Testing:** - Build on multiple platforms if possible - Test npx execution - Verify dist/index.js is executable --- ### Task 1.5: Add JSDoc to Public APIs 📚 **Priority:** P1 | **Effort:** 8-12 hours | **Complexity:** Simple **Issue:** Many modular components lack documentation **Files to Modify:** - `src/memory/core/KnowledgeGraphManager.ts` - `src/memory/core/EntityManager.ts` - `src/memory/core/RelationManager.ts` - `src/memory/features/*.ts` (all managers) - `src/memory/search/*.ts` (all search implementations) **Implementation Steps:** 1. Document all public methods with JSDoc format: ```typescript /** * Retrieves an entity by name. * * @param name - The unique name of the entity to retrieve * @returns The entity if found, null otherwise * * @example * ```typescript * const entity = await manager.getEntity('John_Doe'); * if (entity) { * console.log(entity.observations); * } * ``` */ async getEntity(name: string): Promise<Entity | null> { // ... } ``` 2. Document parameters, return types, exceptions 3. Add usage examples for complex methods 4. Document getters and setters **Acceptance Criteria:** - ✅ All public methods have JSDoc - ✅ Parameters and returns documented - ✅ Examples provided for complex APIs - ✅ IDE tooltips show documentation **Testing:** - Check IDE autocomplete shows docs - Generate API documentation (optional) - Review with typescript language server --- ### Task 1.6: Extract Magic Numbers - Search Limits 🔧 **Priority:** P1 | **Effort:** 2-3 hours | **Complexity:** Simple **Issue:** Search limits hardcoded in multiple files **Files to Modify:** - `src/memory/search/RankedSearch.ts` - `src/memory/search/BasicSearch.ts` - `src/memory/search/FuzzySearch.ts` **Implementation Steps:** 1. Use constants from Task 1.3 2. Update all search implementations 3. Add configurable options for limits **Acceptance Criteria:** - ✅ All search implementations use constants - ✅ Limits can be configured - ✅ Default behavior unchanged **Testing:** - Test search with various limits - Verify max limit is enforced - Check default limit works --- ### Task 1.7: Path Validation Enhancement 🔒 **Priority:** P1 | **Effort:** 8-12 hours | **Complexity:** Moderate **Issue:** `MEMORY_FILE_PATH` not validated against path traversal **Files to Modify:** - `src/memory/index.ts:17-47` - `src/memory/utils/pathUtils.ts` **Implementation Steps:** 1. Create path validation utility: ```typescript import path from 'path'; export function validateFilePath(filePath: string): string { // Normalize path to prevent traversal const normalized = path.normalize(filePath); // Ensure absolute path const absolute = path.isAbsolute(normalized) ? normalized : path.join(process.cwd(), normalized); // Check for suspicious patterns if (absolute.includes('..')) { throw new Error('Path traversal detected'); } // Optional: Restrict to specific directory // if (!absolute.startsWith(ALLOWED_BASE_PATH)) { // throw new Error('Path outside allowed directory'); // } return absolute; } ``` 2. Apply validation to all file path inputs 3. Add tests for path traversal attempts 4. Document security constraints **Acceptance Criteria:** - ✅ Path traversal prevented - ✅ Normalized paths used consistently - ✅ Clear error messages for invalid paths - ✅ Tests cover security scenarios **Testing:** - Test with `../../../etc/passwd` - Test with relative paths - Test with absolute paths - Test with symlinks --- ### Task 1.8: Replace `any` Types - Phase 1 🎯 **Priority:** P1 | **Effort:** 4-6 hours | **Complexity:** Simple **Issue:** Extensive use of `any` defeats type safety **Files to Modify (Phase 1 - Utilities):** - `src/memory/utils/validationUtils.ts:27,74,115` **Implementation Steps:** 1. Replace `any` with proper Entity/Relation types: ```typescript // Before export function validateEntity(entity: any): ValidationResult { // ... } // After export function validateEntity(entity: Entity): ValidationResult { // ... } ``` 2. Use `unknown` with type guards where needed: ```typescript function parseJSON(data: string): unknown { return JSON.parse(data); } function isEntity(obj: unknown): obj is Entity { return typeof obj === 'object' && obj !== null && 'name' in obj && 'entityType' in obj; } ``` 3. Add TypeScript strict flags to tsconfig.json **Acceptance Criteria:** - ✅ No `any` types in validationUtils.ts - ✅ Proper type guards implemented - ✅ TypeScript compilation passes - ✅ Tests still pass **Testing:** - Run TypeScript compiler with strict mode - Verify IDE shows proper types - Run full test suite --- ### Task 1.9: Code Style Consistency - ESLint Setup 🎨 **Priority:** P2 | **Effort:** 4-6 hours | **Complexity:** Simple **Issue:** Inconsistent code style throughout codebase **Files to Create:** - `.eslintrc.json` - `.prettierrc.json` **Implementation Steps:** 1. Install ESLint and Prettier: ```bash npm install --save-dev eslint @typescript-eslint/parser @typescript-eslint/eslint-plugin prettier eslint-config-prettier ``` 2. Create `.eslintrc.json`: ```json { "parser": "@typescript-eslint/parser", "plugins": ["@typescript-eslint"], "extends": [ "eslint:recommended", "plugin:@typescript-eslint/recommended", "prettier" ], "rules": { "@typescript-eslint/no-explicit-any": "error", "@typescript-eslint/explicit-function-return-type": "warn", "@typescript-eslint/no-unused-vars": "error" } } ``` 3. Create `.prettierrc.json`: ```json { "semi": true, "singleQuote": true, "tabWidth": 2, "trailingComma": "es5" } ``` 4. Add npm scripts: ```json { "scripts": { "lint": "eslint src/**/*.ts", "lint:fix": "eslint src/**/*.ts --fix", "format": "prettier --write src/**/*.ts" } } ``` 5. Run `npm run lint:fix` and `npm run format` **Acceptance Criteria:** - ✅ ESLint configured with TypeScript support - ✅ Prettier integrated - ✅ Linting passes with no errors - ✅ Code formatted consistently **Testing:** - Run `npm run lint` - Run `npm run format` - Commit and verify CI passes --- ### Task 1.10: Update CHANGELOG for Sprint 1 📝 **Priority:** P1 | **Effort:** 1 hour | **Complexity:** Simple **Files to Modify:** - `CHANGELOG.md` - `src/memory/package.json` (version bump to 0.11.7) - `README.md` (version badge) **Implementation Steps:** 1. Bump version to 0.11.7 2. Document all Sprint 1 changes in CHANGELOG 3. Run typecheck and tests 4. Commit and push **Acceptance Criteria:** - ✅ Version bumped to 0.11.7 - ✅ CHANGELOG updated with all changes - ✅ All tests passing - ✅ Changes committed to GitHub --- ## Sprint 2: Testing & Documentation **Duration:** 2-3 weeks **Total Effort:** 96-168 hours **Priority:** P0-P1 (Critical to High) **Goal:** Achieve 80%+ test coverage and comprehensive documentation ### Task 2.1: Unit Tests for EntityManager 🧪 **Priority:** P0 | **Effort:** 12-16 hours | **Complexity:** Moderate **Issue:** 0% coverage for modular EntityManager **Files to Create:** - `src/memory/__tests__/unit/core/EntityManager.test.ts` (already exists, expand) **Implementation Steps:** 1. Expand existing tests to cover all methods: - `createEntities()` - success, validation errors, duplicates - `deleteEntities()` - success, non-existent entities, cascade - `getEntity()` - found, not found, case sensitivity - `updateEntity()` - success, non-existent, validation - `searchEntitiesByType()` - exact match, partial, empty - `getEntitiesByTags()` - single tag, multiple tags, no matches 2. Test edge cases: - Empty arrays - Large batches (1000+ entities) - Special characters in names - Unicode handling - Importance boundaries (0, 10, out of range) 3. Test error handling: - Invalid entity structure - Missing required fields - Type mismatches - Storage failures (mock) 4. Use mocks for storage layer: ```typescript const mockStorage = { loadGraph: vi.fn(), saveGraph: vi.fn(), } as unknown as GraphStorage; ``` **Acceptance Criteria:** - ✅ All EntityManager methods covered - ✅ Edge cases tested - ✅ Error paths tested - ✅ Coverage > 90% for EntityManager **Testing:** - Run `npm test -- EntityManager.test.ts` - Check coverage report - All tests pass --- ### Task 2.2: Unit Tests for RelationManager 🧪 **Priority:** P0 | **Effort:** 10-14 hours | **Complexity:** Moderate **Files to Create:** - `src/memory/__tests__/unit/core/RelationManager.test.ts` **Implementation Steps:** 1. Test all relation operations: - `createRelations()` - success, validation, duplicates - `deleteRelations()` - success, non-existent, orphans - `getRelationsByEntity()` - incoming, outgoing, both - `getRelationsByType()` - exact match, multiple types 2. Test graph integrity: - Relations require existing entities - Cascade deletion behavior - Circular relation handling **Acceptance Criteria:** - ✅ All RelationManager methods covered - ✅ Graph integrity tested - ✅ Coverage > 90% --- ### Task 2.3: Unit Tests for CompressionManager 🧪 **Priority:** P0 | **Effort:** 12-16 hours | **Complexity:** Moderate **Files to Create:** - `src/memory/__tests__/unit/features/CompressionManager.test.ts` **Implementation Steps:** 1. Test duplicate detection: - Various similarity thresholds - Different entity types - Name variations (case, punctuation) - Edge cases (empty graphs, single entity) 2. Test merge operations: - Observation combining - Tag deduplication - Importance selection (highest) - Timestamp preservation (earliest) 3. Test compression: - Dry-run vs actual - Statistics accuracy - Relation redirection **Acceptance Criteria:** - ✅ All similarity algorithms tested - ✅ Merge logic verified - ✅ Coverage > 85% --- ### Task 2.4: Unit Tests for SearchManagers 🧪 **Priority:** P0 | **Effort:** 16-20 hours | **Complexity:** Moderate **Files to Create:** - `src/memory/__tests__/unit/search/BasicSearch.test.ts` - `src/memory/__tests__/unit/search/RankedSearch.test.ts` - `src/memory/__tests__/unit/search/BooleanSearch.test.ts` - `src/memory/__tests__/unit/search/FuzzySearch.test.ts` **Implementation Steps:** 1. Test BasicSearch: - Text matching (case insensitive) - Field-specific searches - Tag filtering - Importance filtering 2. Test RankedSearch: - TF-IDF scoring - Result ordering - Limit enforcement - Query tokenization 3. Test BooleanSearch: - AND, OR, NOT operators - Parentheses grouping - Field-specific queries - Quoted phrases - Complex nested queries 4. Test FuzzySearch: - Levenshtein distance - Threshold tuning - Typo tolerance **Acceptance Criteria:** - ✅ All search implementations tested - ✅ Query parsing validated - ✅ Scoring algorithms verified - ✅ Coverage > 85% --- ### Task 2.5: Integration Tests - Full Workflows 🔗 **Priority:** P1 | **Effort:** 16-24 hours | **Complexity:** Moderate **Files to Create:** - `src/memory/__tests__/integration/workflows.test.ts` **Implementation Steps:** 1. Test complete user workflows: - Create entities → Add relations → Search → Export - Import data → Validate → Merge duplicates → Archive - Create hierarchy → Navigate tree → Get subtrees - Create entities → Add tags → Search by tags → Merge tags 2. Test cross-component interaction: - EntityManager + RelationManager - CompressionManager + all managers - Search + Filtering + Pagination 3. Test data persistence: - Operations survive process restart - File corruption handling - Concurrent operation handling **Acceptance Criteria:** - ✅ End-to-end workflows tested - ✅ Component integration verified - ✅ Data persistence validated - ✅ 10+ integration test scenarios --- ### Task 2.6: Edge Case Tests 🧪 **Priority:** P1 | **Effort:** 12-16 hours | **Complexity:** Moderate **Implementation Steps:** 1. Test boundary conditions: - Empty graphs - Single entity/relation - Maximum size graphs (10k+ entities) - Deeply nested hierarchies (100+ levels) 2. Test error scenarios: - Invalid input types - Missing required fields - Circular references - File system errors - Out of memory conditions 3. Test special characters and encoding: - Unicode entities - Emoji in names - Special characters in observations - Very long strings (10k+ chars) **Acceptance Criteria:** - ✅ Boundary conditions tested - ✅ Error paths validated - ✅ Special characters handled - ✅ No crashes on edge cases --- ### Task 2.7: Performance Tests 📊 **Priority:** P1 | **Effort:** 12-16 hours | **Complexity:** Moderate **Files to Create:** - `src/memory/__tests__/performance/benchmarks.test.ts` **Implementation Steps:** 1. Create performance benchmarks: - Entity creation (1, 100, 1000, 10000 entities) - Search operations (various graph sizes) - Duplicate detection (various thresholds) - Graph loading/saving (various file sizes) 2. Set performance budgets: - createEntities(100) < 100ms - searchNodes() < 50ms - loadGraph(10k entities) < 500ms - findDuplicates(1k entities) < 2s 3. Track performance over time: - Store benchmark results - Compare against baselines - Alert on regressions **Acceptance Criteria:** - ✅ Performance benchmarks created - ✅ Budgets defined - ✅ Regression detection in place - ✅ Documentation of results --- ### Task 2.8: Architecture Documentation 📚 **Priority:** P1 | **Effort:** 8-12 hours | **Complexity:** Moderate **Files to Create:** - `docs/ARCHITECTURE.md` - `docs/diagrams/` (C4 model diagrams) **Implementation Steps:** 1. Create architecture document: - System context - Container diagram (MCP Server, Storage, etc.) - Component diagram (managers, services, utils) - Data flow diagrams 2. Document design decisions: - Why modular architecture - Why JSONL format - Why in-memory caching - Why two-level bucketing for duplicates 3. Create ADRs (Architecture Decision Records): - ADR-001: Modular refactoring approach - ADR-002: Zod for validation - ADR-003: File-based vs database storage - ADR-004: Transaction implementation strategy **Acceptance Criteria:** - ✅ ARCHITECTURE.md created - ✅ C4 diagrams included - ✅ ADRs documented - ✅ Design rationale explained --- ### Task 2.9: API Reference Generation 📖 **Priority:** P2 | **Effort:** 6-8 hours | **Complexity:** Simple **Files to Create:** - `docs/API.md` (auto-generated) **Implementation Steps:** 1. Install TypeDoc: ```bash npm install --save-dev typedoc ``` 2. Configure TypeDoc: ```json { "name": "Memory MCP API", "entryPoints": ["src/memory/index.ts"], "out": "docs/api", "excludePrivate": true, "excludeInternal": true } ``` 3. Generate documentation: ```bash npx typedoc ``` 4. Add to CI pipeline **Acceptance Criteria:** - ✅ API documentation generated - ✅ All public APIs documented - ✅ Examples included - ✅ Updated automatically --- ### Task 2.10: Update CHANGELOG for Sprint 2 📝 **Priority:** P1 | **Effort:** 1 hour | **Complexity:** Simple **Files to Modify:** - `CHANGELOG.md` - `src/memory/package.json` (version bump to 0.12.0) - `README.md` (version badge) **Implementation Steps:** 1. Bump version to 0.12.0 (minor - new tests) 2. Document test coverage improvements 3. Document new documentation 4. Commit and push **Acceptance Criteria:** - ✅ Version bumped to 0.12.0 - ✅ CHANGELOG updated - ✅ Test coverage > 80% - ✅ All tests passing --- ## Sprint 3: Performance Improvements **Duration:** 2-3 weeks **Total Effort:** 96-144 hours **Priority:** P1 (High) **Goal:** Optimize search, caching, and eliminate bottlenecks ### Task 3.1: Add Pagination to BasicSearch 📄 **Priority:** P1 | **Effort:** 8-12 hours | **Complexity:** Moderate **Files to Modify:** - `src/memory/search/BasicSearch.ts` **Implementation Steps:** 1. Add pagination parameters: ```typescript interface SearchOptions { query: string; tags?: string[]; minImportance?: number; maxImportance?: number; // New pagination params offset?: number; limit?: number; } async searchNodes(options: SearchOptions): Promise<PaginatedResult> { const offset = options.offset ?? 0; const limit = options.limit ?? SEARCH_LIMITS.DEFAULT; // ... existing search logic ... const total = matches.length; const results = matches.slice(offset, offset + limit); return { results, total, offset, limit, hasMore: offset + limit < total, }; } ``` 2. Update all search implementations 3. Update MCP tool definitions 4. Add tests for pagination **Acceptance Criteria:** - ✅ Pagination works correctly - ✅ Edge cases handled (offset > total) - ✅ Performance improved for large results - ✅ Tests pass --- ### Task 3.2: Add Pagination to BooleanSearch 📄 **Priority:** P1 | **Effort:** 8-12 hours | **Complexity:** Moderate **Files to Modify:** - `src/memory/search/BooleanSearch.ts` **Implementation Steps:** 1. Apply same pagination pattern as BasicSearch 2. Update query parser if needed 3. Test complex queries with pagination **Acceptance Criteria:** - ✅ Boolean search paginated - ✅ Works with complex queries - ✅ Tests pass --- ### Task 3.3: Add Pagination to FuzzySearch 📄 **Priority:** P1 | **Effort:** 6-10 hours | **Complexity:** Moderate **Files to Modify:** - `src/memory/search/FuzzySearch.ts` **Implementation Steps:** 1. Apply pagination pattern 2. Ensure similarity ordering maintained 3. Test with various thresholds **Acceptance Criteria:** - ✅ Fuzzy search paginated - ✅ Result ordering preserved - ✅ Tests pass --- ### Task 3.4: Pre-calculate TF-IDF Indexes 🚀 **Priority:** P1 | **Effort:** 16-24 hours | **Complexity:** Complex **Files to Modify:** - `src/memory/search/RankedSearch.ts` - `src/memory/core/GraphStorage.ts` (index management) **Implementation Steps:** 1. Create index file structure: ```typescript interface TFIDFIndex { version: string; lastUpdated: string; documents: Map<string, DocumentVector>; idf: Map<string, number>; } ``` 2. Build index on graph changes: - After entity creation - After entity updates - After entity deletion - Incremental updates only 3. Store index in `.indexes/tfidf.json` 4. Load index on search: ```typescript async searchNodesRanked(query: string, limit?: number): Promise<SearchResult[]> { // Load pre-calculated index const index = await this.loadTFIDFIndex(); // Query against index (fast) const queryVector = this.calculateQueryVector(query); const scores = this.scoreDocuments(queryVector, index); return scores.slice(0, limit); } ``` **Acceptance Criteria:** - ✅ Index pre-calculated - ✅ Incremental updates working - ✅ Search 10x+ faster - ✅ Tests pass --- ### Task 3.5: Implement Search Result Caching 💾 **Priority:** P1 | **Effort:** 12-16 hours | **Complexity:** Moderate **Files to Create:** - `src/memory/utils/searchCache.ts` **Implementation Steps:** 1. Create LRU cache for search results: ```typescript import { LRUCache } from 'lru-cache'; const searchCache = new LRUCache<string, SearchResult[]>({ max: 500, ttl: 1000 * 60 * 5, // 5 minutes }); export function cacheSearch(query: string, results: SearchResult[]) { const key = hashQuery(query); searchCache.set(key, results); } export function getCachedSearch(query: string): SearchResult[] | undefined { const key = hashQuery(query); return searchCache.get(key); } ``` 2. Integrate into search managers 3. Invalidate cache on graph changes 4. Add cache statistics **Acceptance Criteria:** - ✅ Search results cached - ✅ Cache invalidation working - ✅ Performance improved - ✅ Tests pass --- ### Task 3.6: Batch Operations API 📦 **Priority:** P1 | **Effort:** 16-24 hours | **Complexity:** Moderate **Files to Modify:** - `src/memory/core/EntityManager.ts` - `src/memory/core/RelationManager.ts` - `src/memory/features/TagManager.ts` **Implementation Steps:** 1. Add batch update method to EntityManager: ```typescript async batchUpdate(updates: Array<{ name: string; changes: Partial<Entity>; }>): Promise<Entity[]> { const graph = await this.storage.loadGraph(); const timestamp = new Date().toISOString(); const updated: Entity[] = []; for (const { name, changes } of updates) { const entity = graph.entities.find(e => e.name === name); if (entity) { Object.assign(entity, changes); entity.lastModified = timestamp; updated.push(entity); } } // Single save for all updates await this.storage.saveGraph(graph); return updated; } ``` 2. Add to TagManager: ```typescript async batchAddTags( entityNames: string[], tags: string[] ): Promise<BatchResult> { // Single load, multiple updates, single save } ``` 3. Add MCP tools for batch operations 4. Add tests **Acceptance Criteria:** - ✅ Batch operations implemented - ✅ Single save for multiple operations - ✅ 10x+ faster for bulk updates - ✅ Tests pass --- ### Task 3.7: Graph Size Limits & Quotas 📏 **Priority:** P1 | **Effort:** 8-12 hours | **Complexity:** Moderate **Files to Modify:** - `src/memory/core/EntityManager.ts` - `src/memory/core/RelationManager.ts` - `src/memory/utils/constants.ts` **Implementation Steps:** 1. Add configurable limits: ```typescript export const GRAPH_LIMITS = { MAX_ENTITIES: 100000, MAX_RELATIONS: 1000000, MAX_FILE_SIZE_MB: 500, MAX_ENTITY_NAME_LENGTH: 500, MAX_OBSERVATION_LENGTH: 10000, MAX_OBSERVATIONS_PER_ENTITY: 1000, } as const; ``` 2. Check limits before operations: ```typescript async createEntities(entities: Entity[]): Promise<Entity[]> { const graph = await this.storage.loadGraph(); // Check quota if (graph.entities.length + entities.length > GRAPH_LIMITS.MAX_ENTITIES) { throw new QuotaExceededError('Max entities exceeded'); } // ... proceed with creation ... } ``` 3. Add quota reporting tool 4. Add tests for limit enforcement **Acceptance Criteria:** - ✅ Size limits enforced - ✅ Clear error messages - ✅ Configurable limits - ✅ Tests pass --- ### Task 3.8: Streaming JSON Parser ⚡ **Priority:** P1 | **Effort:** 16-24 hours | **Complexity:** Complex **Files to Modify:** - `src/memory/core/GraphStorage.ts` **Implementation Steps:** 1. Install streaming parser: ```bash npm install stream-json ``` 2. Implement streaming load: ```typescript import { parser } from 'stream-json'; import { streamArray } from 'stream-json/streamers/StreamArray'; async loadGraph(): Promise<KnowledgeGraph> { // Check cache first if (this.cache) { return this.deepCopy(this.cache); } const graph: KnowledgeGraph = { entities: [], relations: [] }; const stream = fs.createReadStream(this.memoryFilePath) .pipe(parser()) .pipe(streamArray()); for await (const { value } of stream) { if (value.type === 'entity') { graph.entities.push(value as Entity); } else if (value.type === 'relation') { graph.relations.push(value as Relation); } } this.cache = graph; return this.deepCopy(graph); } ``` 3. Test with large files (100MB+) 4. Measure performance improvement **Acceptance Criteria:** - ✅ Non-blocking parsing - ✅ Memory-efficient - ✅ Handles large files - ✅ Tests pass --- ### Task 3.9: Query Complexity Limits 🛡️ **Priority:** P1 | **Effort:** 12-16 hours | **Complexity:** Moderate **Files to Modify:** - `src/memory/search/BooleanSearch.ts` **Implementation Steps:** 1. Add query complexity validator: ```typescript interface QueryComplexity { depth: number; // Max nesting level termCount: number; // Total terms operatorCount: number; // AND/OR/NOT count } const QUERY_LIMITS = { MAX_DEPTH: 10, MAX_TERMS: 50, MAX_OPERATORS: 20, } as const; function validateQueryComplexity(ast: QueryNode): void { const complexity = calculateComplexity(ast); if (complexity.depth > QUERY_LIMITS.MAX_DEPTH) { throw new QueryTooComplexError('Query nesting too deep'); } if (complexity.termCount > QUERY_LIMITS.MAX_TERMS) { throw new QueryTooComplexError('Too many search terms'); } // ... other checks ... } ``` 2. Integrate into query parser 3. Add helpful error messages 4. Add tests **Acceptance Criteria:** - ✅ Query complexity limited - ✅ Clear error messages - ✅ No stack overflow - ✅ Tests pass --- ### Task 3.10: Update CHANGELOG for Sprint 3 📝 **Priority:** P1 | **Effort:** 1 hour | **Complexity:** Simple **Files to Modify:** - `CHANGELOG.md` - `src/memory/package.json` (version bump to 0.13.0) - `README.md` (version badge) **Implementation Steps:** 1. Bump version to 0.13.0 (minor - performance improvements) 2. Document all performance optimizations 3. Include before/after metrics 4. Commit and push **Acceptance Criteria:** - ✅ Version bumped to 0.13.0 - ✅ CHANGELOG updated with metrics - ✅ Performance improvements documented - ✅ All tests passing --- ## Sprint 4: Architecture Refactoring **Duration:** 4-6 weeks **Total Effort:** 280-440 hours **Priority:** P0-P1 (Critical to High) **Goal:** Complete modular architecture, remove code duplication ### Task 4.1: Complete Modular Refactoring - Phase 1 🏗️ **Priority:** P0 | **Effort:** 40-60 hours | **Complexity:** Very Complex **Issue:** `index.ts` still 4,188 lines with duplicate implementations **Files to Modify:** - `src/memory/index.ts` (reduce from 4,188 to <200 lines) **Implementation Steps:** **Phase 1a: Remove Implementation Code (Week 1)** 1. Identify all implementation code in index.ts 2. Verify modular equivalents exist 3. Delete implementation functions one by one: - levenshteinDistance() → import from utils/levenshtein.ts - TF-IDF logic → import from utils/tfidf.ts - Boolean parser → import from search/BooleanSearch.ts - Compression logic → import from features/CompressionManager.ts 4. Keep only: - MCP server setup - Tool definitions - Request routing to managers **Phase 1b: Update Imports (Week 2)** 1. Replace internal implementations with imports: ```typescript // Before class KnowledgeGraphManager { private levenshteinDistance(a: string, b: string): number { // 50 lines of implementation } } // After import { levenshteinDistance } from './utils/levenshtein.js'; ``` 2. Update all references 3. Test after each removal 4. Ensure functionality unchanged **Acceptance Criteria:** - ✅ index.ts < 500 lines (interim target) - ✅ No duplicate implementations - ✅ All imports working - ✅ All tests pass --- ### Task 4.2: Complete Modular Refactoring - Phase 2 🏗️ **Priority:** P0 | **Effort:** 40-60 hours | **Complexity:** Very Complex **Phase 2: Extract MCP Server Layer (Week 3-4)** **Files to Create:** - `src/memory/server/MCPServer.ts` - `src/memory/server/toolHandlers.ts` - `src/memory/server/toolDefinitions.ts` **Implementation Steps:** 1. Create MCP server abstraction: ```typescript // server/MCPServer.ts export class MCPServer { constructor(private manager: KnowledgeGraphManager) {} async start() { const server = new Server(/* ... */); server.setRequestHandler( ListToolsRequestSchema, this.handleListTools.bind(this) ); server.setRequestHandler( CallToolRequestSchema, this.handleToolCall.bind(this) ); await server.connect(/* ... */); } private async handleToolCall(request: CallToolRequest) { return toolHandlers[request.params.name]( this.manager, request.params.arguments ); } } ``` 2. Extract tool handlers: ```typescript // server/toolHandlers.ts export const toolHandlers = { create_entities: async (manager, args) => { const entities = await manager.entities.createEntities(args.entities); return formatResponse(entities); }, search_nodes: async (manager, args) => { const results = await manager.search.searchNodes(args); return formatResponse(results); }, // ... all 45 tools ... }; ``` 3. Move tool definitions to separate file 4. Update index.ts to use MCPServer class **Acceptance Criteria:** - ✅ MCP layer separated - ✅ Business logic decoupled - ✅ index.ts < 200 lines - ✅ All tests pass --- ### Task 4.3: Remove Duplicate Type Definitions 🎯 **Priority:** P0 | **Effort:** 16-24 hours | **Complexity:** Moderate **Files to Modify:** - `src/memory/index.ts:53-169` **Implementation Steps:** 1. Remove all type definitions from index.ts: ```typescript // Delete these from index.ts export interface Entity { /* ... */ } export interface Relation { /* ... */ } export interface KnowledgeGraph { /* ... */ } // etc. ``` 2. Import from types module: ```typescript // Add to index.ts export type { Entity, Relation, KnowledgeGraph, SearchOptions, // ... all types } from './types/index.js'; ``` 3. Update all internal references 4. Verify exports work correctly **Acceptance Criteria:** - ✅ No duplicate type definitions - ✅ Single source of truth in types/ - ✅ All imports working - ✅ TypeScript compilation passes --- ### Task 4.4: Implement Storage Abstraction - Interface 🔌 **Priority:** P2 | **Effort:** 20-30 hours | **Complexity:** Complex **Files to Create:** - `src/memory/core/storage/IStorageBackend.ts` - `src/memory/core/storage/FileStorage.ts` - `src/memory/core/storage/MemoryStorage.ts` **Implementation Steps:** 1. Define storage interface: ```typescript // core/storage/IStorageBackend.ts export interface IStorageBackend { /** * Load the entire knowledge graph */ loadGraph(): Promise<KnowledgeGraph>; /** * Save the entire knowledge graph */ saveGraph(graph: KnowledgeGraph): Promise<void>; /** * Clear any caches */ clearCache(): void; /** * Get file path (for file-based backends) */ getFilePath?(): string; } ``` 2. Refactor GraphStorage to FileStorage: ```typescript // core/storage/FileStorage.ts export class FileStorage implements IStorageBackend { // Current GraphStorage implementation } ``` 3. Create MemoryStorage for testing: ```typescript // core/storage/MemoryStorage.ts export class MemoryStorage implements IStorageBackend { private graph: KnowledgeGraph = { entities: [], relations: [] }; async loadGraph(): Promise<KnowledgeGraph> { return { ...this.graph }; } async saveGraph(graph: KnowledgeGraph): Promise<void> { this.graph = graph; } clearCache(): void { // No-op for memory storage } } ``` 4. Update managers to use interface 5. Add dependency injection **Acceptance Criteria:** - ✅ Storage abstraction defined - ✅ FileStorage implemented - ✅ MemoryStorage for tests - ✅ Tests use MemoryStorage (faster) --- ### Task 4.5: Implement Storage Abstraction - Database Backend 🗄️ **Priority:** P3 | **Effort:** 60-80 hours | **Complexity:** Very Complex **Files to Create:** - `src/memory/core/storage/DatabaseStorage.ts` - `src/memory/core/storage/migrations/` **Implementation Steps:** 1. Install better-sqlite3: ```bash npm install better-sqlite3 npm install --save-dev @types/better-sqlite3 ``` 2. Create database schema: ```sql CREATE TABLE entities ( name TEXT PRIMARY KEY, entityType TEXT NOT NULL, observations TEXT, -- JSON array tags TEXT, -- JSON array importance INTEGER, parentId TEXT, createdAt TEXT, lastModified TEXT ); CREATE TABLE relations ( id INTEGER PRIMARY KEY AUTOINCREMENT, "from" TEXT NOT NULL, "to" TEXT NOT NULL, relationType TEXT NOT NULL, createdAt TEXT, lastModified TEXT, FOREIGN KEY ("from") REFERENCES entities(name) ON DELETE CASCADE, FOREIGN KEY ("to") REFERENCES entities(name) ON DELETE CASCADE ); CREATE INDEX idx_entities_type ON entities(entityType); CREATE INDEX idx_entities_tags ON entities(tags); CREATE INDEX idx_relations_from ON relations("from"); CREATE INDEX idx_relations_to ON relations("to"); ``` 3. Implement DatabaseStorage: ```typescript export class DatabaseStorage implements IStorageBackend { private db: Database; async loadGraph(): Promise<KnowledgeGraph> { const entities = this.db.prepare('SELECT * FROM entities').all(); const relations = this.db.prepare('SELECT * FROM relations').all(); return { entities: entities.map(this.deserializeEntity), relations: relations.map(this.deserializeRelation), }; } async saveGraph(graph: KnowledgeGraph): Promise<void> { const tx = this.db.transaction(() => { // Clear and reinsert (or use upsert) this.db.prepare('DELETE FROM entities').run(); this.db.prepare('DELETE FROM relations').run(); for (const entity of graph.entities) { this.insertEntity(entity); } for (const relation of graph.relations) { this.insertRelation(relation); } }); tx(); } } ``` 4. Add migration system 5. Add tests **Acceptance Criteria:** - ✅ Database backend working - ✅ Proper indexing - ✅ Migrations supported - ✅ Performance better than file **Note:** This is P3 (future) - may defer to later sprint --- ### Task 4.6: Replace `any` Types - Phase 2 🎯 **Priority:** P1 | **Effort:** 12-20 hours | **Complexity:** Moderate **Files to Modify:** - `src/memory/index.ts:93,4123,1853,2152` - `src/memory/features/*.ts` - `src/memory/search/*.ts` **Implementation Steps:** 1. Replace all remaining `any` types 2. Use proper union types where needed 3. Add type guards for runtime checks 4. Enable `noImplicitAny` in tsconfig.json **Acceptance Criteria:** - ✅ No `any` types remaining - ✅ Strict TypeScript enabled - ✅ All tests pass - ✅ Type safety improved --- ### Task 4.7: Consistent Error Handling Strategy 🚨 **Priority:** P1 | **Effort:** 16-24 hours | **Complexity:** Moderate **Files to Create:** - `src/memory/utils/result.ts` (Result type) **Implementation Steps:** 1. Create Result/Either type: ```typescript // utils/result.ts export type Result<T, E = Error> = | { success: true; value: T } | { success: false; error: E }; export function Ok<T>(value: T): Result<T, never> { return { success: true, value }; } export function Err<E>(error: E): Result<never, E> { return { success: false, error }; } ``` 2. Update critical operations to use Result: ```typescript async createEntities(entities: Entity[]): Promise<Result<Entity[]>> { try { const validation = BatchCreateEntitiesSchema.safeParse(entities); if (!validation.success) { return Err(new ValidationError('Invalid entities', validation.error.issues)); } const graph = await this.storage.loadGraph(); // ... process ... await this.storage.saveGraph(graph); return Ok(newEntities); } catch (error) { return Err(error as Error); } } ``` 3. Document error handling patterns 4. Update tests **Acceptance Criteria:** - ✅ Consistent error handling - ✅ Result type used for fallible operations - ✅ Clear error messages - ✅ Tests pass --- ### Task 4.8: Proper Logging System 📋 **Priority:** P1 | **Effort:** 8-12 hours | **Complexity:** Moderate **Files to Create:** - `src/memory/utils/logger.ts` (enhanced from Task 1.1) **Implementation Steps:** 1. Install winston or pino: ```bash npm install winston ``` 2. Create logger configuration: ```typescript import winston from 'winston'; export const logger = winston.createLogger({ level: process.env.LOG_LEVEL || 'info', format: winston.format.combine( winston.format.timestamp(), winston.format.errors({ stack: true }), winston.format.json() ), transports: [ new winston.transports.Console({ format: winston.format.simple(), }), new winston.transports.File({ filename: 'memory-mcp-error.log', level: 'error', }), new winston.transports.File({ filename: 'memory-mcp.log', }), ], }); ``` 3. Replace all console.* with logger.* 4. Add structured logging 5. Add log rotation **Acceptance Criteria:** - ✅ Winston/Pino integrated - ✅ Log levels working - ✅ Structured logging - ✅ File rotation configured --- ### Task 4.9: Integration Tests for Refactored Code 🔗 **Priority:** P0 | **Effort:** 24-32 hours | **Complexity:** Complex **Files to Create:** - `src/memory/__tests__/integration/modular-architecture.test.ts` **Implementation Steps:** 1. Test new modular architecture: - MCPServer connects correctly - Tool handlers route properly - Storage abstraction works - All managers integrate 2. Test storage backends: - FileStorage - MemoryStorage - (Optional) DatabaseStorage 3. Test error propagation 4. Test logging **Acceptance Criteria:** - ✅ All integration paths tested - ✅ Modular architecture validated - ✅ Storage backends work - ✅ Tests pass --- ### Task 4.10: Update CHANGELOG for Sprint 4 📝 **Priority:** P1 | **Effort:** 2 hours | **Complexity:** Simple **Files to Modify:** - `CHANGELOG.md` - `src/memory/package.json` (version bump to 1.0.0) - `README.md` (version badge) **Implementation Steps:** 1. Bump to v1.0.0 (major - architecture complete) 2. Document complete modular refactoring 3. Document storage abstraction 4. Document all breaking changes (if any) 5. Commit and push **Acceptance Criteria:** - ✅ Version bumped to 1.0.0 - ✅ Major refactoring documented - ✅ Architecture diagram updated - ✅ All tests passing --- ## Sprint 5: Advanced Features **Duration:** 3-4 weeks **Total Effort:** 200-320 hours **Priority:** P2-P3 (Medium to Future) **Goal:** Add production features (auth, metrics, migrations) ### Task 5.1: Rate Limiting 🛡️ **Priority:** P2 | **Effort:** 24-32 hours | **Complexity:** Complex **Implementation:** Per-client rate limiting with token bucket algorithm --- ### Task 5.2: Metrics & Monitoring 📊 **Priority:** P2 | **Effort:** 40-60 hours | **Complexity:** Complex **Implementation:** Prometheus metrics, health checks, instrumentation --- ### Task 5.3: Schema Migration System 🔄 **Priority:** P2 | **Effort:** 40-60 hours | **Complexity:** Complex **Implementation:** Migration framework, versioning, upgrade/downgrade --- ### Task 5.4: Authentication & Authorization 🔐 **Priority:** P2 | **Effort:** 80-120 hours | **Complexity:** Very Complex **Implementation:** Auth layer, RBAC, audit logging, operation permissions --- ### Task 5.5: Additional Export Formats 📤 **Priority:** P3 | **Effort:** 24-32 hours | **Complexity:** Moderate **Implementation:** RDF, Turtle, N-Triples, JSON-LD --- ### Task 5.6-5.10: Additional Features - Connection pooling - Advanced query optimizer - Multi-tenant support - Distributed architecture (future) - Advanced search engine integration (future) --- ## Future Roadmap (P3) ### Database Backend (120-200 hours) - PostgreSQL/SQLite implementation - Full SQL query support - Advanced indexing - Connection pooling ### Distributed Architecture (200-300 hours) - Multi-node deployment - Leader election - Sharding support - Redis caching layer ### Advanced Search (120-160 hours) - Elasticsearch integration - Full-text search - Faceted search - SPARQL endpoint --- ## Progress Tracking ### Sprint Completion Checklist **Sprint 1: Quick Wins** ⬜ - [ ] Task 1.1: Console logging ✅ - [ ] Task 1.2: Deprecated deps ✅ - [ ] Task 1.3: Magic numbers ✅ - [ ] Task 1.4: Build script ✅ - [ ] Task 1.5: JSDoc ✅ - [ ] Task 1.6: Search limits ✅ - [ ] Task 1.7: Path validation ✅ - [ ] Task 1.8: Replace any ✅ - [ ] Task 1.9: ESLint setup ✅ - [ ] Task 1.10: CHANGELOG ✅ **Sprint 2: Testing** ⬜ - [ ] 10 tasks (testing & documentation) **Sprint 3: Performance** ⬜ - [ ] 10 tasks (optimization) **Sprint 4: Architecture** ⬜ - [ ] 10 tasks (refactoring) **Sprint 5: Advanced** ⬜ - [ ] 10 tasks (features) --- ## Metrics & Goals ### Test Coverage Targets - **Current:** 9.61% overall - **Sprint 1:** No change (code quality only) - **Sprint 2:** 80%+ overall - **Sprint 3:** Maintain 80%+ - **Sprint 4:** Maintain 80%+ - **Sprint 5:** Maintain 80%+ ### Performance Targets - **Search:** <50ms for 10k entities - **Duplicate detection:** <2s for 1k entities - **Graph loading:** <500ms for 10k entities - **Entity creation (batch 100):** <100ms ### Code Quality Targets - **TypeScript strict mode:** Enabled - **ESLint errors:** 0 - **No `any` types:** Achieved - **Lines of code in index.ts:** <200 (from 4,188) --- ## Notes & Assumptions 1. **Backward Compatibility:** Maintain API compatibility through v1.x 2. **Breaking Changes:** Only in major versions (v2.0, etc.) 3. **Testing:** All changes must have tests 4. **Documentation:** Update docs with each sprint 5. **Code Review:** Each task reviewed before merge 6. **CI/CD:** All tests must pass before deployment --- **Plan Status:** Active **Last Updated:** 2025-11-25 **Next Review:** After Sprint 1 completion