Article Manager MCP Server

# Implementation Plan - [x] 1. Set up database schema and core queue infrastructure - Create embedding_tasks table with proper indexes and constraints - Create embedding_worker_status table for worker state tracking - Add database migration script for schema changes - _Requirements: 2.1, 2.4, 2.5_ - [x] 2. Implement Queue Manager Service - [x] 2.1 Create EmbeddingTask interface and core queue operations - Define EmbeddingTask interface with all required fields - Implement task creation, status updates, and retrieval methods - Add database persistence for task operations - _Requirements: 2.1, 2.5_ - [ ]* 2.2 Write property test for task persistence - **Property 4: Task persistence across restarts** - **Validates: Requirements 2.1, 2.4** - [x] 2.3 Implement queue statistics and monitoring - Add methods for queue statistics (pending, processing, completed, failed counts) - Implement task cleanup for old completed tasks - Add queue health check functionality - _Requirements: 5.2_ - [ ]* 2.4 Write property test for queue statistics accuracy - **Property 12: Accurate queue statistics** - **Validates: Requirements 5.2** - [x] 3. Implement Background Worker Service - [x] 3.1 Create worker lifecycle management - Implement worker start/stop functionality with proper state tracking - Add worker status persistence to database - Create worker heartbeat mechanism for monitoring - _Requirements: 2.2, 2.4_ - [x] 3.2 Implement task processing with retry logic - Add task dequeue and processing logic - Implement exponential backoff retry mechanism with 3 attempt limit - Add comprehensive error handling and logging - _Requirements: 2.2, 2.3, 5.1_ - [ ]* 3.3 Write property test for sequential processing - **Property 5: Sequential task processing** - **Validates: Requirements 2.2** - [ ]* 3.4 Write property test for retry behavior - **Property 6: Retry with exponential backoff** - **Validates: Requirements 2.3** - [x] 3.5 Implement embedding generation integration - Connect worker to existing embedding service - Add proper error handling for embedding failures - Implement embedding storage through existing database services - _Requirements: 3.4, 4.4_ - [ ]* 3.6 Write property test for status tracking - **Property 7: Status tracking throughout lifecycle** - **Validates: Requirements 2.5, 3.4** - [x] 4. Enhance Article Service for background processing - [x] 4.1 Modify article creation to use background embedding - Update createArticle method to queue embedding tasks instead of processing synchronously - Ensure article persistence happens before task queuing - Add options parameter for embedding configuration - _Requirements: 1.1, 1.3, 1.4_ - [ ]* 4.2 Write property test for article persistence ordering - **Property 1: Article persistence precedes task queuing** - **Validates: Requirements 1.1, 1.2** - [x] 4.3 Modify article update to use background embedding - Update updateArticle method to queue embedding update tasks - Handle slug changes and embedding cleanup properly - Maintain backward compatibility with existing interfaces - _Requirements: 1.2, 1.3, 1.4_ - [ ]* 4.4 Write property test for immediate response - **Property 2: Immediate response without embedding wait** - **Validates: Requirements 1.3, 1.4, 4.1, 4.2** - [x] 4.5 Add embedding failure isolation - Ensure article operations continue working when embedding tasks fail - Add proper error handling to prevent embedding failures from affecting article CRUD - _Requirements: 1.5_ - [ ]* 4.6 Write property test for embedding failure isolation - **Property 3: Embedding failure isolation** - **Validates: Requirements 1.5** - [x] 5. Implement bulk embedding operations - [x] 5.1 Create bulk embedding update functionality - Implement identification of articles with missing or failed embeddings - Add bulk task queuing for identified articles - Create progress tracking for bulk operations - _Requirements: 3.1, 3.2, 3.3_ - [ ]* 5.2 Write property test for bulk operation completeness - **Property 8: Bulk operation completeness** - **Validates: Requirements 3.1, 3.2** - [x] 5.3 Add bulk operation reporting - Implement progress feedback during bulk operations - Add completion summary with success/failure counts - Create administrative commands for bulk operations - _Requirements: 3.3, 3.5_ - [x] 6. Update MCP server integration - [x] 6.1 Modify MCP article operations for background processing - Update MCP create_article tool to use background embedding - Update MCP update_article tool to queue embedding tasks - Ensure immediate responses without waiting for embedding completion - _Requirements: 4.1, 4.2_ - [x] 6.2 Enhance MCP search operations - Ensure search operations work with available embeddings - Add non-blocking search behavior during pending embedding tasks - Update search results to reflect current embedding status - _Requirements: 4.3, 4.4_ - [ ]* 6.3 Write property test for search availability - **Property 9: Search availability during processing** - **Validates: Requirements 4.3** - [x] 6.4 Add embedding status to MCP responses - Include embedding task status in article metadata responses - Add MCP tools for querying embedding queue status - Provide embedding statistics through MCP interface - _Requirements: 4.5_ - [ ]* 6.5 Write property test for search index consistency - **Property 10: Search index consistency after completion** - **Validates: Requirements 4.4** - [x] 7. Implement logging and monitoring - [x] 7.1 Add comprehensive audit logging - Implement logging for all task lifecycle events with timestamps - Add detailed error logging for failed tasks - Create structured logging for monitoring integration - _Requirements: 5.1, 5.3, 5.5_ - [ ]* 7.2 Write property test for audit logging - **Property 11: Comprehensive audit logging** - **Validates: Requirements 5.1, 5.5** - [x] 7.3 Add performance metrics tracking - Implement processing time tracking for embedding tasks - Add worker performance statistics - Create metrics for queue throughput and success rates - _Requirements: 5.4_ - [x] 8. Add configuration and startup integration - [x] 8.1 Create configuration system for embedding queue - Add environment variables for queue configuration - Implement feature flags for enabling/disabling background processing - Add configuration validation and defaults - _Requirements: All_ - [x] 8.2 Integrate worker startup with main server - Add worker initialization to server startup sequence - Implement graceful shutdown for background worker - Add worker status to health check endpoints - _Requirements: 2.4_ - [x] 9. Checkpoint - Ensure all tests pass - Ensure all tests pass, ask the user if questions arise. - [-] 10. Add administrative tools and utilities - [x] 10.1 Create queue management utilities - Add CLI commands for queue inspection and management - Implement manual task retry and cleanup operations - Create debugging tools for task troubleshooting - _Requirements: 3.1, 3.2_ - [x] 10.2 Add migration and deployment support - Create migration script for existing articles without embeddings - Add deployment documentation for queue system - Implement rollback procedures for emergency situations - _Requirements: All_ - [x] 11. Final Checkpoint - Ensure all tests pass - Ensure all tests pass, ask the user if questions arise.