MCP Console Automation Server

# Enhanced Race Condition Solution Plan ## Event-Driven Architecture + Smart Execute **Version:** 2.0 **Date:** 2025-09-27 **Status:** Implementation Ready --- ## Executive Summary After re-analysis using sequential thinking MCP, the comprehensive solution addresses THREE problems: 1. **Architectural Inefficiency:** `console_execute_command` polls every 100ms despite having event-driven completion detection 2. **User Guidance Problem:** Error messages guide users to wrong APIs (sendInput/getOutput) 3. **UX Complexity:** Users must understand and choose between 3 execution patterns **Solution:** Three-phase implementation that fixes the architecture, updates documentation, and provides intelligent execution. **Total Effort:** 22-30 hours across 3 phases **Risk:** Low-Medium (incremental deployment with value at each phase) --- ## Problem Analysis ### Problem 1: Polling Inefficiency **Current Implementation (ConsoleManager.ts lines 1057-1105):** ```typescript private async waitForCommandCompletion(commandId: string, timeout: number) { const checkInterval = 100; // Check every 100ms return new Promise((resolve) => { const checkCompletion = () => { // Check if command status changed if (currentExecution.status !== 'executing') { resolve({...}); return; } // Continue polling setTimeout(checkCompletion, checkInterval); }; checkCompletion(); }); } ``` **Issues:** - CPU waste: Checks every 100ms even when idle - Latency: Up to 100ms delay in detecting completion - Scalability: Multiple concurrent commands = many polling loops - Not event-driven: Despite having event-based detection at line 7910 **Root Cause:** Completion is detected via events (line 7916: `completeCommandExecution` called from data event), but wrapped in polling. ### Problem 2: User Guidance (Covered in Original Plan) Error messages at lines 1402, 1461 incorrectly suggest using `sendInput` + `waitForOutput`. ### Problem 3: UX Complexity Users must choose between 3 patterns: - `console_execute_command` - might timeout or hit token limits - Streaming session - complex setup, manual pagination - Background job - async, need to poll for completion **Better UX:** One intelligent tool that does the right thing automatically. --- ## Solution Architecture ### Three-Phase Implementation ``` Phase 1: Event-Driven Completion (4-6 hours) ↓ Fixes architectural inefficiency Phase 2: Documentation Fixes (6-8 hours) ↓ Prevents wrong API usage Phase 3: Smart Execute (12-16 hours) ↓ Best UX with intelligent execution ``` **Total:** 22-30 hours, deployable incrementally --- ## Phase 1: Event-Driven Completion Fix **Priority:** HIGH (Architectural Fix) **Effort:** 4-6 hours **Risk:** Low (internal refactoring) ### Current vs. Proposed **Current Flow:** ``` Data arrives → detectCommandCompletion (line 7910) → completeCommandExecution (line 7916) → Updates status to 'completed' Meanwhile: waitForCommandCompletion polls status every 100ms ``` **Proposed Flow:** ``` Data arrives → detectCommandCompletion (line 7910) → completeCommandExecution (line 7916) → Emits 'command-completed' event waitForCommandCompletion awaits event (zero latency, zero CPU) ``` ### Implementation Steps #### Step 1.1: Add Event Emission (ConsoleManager.ts) **File:** `src/core/ConsoleManager.ts` **Location:** `completeCommandExecution` method (around line 917) ```typescript private completeCommandExecution(commandId: string, exitCode?: number): void { const execution = this.commandExecutions.get(commandId); if (!execution) return; execution.status = 'completed'; execution.exitCode = exitCode; execution.completedAt = new Date(); execution.duration = execution.completedAt.getTime() - execution.startedAt.getTime(); const session = this.sessions.get(execution.sessionId); if (session) { session.executionState = 'idle'; session.lastCommandCompletedAt = new Date(); session.currentCommandId = undefined; } // NEW: Emit completion event this.emit('command-completed', { commandId, exitCode, duration: execution.duration, status: execution.status }); } ``` #### Step 1.2: Replace Polling with Event Await **File:** `src/core/ConsoleManager.ts` **Location:** `waitForCommandCompletion` method (lines 1057-1105) **Replace entire method with:** ```typescript private async waitForCommandCompletion(commandId: string, timeout: number): Promise<{ exitCode?: number; duration: number; status: 'completed' | 'failed' | 'timeout'; }> { const commandExecution = this.commandExecutions.get(commandId); if (!commandExecution) { throw new Error(`Command execution ${commandId} not found`); } const startTime = Date.now(); return new Promise((resolve, reject) => { // Setup timeout const timeoutHandle = setTimeout(() => { this.removeListener('command-completed', completionListener); this.completeCommandExecution(commandId, -1); resolve({ exitCode: -1, duration: Date.now() - startTime, status: 'timeout' }); }, timeout); // Setup completion listener const completionListener = (event: any) => { if (event.commandId === commandId) { clearTimeout(timeoutHandle); this.removeListener('command-completed', completionListener); resolve({ exitCode: event.exitCode, duration: event.duration || (Date.now() - startTime), status: event.status === 'completed' ? 'completed' : 'failed' }); } }; this.on('command-completed', completionListener); // Check if already completed (race condition protection) const currentExecution = this.commandExecutions.get(commandId); if (currentExecution && currentExecution.status !== 'executing') { clearTimeout(timeoutHandle); this.removeListener('command-completed', completionListener); resolve({ exitCode: currentExecution.exitCode, duration: currentExecution.duration || (Date.now() - startTime), status: currentExecution.status === 'completed' ? 'completed' : 'failed' }); } }); } ``` #### Step 1.3: Add Event Types **File:** `src/types/index.ts` ```typescript export interface CommandCompletedEvent { commandId: string; exitCode?: number; duration: number; status: 'completed' | 'failed' | 'timeout'; } ``` ### Testing Phase 1 ```typescript // Test 1: Verify zero latency const start = Date.now(); await executeCommand({ sessionId, command: 'echo test' }); const latency = Date.now() - start; assert(latency < 150); // Should be <150ms (previously 100-200ms due to polling) // Test 2: Verify no CPU waste during idle const cpuBefore = process.cpuUsage(); await sleep(1000); // Wait with active session const cpuAfter = process.cpuUsage(); const cpuUsed = (cpuAfter.user - cpuBefore.user) / 1000; assert(cpuUsed < 10); // Should use <10ms CPU during 1s idle // Test 3: Verify concurrent commands scale const commands = Array(100).fill(0).map(() => executeCommand({ sessionId, command: 'echo test' }) ); await Promise.all(commands); // Should complete without performance degradation ``` --- ## Phase 2: Documentation Fixes **Priority:** HIGH (Prevents User Errors) **Effort:** 6-8 hours **Risk:** Low (documentation only) Same as original plan - fix error messages and tool descriptions. --- ## Phase 3: Smart Execute Implementation **Priority:** MEDIUM-HIGH (Best UX) **Effort:** 12-16 hours **Risk:** Medium (new feature) ### Concept **User Experience:** ```typescript // Single tool, works for everything const result = await executeCommand({ sessionId, command: "any-command-here" }); // System automatically: // - Analyzes command // - Picks best strategy // - Switches mid-execution if needed // - Returns unified result ``` **Under the hood:** Intelligent strategy selection and switching. ### Architecture ``` ┌─────────────────────────────────────────┐ │ console_execute_smart │ │ (New tool - recommended for all users) │ └──────────────┬──────────────────────────┘ │ ▼ ┌──────────────────────┐ │ Smart Executor │ │ - Command Analyzer │ │ - Strategy Selector │ │ - Runtime Monitor │ │ - Result Aggregator │ └──────────┬───────────┘ │ ┌──────────┴─────────────┐ │ │ ▼ ▼ ┌────────────┐ ┌────────────────┐ │ Fast Path │ │ Adaptive Path │ │ (execute) │ │ (monitor → │ │ │ │ switch) │ └────────────┘ └────────────────┘ │ │ │ ┌───────────────────┴─────────────┐ │ │ │ ▼ ▼ ▼ ┌──────────────┐ ┌──────────────────┐ ┌──────────────┐ │ execute_cmd │ │ streaming session│ │ background │ │ │ │ │ │ job │ └──────────────┘ └──────────────────┘ └──────────────┘ ``` ### Smart Executor Algorithm ```typescript class SmartExecutor { async execute(command: string, options: ExecuteOptions): Promise<UnifiedResult> { // Step 1: Analyze command const analysis = this.analyzeCommand(command); // Step 2: Select initial strategy const strategy = this.selectStrategy(analysis); // Step 3: Execute with monitoring if (strategy === 'fast') { return await this.executeFastPath(command, options); } else if (strategy === 'streaming') { return await this.executeStreaming(command, options); } else { // 'background' return await this.executeBackground(command, options); } } private analyzeCommand(command: string): CommandAnalysis { // Known long-running patterns const longRunners = [ /npm\s+(install|ci|update)/, /pip\s+install/, /cargo\s+build/, /mvn\s+clean\s+install/, /gradle\s+build/, /(train|build|compile|bundle|webpack)/i ]; // Known quick commands const quickCommands = [ /^(ls|pwd|echo|cat|head|tail|grep|find)/, /^git\s+(status|log|diff|branch)/ ]; // Known streaming commands const streamingCommands = [ /tail\s+-f/, /watch\s+/, /journalctl\s+-f/ ]; return { isLongRunner: longRunners.some(r => r.test(command)), isQuick: quickCommands.some(r => r.test(command)), isStreaming: streamingCommands.some(r => r.test(command)), estimatedDuration: this.estimateFromHistory(command) }; } private selectStrategy(analysis: CommandAnalysis): 'fast' | 'streaming' | 'background' { if (analysis.isStreaming) return 'streaming'; if (analysis.isLongRunner) return 'background'; if (analysis.isQuick) return 'fast'; // Unknown commands: start with fast path return 'fast'; } private async executeFastPath(command: string, options: ExecuteOptions): Promise<UnifiedResult> { // Start monitoring const monitor = this.createOutputMonitor(); try { // Execute with conservative timeout const result = await this.consoleManager.executeCommandInSession( options.sessionId, command, options.args, options.timeout || 30000 // 30s default ); return { success: true, output: result.output, exitCode: result.exitCode, duration: result.duration, strategyUsed: 'fast', switched: false }; } catch (error) { // Check if we should switch strategies if (monitor.outputSize > 10 * 1024) { // Large output detected - switch to streaming return await this.switchToStreaming(command, options, monitor); } if (error.message.includes('timeout') && monitor.isStillRunning) { // Timeout but still running - switch to background return await this.switchToBackground(command, options, monitor); } throw error; } } private async switchToStreaming( command: string, options: ExecuteOptions, monitor: OutputMonitor ): Promise<UnifiedResult> { // Migration logic: switch from execute to streaming // Both share same session, so can access same output buffer const chunks: string[] = []; let sequenceId = 0; while (true) { const stream = await this.getStream({ sessionId: options.sessionId, since: sequenceId, maxLines: 50 }); chunks.push(...stream.chunks.map(c => c.data)); if (!stream.hasMore) break; sequenceId = stream.nextSequenceId; } return { success: true, output: chunks.join(''), strategyUsed: 'streaming', switched: true, switchReason: 'Large output detected' }; } private async switchToBackground( command: string, options: ExecuteOptions, monitor: OutputMonitor ): Promise<UnifiedResult> { // Migration logic: switch from execute to background job const job = await this.executeAsync({ command, args: options.args, options: { sessionId: options.sessionId, timeout: 600000 // 10 minutes } }); // Poll for completion let status; do { await sleep(1000); status = await this.getJobStatus(job.jobId); } while (status.job.status === 'running'); const output = await this.getJobOutput(job.jobId); return { success: status.job.status === 'completed', output: output.output.map(o => o.content).join(''), exitCode: status.job.exitCode, strategyUsed: 'background', switched: true, switchReason: 'Command exceeded timeout' }; } } ``` ### Implementation Files **New File:** `src/core/SmartExecutor.ts` (300-400 lines) - CommandAnalyzer class - StrategySelector class - RuntimeMonitor class - ResultAggregator class - SmartExecutor orchestrator **New File:** `src/mcp/tools/console_execute_smart.ts` (100-150 lines) - MCP tool definition - Parameter validation - SmartExecutor integration **Modified:** `src/mcp/server.ts` - Add console_execute_smart tool registration - Update error messages to recommend smart execute ### Testing Phase 3 ```typescript // Test 1: Quick command stays fast const result1 = await executeSmart({ command: 'echo test' }); assert(result1.strategyUsed === 'fast'); assert(result1.switched === false); // Test 2: Large output switches to streaming const result2 = await executeSmart({ command: 'cat large-file.log' }); assert(result2.strategyUsed === 'streaming'); assert(result2.switched === true); // Test 3: Long command switches to background const result3 = await executeSmart({ command: 'npm install' }); assert(result3.strategyUsed === 'background'); // Test 4: Known patterns pre-selected correctly const result4 = await executeSmart({ command: 'tail -f log.txt' }); assert(result4.strategyUsed === 'streaming'); assert(result4.switched === false); // Pre-selected, not switched ``` --- ## Migration Guide ### For Existing Users **Before (Manual Choice):** ```typescript // Users must know which pattern to use const result = await executeCommand({ command: 'long-task' }); // Might timeout! // OR const session = await createSession({ command: 'long-task', streaming: true }); const stream = await getStream({ sessionId: session.sessionId }); // ... pagination logic ... // OR const job = await executeAsync({ command: 'long-task' }); // ... polling logic ... ``` **After (Smart Execute):** ```typescript // One tool, works for everything const result = await executeSmart({ command: 'any-command', sessionId: 'my-session' }); // System automatically picks best strategy and returns unified result console.log(result.output); console.log(`Used strategy: ${result.strategyUsed}`); ``` ### Backward Compatibility All existing tools remain unchanged: - `console_execute_command` - still works as before (but now event-driven) - `console_create_session` + streaming - still available for power users - `console_execute_async` - still available for explicit async needs **Recommendation:** New users should prefer `console_execute_smart`, power users can still use specific patterns. --- ## Timeline & Effort ### Phase 1: Event-Driven Completion - Implementation: 4 hours - Testing: 1 hour - Documentation: 1 hour - **Total: 6 hours** ### Phase 2: Documentation Fixes - Error messages: 1 hour - Tool descriptions: 30 minutes - USAGE_PATTERNS.md: 3 hours - Integration guide: 1 hour - Examples: 2 hours - **Total: 7.5 hours** ### Phase 3: Smart Execute - SmartExecutor core: 6 hours - Command analyzer: 2 hours - Strategy switcher: 3 hours - MCP tool integration: 2 hours - Testing: 3 hours - Documentation: 2 hours - **Total: 18 hours** **Grand Total: 31.5 hours** Can be deployed incrementally with value at each phase. --- ## Risk Assessment ### Phase 1 Risks: LOW - Internal refactoring only - Backward compatible - Well-tested event system already exists **Mitigation:** Comprehensive testing of edge cases ### Phase 2 Risks: LOW - Documentation changes only - No code changes **Mitigation:** Clear examples and migration guide ### Phase 3 Risks: MEDIUM - New feature with complex logic - Strategy switching needs careful handling - Edge cases in pattern recognition **Mitigation:** - Extensive testing - Graceful degradation if switching fails - User can override strategy if needed - Deploy as opt-in tool first --- ## Success Metrics ### Phase 1 Success - ✅ Zero polling during idle periods - ✅ <50ms latency in completion detection (down from 100ms) - ✅ 100+ concurrent commands without CPU degradation ### Phase 2 Success - ✅ No users guided to sendInput/getOutput pattern - ✅ Clear documentation for all patterns - ✅ Migration guide available ### Phase 3 Success - ✅ 90%+ commands succeed without manual strategy selection - ✅ Automatic switching works for large outputs - ✅ Users report improved UX - ✅ Reduced support questions about "which tool to use" --- ## Future Enhancements ### Machine Learning Command Analysis - Learn from command execution history - Predict duration/output size more accurately - Adapt to user's specific codebase patterns ### User Hints ```typescript await executeSmart({ command: 'my-command', hints: { expectedDuration: 'long', expectedOutputSize: 'large' } }); ``` ### Strategy Override ```typescript await executeSmart({ command: 'my-command', strategyOverride: 'streaming' // Force specific strategy }); ``` --- ## Conclusion This enhanced plan addresses: 1. ✅ **Architectural inefficiency** - Remove 100ms polling 2. ✅ **User guidance problem** - Fix error messages 3. ✅ **UX complexity** - Intelligent execution **Benefits over documentation-only approach:** - Better performance (event-driven vs polling) - Better UX (one tool vs three choices) - Self-optimizing (learns from patterns) - Still backward compatible **Effort:** 31.5 hours total, deployable incrementally **Risk:** Low-Medium, mitigated through phased deployment This is a production-ready plan that significantly improves both architecture and user experience.