MCP Puppet Pipeline

# Comprehensive Remediation Checklist ## Credential Rotation and Schema Alignment **Task ID:** MCP-001 **Date:** September 25, 2025 **Status:** Ready for Implementation --- ## Executive Summary This checklist addresses critical authentication failures and schema misalignment issues identified in the MCP Server. Current problems include: - HTTP 401 "Invalid credentials" errors from Affogato API - Missing Notion database configuration and schema validation - Inconsistent environment variable loading across services - Lack of authentication failure handling and recovery mechanisms --- ## 1. AFFOGATO API CREDENTIAL REFRESH AND ENVIRONMENT LOADING FIXES ### Priority: CRITICAL **Current Issue:** HTTP 401 errors with Affogato API despite valid key in secrets ### Implementation Steps: #### 1.1 Credential Validation and Refresh **File:** `src/integrations/affogato-client.js` **Steps:** 1. **Add credential validation method:** ```javascript // Add to AffogatoClient class async validateCredentials() { try { const response = await this.makeApiRequest('/pub/v1/account', null, 'GET'); console.log('✅ Affogato credentials valid'); return { valid: true, account: response }; } catch (error) { console.error('❌ Affogato credentials invalid:', error.message); return { valid: false, error: error.message }; } } ``` 2. **Implement automatic credential refresh logic:** ```javascript async refreshCredentials() { console.log('🔄 Attempting to refresh Affogato credentials...'); // Check if we have a refresh token or need manual intervention const validation = await this.validateCredentials(); if (!validation.valid) { throw new Error('Credential refresh failed. Manual intervention required.'); } return validation; } ``` 3. **Add credential check before each API call:** ```javascript async makeApiRequest(endpoint, data, method = 'POST') { // Add validation check at start of method if (!this.apiKey) { throw new Error('Affogato API key not configured'); } // Existing implementation... // Add retry logic for 401 errors } ``` **Verification Steps:** - [ ] Run `npm test -- --grep "affogato-credentials"` - [ ] Verify console shows "✅ Affogato credentials valid" - [ ] Confirm no 401 errors in logs after implementation #### 1.2 Environment Loading Consolidation **Files:** `src/config.ts`, `src/integrations/production-orchestrator.js` **Steps:** 1. **Create centralized environment configuration:** ```typescript // Update src/config.ts const configSchema = z.object({ // Existing configs... AFFOGATO_API_KEY: z.string().min(1, 'Affogato API key required'), ANTHROPIC_API_KEY: z.string().min(1, 'Anthropic API key required'), OPENAI_API_KEY: z.string().min(1, 'OpenAI API key required'), ELEVENLABS_API_KEY: z.string().optional(), }); ``` 2. **Update all service constructors to use centralized config:** ```javascript // Update production-orchestrator.js import { CONFIG } from '../config.ts'; constructor() { this.characterPipeline = new CharacterProductionPipeline(CONFIG.AFFOGATO_API_KEY); this.openai = new OpenAI({ apiKey: CONFIG.OPENAI_API_KEY }); } ``` **Verification Steps:** - [ ] All API keys load from single config source - [ ] No undefined API key errors in console - [ ] Environment validation passes on startup #### 1.3 API Key Rotation Management **New File:** `src/utils/credential-manager.js` **Steps:** 1. **Create credential rotation system:** ```javascript export class CredentialManager { constructor() { this.credentials = new Map(); this.rotationCallbacks = new Map(); } async rotateCredential(service, newCredential) { // Validate new credential // Update environment // Notify all dependent services // Log rotation event } onCredentialRotation(service, callback) { // Register callback for when credentials change } } ``` **Verification Steps:** - [ ] Test credential rotation with dummy keys - [ ] Verify dependent services receive rotation notifications - [ ] Confirm production services restart with new credentials --- ## 2. NOTION DATABASE SCHEMA RECONCILIATION ### Priority: HIGH **Current Issue:** Missing Notion API configuration and schema validation ### Implementation Steps: #### 2.1 Notion Connection Establishment **File:** `src/integrations/character-pipeline.js` **Steps:** 1. **Fix Notion client initialization:** ```javascript async getNotionClient() { try { // Validate Replit connector settings first const connectorStatus = await this.validateConnectorSettings(); if (!connectorStatus.valid) { throw new Error(`Notion connector invalid: ${connectorStatus.error}`); } // Existing connection logic... // Test connection immediately await this.testNotionConnection(); return this.notion; } catch (error) { console.error('❌ Notion connection failed:', error.message); throw new Error(`Notion setup failed: ${error.message}`); } } ``` 2. **Add connection validation:** ```javascript async validateConnectorSettings() { const hostname = process.env.REPLIT_CONNECTORS_HOSTNAME; const xReplitToken = process.env.REPL_IDENTITY ? 'repl ' + process.env.REPL_IDENTITY : null; if (!hostname || !xReplitToken) { return { valid: false, error: 'Replit connector environment not configured' }; } return { valid: true }; } ``` **Verification Steps:** - [ ] Notion client initializes without errors - [ ] Connection test passes successfully - [ ] Access token retrieved and validated #### 2.2 Database Schema Validation and Creation **File:** `src/integrations/database-schema-enhancer.js` **Steps:** 1. **Add schema validation method:** ```javascript async validateDatabaseSchema(databaseId) { try { const database = await this.notion.databases.retrieve({ database_id: databaseId }); const requiredFields = this.getRequiredFields(); const missingFields = []; const incorrectTypes = []; for (const [fieldName, expectedConfig] of Object.entries(requiredFields)) { const existingField = database.properties[fieldName]; if (!existingField) { missingFields.push(fieldName); } else if (existingField.type !== expectedConfig.type) { incorrectTypes.push({ field: fieldName, expected: expectedConfig.type, actual: existingField.type }); } } return { valid: missingFields.length === 0 && incorrectTypes.length === 0, missingFields, incorrectTypes, database }; } catch (error) { return { valid: false, error: error.message }; } } ``` 2. **Add automatic schema creation/update:** ```javascript async ensureSchemaCompliance(databaseId) { const validation = await this.validateDatabaseSchema(databaseId); if (!validation.valid) { console.log('🔧 Database schema needs updates...'); await this.updateDatabaseSchema(databaseId, validation.missingFields); // Re-validate after updates const revalidation = await this.validateDatabaseSchema(databaseId); if (!revalidation.valid) { throw new Error('Schema update failed - manual intervention required'); } console.log('✅ Database schema updated successfully'); } return validation; } ``` **Verification Steps:** - [ ] Schema validation runs without errors - [ ] Missing fields are identified correctly - [ ] Schema updates apply successfully - [ ] Re-validation confirms compliance #### 2.3 Character Property Mapping **File:** `src/integrations/database-schema-enhancer.js` **Steps:** 1. **Create property mapping validation:** ```javascript validateCharacterProperties(characterData) { const required = ['Character Name', 'Character Bio', 'Character Type']; const missing = []; const invalid = []; for (const field of required) { if (!characterData.properties[field]) { missing.push(field); } } return { valid: missing.length === 0 && invalid.length === 0, missing, invalid }; } ``` 2. **Add pre-creation validation:** ```javascript async createCharacterWithValidation(characterData, traits) { // Validate required database exists const dbValidation = await this.ensureSchemaCompliance(process.env.CHARACTERS_MASTER_DB); // Create properties with validation const properties = await this.createCharacterWithTraits(characterData, traits); // Validate properties before creation const propValidation = this.validateCharacterProperties({ properties }); if (!propValidation.valid) { throw new Error(`Character properties invalid: ${propValidation.missing.join(', ')}`); } return properties; } ``` **Verification Steps:** - [ ] Character property validation passes - [ ] Required fields are enforced - [ ] Type validation works correctly - [ ] Character creation succeeds with valid data --- ## 3. AUTHENTICATION FAILURE HANDLING IMPROVEMENTS ### Priority: HIGH **Current Issue:** No retry mechanisms or graceful failure handling ### Implementation Steps: #### 3.1 Retry Mechanism Implementation **File:** `src/integrations/affogato-client.js` **Steps:** 1. **Add retry wrapper for API calls:** ```javascript async makeApiRequestWithRetry(endpoint, data, method = 'POST', maxRetries = 3) { let lastError; for (let attempt = 1; attempt <= maxRetries; attempt++) { try { console.log(`🔄 API attempt ${attempt}/${maxRetries}: ${method} ${endpoint}`); const result = await this.makeApiRequest(endpoint, data, method); console.log(`✅ API success on attempt ${attempt}`); return result; } catch (error) { lastError = error; console.log(`⚠️ API attempt ${attempt} failed: ${error.message}`); if (error.message.includes('401') && attempt < maxRetries) { console.log('🔑 Attempting credential refresh...'); await this.refreshCredentials(); } if (attempt < maxRetries) { const delay = Math.pow(2, attempt) * 1000; // Exponential backoff await this.delay(delay); } } } throw new Error(`API failed after ${maxRetries} attempts: ${lastError.message}`); } ``` 2. **Update all API calls to use retry mechanism:** ```javascript // Replace existing makeApiRequest calls async generatePuppetImages(characterId, emotions, characterTraits = null) { // ... existing setup ... for (const emotion of emotionList) { try { const response = await this.makeApiRequestWithRetry('/pub/v1/generations', [data]); // ... rest of implementation } catch (error) { console.log(` ⚠️ ${emotion} expression failed after retries: ${error.message}`); // Graceful degradation logic } } } ``` **Verification Steps:** - [ ] Retry mechanism triggers on 401 errors - [ ] Exponential backoff delays work correctly - [ ] Successful retry after credential refresh - [ ] Final failure logged appropriately #### 3.2 Circuit Breaker Pattern **New File:** `src/utils/circuit-breaker.js` **Steps:** 1. **Implement circuit breaker for external APIs:** ```javascript export class CircuitBreaker { constructor(name, failureThreshold = 5, timeout = 60000) { this.name = name; this.failureThreshold = failureThreshold; this.timeout = timeout; this.state = 'CLOSED'; // CLOSED, OPEN, HALF_OPEN this.failureCount = 0; this.lastFailureTime = null; } async execute(operation) { if (this.state === 'OPEN') { if (Date.now() - this.lastFailureTime >= this.timeout) { this.state = 'HALF_OPEN'; } else { throw new Error(`Circuit breaker ${this.name} is OPEN`); } } try { const result = await operation(); this.onSuccess(); return result; } catch (error) { this.onFailure(); throw error; } } } ``` 2. **Integrate circuit breaker with Affogato client:** ```javascript // Add to AffogatoClient constructor constructor(apiKey) { // ... existing code ... this.circuitBreaker = new CircuitBreaker('affogato-api', 3, 30000); } async makeApiRequestWithRetry(endpoint, data, method = 'POST', maxRetries = 3) { return this.circuitBreaker.execute(async () => { return this.makeApiRequestWithRetryInternal(endpoint, data, method, maxRetries); }); } ``` **Verification Steps:** - [ ] Circuit breaker opens after threshold failures - [ ] Circuit breaker transitions to half-open after timeout - [ ] Successful calls reset circuit breaker state - [ ] Circuit breaker logs state changes #### 3.3 Graceful Degradation **File:** `src/integrations/production-orchestrator.js` **Steps:** 1. **Add fallback strategies:** ```javascript async processCharacterImageWithFallback(imagePath, characterName) { try { return await this.processCharacterImage(imagePath, characterName); } catch (error) { console.log('⚠️ Primary processing failed, attempting fallback...'); if (error.message.includes('Affogato')) { return await this.processCharacterImageOffline(imagePath, characterName); } throw error; } } async processCharacterImageOffline(imagePath, characterName) { console.log('🔄 Using offline character processing...'); // Create character record without generated images // Store for later processing when API is available return { success: true, character_name: characterName, status: 'offline_processing', retry_required: true }; } ``` **Verification Steps:** - [ ] Fallback triggers when primary method fails - [ ] Offline processing creates valid character records - [ ] Retry queue populated for later processing - [ ] User receives meaningful status updates --- ## 4. END-TO-END VALIDATION PROCEDURES ### Priority: MEDIUM **Current Issue:** No comprehensive validation pipeline ### Implementation Steps: #### 4.1 Environment Validation Suite **New File:** `src/utils/environment-validator.js` **Steps:** 1. **Create comprehensive environment check:** ```javascript export class EnvironmentValidator { static async validateComplete() { const results = { environment: await this.validateEnvironmentVariables(), services: await this.validateServices(), databases: await this.validateDatabases(), integrations: await this.validateIntegrations() }; const overallValid = Object.values(results).every(r => r.valid); return { valid: overallValid, results, summary: this.generateSummary(results) }; } static async validateServices() { const services = { affogato: await this.validateAffogatoAPI(), notion: await this.validateNotionConnection(), openai: await this.validateOpenAI() }; return { valid: Object.values(services).every(s => s.valid), services }; } } ``` 2. **Add startup validation:** ```javascript // Add to src/server.ts import { EnvironmentValidator } from './utils/environment-validator.js'; async function startServer() { console.log('🔍 Validating environment...'); const validation = await EnvironmentValidator.validateComplete(); if (!validation.valid) { console.error('❌ Environment validation failed:'); console.error(validation.summary); process.exit(1); } console.log('✅ Environment validation passed'); // Continue with server startup... } ``` **Verification Steps:** - [ ] Environment validation runs on startup - [ ] All service connections tested - [ ] Clear error messages for failures - [ ] Server refuses to start with invalid environment #### 4.2 Health Check Endpoints **File:** `src/server.ts` **Steps:** 1. **Add health check routes:** ```javascript app.get('/health', async (req, res) => { try { const validation = await EnvironmentValidator.validateComplete(); res.status(validation.valid ? 200 : 503).json({ status: validation.valid ? 'healthy' : 'unhealthy', timestamp: new Date().toISOString(), checks: validation.results }); } catch (error) { res.status(500).json({ status: 'error', error: error.message }); } }); app.get('/health/deep', async (req, res) => { // Detailed health checks including API calls const checks = await this.performDeepHealthChecks(); res.json(checks); }); ``` **Verification Steps:** - [ ] Health endpoint returns correct status - [ ] Deep health checks test actual API calls - [ ] Health status reflects current system state - [ ] Health checks complete within timeout #### 4.3 Integration Test Suite **New File:** `tests/integration/full-pipeline.test.js` **Steps:** 1. **Create end-to-end test:** ```javascript describe('Full Character Pipeline Integration', () => { test('should process character from image to Notion', async () => { const orchestrator = new ProductionOrchestrator(); // Test image processing const result = await orchestrator.processCharacterImage( './test-assets/sample-character.jpg', 'Test Character' ); expect(result.success).toBe(true); expect(result.notion_page_id).toBeDefined(); expect(result.blueprint_images).toBeDefined(); // Verify Notion record const notionClient = await orchestrator.characterPipeline.getNotionClient(); const page = await notionClient.pages.retrieve({ page_id: result.notion_page_id }); expect(page.properties['Character Name']).toBeDefined(); }); }); ``` **Verification Steps:** - [ ] Integration tests pass consistently - [ ] Tests cover complete workflow - [ ] Test cleanup removes test data - [ ] Tests run in isolated environment --- ## VERIFICATION CHECKPOINTS ### Phase 1: Credential Fixes (Days 1-2) - [ ] Affogato API returns 200 for credential validation - [ ] Environment variables load consistently - [ ] API key rotation system functional - [ ] Zero 401 errors in production logs ### Phase 2: Schema Alignment (Days 3-4) - [ ] Notion connection established successfully - [ ] Database schema validation passes - [ ] Character creation succeeds with all fields - [ ] Property mapping validates correctly ### Phase 3: Error Handling (Days 5-6) - [ ] Retry mechanisms handle temporary failures - [ ] Circuit breaker prevents cascade failures - [ ] Graceful degradation maintains service - [ ] Error logs provide actionable information ### Phase 4: End-to-End Validation (Days 7) - [ ] Environment validation passes on startup - [ ] Health checks return accurate status - [ ] Integration tests pass consistently - [ ] Full pipeline processes characters successfully --- ## SUCCESS CRITERIA **Primary Goals:** 1. **Zero authentication failures** in production logs 2. **100% schema compliance** for character database 3. **< 5 second recovery time** from transient failures 4. **95% success rate** for character processing pipeline **Secondary Goals:** 1. Comprehensive monitoring and alerting 2. Automated credential rotation 3. Self-healing system capabilities 4. Complete test coverage for critical paths --- ## ROLLBACK PROCEDURES If any implementation causes system instability: 1. **Immediate Rollback:** ```bash git revert <commit-hash> npm run restart-services ``` 2. **Service-Specific Rollback:** - Disable affected integrations - Switch to offline processing mode - Alert operations team 3. **Data Recovery:** - Restore from latest backup - Verify data integrity - Restart validation pipeline --- ## POST-IMPLEMENTATION MONITORING **Key Metrics to Track:** - API success rates per service - Authentication failure frequency - Schema validation results - End-to-end processing times - Error recovery success rates **Alerting Thresholds:** - More than 3 authentication failures per hour - Schema validation failure - Circuit breaker open state - Health check failures --- **Document Version:** 1.0 **Last Updated:** September 25, 2025 **Next Review:** October 2, 2025