MCP Puppet Pipeline

# Overview This is a Model Context Protocol (MCP) server that provides autonomous puppet character production using a hybrid AI workflow. The server combines OpenAI Vision and DALL-E 3 for core puppet creation, Affogato for character consistency across scenes, and ElevenLabs for voice video generation. Built with TypeScript and Node.js, the system offers quantitative quality control, systematic file organization, and end-to-end automation for puppet character development. The server integrates with Claude Desktop via MCP protocol and includes legacy AI coordination tools for multi-agent collaboration. # User Preferences Preferred communication style: Simple, everyday language. # System Architecture ## Core Transport Layer - **WebSocket Server**: Uses the `ws` library to provide real-time bidirectional communication on configurable port (default 3000) - **JSON-RPC 2.0 Protocol**: Implements MCP specification using `jsonrpc-lite` for structured message handling - **Connection Management**: Includes rate limiting, ping/pong keepalive, and connection tracking for stability ## Tool Architecture The server exposes four main tool categories through the MCP interface: ### Hybrid Puppet Production Pipeline - **Core OpenAI Workflow**: OpenAI Vision analyzes reference images → DALL-E 3 generates puppet images with fixed constants (angles, emotions, mouth states) - **Character Consistency**: Affogato creates persistent character profiles from best puppet images for scene-based generations - **Voice Video Creation**: ElevenLabs text-to-speech integration with Affogato's narrator feature for animated videos - **Quantitative QC System**: Threshold-based scoring with automatic retry mechanisms for failed generations - **Systematic Organization**: Structured pipeline with 01_input through 06_delivery directories ### Production Tools Available via MCP - **puppet_production_pipeline**: Core OpenAI DALL-E 3 puppet creation with QC validation - **hybrid_puppet_pipeline**: Full workflow (OpenAI → Affogato character → scene generation → voice videos) - **create_scene_image**: Individual scene generation using existing Affogato characters - **create_voice_video**: Individual ElevenLabs voice video creation from images - **puppet_pipeline_status**: Configuration and integration status checker ### AI Coordination Infrastructure (Legacy) - **Multi-AI Communication**: Structured conversation sessions between Claude, OpenAI, and Replit - **Task Assignment System**: Inter-AI task delegation with progress tracking and status management - **Planning Session Management**: Collaborative planning sessions with decision tracking and consensus building - **Database-Backed Persistence**: PostgreSQL storage for conversation history, task assignments, and planning outcomes - **Session Management**: Unique session IDs for organizing multi-AI collaborative work ### Local LLM Integration (Legacy) - **Dual Provider Support**: Connects to both Ollama (localhost:11434) and LM Studio (localhost:1234) services - **Flexible API Compatibility**: Supports both Ollama's native API and OpenAI-compatible endpoints - **Configuration-Driven**: Uses environment variables to toggle between /api/chat and /api/generate for Ollama ### Code Execution System (Legacy) - **Security-First Design**: Implements executable whitelisting (node, python, npm by default) - **Process Management**: Uses Node.js `spawn` with timeout controls and proper signal handling - **Resource Isolation**: Runs commands in specified working directories with configurable timeouts ## Configuration Management - **Environment-Based**: Uses `dotenv` for configuration with sensible defaults - **Zod Validation**: Strict schema validation for all inputs and configuration - **Type Safety**: Full TypeScript coverage with proper type inference ## Security Measures - **Command Whitelisting**: Only predefined executables can be run through the code execution tool - **Input Validation**: All tool inputs are validated using Zod schemas before execution - **Timeout Protection**: All operations include configurable timeout mechanisms - **URL Filtering**: Planned support for URL allowlisting for HTTP fetch operations ## Error Handling - **Graceful Degradation**: Tools return structured error messages rather than crashing - **Connection Resilience**: WebSocket connections include automatic cleanup and error recovery - **Logging**: Comprehensive logging for debugging and monitoring # External Dependencies ## Core Runtime Dependencies - **axios**: HTTP client for making requests to Ollama and LM Studio APIs - **ws**: WebSocket server implementation for MCP transport layer - **jsonrpc-lite**: JSON-RPC 2.0 protocol implementation - **zod**: Runtime type validation and schema enforcement - **dotenv**: Environment variable configuration management - **pg**: PostgreSQL client for AI coordination database operations - **@types/pg**: PostgreSQL TypeScript type definitions ## Development Dependencies - **TypeScript**: Type system and compilation - **tsx**: TypeScript execution for development - **@types/node**: Node.js type definitions - **@types/ws**: WebSocket type definitions ## External Services - **Ollama**: Local LLM inference service (http://localhost:11434) - **LM Studio**: Alternative local LLM service with OpenAI-compatible API (http://localhost:1234) ## System Requirements - **Node.js**: Version 18.0.0 or higher - **PostgreSQL Database**: Connected via DATABASE_URL environment variable for AI coordination - **Local LLM Services**: At least one of Ollama or LM Studio must be running locally # AI Coordination Features ## Available MCP Tools for AI Communication 1. **initialize_ai_conversation** - Start structured AI collaboration sessions 2. **ai_send_message** - Enable AIs to communicate with structured message types 3. **ai_planning_session** - Coordinate collaborative planning and decision-making 4. **ai_task_assignment** - Delegate tasks between AI participants with tracking 5. **get_ai_conversation_status** - Monitor conversation progress and active tasks 6. **ai_task_update** - Update task status and progress ## Database Schema for AI Coordination - **ai_conversations** - Conversation sessions with orchestrator, participants, and objectives - **ai_conversation_messages** - All AI-to-AI messages with types (discussion, proposal, agreement, etc.) - **ai_planning_sessions** - Collaborative planning sessions with decisions and action items - **ai_task_assignments** - Task delegation between AIs with status tracking - **ai_consensus_tracking** - Agreement and decision tracking across participants ## Usage Flow 1. Initialize AI conversation with Claude as orchestrator 2. AIs communicate through structured message types 3. Planning sessions coordinate collaborative design work 4. Tasks are assigned and tracked between participants 5. Progress is monitored and decisions are recorded