Mnemex

# Prompt Optimization Flow ## Overview This document describes a sophisticated prompt optimization architecture that intercepts, analyzes, enhances, and validates user prompts before they reach Claude. The system uses a multi-stage pipeline involving local LLMs, MCP tool chains, knowledge graph integration, and cloud-based optimization to maximize prompt quality while minimizing API costs. ## Key Benefits - **Zero Initial API Cost**: All optimization happens before hitting paid Claude API endpoints - **Intelligent Complexity Routing**: Simple prompts bypass optimization for speed; complex prompts get full treatment - **Knowledge Graph Integration**: Automatically enriches prompts with relevant context from CortexGraph - **Multi-Model Validation**: Cross-validates optimizations using multiple LLMs to ensure quality - **Flexible Architecture**: Local LLMs can be swapped with cloud providers as needed - **Metadata Enrichment**: Adds confidence scores, similarity metrics, and processing metadata to prompts ## Architecture Components ### 1. **Proxy Server** - Central orchestration layer - Handles routing decisions based on complexity - Manages communication between all components - Tracks confidence/similarity thresholds ### 2. **Local LLMs** - Primary: Prompt optimization and tagging - Validation: Multiple instances for cross-validation - Can be replaced with cloud providers (OpenAI, Anthropic, etc.) ### 3. **MCP Tool Chain** - **CortexGraph**: Knowledge graph for context retrieval - **STOPPER**: Process control and validation - **Custom Tools**: User-defined extensions - **Gemini Optimizer**: Large context window for final assembly ### 4. **Validation Layer** - Semantic similarity checks - Confidence scoring - Iterative refinement below thresholds ## Detailed Flow Description ### Phase 1: Initial Intake 1. **User Input**: User enters prompt in Claude Code interface 2. **Proxy Intercept**: Proxy captures the prompt before it reaches Claude 3. **Complexity Analysis**: NLP-based complexity rating determines routing strategy ### Phase 2: Intelligent Routing 4. **Simple Path** (Low Complexity): - Proxy applies basic formatting rules - Routes directly to Claude with minimal processing - Optimizes for speed and reduces overhead 5. **Complex Path** (High Complexity): - Triggers full optimization pipeline - Proceeds to Phase 3 ### Phase 3: Prompt Optimization 6. **Local LLM Processing**: - Adds semantic tags to categorize intent - Restructures prompt for optimal Claude comprehension - Formats according to Claude best practices - Extracts key entities and concepts ### Phase 4: Validation & Refinement 7. **Multi-Model Validation**: - Routes optimized prompt to 2-n additional local LLMs - Each validator scores the optimization independently - Can use semantic similarity algorithms instead of LLMs - Calculates confidence and similarity metrics 8. **Threshold Check**: - If scores meet threshold: Proceed to Phase 5 - If scores below threshold: Return to Phase 3 for reprocessing - Prevents low-quality optimizations from proceeding 9. **Tool Recommendation**: - Proxy receives validated prompt with metadata - System suggests relevant MCP tools for the query ### Phase 5: MCP Tool Chain Execution 10. **CortexGraph Search**: - Searches knowledge graph for related concepts - Retrieves relevant memories and context - Returns similarity-scored results 11. **STOPPER Validation**: - Process control checks - Safety and constraint validation - Prevents out-of-scope operations 12. **Additional Tools**: - Routes to n other tools based on user preferences - Each tool contributes specialized context - Tools run in parallel for efficiency ### Phase 6: Final Assembly 13. **Gemini Optimization**: - Combines original prompt + optimizations + tool outputs - Leverages Gemini's large context window (2M tokens) - Uses generous free tier for cost optimization - Assembles coherent final prompt 14. **Quality Assurance**: - Compares input to assembled output - Generates similarity score (drift detection) - Calculates final confidence rating - Appends metadata to prompt ### Phase 7: Claude Execution 15. **Final Prompt Delivery**: - Proxy sends optimized prompt to Claude - **First API cost incurred at this step** - Prompt includes: - Original user intent (preserved) - Optimization tags and structure - Knowledge graph context - Tool outputs and recommendations - Confidence/similarity metadata - Processing history 16. **Normal Operation**: - Claude processes the enriched prompt - Claude Code continues standard workflow - User receives high-quality response ## Sequence Diagram \`\`\`mermaid sequenceDiagram actor User participant Claude Code Interface participant Proxy participant NLP Complexity Analyzer participant Local LLM (Optimizer) participant Local LLM 2 (Validator) participant Local LLM N (Validator) participant Semantic Similarity Engine participant MCP Chain participant CortexGraph participant STOPPER participant Custom Tools participant Gemini participant Claude API %% Phase 1: Initial Intake User->>Claude Code Interface: Enter prompt Claude Code Interface->>Proxy: Forward prompt Proxy->>NLP Complexity Analyzer: Analyze complexity NLP Complexity Analyzer-->>Proxy: Complexity rating %% Phase 2: Routing Decision alt Low Complexity (Simple Prompt) Proxy->>Proxy: Apply basic rules Proxy->>Claude API: Route directly to Claude Note over Proxy,Claude API: Fast path for simple queries else High Complexity (Complex Prompt) Note over Proxy: Trigger full optimization pipeline %% Phase 3: Optimization Proxy->>Local LLM (Optimizer): Optimize prompt Note over Local LLM (Optimizer): - Add semantic tags<br/>- Format for Claude<br/>- Extract entities<br/>- Restructure query Local LLM (Optimizer)-->>Proxy: Optimized prompt v1 %% Phase 4: Validation Loop rect rgb(240, 240, 240) Note over Proxy,Semantic Similarity Engine: Validation & Refinement Loop par Parallel Validation Proxy->>Local LLM 2 (Validator): Validate optimization Proxy->>Local LLM N (Validator): Validate optimization Proxy->>Semantic Similarity Engine: Check semantic similarity end Local LLM 2 (Validator)-->>Proxy: Confidence score 2 Local LLM N (Validator)-->>Proxy: Confidence score N Semantic Similarity Engine-->>Proxy: Similarity score Proxy->>Proxy: Aggregate scores alt Below Confidence/Similarity Threshold Note over Proxy,Local LLM (Optimizer): Quality check failed Proxy->>Local LLM (Optimizer): Reprocess with feedback Local LLM (Optimizer)-->>Proxy: Optimized prompt v2 Note over Proxy: Loop until threshold met else Above Threshold Note over Proxy: Quality validated, proceed end end Proxy->>Proxy: Append recommendation metadata %% Phase 5: MCP Tool Chain Proxy->>MCP Chain: Route validated prompt + metadata rect rgb(230, 245, 255) Note over MCP Chain,Custom Tools: MCP Tool Execution (Parallel) par Tool Execution MCP Chain->>CortexGraph: Search knowledge graph MCP Chain->>STOPPER: Validate constraints MCP Chain->>Custom Tools: Execute user-defined tools end CortexGraph-->>MCP Chain: Context + memories (similarity scored) STOPPER-->>MCP Chain: Validation results Custom Tools-->>MCP Chain: Tool outputs end %% Phase 6: Final Assembly MCP Chain->>Gemini: Assemble final prompt Note over Gemini: - Combine all inputs<br/>- Optimize structure<br/>- 2M token context<br/>- Free tier usage Gemini->>Gemini: Compare input vs output Gemini->>Gemini: Calculate similarity & confidence Gemini-->>MCP Chain: Final prompt + metadata MCP Chain-->>Proxy: Return final prompt %% Phase 7: Claude Execution Note over Proxy,Claude API: 💰 First API cost incurred here Proxy->>Claude API: Send final optimized prompt Note over Claude API: Prompt includes:<br/>- Original intent<br/>- Optimizations<br/>- Knowledge graph context<br/>- Tool outputs<br/>- Metadata end %% Normal Operation Claude API-->>Claude Code Interface: Process request Claude Code Interface-->>User: Return response Note over User,Claude Code Interface: Claude Code continues as normal \`\`\` ## Configuration Options ### Complexity Thresholds \`\`\`python # Proxy configuration # Prompts with complexity > COMPLEX_PROMPT_THRESHOLD follow the complex path, otherwise the simple path is used. COMPLEX_PROMPT_THRESHOLD = 0.4 \`\`\` ### Validation Settings \`\`\`python # Validation thresholds CONFIDENCE_THRESHOLD = 0.75 # Minimum confidence to proceed SIMILARITY_THRESHOLD = 0.80 # Minimum semantic similarity MAX_REFINEMENT_ITERATIONS = 3 # Prevent infinite loops \`\`\` ### Model Selection \`\`\`python # Local LLMs (can be replaced with cloud providers) OPTIMIZER_MODEL = "llama-3.1-70b" # Primary optimizer VALIDATOR_MODELS = [ # Validation ensemble "mixtral-8x7b", "qwen-2.5-72b", "deepseek-v2" ] # Example using cloud providers (alternative to local) # OPTIMIZER_MODEL = "openai:gpt-4" # VALIDATOR_MODELS = ["anthropic:claude-3-opus", "openai:gpt-4"] \`\`\` ### MCP Tools \`\`\`python # Tool chain configuration MCP_TOOLS = { "cortex_graph": { "enabled": True, "similarity_threshold": 0.7, "max_results": 10 }, "stopper": { "enabled": True, "strict_mode": False }, "custom": { "user_preferences": True, "context_retrieval": True } } \`\`\` ### Gemini Settings \`\`\`python # Final assembly configuration GEMINI_MODEL = "gemini-2.0-flash-exp" # Free tier, large context GEMINI_MAX_TOKENS = 2000000 # 2M token context window GEMINI_TEMPERATURE = 0.3 # Consistent assembly \`\`\` ## Performance Characteristics ### Latency Profile | Stage | Estimated Time | Notes | |-------|---------------|-------| | Complexity Analysis | 10-50ms | Fast NLP classification | | Simple Path (total) | 50-100ms | Minimal processing overhead | | Optimization | 200-500ms | Local LLM inference | | Validation | 150-300ms | Parallel execution | | MCP Tool Chain | 100-400ms | Depends on tool complexity | | Gemini Assembly | 300-800ms | Large context processing | | **Complex Path (total)** | **1-3 seconds** | Full pipeline | ### Cost Analysis **Traditional Approach** (direct to Claude): - Every prompt hits Claude API immediately - No optimization or context enrichment - Cost: $X per request from first token **Optimized Approach** (this architecture): - Local LLMs: Free (self-hosted) or cheap (cloud) - Gemini: Leverages the generous free tier for final assembly - Claude API: Only hit after full optimization - Cost: $0 until Claude execution, then same $X but better results **Net Effect**: - Same Claude API cost per request - Significantly better prompt quality - Higher success rate (fewer retries needed) - Lower total cost due to reduced iterations ## Implementation Considerations ### 1. **Local LLM Requirements** - GPU: RTX 4090 or better for 70B models - RAM: 64GB+ recommended - Alternative: Use cloud inference APIs (Groq, Together.ai, OpenRouter) ### 2. **Proxy Server** - Needs to be MCP-compatible - Should support WebSocket for streaming - Must handle concurrent validation requests ### 3. **Knowledge Graph Integration** - CortexGraph needs to be populated with relevant data - Index must be kept up-to-date - Consider using CortexGraph for temporal memory ### 4. **Error Handling** - Fallback to simple path if optimization fails - Timeout protection (max 5s total processing) - Graceful degradation if tools unavailable ### 5. **Monitoring & Observability** - Track optimization success rates - Monitor confidence/similarity distributions - Log processing times for each stage - A/B test optimized vs non-optimized prompts ## Future Enhancements 1. **Adaptive Thresholds**: Learn optimal confidence/similarity thresholds per user 2. **Caching Layer**: Cache optimizations for similar prompts 3. **User Feedback Loop**: Incorporate user ratings to improve optimization 4. **Model Selection**: Automatically choose best LLM based on prompt type 5. **Streaming Optimization**: Stream partial results during processing 6. **Cost Tracking**: Detailed cost accounting per stage 7. **A/B Testing Framework**: Compare different optimization strategies ## Security Considerations - **Prompt Injection**: Validate all optimized prompts for injection attempts - **Data Privacy**: Local LLMs keep sensitive data on-premise - **Rate Limiting**: Prevent abuse of free tier services - **Access Control**: Authenticate proxy requests - **Audit Trail**: Log all prompt transformations ## Related Documentation - [CortexGraph Architecture](architecture.md) - Integration with temporal memory - [CortexGraph Documentation](graph_features.md) - Knowledge graph features - [MCP Specification](https://github.com/modelcontextprotocol/specification) - Tool protocol details - [Prompt Injection Prevention](prompt_injection.md) - Security best practices ## Example Workflow ### Input Prompt \`\`\` "Help me write a Python function to process user data" \`\`\` ### After Optimization \`\`\`markdown ## Task: Python Function Development **User Intent**: Create data processing function **Context** (from CortexGraph): - User prefers type hints (from memory: 2025-10-15) - Uses pytest for testing (from memory: 2025-10-20) - Prefers dataclasses over dicts (from memory: 2025-10-12) **Requirements**: 1. Function should process user data 2. Follow user's Python style preferences 3. Include type hints and docstrings 4. Consider testing approach **Metadata**: - Confidence: 0.87 - Similarity: 0.92 - Optimization iterations: 1 - Tools used: CortexGraph, STOPPER - Processing time: 1.2s \`\`\` ### Result Claude receives a rich, contextualized prompt that produces higher-quality output on the first try, reducing the need for follow-up iterations. --- **Built with** [Claude Code](https://claude.com/claude-code) 🤖