MCP PyBoy Emulator Server

# MCP PyBoy MVP Implementation Roadmap ## Overview This document provides a step-by-step implementation guide for building the MCP PyBoy Emulator Server MVP. Designed for solo development with Claude Code assistance, each task includes specific instructions for effective LLM collaboration. ## Progress Tracking - **Total Tasks**: 24 (streamlined from original 45) - **Estimated Time**: 2 weeks (solo developer) - **Current Progress**: Phase 1 completed, Phase 2.4 (Session Management) completed, Phase 2.5 (Web Frontend) in progress --- ## Phase 1: Project Foundation (Days 1-2) ### Project Setup - [x] **1.1** Create Python project structure - [x] Set up `pyproject.toml` with dependencies - [x] Create `src/` directory structure - [x] Initialize git repository - **Claude Code Tip**: "Create a new Python project with pyproject.toml using the dependencies from our technical architecture document" - [x] **1.2** Set up development environment - [x] Install uv for fast dependency management - [x] Create virtual environment with .python-version - [x] Install core dependencies (PyBoy, MCP, etc.) - [x] Install development dependencies (pytest, black, ruff, mypy) - **Claude Code Tip**: "Help me set up a Python development environment for this project with proper dependency management" - [x] **1.3** Complete project documentation and IDE setup - [x] Update README.md with comprehensive setup instructions - [x] Verify CLAUDE.md has complete uv workflow documentation - [x] Ensure all architecture documents are properly organized in docs/ - [x] Configure VS Code settings for Black/Ruff integration - [x] Verify IDE extensions and toolchain integration - [x] Fix all formatting and linting issues - [x] Add py.typed marker for type checking - **Claude Code Tip**: "Update the README with detailed setup instructions and verify all IDE tooling works seamlessly" - [x] **1.4** Establish development workflow - [x] Set up pre-commit hooks for automated quality checks - [x] Create comprehensive test directory structure - [x] Add test fixtures for mock PyBoy instances - [x] Create comprehensive test fixtures and conftest.py - [x] Implement mock PyBoy classes for testing - [x] Verify all dev tools work together (black, ruff, mypy, pytest) - [x] Test complete development workflow end-to-end - **Claude Code Tip**: "Create a robust development workflow with pre-commit hooks and comprehensive testing infrastructure" --- ## Phase 2: Core PyBoy Integration and Basic MCP Server (Days 3-5) ### Basic MCP Server Structure - [x] **2.1** Create MCP server using FastMCP - [x] Create `src/mcp_server/server.py` using FastMCP - [x] Set up basic server with health check tool - [x] Test MCP server connectivity via stdio - **Claude Code Tip**: "Create a basic FastMCP server with a simple health check tool to validate MCP connectivity" ### PyBoy Integration Foundation - [x] **2.2** Implement PyBoy wrapper class - [x] Create `src/mcp_server/emulator.py` with PyBoy wrapper - [x] Implement basic emulator lifecycle (start, stop, load ROM) - [x] Add LLM-friendly error handling for common PyBoy issues - **Claude Code Tip**: "Create a PyBoy wrapper that handles emulator lifecycle and provides LLM-friendly error messages" ### Core Emulation Tools - [x] **2.3** Implement essential MCP tools - [x] Implement `load_rom` tool with path validation - [x] Implement `get_screen` tool (returns base64 screenshot) - [x] Implement `press_button` tool with Game Boy button mapping - [x] Test all tools work end-to-end with test ROMs - **Claude Code Tip**: "Create the three core tools that allow LLMs to load games, see the screen, and interact with controls" ### Session Management - [x] **2.4** Create session management - [x] Create singleton pattern for active game session - [x] Add ROM validation and loading with error recovery - [x] Implement graceful session cleanup and crash recovery - [x] Add session state tracking and metrics - [x] Implement thread-safe concurrent access - **Claude Code Tip**: "Implement a robust session manager that maintains one active game session and handles ROM lifecycle with recovery" ### Web Frontend for Debugging - [ ] **2.5** Create minimal web frontend - [x] Create simple FastAPI debug server - [ ] Live screen display (auto-refreshing base64 image) - [ ] Visual button press indicators - [ ] Current ROM and session status display - [ ] Basic error/crash notifications - [ ] MCP tool call log viewer - **Claude Code Tip**: "Create a minimal web UI for debugging screen capture and session state during development" ### Logging Infrastructure - [ ] **2.6** Add structured logging - [ ] Structured logging for all MCP tool calls - [ ] Performance metrics (screen capture time, input latency) - [ ] Debug mode with verbose PyBoy output - [ ] Log rotation and management - **Claude Code Tip**: "Add comprehensive logging to help debug MCP interactions and performance issues" --- ## Phase 3: Enhanced Tools with Proper Foundation (Days 6-8) ### Input Queue System (Foundation) - [ ] **3.1** Create robust input handling - [ ] Implement async input queue to prevent race conditions - [ ] Add input validation and timing controls - [ ] Handle input cancellation and error recovery - [ ] Queue overflow handling - **Claude Code Tip**: "Create an input queue system that ensures button presses are processed safely without corrupting game state" ### Advanced Input Controls - [ ] **3.2** Implement enhanced input tools - [ ] Add input sequence and timing controls - [ ] Implement hold/release button functionality - [ ] Add frame-by-frame advancement (`tick` tool) - [ ] Macro recording/playback - **Claude Code Tip**: "Build on the basic press_button tool to add sequence controls and precise timing for complex game interactions" ### Save State System - [ ] **3.3** Implement save state functionality - [ ] Add save/load state tools with validation - [ ] Implement state file management and organization - [ ] Create state persistence across sessions - [ ] Quick save/load slots - **Claude Code Tip**: "Create save state tools that allow LLMs to create checkpoints and experiment with different approaches" ### Enhanced Screen Capture - [ ] **3.4** Add advanced screen features - [ ] Multiple output formats (base64, raw arrays) - [ ] Screen region extraction for analysis - [ ] Smart caching to reduce CPU usage - [ ] Frame differencing for change detection - **Claude Code Tip**: "Enhance the basic get_screen tool with different formats and region selection for more sophisticated analysis" ### Debug Tools - [ ] **3.5** Add debugging and monitoring tools - [ ] MCP tool to dump emulator state - [ ] Session reset and recovery tools - [ ] Memory usage monitoring - [ ] Performance profiling commands - **Claude Code Tip**: "Create debug tools to help diagnose issues and monitor system health during development" --- ## Phase 4: Game-Specific Notebook System (Days 9-10) ### Game-Specific Notebook Implementation - [ ] **4.1** Create notebook manager - [ ] Implement `src/mcp_server/notebook.py` with game-specific storage - [ ] One notebook per ROM (identified by ROM hash) - [ ] Sections: Current Objectives, Progress Log, Important Discoveries, Strategy Notes - [ ] Auto-clear temporary notes when loading different ROM - **Claude Code Tip**: "Create a notebook system focused on helping LLMs remember objectives and progress, not observable game state" ### Notebook MCP Tools - [ ] **4.2** Implement notebook tools - [ ] Add `save_notes` with section-based organization - [ ] Implement `get_notes` returning current game's relevant information - [ ] Create `update_objectives` for current goals management - [ ] Add `list_sections` showing available note categories - **Claude Code Tip**: "Create notebook tools that guide LLMs toward useful objective tracking rather than redundant state recording" ### Memory Guidance System - [ ] **4.3** Add intelligent note filtering - [ ] Tool descriptions guide LLM on what to record vs. observe fresh - [ ] Validation prevents storing obviously visible information - [ ] Focus on decision-making context and non-obvious mechanics - [ ] Size limiting with feedback on what to include/exclude - **Claude Code Tip**: "Build validation that encourages useful memory while preventing redundant information storage" --- ## Phase 5: Integration and Testing (Days 11-12) ### Enhanced Testing Infrastructure - [ ] **5.1** Create comprehensive integration tests - [ ] Document test ROM selection criteria and acquisition strategy - [ ] Create test suite with specific ROMs for each feature - [ ] Mock PyBoy implementation for faster unit tests - [ ] Performance benchmarks with target metrics - [ ] Integration tests using `tests/fixtures/test_roms/` - [ ] Test complete ROM loading → gameplay → note taking flow - [ ] Error handling validation with LLM-friendly messages - [ ] Notebook persistence testing across sessions - **Claude Code Tip**: "Create integration tests that simulate real LLM usage patterns and validate the complete system flow" ### CLI and Basic Documentation - [ ] **5.2** Add command-line interface - [ ] Create `src/mcp_server/cli.py` for easy server startup - [ ] Add configuration options and help text - [ ] Implement proper signal handling - **Claude Code Tip**: "Create a user-friendly CLI that makes it easy to start and configure the MCP server" ### Performance and Reliability - [ ] **5.3** Optimize critical paths - [ ] Profile screen capture and input processing - [ ] Optimize memory usage and caching - [ ] Test concurrent operations and edge cases - **Claude Code Tip**: "Profile and optimize the performance bottlenecks while ensuring system reliability" ### Usage Documentation - [ ] **5.4** Create usage examples - [ ] Write example MCP interactions - [ ] Document notebook best practices - [ ] Create basic troubleshooting guide - **Claude Code Tip**: "Generate usage examples that help users understand how to interact effectively with the MCP server" --- ## Phase 6: MVP Validation and Polish (Days 13-14) ### Real-World Testing - [ ] **6.1** Test with actual Game Boy ROMs - [ ] Validate with classic games (Tetris, Pokemon, etc.) - [ ] Test edge cases and error conditions - [ ] Verify notebook system helps LLM maintain context effectively - **Claude Code Tip**: "Test the complete system with real Game Boy ROMs to validate MVP functionality and notebook effectiveness" ### LLM Integration Testing - [ ] **6.2** Validate actual LLM interactions - [ ] Test actual Claude interactions via Claude Desktop - [ ] Validate tool discovery and usage patterns - [ ] Check error message clarity and recovery guidance - [ ] Test notebook system prevents redundant information storage - **Claude Code Tip**: "Test the actual LLM integration by simulating realistic Claude interactions with the MCP server" ### Packaging and Distribution - [ ] **6.3** Create installation package - [ ] Set up PyPI-ready package structure - [ ] Create installation and setup documentation - [ ] Add example configuration files - [ ] Test installation process end-to-end - **Claude Code Tip**: "Create a professional Python package with proper setup for PyPI distribution" ### Final Documentation Review - [ ] **6.4** Polish documentation and examples - [ ] Update all documentation for accuracy - [ ] Create quick start guide with notebook usage examples - [ ] Add API reference documentation - [ ] Include troubleshooting guide for common issues - **Claude Code Tip**: "Review and update all project documentation to ensure it's accurate and helpful for end users" --- ## Success Criteria for MVP ### Core Functionality - [ ] **Load and play Game Boy ROMs successfully** - [ ] **LLM can discover and use all MCP tools** - [ ] **Error handling provides LLM-friendly actionable feedback** - [ ] **Session management handles crashes and recovery gracefully** - [ ] **Game-specific notebook system helps LLM track objectives across sessions** - [ ] **Screen capture works with base64 output for LLM vision** - [ ] **Input controls respond accurately with proper timing** - [ ] **Save states allow experimentation and checkpointing** - [ ] **Web frontend provides effective debugging capabilities** ### LLM Integration Quality - [ ] **FastMCP server properly exposes tools via stdio transport** - [ ] **Tool descriptions guide effective LLM usage** - [ ] **Notebook system prevents redundant information storage** - [ ] **Error messages suggest corrective actions** - [ ] **All tools validate parameters and provide helpful feedback** ### Technical Quality - [ ] **All tests pass with solid coverage** - [ ] **Code follows style guidelines (Black, Ruff)** - [ ] **Type checking passes (MyPy)** - [ ] **No memory leaks during extended gameplay** - [ ] **Emulation runs smoothly without performance issues** ### Documentation Quality - [ ] **README provides clear setup instructions** - [ ] **API documentation covers all MCP tools** - [ ] **Examples demonstrate effective LLM usage patterns** - [ ] **Notebook best practices guide LLM behavior** --- ## Next Steps After MVP Once MVP is complete, consider these enhancements: - Enhanced web frontend with human takeover capabilities - Advanced screen analysis tools (object detection, OCR) - Multi-emulator support (GBA, NES, SNES) - Cloud save synchronization - Advanced performance optimizations - Additional Game Boy features (sound, multiplayer) - Reader-writer lock improvements for session management **Remember**: The goal is a working, demonstrable MVP that showcases the core concept. Perfect is the enemy of done!