Sherlock MCP Server

# Development Log: Sherlock MCP Server ## Introduction The Sherlock MCP Server project represents an innovative integration of the popular Sherlock OSINT tool with the emerging Model Context Protocol (MCP). This initiative aims to bridge the gap between open-source intelligence gathering and AI-powered workflows, enabling AI assistants to seamlessly perform username enumeration across hundreds of social media platforms. ### Project Motivation As an OSINT enthusiast and developer, I recognized the potential of combining Sherlock's comprehensive username searching capabilities with MCP's standardized interface for AI tools. The goal was to create a production-ready MCP server that could be easily integrated into AI workflows while maintaining ethical OSINT practices. ### Target Audience - **OSINT Researchers**: Professionals needing automated social media reconnaissance - **AI Developers**: Those building MCP-compatible applications - **Cybersecurity Teams**: Organizations requiring username enumeration tools - **Ethical Hackers**: Individuals performing responsible security research - **Journalist**: Individuals using AI to find the truth about current events ## Development Phases ### Phase 1: Initial Planning and MCP Server Setup **Duration**: 1-2 hours **Objectives**: Establish project structure and basic MCP server functionality - Set up Python project with `pyproject.toml` - Installed FastMCP library and configured basic server - Created initial `main.py` with placeholder functionality - Established project directory structure **Key Decisions**: - Chose FastMCP for its Pythonic API and comprehensive MCP support - Used uv for dependency management to ensure reproducible builds - Structured code for easy testing and maintenance ### Phase 2: Sherlock CLI Integration **Duration**: 2-3 hours **Objectives**: Implement core username search functionality - Integrated subprocess calls to Sherlock CLI - Added JSON output parsing for structured results - Implemented comprehensive error handling - Created the `search_username` MCP tool **Challenges Faced**: - Sherlock's output formats required careful parsing - Handling timeouts and missing CLI installations - Balancing performance with reliability **Solutions**: - Used temporary files for JSON output to avoid stdout complexity - Implemented multiple error checks and graceful degradation - Added timeout handling to prevent hanging processes ### Phase 3: Testing and Docker Containerization **Duration**: 3-4 hours **Objectives**: Ensure code quality and enable easy deployment - Set up pytest with comprehensive test coverage - Created Docker container using Python 3.13 base image - Implemented mocking for external dependencies - Added test cases for success, error, and edge scenarios **Technical Highlights**: - Separated implementation from decoration for testability - Used pytest-mock for subprocess simulation - Ensured Docker compatibility with Sherlock CLI installation ### Phase 4: SEO Optimization and Community Setup **Duration**: 1-2 hours **Objectives**: Improve discoverability and encourage contributions - Optimized GitHub repository metadata - Enhanced README with badges and keywords - Created CONTRIBUTING.md and issue templates - Added relevant topics and description **SEO Strategies**: - Incorporated keywords like "OSINT", "MCP", "social media search" - Added visual badges for credibility - Created structured contribution guidelines ### Phase 5: Project Planning and Issue Creation **Duration**: 1 hour **Objectives**: Establish roadmap for future development - Identified 6 key areas for improvement - Created detailed GitHub issues for each enhancement - Prioritized based on impact and feasibility ## Technical Implementation ### Core Technologies - **FastMCP**: Python SDK for building MCP servers - **Sherlock CLI**: Open-source username enumeration tool - **Pytest**: Testing framework with mocking capabilities - **Docker**: Containerization for deployment - **GitHub CLI**: Repository management and issue creation ### Architecture ``` MCP Client → FastMCP Server → Subprocess → Sherlock CLI → Social Media APIs ``` ### Key Code Patterns - **Tool Definition**: Used decorators for clean MCP tool registration - **Error Handling**: Comprehensive try-catch blocks with specific error types - **Testing**: Mocked external dependencies for reliable unit tests - **Configuration**: Environment-based settings for flexibility ## Challenges and Solutions ### FastMCP Import Issues **Problem**: Initial dependency resolution conflicts with FastMCP installation. **Solution**: Switched to manual pip installation in Docker, ensuring compatibility with Python 3.13. ### Testing Subprocess Calls **Problem**: Difficulty mocking complex subprocess interactions. **Solution**: Separated implementation from decoration, allowing direct function testing with mocked subprocess.run calls. ### Docker Base Image Selection **Problem**: Sherlock's official Docker image lacked Python package management tools. **Solution**: Used python:3.13-slim as base, manually installing Sherlock via pipx for compatibility. ### SEO Implementation **Problem**: Limited direct access to GitHub repository settings. **Solution**: Leveraged GitHub CLI for metadata updates and focused on content optimization. ## Features Delivered ### ✅ Core Functionality - Username search across 400+ social media platforms - JSON-formatted results with site names and URLs - Real-time search execution via subprocess ### ✅ Error Handling - Sherlock CLI availability checks - Timeout protection (300-second limit) - JSON parsing error recovery - Graceful failure responses ### ✅ Testing Infrastructure - 100% test coverage for core functions - Mocked external dependencies - Edge case handling (timeouts, invalid JSON, missing CLI) ### ✅ Deployment Ready - Docker containerization - Minimal image size with efficient layering - Production-ready error logging ### ✅ Developer Experience - Comprehensive documentation - Contribution guidelines - Issue templates for structured feedback - SEO-optimized repository ## Lessons Learned ### Technical Insights 1. **Planning Pays Off**: Thorough planning in read-only mode prevented costly refactoring. 2. **Testing First**: Implementing tests early ensured code reliability and easier maintenance. 3. **Dependency Management**: Uv provided excellent dependency resolution, but Docker compatibility required adjustments. 4. **Error Handling**: Comprehensive error checking significantly improved user experience. ### Project Management 1. **Incremental Development**: Breaking work into phases allowed for steady progress and early feedback. 2. **Documentation Importance**: Investing time in README and contributing guidelines boosted perceived project quality. 3. **Community Focus**: Even for solo projects, preparing for collaboration improves structure and scalability. ### Personal Growth 1. **MCP Protocol Understanding**: Deep dive into MCP revealed its potential for AI tool integration. 2. **OSINT Ethics**: Reinforced importance of responsible tool development and clear usage guidelines. 3. **Open-Source Mindset**: Learned the value of thorough documentation and community engagement strategies. ## Future Roadmap Based on the 6 GitHub issues created, here's the prioritized development path: ### High Priority (Next 1-2 Months) 1. **CI/CD Setup**: Automated testing and deployment pipelines 2. **Feature Expansion**: HTTP/SSE transport and advanced MCP capabilities ### Medium Priority (2-4 Months) 1. **Security Enhancements**: Vulnerability scanning and compliance improvements 2. **Documentation**: Video tutorials and multi-language support ### Long-term (4+ Months) 1. **Community Building**: Discord/Slack integration and conference participation 2. **Marketing**: Blog posts, featured listings, and broader visibility ## Resources and Credits ### Libraries Used - [FastMCP](https://gofastmcp.com/) - MCP server framework - [Sherlock](https://github.com/sherlock-project/sherlock) - Username enumeration tool - [Pytest](https://pytest.org/) - Testing framework - [Docker](https://www.docker.com/) - Containerization platform ### Development Tools - uv - Python package manager - GitHub CLI - Repository management - Python 3.13 - Runtime environment ### Special Thanks - Sherlock Project maintainers for their excellent OSINT tool - FastMCP developers for simplifying MCP implementation - Open-source community for inspiration and resources ## Conclusion The Sherlock MCP Server project successfully demonstrates the power of combining established OSINT tools with modern AI protocols. From initial concept to production-ready implementation, the journey highlighted the importance of thorough planning, comprehensive testing, and community-oriented development practices. This project serves as both a functional tool for OSINT workflows and a template for integrating legacy command-line tools with contemporary AI ecosystems. The established roadmap ensures continued evolution and community adoption. For the latest updates, visit [https://github.com/Burnsedia/sherlock-mcp](https://github.com/Burnsedia/sherlock-mcp) and check the issues for upcoming features! --- *Development Log created on January 9, 2026* *Total development time: ~15-20 hours across 5 phases*