LangSmith MCP Server

# LangSmith MCP Server - Deep Technical Analysis ## Project Overview The LangSmith MCP Server is a production-ready Model Context Protocol (MCP) server that provides seamless integration with the LangSmith observability platform. This server enables AI language models to programmatically interact with LangSmith's suite of tools for conversation tracking, prompt management, dataset operations, and trace analytics. ### What is LangSmith? LangSmith is LangChain's comprehensive observability and evaluation platform for LLM applications, providing: - Conversation thread tracking and history - Prompt template management and versioning - Dataset creation and management for testing - Trace collection and analysis for debugging - Performance analytics and monitoring ### What is MCP? The Model Context Protocol (MCP) is an open standard that enables secure connections between AI assistants and external data sources and tools. This server implements the MCP specification to bridge AI models with LangSmith's capabilities. ## Architecture Deep Dive ### Core Components #### 1. Server Architecture (`langsmith_mcp_server/server.py`) The server is built on FastMCP, a high-performance MCP framework that provides: - **Transport Layer**: Uses stdio transport for communication with MCP clients - **Tool Registration**: Modular system for registering LangSmith tools, prompts, and resources - **Client Management**: Centralized LangSmith API client initialization and management - **Error Handling**: Comprehensive exception handling across all operations ```python # Core server initialization mcp = FastMCP("LangSmith API MCP Server") langsmith_client = LangSmithClient(api_key) if api_key else None # Modular registration system register_tools(mcp, langsmith_client) register_prompts(mcp, langsmith_client) register_resources(mcp, langsmith_client) ``` #### 2. Client Wrapper (`langsmith_mcp_server/langsmith_client.py`) A thin wrapper around the LangSmith Python SDK that: - Manages API key configuration and environment setup - Provides a consistent interface for all LangSmith operations - Handles authentication and connection management - Abstracts the underlying LangSmith Client complexity #### 3. Service Layer Architecture The services are organized into three distinct modules: **Tools (`langsmith_mcp_server/services/tools/`)** - `datasets.py`: Dataset CRUD operations and example management - `prompts.py`: Prompt discovery, retrieval, and template access - `traces.py`: Conversation history, trace analysis, and run statistics - `experiments.py`: Experimental features and advanced analytics - `workspaces.py`: Workspace and project management **Prompts (`langsmith_mcp_server/services/prompts/`)** - Predefined prompt templates for common LangSmith operations - Contextual prompts for AI model interactions **Resources (`langsmith_mcp_server/services/resources/`)** - `langsmith_docs.py`: Dynamic access to LangSmith documentation - `langgraph_docs.py`: Integration with LangGraph documentation ### Tool Specifications #### Conversation & Thread Management ```python @mcp.tool() def get_thread_history(thread_id: str, project_name: str) -> Dict[str, Any]: ``` - **Purpose**: Retrieves complete message history for conversation threads - **Use Cases**: Context restoration, conversation analysis, debugging chat flows - **Data Format**: Returns chronologically ordered messages with metadata - **Project Formats**: Supports both "owner/project" and standalone project names #### Prompt Management System ```python @mcp.tool() def list_prompts(is_public: str = "false", limit: int = 20) -> Dict[str, Any]: @mcp.tool() def get_prompt_by_name(prompt_name: str) -> Dict[str, Any]: ``` - **Advanced Filtering**: Public/private visibility, limit controls, search capabilities - **Template Access**: Full prompt templates with variable interpolation - **Metadata Retrieval**: Creation dates, authors, usage statistics, versioning info - **Use Cases**: Prompt discovery, template reuse, prompt library management #### Dataset Operations ```python @mcp.tool() def list_datasets(dataset_ids: List[str] = None, data_type: str = None, ...) -> Dict[str, Any]: @mcp.tool() def list_examples(dataset_id: str = None, dataset_name: str = None, ...) -> Dict[str, Any]: ``` - **Flexible Filtering**: By ID, name, type, metadata, creation date - **Example Management**: Pagination, filtering, version control - **Data Types**: Support for chat datasets, key-value pairs, and custom formats - **Versioning**: Access historical versions via `as_of` parameter #### Analytics & Monitoring ```python @mcp.tool() def get_project_runs_stats(project_name: str, is_last_run: str = "true") -> Dict[str, Any]: @mcp.tool() def fetch_trace(project_name: str = None, trace_id: str = None) -> Dict[str, Any]: ``` - **Performance Metrics**: Response times, error rates, token usage - **Trace Analysis**: Complete execution traces for debugging - **Statistical Aggregation**: Project-level and run-level statistics - **Debug Support**: Detailed trace information for troubleshooting ## Development Environment Setup ### Prerequisites - **Python**: 3.10+ (type-checked and tested) - **uv**: Fast Python package manager and resolver - **LangSmith API Key**: Required for all operations ### Installation Methods #### Development Setup ```bash # Clone and setup development environment git clone https://github.com/langchain-ai/langsmith-mcp-server.git cd langsmith-mcp-server # Create isolated environment with all dependencies uv sync # Include test dependencies for development uv sync --group test # Verify installation uvx langsmith-mcp-server ``` #### Production Deployment ```bash # Install from PyPI for production use uv run pip install --upgrade langsmith-mcp-server # Direct execution uvx langsmith-mcp-server ``` ### Development Tools & Workflows #### MCP Inspector Integration ```bash # Start development server with inspector uv run mcp dev langsmith_mcp_server/server.py ``` - **Real-time Testing**: Interactive tool testing in browser interface - **Environment Configuration**: Set API keys and test configurations - **Tool Discovery**: Browse all available tools and their schemas - **Request/Response Debugging**: Inspect tool calls and responses #### Code Quality & Testing ```bash # Linting and formatting (configured via pyproject.toml) make lint # Check code style with ruff make format # Auto-format code with ruff # Testing suite make test # Run pytest with socket restrictions make test_watch # Continuous testing during development # Type checking uv run mypy langsmith_mcp_server/ # Static type analysis ``` #### Configuration Files Analysis **pyproject.toml** - **Build System**: Uses pdm-backend for modern Python packaging - **Dependencies**: Core MCP framework, LangSmith SDK, LangChain Core - **Development Tools**: pytest, ruff (linting), mypy (type checking) - **Entry Points**: Command-line interface via `langsmith-mcp-server` - **Test Configuration**: Async test support, socket restrictions for security **Dockerfile** - **Base Image**: Python 3.12 Alpine for minimal footprint - **Build Dependencies**: gcc, musl-dev, libffi-dev for compiled extensions - **Security**: Non-root execution, minimal attack surface - **Entry Point**: Direct execution of langsmith-mcp-server command ## Integration Patterns ### MCP Client Configuration #### Cursor IDE Integration ```json { "mcpServers": { "LangSmith API MCP Server": { "command": "/Users/username/.local/bin/uvx", "args": ["langsmith-mcp-server"], "env": { "LANGSMITH_API_KEY": "lsv2_pt_your_key_here" } } } } ``` #### Claude Desktop Integration ```json { "mcpServers": { "LangSmith API MCP Server": { "command": "uvx", "args": ["langsmith-mcp-server"], "env": { "LANGSMITH_API_KEY": "lsv2_pt_your_key_here" } } } } ``` #### Source-based Integration ```json { "command": "/path/to/uvx", "args": [ "--directory", "/path/to/langsmith-mcp-server", "run", "langsmith_mcp_server/server.py" ], "env": { "LANGSMITH_API_KEY": "lsv2_pt_your_key_here" } } ``` ### Smithery Platform Integration The project includes `smithery.yml` configuration for deployment on the Smithery platform: - **Schema Definition**: JSON Schema for configuration validation - **Environment Setup**: Automated API key configuration - **Command Generation**: Dynamic CLI command construction - **Example Configuration**: Template for quick setup ## Advanced Use Cases & Workflows ### 1. Conversation Analytics Pipeline ```python # Retrieve conversation history history = get_thread_history("thread-abc123", "my-chatbot-project") # Analyze conversation patterns stats = get_project_runs_stats("my-chatbot-project", "false") # Overall stats # Debug specific issues trace = fetch_trace(trace_id="trace-xyz789") ``` ### 2. Prompt Library Management ```python # Discover available prompts prompts = list_prompts(is_public="true", limit=50) # Retrieve specific prompt templates template = get_prompt_by_name("legal-case-summarizer") # Access prompt configuration and variables system_message = template['prompt']['messages'][0]['content'] ``` ### 3. Dataset-Driven Development ```python # Find datasets for testing datasets = list_datasets(data_type="chat", limit=10) # Load test examples examples = list_examples(dataset_name="customer-support-qa", limit=100) # Access specific test case example = read_example("example-uuid-here") ``` ### 4. Performance Monitoring ```python # Monitor recent performance recent_stats = get_project_runs_stats("production-bot", "true") # Last run only # Historical analysis all_stats = get_project_runs_stats("production-bot", "false") # All runs # Debug performance issues problematic_trace = fetch_trace(project_name="production-bot") ``` ## Error Handling & Reliability ### Exception Management - **Graceful Degradation**: All tools return error dictionaries rather than raising exceptions - **Client Validation**: API key validation and client initialization checks - **Input Sanitization**: Parameter validation and type coercion - **Timeout Handling**: Network timeout management for long-running operations ### Security Considerations - **API Key Protection**: Environment-based API key management - **Input Validation**: Parameter sanitization and type checking - **Network Security**: Socket restrictions in testing environment - **Minimal Dependencies**: Reduced attack surface through careful dependency selection ### Performance Optimizations - **Lazy Loading**: Client initialization only when needed - **Pagination Support**: Limit controls for large dataset operations - **Caching Opportunities**: Client-side caching of frequently accessed prompts - **Async Support**: Full asyncio compatibility for concurrent operations ## Future Development Roadmap ### Planned Features - **Enhanced Filtering**: Advanced query capabilities for datasets and prompts - **Batch Operations**: Multi-item CRUD operations for efficiency - **Real-time Streaming**: Live trace and conversation monitoring - **Advanced Analytics**: Statistical analysis and trend detection - **Workflow Integration**: LangGraph workflow management and execution ### Extension Points - **Custom Tools**: Framework for adding domain-specific tools - **Plugin Architecture**: Extensible resource and prompt providers - **Integration Adapters**: Connectors for other observability platforms - **Authentication Methods**: Support for additional auth mechanisms ## Troubleshooting Guide ### Common Issues 1. **API Key Problems**: Ensure LANGSMITH_API_KEY is properly set in environment 2. **Client Connection**: Verify LangSmith service availability and network connectivity 3. **Permission Issues**: Check API key permissions for requested operations 4. **Tool Discovery**: Use MCP inspector to debug tool registration and schemas ### Development Debugging 1. **Use MCP Inspector**: Interactive debugging of tool calls and responses 2. **Enable Verbose Logging**: Set appropriate log levels for detailed operation tracing 3. **Test Individual Tools**: Isolate issues by testing tools individually 4. **Check Network Connectivity**: Verify LangSmith API accessibility ### Performance Tuning 1. **Limit Result Sets**: Use appropriate limits for large dataset operations 2. **Batch Related Calls**: Group related operations to reduce round trips 3. **Cache Frequently Used Data**: Store commonly accessed prompts and datasets locally 4. **Monitor Resource Usage**: Track memory and CPU usage during heavy operations This comprehensive technical analysis provides the foundation for understanding, deploying, and extending the LangSmith MCP Server in production environments.