Shepherd MCP

# 🐑 Shepherd MCP MCP (Model Context Protocol) server for Shepherd - Debug your AI agents like you debug your code. This MCP server allows AI assistants (Claude, Cursor, etc.) to query and analyze your AI agent sessions from multiple observability providers. ## Supported Providers - **AIOBS** (Shepherd backend) - Native Shepherd observability - **Langfuse** - Open-source LLM observability platform ## Installation ```bash pip install shepherd-mcp ``` Or run directly with uvx: ```bash uvx shepherd-mcp ``` ## Configuration ### Environment Variables #### AIOBS (Shepherd) - `AIOBS_API_KEY` (required) - Your Shepherd API key - `AIOBS_ENDPOINT` (optional) - Custom API endpoint URL #### Langfuse - `LANGFUSE_PUBLIC_KEY` (required) - Your Langfuse public API key - `LANGFUSE_SECRET_KEY` (required) - Your Langfuse secret API key - `LANGFUSE_HOST` (optional) - Custom Langfuse host URL (defaults to cloud.langfuse.com) ### .env File Support shepherd-mcp automatically loads `.env` files from the current directory or any parent directory. This means if you have a `.env` file in your project root: ```bash # .env # AIOBS AIOBS_API_KEY=aiobs_sk_xxxx # Langfuse LANGFUSE_PUBLIC_KEY=pk-lf-xxxx LANGFUSE_SECRET_KEY=sk-lf-xxxx LANGFUSE_HOST=https://cloud.langfuse.com ``` It will be automatically loaded when the MCP server starts. ### Claude Desktop Add to your `claude_desktop_config.json`: ```json { "mcpServers": { "shepherd": { "command": "uvx", "args": ["shepherd-mcp"], "env": { "AIOBS_API_KEY": "aiobs_sk_xxxx", "LANGFUSE_PUBLIC_KEY": "pk-lf-xxxx", "LANGFUSE_SECRET_KEY": "sk-lf-xxxx", "LANGFUSE_HOST": "https://cloud.langfuse.com" } } } } ``` ### Cursor Add to your `.cursor/mcp.json`: ```json { "mcpServers": { "shepherd": { "command": "uvx", "args": ["shepherd-mcp"], "env": { "AIOBS_API_KEY": "aiobs_sk_xxxx", "LANGFUSE_PUBLIC_KEY": "pk-lf-xxxx", "LANGFUSE_SECRET_KEY": "sk-lf-xxxx", "LANGFUSE_HOST": "https://cloud.langfuse.com" } } } } ``` Or if installed via pip: ```json { "mcpServers": { "shepherd": { "command": "shepherd-mcp", "env": { "AIOBS_API_KEY": "aiobs_sk_xxxx", "LANGFUSE_PUBLIC_KEY": "pk-lf-xxxx", "LANGFUSE_SECRET_KEY": "sk-lf-xxxx", "LANGFUSE_HOST": "https://cloud.langfuse.com" } } } } ``` ## Available Tools ### AIOBS (Shepherd) Tools #### `aiobs_list_sessions` List all AI agent sessions from Shepherd. **Parameters:** - `limit` (optional): Maximum number of sessions to return **Example prompt:** > "List my recent AI agent sessions from AIOBS" #### `aiobs_get_session` Get detailed information about a specific session including the full trace tree, LLM calls, function events, and evaluations. **Parameters:** - `session_id` (required): The UUID of the session to retrieve **Example prompt:** > "Get AIOBS session details for abc123-def456" #### `aiobs_search_sessions` Search and filter sessions with multiple criteria. **Parameters:** - `query` (optional): Text search (matches name, ID, labels, metadata) - `labels` (optional): Filter by labels as key-value pairs - `provider` (optional): Filter by LLM provider (e.g., 'openai', 'anthropic') - `model` (optional): Filter by model name (e.g., 'gpt-4o-mini', 'claude-3') - `function` (optional): Filter by function name - `after` (optional): Sessions started after date (YYYY-MM-DD) - `before` (optional): Sessions started before date (YYYY-MM-DD) - `has_errors` (optional): Only return sessions with errors - `evals_failed` (optional): Only return sessions with failed evaluations - `limit` (optional): Maximum number of sessions to return **Example prompts:** > "Find all AIOBS sessions that used OpenAI with errors" > "Search for sessions from yesterday that failed evaluations" #### `aiobs_diff_sessions` Compare two sessions and show their differences including: - **Metadata**: Duration, labels, timestamps - **LLM calls**: Count, tokens (input/output/total), average latency, errors - **Provider/Model distribution**: Which providers and models were used - **Function events**: Total calls, unique functions, function-specific counts - **Trace structure**: Trace depth, root nodes - **Evaluations**: Pass/fail counts and rates - **System prompts**: Compare system prompts across sessions - **Request parameters**: Temperature, max_tokens, tools used - **Response content**: Content length, tool calls, stop reasons **Parameters:** - `session_id_1` (required): First session UUID to compare - `session_id_2` (required): Second session UUID to compare **Example prompt:** > "Compare AIOBS sessions abc123 and def456" --- ### Langfuse Tools #### `langfuse_list_traces` List traces with pagination and filters. Traces represent complete workflows or conversations. **Parameters:** - `limit` (optional): Maximum results per page (default: 50) - `page` (optional): Page number (1-indexed) - `user_id` (optional): Filter by user ID - `name` (optional): Filter by trace name - `session_id` (optional): Filter by session ID - `tags` (optional): Filter by tags - `from_timestamp` (optional): Filter after timestamp - `to_timestamp` (optional): Filter before timestamp **Example prompt:** > "List the last 20 Langfuse traces" #### `langfuse_get_trace` Get a specific trace with its observations (generations, spans, events). **Parameters:** - `trace_id` (required): The trace ID to fetch **Example prompt:** > "Get Langfuse trace details for trace-id-123" #### `langfuse_list_sessions` List sessions with pagination. Sessions group related traces together. **Parameters:** - `limit` (optional): Maximum results per page - `page` (optional): Page number - `from_timestamp` (optional): Filter after timestamp - `to_timestamp` (optional): Filter before timestamp **Example prompt:** > "Show me Langfuse sessions from the last week" #### `langfuse_get_session` Get a specific session with its metrics and traces. **Parameters:** - `session_id` (required): The session ID to fetch **Example prompt:** > "Get Langfuse session details for session-123" #### `langfuse_list_observations` List observations (generations, spans, events) with filters. **Parameters:** - `limit` (optional): Maximum results per page - `page` (optional): Page number - `name` (optional): Filter by observation name - `user_id` (optional): Filter by user ID - `trace_id` (optional): Filter by trace ID - `type` (optional): Filter by type (GENERATION, SPAN, EVENT) - `from_timestamp` (optional): Filter after timestamp - `to_timestamp` (optional): Filter before timestamp **Example prompt:** > "List all GENERATION type observations from Langfuse" #### `langfuse_get_observation` Get a specific observation with full details including input, output, usage, and costs. **Parameters:** - `observation_id` (required): The observation ID to fetch **Example prompt:** > "Get details for Langfuse observation obs-123" #### `langfuse_list_scores` List scores/evaluations with filters. **Parameters:** - `limit` (optional): Maximum results per page - `page` (optional): Page number - `name` (optional): Filter by score name - `user_id` (optional): Filter by user ID - `trace_id` (optional): Filter by trace ID - `from_timestamp` (optional): Filter after timestamp - `to_timestamp` (optional): Filter before timestamp **Example prompt:** > "Show me Langfuse scores for trace trace-123" #### `langfuse_get_score` Get a specific score/evaluation with full details. **Parameters:** - `score_id` (required): The score ID to fetch **Example prompt:** > "Get Langfuse score details for score-123" --- ### Legacy Tools (Deprecated) For backwards compatibility, the following tools are still available but will be removed in a future version: - `list_sessions` → Use `aiobs_list_sessions` - `get_session` → Use `aiobs_get_session` - `search_sessions` → Use `aiobs_search_sessions` - `diff_sessions` → Use `aiobs_diff_sessions` ## Use Cases ### 1. Debugging Failed Runs > "Show me all AIOBS sessions that had errors in the last 24 hours" ### 2. Performance Analysis > "Compare AIOBS session abc123 with session def456 and tell me which one was more efficient" ### 3. Prompt Regression Detection > "Find Langfuse traces with failed evaluations" ### 4. Cost Tracking > "List Langfuse observations and summarize the total cost" ### 5. Session Inspection > "Get the full trace tree for the most recent Langfuse trace and explain what happened" ### 6. Cross-Provider Analysis > "Show me both AIOBS sessions and Langfuse traces from today" ## Development ### Setup ```bash git clone https://github.com/neuralis/shepherd-mcp cd shepherd-mcp python -m venv venv source venv/bin/activate pip install -e ".[dev]" ``` ### Running Tests ```bash pytest ``` ### Running Locally ```bash export AIOBS_API_KEY=aiobs_sk_xxxx export LANGFUSE_PUBLIC_KEY=pk-lf-xxxx export LANGFUSE_SECRET_KEY=sk-lf-xxxx python -m shepherd_mcp ``` ### Publishing to PyPI Releases are automatically published to PyPI via GitHub Actions when a release is created. To publish manually: ```bash # Build the package pip install build twine python -m build # Upload to PyPI twine upload dist/* ``` ## Architecture ``` src/shepherd_mcp/ ├── __init__.py # Package exports ├── __main__.py # Entry point ├── server.py # MCP server with tool handlers ├── models/ # Data models │ ├── __init__.py │ ├── aiobs.py # AIOBS-specific models │ └── langfuse.py # Langfuse-specific models └── providers/ # Provider clients ├── __init__.py ├── base.py # Base provider interface ├── aiobs.py # AIOBS client implementation └── langfuse.py # Langfuse client implementation ``` ``` ┌─────────────────┐ stdio ┌─────────────────┐ │ Cursor/Claude │ ◄────────────► │ shepherd-mcp │ │ (Client) │ stdin/stdout │ (subprocess) │ └─────────────────┘ └────────┬────────┘ │ HTTPS ┌─────────┴─────────┐ │ │ ▼ ▼ ┌─────────────┐ ┌─────────────┐ │ Shepherd API│ │ Langfuse API│ │ (AIOBS) │ │ (Cloud) │ └─────────────┘ └─────────────┘ ``` ## License MIT