# Clerk Chat MCP Server

MCP server for Clerk Chat voice AI tools and skills.

## Features

### Prompt Improvement
Autonomous prompt improvement loop for voice AI agents:
- Analyze call transcripts and feedback
- Generate improved prompts
- Create regression tests with LLM evaluation
- Track improvement runs in database
- Iterate until tests pass

## Setup

### 1. Install Dependencies

```bash
npm install
```

### 2. Configure API Access

The server requires a Clerk Chat API key for full functionality. You have two options:

#### Option A: Using .env file (Development)

1. Copy the example configuration:
   ```bash
   cp .env.example .env
   ```

2. Edit `.env` and add your API key:
   ```
   CLERK_CHAT_API_KEY=your_api_key_here
   ```

#### Option B: Using Claude Desktop config (Production)

Add environment variables directly to your Claude Desktop config at:
`~/Library/Application Support/Claude/claude_desktop_config.json`

```json
{
  "mcpServers": {
    "clerk-chat": {
      "command": "node",
      "args": ["/absolute/path/to/clerk-chat-mcp/src/index.js"],
      "env": {
        "CLERK_CHAT_API_KEY": "07fe6d3f658d65d7fd906068e21eef5f5182fd2438e66c78e6786b661e668b2e",
        "CLERK_CHAT_API_BASE_URL": "https://puxgxqdkizwdzqyuaitm.supabase.co/functions/v1"
      }
    }
  }
}
```

**Note**: The API key shown above is for Technical Life Care company (testing). Each company has its own API key hash.

### 3. Start the Server

```bash
npm start
```

The server will validate your API configuration on startup and report if the API is enabled or disabled.

## Skills

Skills are exposed as MCP resources. Available skills:

### prompt-improvement/
- `feedback-analysis` - Structure user feedback into actionable specs
- `prompt-improvement` - Generate improved prompts
- `test-generation` - Create regression tests
- `test-analysis` - Diagnose test failures
- `improvement-loop` - Orchestrate the full improvement cycle

## Tools

### Tools Management

#### `list_tools`
Get all available tools for a company. Use this when generating tests to determine if tool calls should be tested.

**Parameters**:
- `company_name` (string): Company name in kebab-case (e.g., 'technical-life-care', 'tetrix')

**Returns**: List of tool definitions with IDs, names, descriptions, and parameters.

**Example**:
```javascript
list_tools(company_name: "tetrix")
// Returns:
[
  {
    "id": "tool-uuid-123",
    "name": "search_knowledge_base",
    "description": "Search the knowledge base",
    "parameters": [{"name": "query", "type": "string"}]
  }
]
```

#### `get_tool`
Get details of a specific tool by ID.

**Parameters**:
- `company_name` (string): Company name in kebab-case
- `tool_id` (string): UUID of the tool

**Returns**: Tool definition with full details.

### Test Management

#### `list_test_cases`
Get all test cases from the database for a specific company.

**Parameters**:
- `company_name` (string): Company name in kebab-case

**Returns**: List of test cases with IDs, names, conversations, and expected outputs.

#### `create_test_case`
Create a new test case in the database.

**Parameters**:
- `name` (string): Test case name
- `conversation` (array): Array of {role, content} messages
- `expected_output` (string): Expected AI response
- `tool_mocks` (array, optional): Tool call mocks
- `expected_tool_call` (object, optional): Expected tool call

**Returns**: Created test case with UUID.

#### `run_tests`
Run actual LLM evaluation against test cases. Supports two modes: direct prompt text or saved prompt ID (faster).

**Parameters** (use one of the first two):
- `system_prompt` (string, optional): Direct prompt text to test
- `prompt_id` (string, optional): UUID of saved draft prompt (from `save_draft_prompt`) - **Preferred for performance**
- `test_case_ids` (array of strings, optional): Test case UUIDs to run. If omitted, runs all tests.
- `test_model` (string, optional): Model to use for testing
- `tools` (array, optional): Tool definitions

**Returns**: Test results with summary (total, passed, failed) and individual results.

**Example (with prompt_id - faster)**:
```javascript
{
  "prompt_id": "550e8400-e29b-41d4-a716-446655440000",
  "test_case_ids": ["uuid-1", "uuid-2"]
}
```

**Example (with direct text - slower)**:
```javascript
{
  "system_prompt": "You are a helpful assistant...",
  "test_case_ids": ["uuid-1", "uuid-2"]
}
```

### Prompt Management

#### `save_draft_prompt`
Save a draft prompt to the database for later use in testing. Returns a prompt_id that avoids streaming full prompts during iterations.

**Parameters**:
- `prompt` (string): The full prompt text to save
- `label` (string, optional): Label like "iteration-1" or "baseline"

**Returns**: Saved prompt with UUID and timestamp.

**Benefits**:
- Faster performance (no streaming of full prompts)
- Automatic version history
- Can re-run tests against old versions

**Example**:
```javascript
{
  "prompt": "You are a helpful assistant...",
  "label": "iteration-1"
}
// Returns: { id: "550e8400-...", created_at: "2026-02-02T14:30:00Z" }
```

#### `get_draft_prompt`
Retrieve a previously saved draft prompt by ID.

**Parameters**:
- `prompt_id` (string): UUID of the draft prompt

**Returns**: Prompt text, label, and metadata.

#### `list_draft_prompts`
List all saved draft prompts with their IDs and labels.

**Returns**: Array of draft prompts.

### Improvement Tracking

#### `save_improvement_run`
Store a complete improvement cycle with prompts, analysis, and test results.

**Parameters**:
- `company_name` (string): Company name in kebab-case
- `original_prompt` (string): Starting system prompt text
- `new_prompt` (string): Final improved system prompt text
- `client_feedback` (string): User's description of what went wrong
- `analysis` (object): Structured feedback analysis
  - `what_went_wrong` (string): Specific behavior that failed
  - `why_it_went_wrong` (string): Root cause analysis
  - `recommended_fix` (string): What changes were made to fix it
- `model_used` (string, optional): Model used for testing (e.g., 'google/gemini-2.5-flash')
- `test_results` (array): Test execution results with full details
  - `test_name` (string): Test case name
  - `passed` (boolean): Whether the test passed
  - `is_generated` (boolean): true for new tests from feedback, false for existing tests
  - `expected` (string): What the response should be
  - `response` (string): What the AI actually responded
  - `conversation` (array): Full conversation for this test
- `metadata` (object, optional): Additional context (iterations, timestamps, etc.)

**Returns**: Saved improvement run with UUID and timestamp.

**Example**:
```javascript
save_improvement_run(
  company_name: "tetrix",
  original_prompt: "You are a helpful assistant...",
  new_prompt: "You are a helpful assistant. Always confirm existing data...",
  client_feedback: "AI keeps re-asking for customer email even when on file",
  analysis: {
    what_went_wrong: "AI re-requests known customer information",
    why_it_went_wrong: "System prompt didn't specify to confirm existing data",
    recommended_fix: "Added explicit instruction to confirm rather than re-request"
  },
  model_used: "google/gemini-2.5-flash",
  test_results: [
    {
      test_name: "Confirm existing email",
      passed: true,
      is_generated: true,
      expected: "AI should confirm existing email",
      response: "I have john@example.com on file — is that current?",
      conversation: [
        { role: "user", content: "Hi, I have a question" },
        { role: "assistant", content: "I have john@example.com on file — is that current?" }
      ]
    }
  ]
)
```

### Skills

#### `list_skills`
List all available skills (filesystem-only, no API required).

## Usage Flow

### Basic Improvement Loop

1. Provide Claude with: transcript + feedback + current prompt
2. Claude reads relevant skills
3. Claude runs the improvement loop:
   - Analyze feedback (`feedback-analysis` skill)
   - Improve prompt (`prompt-improvement` skill)
   - Generate tests (`test-generation` skill → `create_test_case` tool)
   - Save prompt version (`save_draft_prompt` tool → returns prompt_id)
   - Run tests (`run_tests` tool with prompt_id - fast, no streaming)
   - Analyze failures (`test-analysis` skill)
   - Iterate until pass or stop condition
   - Save run (`save_improvement_run` tool)

### Example Workflow

```
User: "Here's a transcript where the AI was too verbose. Current prompt: [...]"

Claude:
1. Uses feedback-analysis skill to structure the feedback
2. Uses prompt-improvement skill to generate new prompt
3. Uses test-generation skill to create test cases
4. Calls create_test_case for each test
5. Calls save_draft_prompt(new_prompt, "iteration-1") → gets prompt_id
6. Calls run_tests(prompt_id, test_ids) → fast, no streaming
7. If failures: uses test-analysis skill, improves prompt, repeats from step 5
8. If success: calls save_improvement_run to persist results
```

## Architecture

```
src/
├── index.js              # Main MCP server
├── config.js             # Configuration management
├── api/
│   ├── client.js         # HTTP client with auth
│   ├── test-cases.js     # Test CRUD operations
│   ├── test-runner.js    # Test execution API
│   ├── prompts.js        # Draft prompt management
│   ├── tools.js          # Tool definitions API
│   └── improvement-runs.js # Improvement tracking
└── tools/
    ├── test-tools.js     # Test management tools
    ├── improvement-tools.js # Improvement tracking tools
    ├── prompt-tools.js   # Draft prompt tools
    ├── tools-management.js # Tool fetching tools
    └── skill-tools.js    # Skill listing tools

skills/
└── prompt-improvement/   # Markdown skills for Claude
    ├── feedback-analysis.md
    ├── prompt-improvement.md
    ├── test-generation.md
    ├── test-analysis.md
    └── improvement-loop.md
```

## Error Handling

The server fails gracefully with clear error messages:

- **Missing API key**: "API authentication failed. Set CLERK_CHAT_API_KEY in .env or Claude Desktop config."
- **Network error**: "Unable to reach API. Check internet connection."
- **404 Not Found**: "Test case 'abc123' not found. Use list_test_cases to see available tests."
- **422 Validation**: "Invalid test case: 'name' is required."
- **500 Server Error**: "API error. Try again or check API status."

The MCP server never crashes - all errors are returned as tool results to Claude.

## Development

### Running Without API

The server can run without API configuration for skill-only functionality:
- Skills will still be available as resources
- `list_skills` tool will work
- API-dependent tools (test management, improvement tracking) will not be registered

### Testing API Integration

1. Configure API key in .env or Claude Desktop config
2. Restart Claude Desktop (if using config option)
3. Test each tool:
   ```
   list_test_cases → Should return test cases from database
   create_test_case → Should create test with UUID
   run_tests → Should execute with real LLM evaluation
   save_improvement_run → Should persist to database
   ```

### API Endpoints

The server integrates with these Supabase Edge Functions:

**Tool Definitions:**
- `GET /api-tools` - List all tools for authenticated company
- `GET /api-tools?id=uuid` - Get specific tool by ID

**Test Execution:**
- `POST /api-run-tests` - Execute tests with LLM evaluation (supports prompt_id or system_prompt)

**Test Cases:**
- `GET /api-test-cases` - List all test cases
- `POST /api-test-cases` - Create new test case
- `PUT /api-test-cases?id=uuid` - Update test case
- `DELETE /api-test-cases?id=uuid` - Delete test case

**Draft Prompts:**
- `POST /api-prompts` - Save draft prompt (returns prompt_id)
- `GET /api-prompts/:id` - Get specific draft prompt
- `GET /api-prompts` - List all draft prompts

**Improvement Tracking:**
- `POST /api-improvement-runs` - Save improvement run
- `GET /api-improvement-runs` - List improvement runs

## Security

- API keys are never logged
- `.env` is gitignored
- All credentials use environment variables
- Input validation with Zod schemas
- Sanitized error messages

## License

MIT