testing.mdc•8.44 kB
# Testing
Testing strategies and patterns for the Sentry MCP server.
## Testing Philosophy
Our testing approach prioritizes **functional coverage** over implementation details:
### Core Principles
1. **Favor Integration Over Unit Tests**
- Tests should verify actual functionality, not implementation details
- Focus on "does this feature work?" rather than "does this internal method work?"
- Prefer testing through public APIs rather than testing internals directly
2. **Minimize Mocking**
- **Only mock external network APIs** (Sentry API, OpenAI, etc.) using MSW
- **Don't mock internal code** - use real implementations
- **Don't test third-party library behavior** - trust that Node.js, npm packages, etc. work correctly
- Example: Don't test that Promise.all handles concurrent operations - that's Node.js's job
3. **Test What Matters**
- Test the functionality users/integrators interact with
- Test edge cases that could cause real problems
- Don't test obvious behavior or standard library functionality
- Don't test that dependencies work as documented
4. **Keep Tests Focused**
- Each test should verify one clear functional behavior
- Avoid testing multiple unrelated things in one test
- Remove tests that don't catch meaningful bugs
### What to Test
✅ **DO test:**
- Tool functionality: "Does find_issues return correct data?"
- API integration: "Does our client handle API errors correctly?"
- Error handling: "Do we provide helpful error messages?"
- Edge cases: "What happens with empty results, special characters, etc.?"
- Configuration: "Can the server be configured for different deployment modes?"
❌ **DON'T test:**
- Standard library behavior (Promise.all, Array.map, etc.)
- Third-party package internals (Zod validation, MSW mocking, etc.)
- Implementation details (private methods, internal state)
- Obvious behavior that will break immediately if wrong
### Example: Context Passing
**Bad approach** (testing language features):
```typescript
// ❌ Don't test that closures capture variables
it("closures capture context", async () => {
const context = { value: 42 };
const fn = () => context;
expect(fn().value).toBe(42);
});
```
**Good approach** (testing our functionality):
```typescript
// ✅ Test that server configuration works
it("builds server with context for tool handlers", async () => {
const server = buildServer({ context });
expect(server).toBeDefined();
});
```
## Testing Levels
### 1. Functional Tests
Fast, focused tests of actual functionality:
- Located alongside source files (`*.test.ts`)
- Use Vitest with inline snapshots
- Mock external APIs only (Sentry API, OpenAI) with MSW
- Use real implementations for internal code
- Test through public APIs rather than implementation details
### 2. Evaluation Tests
Real-world scenarios with LLM:
- Located in `packages/mcp-server-evals`
- Use actual AI models
- Verify end-to-end functionality
- Test complete workflows
### 3. Manual Testing
Interactive testing with the MCP test client (preferred for testing MCP changes):
```bash
# Test with local dev server (default: http://localhost:5173)
pnpm -w run cli "who am I?"
# Test agent mode (use_sentry tool only) - approximately 2x slower
pnpm -w run cli --agent "who am I?"
# Test against production
pnpm -w run cli --mcp-host=https://mcp.sentry.dev "query"
# Test with local stdio mode (requires SENTRY_ACCESS_TOKEN)
pnpm -w run cli --access-token=TOKEN "query"
```
**When to use manual testing:**
- Verifying end-to-end MCP server behavior
- Testing OAuth flows
- Debugging tool interactions
- Validating real API responses
- Testing AI-powered tools (search_events, search_issues, use_sentry)
**Note:** The CLI defaults to `http://localhost:5173` for easier local development. Override with `--mcp-host` or set `MCP_URL` environment variable to test against different servers.
## Functional Testing Patterns
See `adding-tools.mdc#step-3-add-tests` for the complete tool testing workflow.
### Basic Test Structure
```typescript
describe("tool_name", () => {
it("returns formatted output", async () => {
const result = await TOOL_HANDLERS.tool_name(mockContext, {
organizationSlug: "test-org",
param: "value"
});
expect(result).toMatchInlineSnapshot(`
"# Expected Output
Formatted markdown response"
`);
});
});
```
**NOTE**: Follow error handling patterns from `common-patterns.mdc#error-handling` when testing error cases.
### Testing Error Cases
```typescript
it("validates required parameters", async () => {
await expect(
TOOL_HANDLERS.tool_name(mockContext, {})
).rejects.toThrow(UserInputError);
});
it("handles API errors gracefully", async () => {
server.use(
http.get("*/api/0/issues/*", () =>
HttpResponse.json({ detail: "Not found" }, { status: 404 })
)
);
await expect(handler(mockContext, params))
.rejects.toThrow("Issue not found");
});
```
## Mock Server Setup
Use MSW patterns from `api-patterns.mdc#mock-patterns` for API mocking.
### Test Configuration
```typescript
// packages/mcp-server/src/test-utils/setup.ts
import { setupMockServer } from "@sentry-mcp/mocks";
export const mswServer = setupMockServer();
// Global test setup
beforeAll(() => mswServer.listen({ onUnhandledRequest: "error" }));
afterEach(() => mswServer.resetHandlers());
afterAll(() => mswServer.close());
```
### Mock Context
```typescript
export const mockContext: ServerContext = {
host: "sentry.io",
accessToken: "test-token",
organizationSlug: "test-org"
};
```
## Snapshot Testing
### When to Use Snapshots
Use inline snapshots for:
- Tool output formatting
- Error message text
- Markdown responses
- JSON structure validation
### Updating Snapshots
When output changes are intentional:
```bash
cd packages/mcp-server
pnpm vitest --run -u
```
**Always review snapshot changes before committing!**
### Snapshot Best Practices
```typescript
// Good: Inline snapshot for output verification
expect(result).toMatchInlineSnapshot(`
"# Issues in **my-org**
Found 2 unresolved issues"
`);
// Bad: Don't use snapshots for dynamic data
expect(result.timestamp).toMatchInlineSnapshot(); // ❌
```
## Evaluation Testing
### Eval Test Structure
```typescript
import { describeEval } from "vitest-evals";
import { TaskRunner, Factuality } from "./utils";
describeEval("tool-name", {
data: async () => [
{
input: "Natural language request",
expected: "Expected response content"
}
],
task: TaskRunner(), // Uses AI to call tools
scorers: [Factuality()], // Validates output
threshold: 0.6,
timeout: 30000
});
```
### Running Evals
```bash
# Requires OPENAI_API_KEY in .env
pnpm eval
# Run specific eval
pnpm eval tool-name
```
## Test Data Management
### Using Fixtures
```typescript
import { issueFixture } from "@sentry-mcp/mocks";
// Modify fixture for test case
const customIssue = {
...issueFixture,
status: "resolved",
id: "CUSTOM-123"
};
```
### Dynamic Test Data
```typescript
// Generate test data
function createTestIssues(count: number) {
return Array.from({ length: count }, (_, i) => ({
...issueFixture,
id: `TEST-${i}`,
title: `Test Issue ${i}`
}));
}
```
## Performance Testing
### Timeout Configuration
```typescript
it("handles large datasets", async () => {
const largeDataset = createTestIssues(1000);
const result = await handler(mockContext, params);
expect(result).toBeDefined();
}, { timeout: 10000 }); // 10 second timeout
```
### Memory Testing
```typescript
it("streams large responses efficiently", async () => {
const initialMemory = process.memoryUsage().heapUsed;
await processLargeDataset();
const memoryIncrease = process.memoryUsage().heapUsed - initialMemory;
expect(memoryIncrease).toBeLessThan(50 * 1024 * 1024); // < 50MB
});
```
## Common Testing Patterns
See `common-patterns.mdc` for:
- Mock server setup
- Error handling tests
- Parameter validation
- Response formatting
## CI/CD Integration
Tests run automatically on:
- Pull requests
- Main branch commits
- Pre-release checks
Coverage requirements:
- Statements: 80%
- Branches: 75%
- Functions: 80%
## References
- Test setup: `packages/mcp-server/src/test-utils/`
- Mock server: `packages/mcp-server-mocks/`
- Eval tests: `packages/mcp-server-evals/`
- Vitest docs: https://vitest.dev/