# Test Runner MCP
A Model Context Protocol (MCP) server for running and parsing test results from multiple testing frameworks. This server provides a unified interface for executing tests and processing their outputs, supporting:
- Bats (Bash Automated Testing System)
- Pytest (Python Testing Framework)
- Flutter Tests
- Jest (JavaScript Testing Framework)
- Go Tests
- Rust Tests (Cargo test)
- Generic (for arbitrary command execution)
## Installation
```bash
npm install test-runner-mcp
```
## Prerequisites
The following test frameworks need to be installed for their respective test types:
- Bats: `apt-get install bats` or `brew install bats`
- Pytest: `pip install pytest`
- Flutter: Follow [Flutter installation guide](https://flutter.dev/docs/get-started/install)
- Jest: `npm install --save-dev jest`
- Go: Follow [Go installation guide](https://go.dev/doc/install)
- Rust: Follow [Rust installation guide](https://www.rust-lang.org/tools/install)
## Usage
### Configuration
Add the test-runner to your MCP settings (e.g., in `claude_desktop_config.json` or `cline_mcp_settings.json`):
```json
{
"mcpServers": {
"test-runner": {
"command": "node",
"args": ["/path/to/test-runner-mcp/build/index.js"],
"env": {
"NODE_PATH": "/path/to/test-runner-mcp/node_modules",
// Flutter-specific environment (required for Flutter tests)
"FLUTTER_ROOT": "/opt/homebrew/Caskroom/flutter/3.27.2/flutter",
"PUB_CACHE": "/Users/username/.pub-cache",
"PATH": "/opt/homebrew/Caskroom/flutter/3.27.2/flutter/bin:/usr/local/bin:/usr/bin:/bin"
}
}
}
}
```
Note: For Flutter tests, ensure you replace:
- `/opt/homebrew/Caskroom/flutter/3.27.2/flutter` with your actual Flutter installation path
- `/Users/username/.pub-cache` with your actual pub cache path
- Update PATH to include your system's actual paths
You can find these values by running:
```bash
# Get Flutter root
flutter --version
# Get pub cache path
echo $PUB_CACHE # or default to $HOME/.pub-cache
# Get Flutter binary path
which flutter
```
### Running Tests
Use the `run_tests` tool with the following parameters:
```json
{
"command": "test command to execute",
"workingDir": "working directory for test execution",
"framework": "bats|pytest|flutter|jest|go|rust|generic",
"outputDir": "directory for test results",
"timeout": "test execution timeout in milliseconds (default: 300000)",
"env": "optional environment variables",
"securityOptions": "optional security options for command execution"
}
```
Example for each framework:
```json
// Bats
{
"command": "bats test/*.bats",
"workingDir": "/path/to/project",
"framework": "bats",
"outputDir": "test_reports"
}
// Pytest
{
"command": "pytest test_file.py -v",
"workingDir": "/path/to/project",
"framework": "pytest",
"outputDir": "test_reports"
}
// Flutter
{
"command": "flutter test test/widget_test.dart",
"workingDir": "/path/to/project",
"framework": "flutter",
"outputDir": "test_reports",
"FLUTTER_ROOT": "/opt/homebrew/Caskroom/flutter/3.27.2/flutter",
"PUB_CACHE": "/Users/username/.pub-cache",
"PATH": "/opt/homebrew/Caskroom/flutter/3.27.2/flutter/bin:/usr/local/bin:/usr/bin:/bin"
}
// Jest
{
"command": "jest test/*.test.js",
"workingDir": "/path/to/project",
"framework": "jest",
"outputDir": "test_reports"
}
// Go
{
"command": "go test ./...",
"workingDir": "/path/to/project",
"framework": "go",
"outputDir": "test_reports"
}
// Rust
{
"command": "cargo test",
"workingDir": "/path/to/project",
"framework": "rust",
"outputDir": "test_reports"
}
// Generic (for arbitrary commands, CI/CD tools, etc.)
{
"command": "act -j build",
"workingDir": "/path/to/project",
"framework": "generic",
"outputDir": "test_reports"
}
// Generic with security overrides
{
"command": "sudo docker-compose -f docker-compose.test.yml up",
"workingDir": "/path/to/project",
"framework": "generic",
"outputDir": "test_reports",
"securityOptions": {
"allowSudo": true
}
}
```
### Security Features
The test-runner includes built-in security features to prevent execution of potentially harmful commands, particularly for the `generic` framework:
1. **Command Validation**
- Blocks `sudo` and `su` by default
- Prevents dangerous commands like `rm -rf /`
- Blocks file system write operations outside of safe locations
2. **Environment Variable Sanitization**
- Filters out potentially dangerous environment variables
- Prevents overriding critical system variables
- Ensures safe path handling
3. **Configurable Security**
- Override security restrictions when necessary via `securityOptions`
- Fine-grained control over security features
- Default safe settings for standard test usage
Security options you can configure:
```json
{
"securityOptions": {
"allowSudo": false, // Allow sudo commands
"allowSu": false, // Allow su commands
"allowShellExpansion": true, // Allow shell expansion like $() or backticks
"allowPipeToFile": false // Allow pipe to file operations (> or >>)
}
}
```
### Flutter Test Support
The test runner includes enhanced support for Flutter tests:
1. Environment Setup
- Automatic Flutter environment configuration
- PATH and PUB_CACHE setup
- Flutter installation verification
2. Error Handling
- Stack trace collection
- Assertion error handling
- Exception capture
- Test failure detection
3. Output Processing
- Complete test output capture
- Stack trace preservation
- Detailed error reporting
- Raw output preservation
### Rust Test Support
The test runner provides specific support for Rust's `cargo test`:
1. Environment Setup
- Automatically sets RUST_BACKTRACE=1 for better error messages
2. Output Parsing
- Parses individual test results
- Captures detailed error messages for failed tests
- Identifies ignored tests
- Extracts summary information
### Generic Test Support
For CI/CD pipelines, GitHub Actions via `act`, or any other command execution, the generic framework provides:
1. Automatic Output Analysis
- Attempts to segment output into logical blocks
- Identifies section headers
- Detects pass/fail indicators
- Provides reasonable output structure even for unknown formats
2. Flexible Integration
- Works with arbitrary shell commands
- No specific format requirements
- Perfect for integration with tools like `act`, Docker, and custom scripts
3. Security Features
- Command validation to prevent harmful operations
- Can be configured to allow specific elevated permissions when necessary
## Output Format
The test runner produces structured output while preserving complete test output:
```typescript
interface TestResult {
name: string;
passed: boolean;
output: string[];
rawOutput?: string; // Complete unprocessed output
}
interface TestSummary {
total: number;
passed: number;
failed: number;
duration?: number;
}
interface ParsedResults {
framework: string;
tests: TestResult[];
summary: TestSummary;
rawOutput: string; // Complete command output
}
```
Results are saved in the specified output directory:
- `test_output.log`: Raw test output
- `test_errors.log`: Error messages if any
- `test_results.json`: Structured test results
- `summary.txt`: Human-readable summary
## Development
### Setup
1. Clone the repository
2. Install dependencies:
```bash
npm install
```
3. Build the project:
```bash
npm run build
```
### Running Tests
```bash
npm test
```
The test suite includes tests for all supported frameworks and verifies both successful and failed test scenarios.
### CI/CD
The project uses GitHub Actions for continuous integration:
- Automated testing on Node.js 18.x and 20.x
- Test results uploaded as artifacts
- Dependabot configured for automated dependency updates
## Contributing
1. Fork the repository
2. Create your feature branch
3. Commit your changes
4. Push to the branch
5. Create a Pull Request
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.