xCOMET MCP Server

日本語版 README はこちら

⚠️ This is an unofficial community project, not affiliated with Unbabel.

Translation quality evaluation MCP Server powered by xCOMET (eXplainable COMET).

🎯 Overview

xCOMET MCP Server provides AI agents with the ability to evaluate machine translation quality. It integrates with the xCOMET model from Unbabel to provide:

• Quality Scoring: Scores between 0-1 indicating translation quality

• Error Detection: Identifies error spans with severity levels (minor/major/critical)

• Batch Processing: Evaluate multiple translation pairs efficiently (optimized single model load)

• GPU Support: Optional GPU acceleration for faster inference

🔧 Prerequisites

Python Environment

xCOMET requires Python with the following packages:

Model Download

The first run will download the xCOMET model (~14GB for XL, ~42GB for XXL):

Node.js

• Node.js >= 18.0.0

• npm or yarn

📦 Installation

🚀 Usage

With Claude Desktop (npx)

Add to your Claude Desktop configuration (claude_desktop_config.json):

With Claude Code

Local Installation

If you prefer a local installation:

Then configure:

HTTP Mode (Remote Access)

Then connect to http://localhost:3000/mcp

🛠️ Available Tools

xcomet_evaluate

Evaluate translation quality for a single source-translation pair.

Parameters:

Example:

Response:

xcomet_detect_errors

Focus on detecting and categorizing translation errors.

Parameters:

xcomet_batch_evaluate

Evaluate multiple translation pairs in a single request.

Performance Note: With the persistent server architecture (v0.3.0+), the model stays loaded in memory. Batch evaluation processes all pairs efficiently without reloading the model.

Parameters:

Example:

🔗 Integration with Other MCP Servers

xCOMET MCP Server is designed to work alongside other MCP servers for complete translation workflows:

Recommended Workflow

1. Translate using DeepL MCP Server (official)

2. Evaluate using xCOMET MCP Server

3. Iterate if quality is below threshold

Example: DeepL + xCOMET Integration

Configure both servers in Claude Desktop:

Then ask Claude:

"Translate this text to Japanese using DeepL, then evaluate the translation quality with xCOMET. If the score is below 0.8, suggest improvements."

⚙️ Configuration

Environment Variables

Model Selection

Choose the model based on your quality/performance needs:

Important: wmt22-comet-da requires a reference translation for evaluation. XCOMET models support referenceless evaluation.

Tip: If you experience memory issues or slow model loading, try Unbabel/wmt22-comet-da for faster performance with slightly lower accuracy (but remember to provide reference translations).

To use a different model, set the XCOMET_MODEL environment variable:

Python Path Auto-Detection

The server automatically detects a Python environment with unbabel-comet installed:

1. XCOMET_PYTHON_PATH environment variable (if set)

2. pyenv versions (~/.pyenv/versions/*/bin/python3) - checks for comet module

3. Homebrew Python (/opt/homebrew/bin/python3, /usr/local/bin/python3)

4. Fallback: python3 command

This ensures the server works correctly even when the MCP host (e.g., Claude Desktop) uses a different Python than your terminal.

Example: Explicit Python path configuration

⚡ Performance

Persistent Server Architecture (v0.3.0+)

The server uses a persistent Python FastAPI server that keeps the xCOMET model loaded in memory:

This provides a 177x speedup for consecutive evaluations compared to reloading the model each time.

Eager Loading (v0.3.1+)

Enable XCOMET_PRELOAD=true to pre-load the model at server startup:

With preload enabled, all requests are fast (~500ms), including the first one.

Batch Processing Optimization

The xcomet_batch_evaluate tool processes all pairs with a single model load:

GPU vs CPU Performance

Note: GPU requires CUDA-compatible hardware and PyTorch with CUDA support. If GPU is not available, set use_gpu: false (default).

Best Practices

1. Let the persistent server do its job

With v0.3.0+, the model stays in memory. Multiple xcomet_evaluate calls are now efficient:

2. For many pairs, use batch evaluation

3. Memory considerations

• XCOMET-XL requires ~8-10GB RAM

• For large batches (500 pairs), ensure sufficient memory

• If memory is limited, split into smaller batches (100-200 pairs)

Auto-Restart (v0.3.1+)

The server automatically recovers from failures:

• Monitors health every 30 seconds

• Restarts after 3 consecutive health check failures

• Up to 3 restart attempts before giving up

📊 Quality Score Interpretation

🔍 Troubleshooting

Common Issues

"No module named 'comet'"

Cause: Python environment without unbabel-comet installed.

Solution:

Model download fails or times out

Cause: Large model files (~14GB for XL) require stable internet connection.

Solution:

GPU not detected

Cause: PyTorch not installed with CUDA support.

Solution:

Slow performance on Mac (MPS)

Cause: Mac MPS (Metal Performance Shaders) has compatibility issues with some operations.

Solution: The server automatically uses num_workers=1 for Mac MPS compatibility. For best performance on Mac, use CPU mode (use_gpu: false).

High memory usage or crashes

Cause: XCOMET-XL requires ~8-10GB RAM.

Solutions:

1. Use the persistent server (v0.3.0+): Model loads once and stays in memory, avoiding repeated memory spikes

2. Use a lighter model: Set XCOMET_MODEL=Unbabel/wmt22-comet-da for lower memory usage (~3GB)

3. Reduce batch size: For large batches, process in smaller chunks (100-200 pairs)

4. Close other applications: Free up RAM before running large evaluations

VS Code or IDE crashes during evaluation

Cause: High memory usage from the xCOMET model (~8-10GB for XL).

Solution:

• With v0.3.0+, the model loads once and stays in memory (no repeated loading)

• If memory is still an issue, use a lighter model: XCOMET_MODEL=Unbabel/wmt22-comet-da

• Close other memory-intensive applications before evaluation

Getting Help

If you encounter issues:

1. Check the GitHub Issues

2. Enable debug logging by checking Claude Desktop's Developer Mode logs

3. Open a new issue with:

   • Your OS and Python version

   • The error message

   • Your configuration (without sensitive data)

🧪 Development

📋 Changelog

See CHANGELOG.md for version history and updates.

📝 License

MIT License - see LICENSE for details.

🙏 Acknowledgments

• Unbabel for the xCOMET model

• Anthropic for the MCP protocol

• Model Context Protocol community

📚 References

• xCOMET Paper

• COMET Framework

• MCP Specification