# Token Analyzer MCP

[![CI/CD Pipeline](https://github.com/cordlesssteve/token-analyzer-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/cordlesssteve/token-analyzer-mcp/actions/workflows/ci.yml)
[![npm version](https://badge.fury.io/js/token-analyzer-mcp.svg)](https://www.npmjs.com/package/token-analyzer-mcp)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

A tool for independent verification of token counts and context volume in Model Context Protocol (MCP) server configurations. Provides accurate token consumption analysis to validate and optimize your Claude Code setup.

## Purpose

This tool serves as an **independent verification system** for token usage in MCP configurations. Rather than relying on estimates or built-in metrics, it provides precise token counting and context volume analysis to:

- Verify actual token consumption vs. expected usage
- Validate context window utilization against the 200k token limit
- Identify servers consuming excessive tokens
- Provide independent confirmation of token optimization efforts

## Features

- **Independent Token Verification** - Accurate token counting separate from MCP server self-reporting
- **Context Volume Analysis** - Precise measurement of total context consumption
- **MCP Configuration Scanning** - Automatic discovery and analysis of configured servers
- **Overhead Calculation** - Breakdown of token usage by server and tool definitions
- **Verification Reports** - Detailed analysis for validating token usage claims
- **Quick Estimation Mode** - Fast overhead checking without full server connections

## Installation

```bash
npm install -g token-analyzer-mcp
```

## Usage

### Basic Verification
```bash
# Perform independent token count verification
token-analyzer-mcp analyze

# Quick context overhead verification
token-analyzer-mcp quick

# Validate MCP configuration setup
token-analyzer-mcp config

# Verify analyzer dependencies
token-analyzer-mcp doctor
```

### Verification Reports
```bash
# Generate detailed verification report
token-analyzer-mcp analyze \
  --output verification.json \
  --report token-analysis.txt \
  --csv usage-breakdown.csv
```

## Configuration Discovery

The analyzer automatically discovers MCP configurations from standard locations:

- `~/.claude/claude_desktop_config.json`
- `~/.config/claude-desktop/claude_desktop_config.json`
- Windows: `%APPDATA%/Claude/claude_desktop_config.json`
- macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`

### Supported Configuration Format

```json
{
  "mcpServers": {
    "server-name": {
      "command": "node",
      "args": ["path/to/server.js"],
      "disabled": false
    }
  }
}
```

## Sample Verification Output

```
Token Analyzer - Independent Verification v1.0.0
Verifying MCP server token consumption...

Phase 1: Configuration Discovery
✓ Found 6 servers in ~/.claude/claude_desktop_config.json
✓ 4 active servers to verify

Phase 2: Schema Extraction  
✓ document-organizer: 12 tools extracted
✓ conversation-search: 15 tools extracted
✓ claude-telemetry: 8 tools extracted

Phase 3: Independent Token Verification
✓ Token verification complete

Verification Results:
Built-in Tools:     2,250 tokens (1.13%)
MCP Servers:        8,540 tokens (4.27%)
Total Overhead:    10,790 tokens (5.40%)
Available Context: 189,210 tokens (94.60%)

Status: ✅ Context usage within acceptable limits
```

## Commands

- `analyze` - Full independent verification with detailed breakdown
- `quick` - Fast context overhead verification without server connections
- `config` - Validate MCP configuration accessibility and format
- `doctor` - Verify analyzer dependencies and permissions

## Verification Accuracy

- **Connection Timeout**: 10 seconds per server to ensure accurate results
- **Retry Logic**: Up to 2 retry attempts for reliable token counting
- **Independent Counting**: Separate token calculation engine from MCP servers
- **Cross-Validation**: Multiple counting methods for accuracy verification

## Troubleshooting

### Configuration Not Found
```bash
# Verify configuration locations
token-analyzer-mcp doctor

# Check file permissions and accessibility
ls -la ~/.claude/claude_desktop_config.json
```

### Server Connection Issues
```bash
# Test individual server accessibility
node /path/to/server/index.js

# Run verification with debug output
token-analyzer-mcp analyze --debug
```

### Token Count Discrepancies
```bash
# Generate detailed verification report
token-analyzer-mcp analyze --summary

# Compare with previous verification results
token-analyzer-mcp analyze --report comparison.txt
```

## Development

```bash
# Clone repository
git clone https://github.com/cordlesssteve/token-analyzer-mcp.git
cd token-analyzer-mcp

# Install dependencies
npm install

# Run tests
npm test

# Run analyzer locally
node src/index.js analyze
```

## API Reference

### TokenVerificationEngine
- `performIndependentVerification()` - Execute comprehensive token verification
- `verifyServerTokens(servers)` - Independently verify token consumption
- `validateContextUsage(tokens)` - Check context volume against limits
- `generateVerificationReport(analysis)` - Create verification documentation

### ConfigurationValidator
- `discoverConfigurations()` - Find and validate MCP configuration files
- `verifyServerAccessibility(server)` - Test server connection and response
- `validateConfigurationFormat(config)` - Check configuration file structure
- `calculateBaselineOverhead()` - Measure built-in tool token consumption

## Contributing

1. Fork the repository
2. Create a feature branch: `git checkout -b feature-name`
3. Add verification improvements and tests
4. Run the test suite: `npm test`
5. Submit a pull request

## License

MIT License - see [LICENSE](LICENSE) file for details.

## Support

- [Issues](https://github.com/cordlesssteve/token-analyzer-mcp/issues)
- [Discussions](https://github.com/cordlesssteve/token-analyzer-mcp/discussions)
- [Documentation](https://github.com/cordlesssteve/token-analyzer-mcp/wiki)
- [Changelog](CHANGELOG.md)