# mcp-meeting-analyzer

MCP server for analysing recorded meetings, optimised for efficient context delivery to an LLM.

## Problem

Recorded meetings are long and sending the full video to an LLM is inefficient. Most of the time frames contain only people, with no additional informational value beyond what is already in the audio.

## Solution

A three-stage processing pipeline:

### 1. Transcription with timestamps
- Extracts audio from the video
- Transcribes with `faster-whisper` (local, open-source, 3–4× faster than the original Whisper)
- Produces a segmented transcript with precise timestamps
- Optional diarisation (speaker identification) via `pyannote.audio`

### 2. Frame extraction and classification
- Extracts frames from the video with `opencv-python`
- Detects and discards frames containing people via `mediapipe` (face/person detection)
- Keeps frames with screens, slides or presentations
- Optimisation: only processes frames with significant change relative to the previous one

### 3. Temporal association
- Associates each relevant frame with the corresponding transcript segment
- Produces a structured document combining text and images

### Output

```
[00:05:32] "...the decision was to go with option B..."
[FRAME: slide comparing option A vs B]

[00:12:10] "...the deadline is the 20th..."
[FRAME: timeline on screen]
```

This compact output is sent to the LLM to extract a summary, decisions and action items.

## MCP Tools

### Main pipeline

| Tool | Mode | Description |
|------|------|-------------|
| `process_meeting` | synchronous | Full pipeline — transcription + frames + temporal association |
| `start_process_meeting` | asynchronous | Launches the pipeline in the background; returns `job_id` immediately |
| `get_process_meeting_status` | — | Polls pipeline status; returns result when complete |

### Individual tools

| Tool | Mode | Description |
|------|------|-------------|
| `transcribe` | synchronous | Transcription only, no frames |
| `start_transcription` | asynchronous | Transcription in background; returns `job_id` immediately |
| `get_transcription_status` | — | Polls transcription status |
| `extract_frames` | synchronous | Frame extraction only, no transcription |

### Session management

| Tool | Description |
|------|-------------|
| `get_frame` | Retrieves a specific frame as a base64 JPEG |
| `close_session` | Removes session from the registry and deletes files from disk |

## Tech Stack

- **MCP SDK:** `mcp` (Python, official Anthropic)
- **Transcription:** `faster-whisper`
- **Diarisation:** `pyannote.audio` (optional)
- **Video/frames:** `opencv-python` (cv2)
- **Person detection:** `mediapipe`
- **Text/slide detection:** `opencv-python` + `easyocr`

## System Requirements

Binaries that must be installed **before** using the MCP:

| Binary | Purpose | Installation (Ubuntu/Debian) |
|--------|---------|-------------------------------|
| `ffmpeg` | Audio extraction from video (used by `faster-whisper` via PyAV) | `sudo apt install ffmpeg` |
| `libgles2` | Required by mediapipe for face detection | `sudo apt install libgles2` |
| `python3.12+` | Runtime | `sudo apt install python3.12` |
| `uv` | Python package/environment manager | `curl -LsSf https://astral.sh/uv/install.sh \| sh` |

The MCP checks these requirements at startup and reports any that are missing.

## Installation

```bash
git clone https://github.com/mourabraz/mcp-meeting-analyzer
cd mcp-meeting-analyzer
uv sync
```

Download the mediapipe face detection model:

```bash
mkdir -p models
curl -L -o models/blaze_face_short_range.tflite \
  https://storage.googleapis.com/mediapipe-models/face_detector/blaze_face_short_range/float16/1/blaze_face_short_range.tflite
```

## Claude Desktop Configuration

Add to your `claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "meeting-analyzer": {
      "command": "uv",
      "args": [
        "run",
        "--directory",
        "/path/to/mcp-meeting-analyzer",
        "python",
        "main.py"
      ]
    }
  }
}
```

## Project Structure

```
mcp-meeting-analyzer/
├── meeting_analyzer/          # main package
│   ├── core/                  # pure domain logic — no MCP, no I/O side effects
│   │   ├── transcriber.py     # transcribe_video() — audio extraction + Whisper
│   │   ├── frame_extractor.py # extract_frames() — capture + classify + dedup pipeline
│   │   ├── frame_classifier.py# classify_frame() — screen vs people via mediapipe
│   │   ├── frame_dedup.py     # deduplicate_frames(), compute_ssim() — SSIM-based dedup
│   │   └── pipeline.py        # process_meeting() — full pipeline orchestration
│   ├── tools/                 # MCP tool handlers (register(mcp) pattern)
│   │   ├── pipeline.py        # process_meeting, start_process_meeting, get_process_meeting_status
│   │   ├── transcription.py   # transcribe, start_transcription, get_transcription_status
│   │   ├── frames.py          # extract_frames, get_frame
│   │   └── session.py         # close_session
│   ├── infra/                 # infrastructure concerns
│   │   ├── checks.py          # startup dependency checks (ffmpeg, libgles2, …)
│   │   ├── debug.py           # MCP_DEBUG=1 structured logging
│   │   └── sessions.py        # in-memory session registry + disk persistence
│   └── server.py              # FastMCP server entry point
├── tests/
│   ├── unit/                  # pytest unit tests (fast, no video files needed)
│   │   ├── conftest.py        # shared fixtures (synthetic numpy frames)
│   │   ├── test_pipeline.py   # _associate_frames_to_segments, _split_inline_on_demand
│   │   ├── test_frame_dedup.py# compute_ssim, deduplicate_frames
│   │   └── test_frame_classifier.py # classify_frame (mocked detector), _compute_content_score
│   ├── test_transcribe.py     # manual integration script — run with uv run python
│   ├── test_e2e_frames.py     # manual integration script — frame extraction smoke test
│   └── test_dedup.py          # manual integration script — deduplication smoke test
├── models/
│   └── blaze_face_short_range.tflite  # mediapipe model (gitignored, download separately)
├── examples/                  # sample video files for manual testing
└── pyproject.toml
```

## Debug Mode

Set `MCP_DEBUG=1` to enable structured logging of all tool calls to `debug-logs/`.

## Status

- [x] Project structure
- [x] Python environment setup
- [x] Tool: transcription (sync and async)
- [x] Tool: frame extraction
- [x] Tool: full pipeline (sync and async)
- [x] Tool: get_frame (on-demand retrieval)
- [x] Tool: close_session (session cleanup)
- [x] Persistent sessions (survive server restart)

## Tests

There are two kinds of tests: **unit tests** (pytest, fast) and **manual integration scripts** (run directly with `uv run python`).

### Unit tests

```bash
# Run all unit tests
uv run pytest tests/unit/

# Verbose output — shows each test name
uv run pytest tests/unit/ -v

# Filter by name
uv run pytest tests/unit/ -k "test_ssim"

# Short traceback on failures
uv run pytest tests/unit/ --tb=short
```

The unit tests cover `core/` only. They use synthetic numpy arrays instead of real video files, so they run in under 2 seconds and require no external files. The face detector (`_create_face_detector`) is automatically skipped if the `.tflite` model is absent.

| Test file | Module under test | Strategy |
|---|---|---|
| `test_pipeline.py` | `core/pipeline.py` | Pure functions — no I/O, no mocking |
| `test_frame_dedup.py` | `core/frame_dedup.py` | `compute_ssim` with numpy arrays; `deduplicate_frames` with `tmp_path` |
| `test_frame_classifier.py` | `core/frame_classifier.py` | `MagicMock` to replace the mediapipe detector |

### Manual integration scripts

These scripts require real video files and a working environment (model downloaded, ffmpeg installed).

```bash
# Transcription smoke test
uv run python tests/test_transcribe.py examples/electrao_part1.mp4 pt small

# Frame extraction smoke test (writes frames to output/)
uv run python tests/test_e2e_frames.py

# Deduplication smoke test (requires frames already extracted under output/frames_classified/)
uv run python tests/test_dedup.py
```

### Linting

```bash
uv run ruff check .
```

## Contributing

Contributions are welcome. Please open an issue first to discuss what you would like to change.

When submitting a pull request:
- Keep changes focused — one concern per PR
- Follow the existing code style (enforced by `ruff`)
- Run `uv run ruff check .` before submitting

## Development

Co-authored with [Claude Code](https://claude.ai/code).  
Every design decision was a conversation.

## License

MIT