# ads-learnings — eyes & ears for Claude

Give Claude the ability to **see and hear short-form video ads**, so it can give grounded
advice on (a) enhancing future content and (b) designing incrementality tests for a
content/series format.

Claude is the **brain**. The pipeline only does **perception**, delivered as one MCP tool:

```
analyze_ad(source)                                   ← MCP tool (CLI + Desktop app)
   │  ingest   → yt-dlp (URLs) → local mp4
   │  ffmpeg   → frames @ 3fps → Pillow contact sheets (labeled) + manifest   (EYES)
   │  ffmpeg   → mono wav → Groq whisper-large-v3 → transcript + segments      (EARS)
   │  numpy    → audio energy/rhythm: tempo, energy curve, onsets, dynamics    (EARS+)
   │  YAMNet   → audio-event tags: music/speech/instruments/genre/SFX          (EARS+)
   ▼
returns: [ text (timestamp legend + transcript), sheet image, sheet image, … ]
   ▼
Claude sees the sheets + reads the transcript → teardown + 2 deliverables
```

No model runs on your Mac — locally it's just `ffmpeg` (bundled), `yt-dlp`, and Pillow,
managed by `uv`. The only hosted call is Whisper on Groq.

## Where it's wired

The `2Sense` MCP server exposes four tools, available in **both** surfaces:
- **`analyze_ad(source, language="auto")`** — full eyes + ears: contact-sheet images,
  transcript, audio energy/rhythm profile, AND YAMNet audio-event tags.
- **`transcribe(path, language="auto")`** — speech transcript only (Groq Whisper).
- **`audio_profile(path)`** — music/energy signals (free numpy): tempo, energy curve, onsets.
- **`audio_events(path)`** — YAMNet tags (free, local): music/speech/instruments/genre/SFX
  + a coarse timeline. No mood (happy/sad) — YAMNet covers events, not affect.

| Surface | How | Tool namespace |
|---|---|---|
| Claude Code (CLI + Code apps, any dir) | user-scope MCP in `~/.claude.json` | `mcp__2Sense__analyze_ad` |
| Claude Desktop app | `claude_desktop_config.json` | `analyze_ad` |
| This repo (portable) | project `.mcp.json` | — |

Claude Code also gets the **`ad-learnings` skill** + **`ad-eyes` sub-agent** (symlinked into
`~/.claude/`) for the guided teardown workflow. The Desktop app uses the MCP tool directly.

## Setup

Prereq: [`uv`](https://docs.astral.sh/uv/). Then one command (idempotent — installs deps,
writes a local `.mcp.json`, wires the MCP into Claude Code + the Desktop app, links the
skill/agent, runs a health check):

```bash
bin/setup
```

Then:
1. Paste your free Groq key into `.env` → `GROQ_API_KEY=...` ([get one](https://console.groq.com/keys))
2. **Restart Claude Code and the Claude Desktop app** so `2Sense` loads.
3. Verify: `bin/ee doc --ping`

> Notes: the project pins **Python 3.12** (TensorFlow/YAMNet). The first `audio_events` /
> `analyze_ad` call downloads the ~15 MB YAMNet model once (then cached). `.mcp.json` is
> generated by `bin/setup` (gitignored — see `.mcp.json.example`). Disable the audio layers
> in `config.toml` (`[audio] profile`, `yamnet`) for speech-only ears.

## Use

- **Claude Code or Desktop:** *"analyze this ad: <path or URL>"* → Claude calls
  `analyze_ad`, sees the sheets, reads the transcript, and produces the analysis.
- **CLI prep only (no LLM):** `bin/ee prep "<path-or-url>"` → `data/out/<slug>/`.

Output per video: `frames/`, `sheets/`, `manifest.json`, `audio.wav`, `prep.json`
(+ `ears.json` when transcribed).

## Config
Edit `config.toml`: `fps`, `max_frames`, sheet grid (`cols`/`rows`), Whisper `model`,
`language`.

## Pieces
- `src/eyesears/` — prep CLI (`ee`) + 2Sense MCP (`ee-ears`: `analyze_ad`, `transcribe`, `audio_profile`)
- `src/eyesears/audio_features.py` — free numpy music/energy analysis
- `src/eyesears/yamnet.py` — free YAMNet audio-event tagging (TensorFlow, lazy-loaded)
- `.claude/agents/ad-eyes.md` — vision sub-agent (Claude Code batch optimization)
- `.claude/skills/ad-learnings/` — orchestration skill
- `bin/setup` — one-time installer/wiring (generates `.mcp.json`, registers the MCP, links skill/agent)
- `.mcp.json.example` — template; `bin/setup` writes the real `.mcp.json` (gitignored) with local paths

## License

MIT — see [LICENSE](LICENSE).