ads-learnings — eyes & ears for Claude

Give Claude the ability to see and hear short-form video ads, so it can give grounded advice on (a) enhancing future content and (b) designing incrementality tests for a content/series format.

Claude is the brain. The pipeline only does perception, delivered as one MCP tool:

analyze_ad(source)                                   ← MCP tool (CLI + Desktop app)
   │  ingest   → yt-dlp (URLs) → local mp4
   │  ffmpeg   → frames @ 3fps → Pillow contact sheets (labeled) + manifest   (EYES)
   │  ffmpeg   → mono wav → Groq whisper-large-v3 → transcript + segments      (EARS)
   │  numpy    → audio energy/rhythm: tempo, energy curve, onsets, dynamics    (EARS+)
   │  YAMNet   → audio-event tags: music/speech/instruments/genre/SFX          (EARS+)
   ▼
returns: [ text (timestamp legend + transcript), sheet image, sheet image, … ]
   ▼
Claude sees the sheets + reads the transcript → teardown + 2 deliverables

No model runs on your Mac — locally it's just ffmpeg (bundled), yt-dlp, and Pillow, managed by uv. The only hosted call is Whisper on Groq.

Where it's wired

The 2Sense MCP server exposes four tools, available in both surfaces:

• analyze_ad(source, language="auto") — full eyes + ears: contact-sheet images, transcript, audio energy/rhythm profile, AND YAMNet audio-event tags.

• transcribe(path, language="auto") — speech transcript only (Groq Whisper).

• audio_profile(path) — music/energy signals (free numpy): tempo, energy curve, onsets.

• audio_events(path) — YAMNet tags (free, local): music/speech/instruments/genre/SFX

  • a coarse timeline. No mood (happy/sad) — YAMNet covers events, not affect.

Claude Code also gets the ad-learnings skill + ad-eyes sub-agent (symlinked into ~/.claude/) for the guided teardown workflow. The Desktop app uses the MCP tool directly.

Related MCP server: ffmpeg-mcp-server

Setup

Prereq: uv. Then one command (idempotent — installs deps, writes a local .mcp.json, wires the MCP into Claude Code + the Desktop app, links the skill/agent, runs a health check):

bin/setup

Then:

1. Paste your free Groq key into .env → GROQ_API_KEY=... (get one)

2. Restart Claude Code and the Claude Desktop app so 2Sense loads.

3. Verify: bin/ee doc --ping

Notes: the project pins Python 3.12 (TensorFlow/YAMNet). The first audio_events / analyze_ad call downloads the ~15 MB YAMNet model once (then cached). .mcp.json is generated by bin/setup (gitignored — see .mcp.json.example). Disable the audio layers in config.toml ([audio] profile, yamnet) for speech-only ears.

Use

• Claude Code or Desktop: "analyze this ad: " → Claude calls analyze_ad, sees the sheets, reads the transcript, and produces the analysis.

• CLI prep only (no LLM): bin/ee prep "<path-or-url>" → data/out/<slug>/.

Output per video: frames/, sheets/, manifest.json, audio.wav, prep.json (+ ears.json when transcribed).

Config

Edit config.toml: fps, max_frames, sheet grid (cols/rows), Whisper model, language.

Pieces

• src/eyesears/ — prep CLI (ee) + 2Sense MCP (ee-ears: analyze_ad, transcribe, audio_profile)

• src/eyesears/audio_features.py — free numpy music/energy analysis

• src/eyesears/yamnet.py — free YAMNet audio-event tagging (TensorFlow, lazy-loaded)

• .claude/agents/ad-eyes.md — vision sub-agent (Claude Code batch optimization)

• .claude/skills/ad-learnings/ — orchestration skill

• bin/setup — one-time installer/wiring (generates .mcp.json, registers the MCP, links skill/agent)

• .mcp.json.example — template; bin/setup writes the real .mcp.json (gitignored) with local paths

License

MIT — see LICENSE.