Skip to main content
Glama
spxrtiat111

2Sense

by spxrtiat111
OverviewSchemaRelated ServersScoreDiscussions
Python
Local

ads-learnings — eyes & ears for Claude

Give Claude the ability to see and hear short-form video ads, so it can give grounded advice on (a) enhancing future content and (b) designing incrementality tests for a content/series format.

Claude is the brain. The pipeline only does perception, delivered as one MCP tool:

analyze_ad(source)                                   ← MCP tool (CLI + Desktop app)
   │  ingest   → yt-dlp (URLs) → local mp4
   │  ffmpeg   → frames @ 3fps → Pillow contact sheets (labeled) + manifest   (EYES)
   │  ffmpeg   → mono wav → Groq whisper-large-v3 → transcript + segments      (EARS)
   │  numpy    → audio energy/rhythm: tempo, energy curve, onsets, dynamics    (EARS+)
   │  YAMNet   → audio-event tags: music/speech/instruments/genre/SFX          (EARS+)
   ▼
returns: [ text (timestamp legend + transcript), sheet image, sheet image, … ]
   ▼
Claude sees the sheets + reads the transcript → teardown + 2 deliverables

No model runs on your Mac — locally it's just ffmpeg (bundled), yt-dlp, and Pillow, managed by uv. The only hosted call is Whisper on Groq.

Where it's wired

The 2Sense MCP server exposes four tools, available in both surfaces:

  • analyze_ad(source, language="auto") — full eyes + ears: contact-sheet images, transcript, audio energy/rhythm profile, AND YAMNet audio-event tags.

  • transcribe(path, language="auto") — speech transcript only (Groq Whisper).

  • audio_profile(path) — music/energy signals (free numpy): tempo, energy curve, onsets.

  • audio_events(path) — YAMNet tags (free, local): music/speech/instruments/genre/SFX

    • a coarse timeline. No mood (happy/sad) — YAMNet covers events, not affect.

Surface

How

Tool namespace

Claude Code (CLI + Code apps, any dir)

user-scope MCP in ~/.claude.json

mcp__2Sense__analyze_ad

Claude Desktop app

claude_desktop_config.json

analyze_ad

This repo (portable)

project .mcp.json

—

Claude Code also gets the ad-learnings skill + ad-eyes sub-agent (symlinked into ~/.claude/) for the guided teardown workflow. The Desktop app uses the MCP tool directly.

Related MCP server: video-reader-mcp

Setup

Prereq: uv. Then one command (idempotent — installs deps, writes a local .mcp.json, wires the MCP into Claude Code + the Desktop app, links the skill/agent, runs a health check):

bin/setup

Then:

  1. Paste your free Groq key into .env → GROQ_API_KEY=... (get one)

  2. Restart Claude Code and the Claude Desktop app so 2Sense loads.

  3. Verify: bin/ee doc --ping

Notes: the project pins Python 3.12 (TensorFlow/YAMNet). The first audio_events / analyze_ad call downloads the ~15 MB YAMNet model once (then cached). .mcp.json is generated by bin/setup (gitignored — see .mcp.json.example). Disable the audio layers in config.toml ([audio] profile, yamnet) for speech-only ears.

Use

  • Claude Code or Desktop: "analyze this ad: " → Claude calls analyze_ad, sees the sheets, reads the transcript, and produces the analysis.

  • CLI prep only (no LLM): bin/ee prep "<path-or-url>" → data/out/<slug>/.

Output per video: frames/, sheets/, manifest.json, audio.wav, prep.json (+ ears.json when transcribed).

Config

Edit config.toml: fps, max_frames, sheet grid (cols/rows), Whisper model, language.

Pieces

  • src/eyesears/ — prep CLI (ee) + 2Sense MCP (ee-ears: analyze_ad, transcribe, audio_profile)

  • src/eyesears/audio_features.py — free numpy music/energy analysis

  • src/eyesears/yamnet.py — free YAMNet audio-event tagging (TensorFlow, lazy-loaded)

  • .claude/agents/ad-eyes.md — vision sub-agent (Claude Code batch optimization)

  • .claude/skills/ad-learnings/ — orchestration skill

  • bin/setup — one-time installer/wiring (generates .mcp.json, registers the MCP, links skill/agent)

  • .mcp.json.example — template; bin/setup writes the real .mcp.json (gitignored) with local paths

License

MIT — see LICENSE.

Available Tools

4 tools
analyze_adA

Give Claude EYES + EARS on a short-form video ad.

Runs the local prep pipeline (download if a URL, sample frames at the configured
fps, tile them into labeled contact sheets, extract audio), transcribes the audio,
and returns: a text block (timestamp legend + transcript) followed by the contact
sheet IMAGES. Look at the images and read the transcript to analyze the ad.

Args:
    source: A local video path OR a TikTok / Reels / YouTube / Meta URL.
    language: "auto", or an ISO code like "fr" / "en" to force transcription language.
    with_audio: set false to skip transcription (eyes only).

Returns: [text, image, image, ...] — contact sheets each have per-cell timestamps
burned into the top-left as `t=SECONDS`, matching the legend in the text block.
ParametersJSON Schema
NameRequiredDescriptionDefault
sourceYes
languageNoauto
with_audioNo

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden. It discloses the heavy processing pipeline: 'download if a URL, sample frames at configured fps, tile them into labeled contact sheets, extract audio, transcribe.' It also describes the return format. No contradictions or hidden behaviors are evident.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is relatively long but each sentence adds value. It front-loads the main purpose and then details the process and parameters. Minor redundancy (e.g., 'Runs the local prep pipeline' followed by specifics) but overall efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no annotations and no output schema, the description is remarkably complete. It covers the tool's operation, parameter options, return format, and even hints at output structure. The agent has all needed information to invoke and process the result.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must explain all parameters. It does so thoroughly: source (local path or multiple URL types), language (auto or ISO code), with_audio (boolean to skip transcription). Defaults are included, making the semantics clear.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Give Claude EYES + EARS on a short-form video ad.' It specifies the verb (analyze), resource (short-form video ad), and scope (by returning images and transcript). This distinguishes it from sibling tools like transcribe, which only provides transcription.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for analyzing short-form video ads and tells the agent to 'Look at the images and read the transcript to analyze the ad.' However, it does not explicitly state when to use this tool versus alternatives like transcribe or when not to use it (e.g., for long-form content).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

audio_eventsA

Tag audio events with YAMNet (free, local): Music, Speech, instruments, genres, SFX (whoosh/beep/...). Works on an audio OR video file.

Returns {top:[{label,score}], rollup:{Music,Speech,SFX}, timeline:[{start,end,label}]}.
No mood (happy/sad) — YAMNet covers events/instruments/genres, not affect.
ParametersJSON Schema
NameRequiredDescriptionDefault
pathYes
top_kNo

TDQS

A4.4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description fully carries the transparency burden. It discloses the return format ({top, rollup, timeline}) and explicitly notes the limitation regarding mood analysis. It also mentions YAMNet is free and local, which implies no external API calls or costs.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise with two well-structured paragraphs. The first line captures the core purpose and categories, and the second adds output format and limitation. No unnecessary words, and key information is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given lack of annotations and output schema, the description provides solid context: input type, output structure, tool capability, and limitation. It could mention that path is required and top_k is optional with a default, but overall it is sufficient for an agent to use the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has two parameters (path, top_k) with no descriptions. The description explains the purpose of the tool but does not explain top_k's role or default value. While schema coverage is 0%, the description partially compensates by explaining the output structure, leaving top_k inferred but not explicit.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool tags audio events using YAMNet, listing specific categories (Music, Speech, instruments, genres, SFX). It explicitly differentiates from mood analysis, which helps distinguish from sibling tools like audio_profile or transcribe.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context on when to use: for event/instrument/genre tagging on audio or video files. It explicitly states that YAMNet does not cover mood, guiding agents away from misuse. While it doesn't name alternatives, the context and sibling tools list imply differentiation.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

audio_profileA

Analyze music/energy/rhythm of an audio OR video file (free, local, numpy-based).

Returns tempo (BPM), energy curve, onset 'hits' (useful vs. visual cuts), loudness dynamics, brightness, and a music-vs-speech estimate. No genre/mood/song-ID.

ParametersJSON Schema
NameRequiredDescriptionDefault
pathYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden and does well: it states the tool is free, local, numpy-based, lists returned data (tempo, energy curve, onset hits, loudness dynamics, brightness, music-vs-speech estimate), and explicitly notes what it does not provide (genre/mood/song-ID). No contradictions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise, uses bullet points efficiently, front-loads the key purpose, and every sentence adds value without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has only one parameter and no output schema, the description thoroughly explains both the input (file path) and all outputs (tempo, energy curve, onset hits, etc.), along with limitations. It is complete for the tool's complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has one required parameter 'path' with 0% description coverage. The description adds meaning by specifying it accepts audio or video file paths, which clarifies the parameter's purpose. For a single parameter, this is adequate, though it could provide more detail on supported formats.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool analyzes music/energy/rhythm of audio or video files, listing specific outputs like tempo, BPM, energy curve, onset hits, etc., distinguishing it from siblings like transcribe or audio_events which focus on speech or event detection.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implicitly indicates usage for music/energy analysis by listing what it returns and explicitly stating what it does not provide (no genre/mood/song-ID). It could be more explicit about when to use over alternatives, but the sibling tools have different purposes (ad analysis, audio events, transcription).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

transcribeA

Transcribe speech from a local audio OR video file using Groq Whisper.

Args:
    path: Absolute path to an audio or video file (video audio is auto-extracted).
    language: "auto" to detect, or an ISO code like "fr" / "en" to force.

Returns: {text, language, duration, segments:[{start,end,text}]}.
ParametersJSON Schema
NameRequiredDescriptionDefault
pathYes
languageNoauto

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses use of Groq Whisper, auto-extraction of video audio, and return format. However, it does not mention potential side effects, rate limits, supported file formats, or error behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise (4 lines), uses clear sections (Args, Returns), and contains no redundant information. Every sentence adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple two-parameter tool without output schema, the description covers inputs and outputs adequately. However, it lacks context on supported file extensions, maximum file size, and error handling, which would be helpful for an agent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description compensates well: it clarifies that 'path' must be an absolute path and 'language' accepts 'auto' or ISO codes. This adds meaningful context beyond the bare schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly specifies the verb 'transcribe' and the resource 'speech from a local audio or video file', and distinguishes itself from siblings (analyze_ad, audio_events, audio_profile) by focusing on transcription.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives; no mention of prerequisites, limitations, or specific scenarios. The description only states what it does, not when to prefer it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Install Server
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A3.9/5.0
Disambiguation4/5

Tools have largely distinct purposes: analyze_ad combines visual and audio ad analysis, audio_events focuses on event detection, audio_profile on musical attributes, and transcribe on speech transcription. Some overlap exists (analyze_ad includes transcription), but descriptions clarify boundaries.

Naming Consistency2/5

Naming is inconsistent: analyze_ad uses verb_noun with underscore, audio_events and audio_profile use noun_noun, and transcribe is a single verb. No consistent pattern emerges.

Tool Count4/5

With 4 tools, the set is slightly lean but covers essential analysis tasks for audio/video. Each tool serves a clear purpose, though there is room for additional utilities.

Completeness4/5

The tool surface covers core ad analysis (visual frames, transcription, audio events, music profiling). Missing features like emotional analysis or speaker identification are minor gaps not critical for the primary use case.

Maintenance

ActivityStale
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/spxrtiat111/2sense'

If you have feedback or need assistance with the MCP directory API, please join our Discord server