Skip to main content
Glama
Podcast Ingestion Core MCP Server

Podcast Ingestion Core MCP Server

by norton77930

Corpus Ingestion Core

Turns podcast audio into verifiable, searchable knowledge. It takes an RSS feed (or an X / YouTube video), transcribes it locally with faster-whisper, and produces timestamped transcripts, summaries, entity mentions, and research artifacts on disk — exposed to AI agents through an MCP server and Skills.

tests license: MIT python: 3.11+

繁體中文說明

In: a podcast RSS feed, or a video URL. Out: transcripts, subtitles, summaries, mention indexes, and research report bundles under data/, plus a SQLite index for cross-episode search.

Nothing this project produces is investment advice.

Why it exists

Podcast audio is hard to cite. A claim you half-remember from an episode three months ago is effectively unrecoverable. This project makes that content addressable: every extracted claim keeps a timestamp back to the audio, so an answer can be traced to the second it was said.

Two properties follow from that goal and shape the whole design:

  • Local-first. Transcription runs on your machine. Audio and transcripts do not leave it unless you explicitly opt into an LLM step.

  • Evidence over inference. The deterministic path is the default. LLM interpretation is a separate, opt-in layer that never overwrites it.

Related MCP server: Podcast Index MCP Server

Quickstart

Requires Python 3.11+. Examples use PowerShell; any shell works.

git clone https://github.com/norton77930/corpus-ingest-core.git
cd corpus-ingest-core
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -e .[dev]

See it work first, in seconds

A synthetic corpus ships already transcribed and indexed, so the search and evidence tools answer before you have downloaded anything:

$env:CORPUS_INGEST_DATA_DIR = "examples/sample-corpus/data"
$env:CORPUS_INGEST_CONFIG   = "examples/sample-corpus/podcasts.yaml"
python scripts/search_transcripts.py --podcast sample --query harbour --limit 5

Unset both variables before running the real pipeline below.

The real pipeline


python scripts/list_episodes.py --podcast gooaye --limit 5
python scripts/download_episode.py --podcast gooaye --episode latest
python scripts/transcribe_episode.py --podcast gooaye --episode latest --model tiny --device cpu --compute-type int8
python scripts/summarize_episode.py --podcast gooaye --episode latest --mode extractive

Start with --model tiny to verify the pipeline end to end; a full episode on CPU is slow. Switch to --model small --device cuda --compute-type float16 once you know it works. Then build the index and search across episodes:

python scripts/rebuild_cache.py --podcast gooaye --force
python scripts/search_transcripts.py --podcast gooaye --query TSMC --limit 10

Add your own podcast by appending a profile to config/podcasts.yaml. The core never hard-codes a specific show.

Connecting an agent, without cloning

The MCP server installs as a command, so an agent can reach it in one line:

claude mcp add corpus-ingest-core -- uvx --from git+https://github.com/norton77930/corpus-ingest-core.git@v0.2.0 corpus-ingest-mcp

examples/ carries ready-to-copy configs for Claude Desktop, Claude Code and Codex, a set of prompts to try, and a small synthetic sample corpus so the search and evidence tools return real results before you have transcribed anything of your own.

What it produces

One episode yields transcripts (.txt, .srt, .json), a summary, a mention index, and a row in the SQLite index. The claim in "Why it exists" is not a slogan -- it is the shape of the data. Every extracted mention carries the segments it came from:

{
  "type": "industry",
  "text": "AI",
  "evidence": [
    {
      "segment_id": 9,
      "timestamp": "[00:02:30 - 00:02:58]",
      "text": "The interesting part is not the bill. It is that they moved the AI workload off rented capacity and onto hardware they own."
    }
  ]
}

So "the show discussed AI infrastructure costs" is never something you have to take on trust. It resolves to a segment id, a timestamp, and the sentence that was actually said -- and from there back to that second of audio.

The deterministic summary is built the same way, quoting segments rather than paraphrasing them:

  ## Timeline Summary

  ### 00:00:00 - 00:05:00

  - Representative segment:
    > Today we are talking about Harbour Robotics, a fictional company that
    > builds picking arms for fictional warehouses.

Both samples above are real output, taken from examples/sample-corpus/ -- a synthetic corpus that ships already indexed, so the search and evidence tools return results before you have transcribed anything of your own.

What works today

Implemented: RSS episode listing and lookup, audio download, local faster-whisper transcription, transcript validation, deterministic extractive summaries, opt-in LLM semantic summaries with a review gate, deterministic mention extraction, SQLite metadata cache and search, X and YouTube video ingest, deterministic research artifacts (episode intelligence, industry chain mapping, external data boundary, stock lens), verified research report bundles with content-digest versioning, and the MCP server over both transports.

Not implemented: web UI, scheduling, embeddings, and vector search. External market data is deliberately bounded to local fixtures — there is no live market API, and adding one would be an explicit, reviewed decision rather than a feature.

Architecture

flowchart TD
    RSS[RSS feed] --> DL
    VID[X / YouTube video] --> DL[Audio download]
    DL --> ASR[faster-whisper<br/>local transcription]
    ASR --> VAL{Transcript<br/>validation}
    VAL -->|complete| DET[Deterministic<br/>extractive summary]
    VAL -->|complete| MEN[Mention extraction<br/>rule-based]
    VAL -->|opt-in| LLM[LLM semantic summary<br/>OpenAI-compatible]
    DET --> IDX[(SQLite index<br/>FTS5 + LIKE fallback)]
    MEN --> IDX
    LLM --> REV[Deterministic<br/>review gate]
    REV --> IDX
    IDX --> MCP[FastMCP server<br/>stdio + loopback HTTP]
    MCP --> AGENT[AI agents<br/>Codex / Claude]

Every stage reads artifacts the previous stage wrote to data/ and writes its own. There is no hidden state: delete the SQLite cache and it rebuilds from the files. The transcripts, summaries, and mentions on disk are the source of truth.

Two summary paths, kept apart on purpose

This is the central design decision, and it is not an implementation detail.

Deterministic path

LLM path

Entry point

summarize_episode

semantic_summarize_episode

Method

rule-based extraction from transcript segments

OpenAI-compatible API over transcript chunks

Network

none

sends transcript text off the machine

Credentials

none

API key, plus an exact cost acknowledgement

Reproducibility

same input, same output, forever

not reproducible

Output file

.md

.semantic.md

They are never substituted for each other. Anything the deterministic path produces can be re-derived offline from the transcript alone; anything the LLM path produces cannot, so it is labelled as an LLM intermediate artifact rather than as podcast evidence, and it passes a deterministic review gate before downstream steps may consume it.

The reason is auditability. When a research artifact cites an episode, you need to know whether that claim came from the audio or from a model's reading of the audio. Merging the two paths would destroy that distinction permanently, and no amount of prompt engineering gets it back.

Mention extraction follows the same rule: it scans for companies, tickers, industries, macro topics, crypto, and places using deterministic rules, and each mention keeps timestamp evidence. It is not semantic understanding and does not claim to be.

Agent interface

The registry has 35 reviewed tools. Learning workflows generate lectures and05/06 examples, suggest the next step, and provide a separate read-only lineage query via inspect_workflow_derivation_lineage (Tool28). New derivations record their consumed inputs/output bytes; legacy pairs remain untracked. See the tool contract.

The MCP server exposes the same core functions to AI agents over a single FastMCP instance: stdio for local clients, and Streamable HTTP bound to 127.0.0.1:8767/mcp only. Same registry, same guards, two transports.

Every tool that writes, downloads, or spends money defaults to confirm=false and returns an action plan instead of acting. Tools that would send transcript text to an external provider require an exact acknowledgement string on top of that. No tool rebuilds the search index behind your back.

That default is deliberately inconvenient, so seven Skills under .agents/skills/ carry the protocol for turning one plain request into one authorised run: preview, explain the risks, ask once, act once, stop. They cover processing the latest episode, advancing an episode a single step at a time, ingesting an X or a YouTube video, and the verified research report paths. An agent without them can still call every tool -- it just has to ask you twice.

docs/usage.md pairs each task with its CLI command, what to say to an agent, and which Skill applies.

python scripts/validate_mcp_setup.py
python scripts/run_mcp_server.py

Documentation

Start with the usage guide if you know what you want to do but not which command or tool does it; the rest of the list is reference material.

Evaluation suites, for checking that an agent uses the tools within their declared boundaries: docs/mcp-tool-use-eval.md, docs/mcp-eval-prompts.md, docs/mcp-eval-report-template.md, docs/research-safety-eval.md, docs/research-eval-prompts.md, docs/research-llm-smoke.md.

Development

python -m pytest
python -m compileall src scripts

There is no --ignore list: the whole suite runs and is expected to be green.

Scripts stay thin: they parse arguments and call corpus_ingest_core. New behaviour is developed test-first. .env is local-only and must never be committed.

Disclaimer

This project organizes podcast content for research and does not provide investment advice. It produces no buy, sell, or hold recommendations, no target prices, no guaranteed returns, and nothing personalized to your situation. Summaries and extracted mentions can be incomplete or wrong, and LLM-generated content can be confidently mistaken. Verify anything that matters against the original audio and a primary source.

License

MIT — see LICENSE. No third-party source is vendored on main; an archived tag still carries one MIT-licensed snapshot, noted in THIRD-PARTY-NOTICES.md.

Tool29 inspect_study_guide_lineage(podcast_id, episode_ref) adds an offline read-only comparison of lecture03/04/07 against its recorded semantic summary. Tool26 generation declares separate metadata_writes for study_guide.lineage.json; ship with the updated study-guide Skill. Cover-only/reuse preserve provenance or legacy absence. No summary-to-transcript freshness claim; Tool27/28 behavior stays unchanged. See SPEC049.

Tool30 inspect_learning_bundle_recovery(podcast_id, episode_ref) provides offline, read-only recovery diagnosis for five fixed bundle locations and both lineage records. Recovery entries and uncertain publication require manual review; no cleanup, repair, latest-winner or source-freshness claim. Tools1-29 and Skills retain their behavior. See SPEC050 and docs/api.md.

Tool31 inspect_learning_workflow_status(podcast_id, episode_ref) provides a single offline read-only overview of Tools27-30:progress, both lineage scopes and recovery. Recovery gates further diagnostics; legacy/custom/stale observations retain distinct attention reasons. No executable suggested_call, action authorization or end-to-end freshness claim. Existing30 tool contracts/Skills remain unchanged. See SPEC051 and docs/api.md.

Tool32 advance_learning_workflow previews and confirms one explicit-episode learning action with metadata action/plan binding; no automatic chain or repair. See docs/api.md and SPEC052. Existing Tools1-31 retain their contracts.

Tool35 query_source_content reads one prepared RSS/YouTube/X transcript without SQLite. Pair with source-content-qa Skill for timed evidence answers and complete/partial notes; host AI privacy/billing applies. Quickstart.

Available Tools

25 tools
derive_workflow_bundleC

Side-effect tool:confirm=false 是 preview(零寫入、零網路)。

Inputs: force, confirm, podcast_id, episode_ref, +1 more. Returns: object

download_audioC

Side-effect tool:需要 confirm=true 才會下載 podcast audio。

Inputs: force, confirm, podcast_id, episode_ref. Returns: object

extract_mentionsC

Side-effect tool:需要 confirm=true 才會寫入 deterministic mention artifacts。

Inputs: force, confirm, podcast_id, episode_ref, +2 more. Returns: object

generate_stock_lens_reportC

Side-effect tool:需要 confirm=true 才會寫入 deterministic stock lens report。

Inputs: force, confirm, podcast_id, stock_query, +2 more. Returns: object

get_episodeA

查詢單一 podcast episode metadata;支援 latest 與大小寫不敏感 EP ref。

Inputs: podcast_id, episode_ref. Returns: object

ingest_x_videoC

Side-effect tool:confirm=false 是 preview(零寫入,會讀公開 metadata)。

Inputs: url, force, title, confirm. Returns: object

ingest_youtube_videoC

Side-effect tool:confirm=false 是 preview(零寫入,會讀公開 metadata)。

Inputs: url, force, title, confirm. Returns: object

list_episodesB

列出指定 podcast 最近集數;不回傳完整 audio_url。

Inputs: limit, podcast_id. Returns: object

list_verified_report_gap_backlogB

List inventory episodes missing a verified research report bundle.

Inputs: podcast_id (required), limit. Returns: object

query_verified_research_report_catalogC

List, search, or inspect read-only verified research report bundles.

Inputs: limit, query, action, podcast_id, +2 more. Returns: object

query_verified_research_report_coverageC

List episode-centric coverage of local verified research report bundles.

Inputs: podcast_id (required), limit, has_bundle. Returns: object

rebuild_cacheA

Maintenance tool:重建 SQLite cache;只索引既有 artifacts,不下載、不轉錄、不摘要。

Inputs: force, podcast_id. Returns: object

revalidate_verified_research_report_sourcesC

Revalidate one exact verified-report bundle's local source metadata.

Inputs: podcast_id (required), episode_ref (required), source_digest (required). Returns: object

run_corpus_episode_completion_workflowC

Preview or advance one episode by one explicitly confirmed action.

Inputs: podcast_id (required), action, confirm, episode_ref, +11 more. Returns: object

run_corpus_latest_episode_deterministic_workflowC

Preview or process one current latest episode through local deterministic stages.

Inputs: podcast_id (required), confirm, transcription_model, transcription_device, +2 more. Returns: object

run_episode_verified_research_report_workflowC

Preview or publish one explicit-episode verified research report (assemble only).

Inputs: podcast_id (required), episode_ref (required), confirm, stock_query, +1 more. Returns: object

run_latest_episode_verified_research_report_workflowC

Preview or complete one approved latest verified research report workflow.

Inputs: podcast_id (required), confirm, stock_query, api_cost_ack, +10 more. Returns: object

run_research_workflowC

Side-effect workflow tool:dry-run first,confirmed LLM steps require exact ack。

Inputs: force, confirm, podcast_id, episode_ref, +22 more. Returns: object

search_mentionsC

搜尋 SQLite cache 中的 deterministic mentions 與 timestamp evidence。

Inputs: query (required), limit, podcast_id, mention_type, +1 more. Returns: object

search_transcriptsA

搜尋 SQLite cache 中的 transcript segments;不會自動 rebuild cache。

Inputs: query (required), limit, podcast_id, search_mode, +2 more. Returns: object

semantic_summarize_episodeC

API-cost side-effect tool:需要 confirm=true 與 exact api_cost_ack 才會呼叫外部 LLM。

Inputs: force, model, confirm, base_url, +8 more. Returns: object

suggest_historical_verified_report_next_stepC

Suggest one next human-gated step for a named historical episode.

Inputs: podcast_id (required), episode_ref (required). Returns: object

summarize_episode_extractiveA

Side-effect tool:需要 confirm=true 才會寫入 deterministic extractive summary。

Inputs: force, confirm, max_quotes, podcast_id, +3 more. Returns: object

transcribe_episodeA

Long-running side-effect tool:需要 confirm=true 才會下載/取得音檔並轉錄。

Inputs: force, model, device, confirm, +4 more. Returns: object

validate_transcriptC

檢查既有 transcript artifacts 是否完整、有效或疑似 partial。

Inputs: podcast_id, episode_ref. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    C2.7/5.0

    Scored across 25 tools

    Disambiguation2/5

    The base episode/transcript/search tools are distinct, but the workflow surface is crowded: run_research_workflow and the four run_*_workflow variants, plus two query_verified_research_report_* tools, have highly similar names and only subtle distinctions. An agent would likely struggle to select the correct workflow tool without very careful reading.

    Naming Consistency4/5

    Tool names overwhelmingly follow a snake_case verb_noun pattern and form a recognizable family. Minor inconsistencies like summarize_episode_extractive vs semantic_summarize_episode, and some very long workflow names with awkward modifier placement, keep it from being perfectly consistent.

    Tool Count3/5

    25 tools sits at the high end of the heavy-but-acceptable range. The core ingestion operations are well represented, but the many workflow/report variants make the count feel bloated for a server named 'Podcast Ingestion Core'.

    Completeness4/5

    The tool set covers the core ingestion lifecycle well: list/get, download, transcribe, summarize, extract mentions, search, validate, cache rebuild, and extended X/YouTube ingestion and report workflows. Notable minor gaps are the lack of a direct transcript retrieval tool and no delete/update artifact operations, but agents can work around these.

    Maintenance

    ActivitySlowing
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers