SanTech Content Engine

Local-first YouTube research-to-script engine for SanTech010.

The backend performs deterministic work: YouTube fetching, storage, outlier math, pack building, validation, save/select/export, and MCP tool exposure. Connected AI clients perform creative and judgment work using backend packs as source of truth.

Current status: offline backend, MCP backbone, quota guard, resource import, recent-only outliers, early velocity signals, combined signal reports, and signal-target transcript acquisition implemented.

Important MCP rule: get_signal_research_pack reads the current local database. It does not pull fresh YouTube data. Use get_project_status to check freshness, then use refresh_research_data when recommended_next_action says refresh_youtube_data or calculate_signals.

Verified:

python -m pytest tests -q
python -m compileall -q src

New Laptop Setup

On a fresh Windows laptop:

git clone git@github.com:budhasantosh010/san-tech-yt-research-titles-scripts.git
Set-Location -LiteralPath ".\san-tech-yt-research-titles-scripts"
powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\bootstrap-windows.ps1

Then edit .env locally and add:

• YOUTUBE_API_KEYS

• MCP_ROUTE_SECRET

• ALLOWED_HOSTS with your Tailscale Funnel hostname

Do not commit .env.

Before pushing from any laptop:

powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\check-github-ready.ps1

Full reproducibility instructions live in DOCS/FUTURE_PROOF_SETUP.md.

Related MCP server: YouTube Insights MCP Server

Daily MCP Startup

Use two PowerShell terminals.

Terminal 1:

$project = "C:\Users\Lenovo\Music\Startups\YT Automations\1 San Tech Channel Research, Title & Script with Codex"
Set-Location -LiteralPath $project
.\scripts\run-mcp-http.ps1

Terminal 2:

$project = "C:\Users\Lenovo\Music\Startups\YT Automations\1 San Tech Channel Research, Title & Script with Codex"
Set-Location -LiteralPath $project
.\scripts\start-funnel.ps1

Good Funnel output should show:

proxy http://127.0.0.1:3001

ChatGPT Connector URL Format

Use this shape:

https://<machine>.<tailnet>.ts.net/mcp/YOUR_ROUTE_SECRET/

Never include brackets around route secret.

Correct:

https://desktop-fdce9ak.taila47816.ts.net/mcp/abc123.../

Wrong:

https://desktop-fdce9ak.taila47816.ts.net/mcp/<abc123...>/

Use one tool call at a time in ChatGPT. No parallel tool calls. For connection tests, call only get_project_status.

Implemented:

• SQLite schema

• config loader

• competitor storage

• video/snapshot storage

• recent-only outlier math and persistence

• early velocity signal math and persistence

• YouTube Data API adapter with fake-client tests

• viral title collection importer for CSV/XLSX

• video idea save/list/select

• title idea save/list/select

• script points save/select

• context pack builders

• transcript save/get/list-missing storage

• signal-target transcript acquisition adapters

• transcript acquisition attempt tracking

• transcript-aware outlier research packs

• script save/export

• FastMCP server wrapper

• workflow prompt templates

Still needs live/user-provided inputs:

• YOUTUBE_API_KEY

• YOUTUBE_API_KEYS for comma-separated multiple keys

• competitor channel list

• real viral title/video collection file

• ChatGPT MCP connector/tunnel setup

Quota safety:

• YOUTUBE_MAX_DAILY_QUOTA_FRACTION=0.05 limits the app to 5% of each key's configured daily quota.

• YOUTUBE_CACHE_MAX_AGE_HOURS=24 prevents refetching fresh video details/snapshots.

• RESEARCH_STALE_AFTER_HOURS=24 controls when get_project_status warns that research data is stale.

• API keys belong in .env, which is ignored by Git.

Resource folders:

• Competitors: resources/competitors channels list

• Viral titles: resources/viral titles in adjacent niches

Signal workflow

Current deterministic workflow:

1. Sync competitor list from resources/competitors channels list when the sheet/file changes.

2. Sync viral title collection from resources/viral titles in adjacent niches when the sheet/file changes.

3. Refresh YouTube data and save a new snapshot row per collection run.

4. Calculate recent outliers.

5. Calculate early velocity.

6. Dump a combined signal report.

7. Fetch transcripts for selected signal targets only.

8. Build transcript-aware research packs.

9. Later, ChatGPT analyzes and generates ideas/titles/scripts.

MCP tool boundaries:

• sync_competitors_from_file treats the competitor sheet/file as source of truth and deactivates missing channels by default. It does not delete old rows.

• sync_viral_collection treats the viral-title sheet/file as source of truth and archives missing rows by default. It does not delete old rows.

• refresh_research_data is the daily action tool. It can collect YouTube stats and recalculate outlier/velocity signals. It does not fetch transcripts.

• get_signal_research_pack is the read tool ChatGPT should use after freshness is acceptable.

• mark_signal_reviewed can mark a signal as reviewed, approved, rejected, used, or suppressed; default reports exclude reviewed/rejected/used/suppressed signals.

Recent outlier means a recent video already beat that channel's recent baseline.

Early velocity means a new video is moving unusually fast and may become an outlier.

True recent velocity requires at least two snapshots. If only one snapshot exists, the system uses lifetime velocity as a rough fallback.

Seed competitor file:

python -m santech_engine.cli import-competitors --file resources/competitors/santech_competitors.txt

Signal commands:

python -m santech_engine.cli collect-all --limit 10
python -m santech_engine.cli calculate-outliers --recent-only --all-competitors
python -m santech_engine.cli list-outliers --limit 5
python -m santech_engine.cli dump-outlier-report --limit 20 --output exports/recent-outliers.json
python -m santech_engine.cli calculate-velocity --all-competitors
python -m santech_engine.cli list-velocity-signals --limit 10
python -m santech_engine.cli dump-velocity-report --limit 20 --output exports/early-velocity.json
python -m santech_engine.cli dump-signal-report --output exports/signal-report.json
python -m santech_engine.cli fetch-transcripts-for-signal-targets --adapter youtube_transcript_ai --limit-outliers 5 --limit-velocity 5

Transcript acquisition

This project supports transcript-aware research packs.

The system does not fetch transcripts for every competitor video.

Flow:

1. Collect competitor videos.

2. Calculate recent outliers and early velocity.

3. Fetch transcripts only for combined signal targets.

4. Cache successful transcripts.

5. Mark missing/failed transcripts honestly.

6. Include real transcript text in outlier packs only when available.

Default adapter:

• youtube_transcript_ai

Backup adapter:

• youtube_transcript_io if YOUTUBE_TRANSCRIPT_IO_API_TOKEN is set

Manual fallback:

• If automatic fetching fails for an important outlier, the user can paste transcript text into ChatGPT.

• ChatGPT can then call save_video_transcript through MCP to cache it.

Warnings:

• Transcript providers are not the official YouTube Data API.

• They may fail or change.

• Do not use aggressive scraping/proxy bypassing.

• Keep fetches low-volume and signal-first.

• The system must not infer or invent what a video said from title/metrics alone.

Useful local commands:

python -m santech_engine.cli import-competitors
python -m santech_engine.cli import-viral-collection
python -m santech_engine.cli doctor
python -m santech_engine.cli setup-env
python -m santech_engine.cli test-transcript-provider --adapter youtube_transcript_ai
python -m santech_engine.cli smoke-test
python -m santech_engine.cli collect-all --limit 5
python -m santech_engine.cli calculate-outliers --recent-only --all-competitors
python -m santech_engine.cli calculate-velocity --all-competitors
python -m santech_engine.cli dump-signal-report --output .\exports\signal-report.json
python -m santech_engine.cli fetch-transcripts-for-signal-targets --adapter youtube_transcript_ai --limit-outliers 5 --limit-velocity 5
python -m santech_engine.cli transcript-status --video-id 123
python -m santech_engine.cli list-transcript-attempts --limit 20
python -m santech_engine.cli save-transcript --video-id 123 --file .\transcript.txt
python -m santech_engine.cli dump-outlier-pack --include-transcripts --output .\exports\outlier-pack.json
python -m santech_engine.cli dump-title-pack --idea-id 1 --output .\exports\title-pack.json
python -m santech_engine.cli dump-script-pack --title-id 1 --output .\exports\script-pack.json

Run MCP server locally:

python -m santech_engine.mcp_server

ChatGPT MCP via Tailscale Funnel

Use this only after the local backend packs are truthful.

1. Create local .env:

Copy-Item .env.example .env

2. Set the MCP bridge values in .env.

MCP_HOST=127.0.0.1
MCP_PORT=3001
MCP_ROUTE_SECRET=<your-long-secret>
MCP_BEARER_TOKEN=
ALLOWED_ORIGINS=https://chatgpt.com,https://chat.openai.com
ALLOWED_HOSTS=127.0.0.1,localhost,<your-machine>.<your-tailnet>.ts.net
FUNNEL_BASE_URL=

Rules:

• At least 32 characters.

• Do not commit .env.

• Do not paste the secret into docs.

• Keep MCP_BEARER_TOKEN blank if the ChatGPT connector uses Authentication: No Auth.

• Add your real Tailscale Funnel hostname to ALLOWED_HOSTS.

3. Run the HTTP bridge doctor:

.\scripts\doctor-mcp-http.ps1

4. Start the local MCP HTTP server:

.\scripts\run-mcp-http.ps1

5. In another terminal, start Tailscale Funnel:

.\scripts\start-funnel.ps1

6. Add this URL as the ChatGPT custom connector URL:

https://<machine>.<tailnet>.ts.net/mcp/<MCP_ROUTE_SECRET>

Use your real Tailscale Funnel hostname and your local route secret value.

7. In ChatGPT, test:

get_project_status

8. Then test the main research pack:

get_signal_research_pack

Expected route:

ChatGPT custom connector
-> Tailscale Funnel HTTPS URL
-> http://127.0.0.1:3001/mcp/<MCP_ROUTE_SECRET>
-> SanTech MCP tools
-> SQLite / YouTube API / transcript providers

Security notes:

• The Python server binds only to 127.0.0.1.

• The default SanTech MCP HTTP port is 3001.

• The MCP route is mounted under /mcp/<MCP_ROUTE_SECRET>.

• Allowed origins default to https://chatgpt.com and https://chat.openai.com.

• Allowed hosts default to 127.0.0.1 and localhost; add the Tailscale Funnel hostname through ALLOWED_HOSTS or FUNNEL_BASE_URL.

• If another MCP bridge, such as the X/Spectre bridge, is using the same Tailscale Funnel root hostname, run only one of them through that public root at a time unless you add a gateway/router.

• Scripts and doctor checks do not print API keys, bearer tokens, or the MCP route secret.

• Stop Funnel with:

.\scripts\stop-funnel.ps1