Skip to main content
Glama
tg-archive
by myshli

tg-ml-archive

tg-ml-archive copies the posts of public Telegram channels into a SQLite database. An OpenAI model tags each post by topic and writes a one-line English summary. An MCP server lets Claude search the archive by keywords and by meaning. A digest page shows the news of the last 3 days.

systemd timer (hourly) ── ingest.py ──► archive.db (SQLite: posts + FTS5 + sqlite-vec)
                             ├─ enrich.py ─┘  (OpenAI: tags + summary + embeddings)
                             └─ media.py ──► /var/www/tg-digest/media (pictures and videos for the digest)
systemd timer (daily) ──── digest.py ──► /var/www/tg-digest/index.html + feed.xml
Claude ──HTTPS + token──► Caddy ──► mcp_server.py --http

File

What it does

ingest.py

Gets new posts of each channel in channels.toml with Telethon. It logs in as your Telegram user account.

enrich.py

Tags each post with OpenAI (topics, kind, availability, entities, summary) and stores an embedding.

media.py

Downloads the pictures and videos of new posts for the digest page.

digest.py

Builds the digest page for the latest finished 3-day window.

mcp_server.py

MCP tools: search_posts, get_post, recent, list_topics, stats.

channels.toml

The channel list, the channel descriptions, the topic list, and the digest settings.

deploy/

Scripts and systemd units for the VPS.

Setup guide

This guide is written so that a person or an AI agent can do it from start to end. Do the parts in this order:

  1. Collect the accounts and values (Part A).

  2. Install the project on your computer and fill the archive (Part B).

  3. Deploy to the VPS (Part C).

  4. Make the MCP server and the digest public over HTTPS (Part D).

  5. Connect Claude (Part E).

If you only want local search on your computer, do Part A, Part B, and step E1.

Rules for AI agents

  • Do not print, copy into chat, or commit .env, tg.session, or /etc/tg-archive-mcp.env.

  • tg.session gives full access to the Telegram account. Treat it as a password.

  • Only the owner of the Telegram account can type the phone number and the login code. Ask the owner to run step B4 in a terminal.

  • After the VPS has tg.session, do not run ingest.py on the local computer. Telegram can revoke a session that runs in two places at the same time.

Part A. Collect the accounts and values

You need these items before you start:

Item

Where to get it

Goes into

Telegram API ID and API hash

https://my.telegram.org → API development tools

TG_API_ID, TG_API_HASH in .env

A Telegram account with a phone number

Your own account. The code reads channels as this user.

Login in step B4

OpenAI API key

https://platform.openai.com/api-keys

OPENAI_API_KEY in .env

A VPS with Ubuntu and root SSH key login

Any provider. Tested on Ubuntu 24.04 with 2 GB of memory.

DEPLOY_HOST in .env, for example root@203.0.113.10

A domain or subdomain that you control

Your DNS provider

DOMAIN in .env, for example archive.example.com

The public channels to archive

The @username of each channel

channels.toml

The local computer needs git, ssh, rsync, sqlite3, and uv. macOS and Linux both work.

Part B. Install locally and fill the archive

B1. Clone the repo and install the dependencies:

git clone <repo-url> tg-ml-archive
cd tg-ml-archive
uv sync

B2. Make the .env file:

cp .env.example .env

Then open .env and set these keys: TG_API_ID, TG_API_HASH, OPENAI_API_KEY, DEPLOY_HOST, DOMAIN. Leave MCP_TOKEN empty. Step D1 makes the token on the server.

B3. Edit channels.toml:

  1. Put your channels in the channels list (public usernames, with or without @).

  2. In [about], write one description for each channel. Use the username in lowercase as the key. The tagger reads this text, so describe what the channel posts and how its posts look.

  3. In [strip], remove the entries of channels that you do not archive. Add a regular expression only if a channel repeats an ad footer.

  4. Change [topics], [kinds], and [availability] only if you need a different taxonomy.

B4. Get the posts (the account owner must do this step):

uv run ingest.py

On the first run, Telethon asks for the phone number and the login code. Then it saves tg.session and downloads the full history of each channel. A large channel can take some minutes.

Expected result: ingest.py prints the number of new posts for each channel.

B5. Tag a small sample, then examine the tags:

uv run enrich.py --sample 20
sqlite3 archive.db "select summary, topics, kind from posts where enriched_at is not null order by random() limit 20"

If the tags are wrong, improve [about] and [topics] in channels.toml. Then run the sample again.

B6. Tag and embed all posts:

uv run enrich.py

This step sends every post to OpenAI once. The cost depends on the number of posts and on TAG_MODEL (default gpt-5-mini).

B7. Do a test of the local search:

claude mcp add tg-archive -- uv run --directory "$PWD" mcp_server.py

Then ask Claude a question about the channels. Claude must call the tg-archive tools.

Part C. Deploy to the VPS

C1. Make sure that SSH key login to the VPS works without a password prompt:

ssh -o BatchMode=yes "$(grep '^DEPLOY_HOST=' .env | cut -d= -f2)" true && echo ok

C2. Copy the code, the secrets, and the database to the VPS:

deploy/push.sh --with-data

The script does these steps:

  • It makes the system users tgarchive and tgmcp, and the group tgarchive-db.

  • It installs ffmpeg and uv on the VPS if they are not there.

  • It copies the files that git tracks to /opt/tg-ml-archive.

  • It copies .env and tg.session to /opt/tg-ml-archive (mode 600), and archive.db to /var/lib/tg-archive/.

  • It installs and starts the hourly sync timer and the daily digest timer.

Run --with-data only one time. Later runs of --with-data overwrite the server database with your old local copy.

C3. Make sure that the sync works on the VPS:

ssh <DEPLOY_HOST> 'systemctl start tg-archive-sync && systemctl show -p Result --value tg-archive-sync'

Expected result: success. To read the log, run journalctl -u tg-archive-sync -n 50 on the VPS.

From now on, the VPS is the only place that runs ingest.py. Your local archive.db does not get new posts.

Part D. Public HTTPS endpoint and digest page

D1. Run the setup script one time:

deploy/go-public.sh

The script does these steps on the VPS:

  1. It stops the sync for a short time. If the database is still in /opt/tg-ml-archive, it makes a backup in /root and moves the database to /var/lib/tg-archive/.

  2. It makes /etc/tg-archive-mcp.env (root only) with a new random MCP_TOKEN and the OpenAI key. The script does not print the token.

  3. It deploys the code again and starts tg-archive-mcp.service as user tgmcp. This user cannot read tg.session or .env.

  4. It installs Caddy and writes /etc/caddy/sites/tg-ml-archive.caddy for DOMAIN. The main Caddyfile only imports /etc/caddy/sites/*.caddy.

  5. It does a test of the MCP server: 401 without the token, 200 with the token.

You can run the script again. Each step checks the state before it changes something.

D2. At your DNS provider, add an A record: DOMAIN → the IP address of the VPS.

D3. If the VPS has a firewall, open TCP ports 80 and 443. Caddy needs them to get the TLS certificate.

D4. When the DNS record works, do a test of the endpoint:

curl -s -o /dev/null -w '%{http_code}\n' "https://<DOMAIN>/mcp"

Expected result: 401. This shows that HTTPS works and that the server refuses requests without the token.

D5. The first digest page comes after the first 3-day window is finished. The windows start on DIGEST_EPOCH (default 2026-10-17) at 00:00 UTC. To build a page now for a past window, run this on the VPS:

cd /opt/tg-ml-archive && runuser -u tgarchive -- env ARCHIVE_DB=/var/lib/tg-archive/archive.db .venv/bin/python digest.py --window 2026-10-17

Change the date to the first day of a finished window. Then open https://<DOMAIN>/.

Part E. Connect Claude

E1. Claude Code on the local computer, local database: see step B7. This uses your local archive.db, which stops getting new posts after Part C.

E2. Claude Code, VPS database (recommended): get the token from the VPS into a shell variable, then add the server:

TOKEN=$(ssh <DEPLOY_HOST> "grep '^MCP_TOKEN=' /etc/tg-archive-mcp.env | cut -d= -f2")
claude mcp add --transport http -s user tg-archive "https://<DOMAIN>/mcp" --header "Authorization: Bearer $TOKEN"
unset TOKEN

E3. claude.ai (web, desktop, and phone): go to Settings → Connectors → Add custom connector. Use this URL:

https://<DOMAIN>/mcp?token=<MCP_TOKEN>

claude.ai connectors cannot send headers, so the token goes in the URL. Caddy keeps no access log, so the token is not written to disk on the server.

E4. Do a test: ask Claude "Show the archive stats". Claude must call the stats tool and show the post count of each channel.

Related MCP server: Telegram Bot MCP Server

Daily operation

Task

Command (run on the VPS unless noted)

Deploy code changes (on the local computer)

deploy/push.sh

Show the timers

systemctl list-timers 'tg-archive-*'

Run a sync now

systemctl start tg-archive-sync

Read the sync log

journalctl -u tg-archive-sync -n 50

Read the MCP server log

journalctl -u tg-archive-mcp -n 50

Add a channel

Edit channels.toml on the local computer, then run deploy/push.sh. The next sync gets its full history.

Change the topics

Edit channels.toml, deploy, then run enrich.py --retag on the VPS as tgarchive.

Rotate the MCP token

Edit /etc/tg-archive-mcp.env, run systemctl restart tg-archive-mcp, then update the clients (E2, E3).

Run Python commands on the VPS as the user tgarchive, with the database path:

cd /opt/tg-ml-archive && runuser -u tgarchive -- env ARCHIVE_DB=/var/lib/tg-archive/archive.db .venv/bin/python <script>.py <flags>

Configuration

All settings are in .env. .env.example lists them with their defaults.

Key

Default

Use

TG_API_ID, TG_API_HASH

(required)

Telegram API credentials

TG_SESSION

./tg

Session file path without the .session extension

OPENAI_API_KEY

(required)

Tagging, embeddings, and the digest

TAG_MODEL, TAG_EFFORT, TAG_WORKERS

gpt-5-mini, low, 8

The tagger

EMBED_MODEL, EMBED_DIM

text-embedding-3-small, 512

Embeddings. Set EMBED_DIM before the first run. A change later needs a new database.

ARCHIVE_DB

./archive.db

Database path. The systemd units set /var/lib/tg-archive/archive.db.

MCP_TOKEN, MCP_HOST, MCP_PORT

(required for --http), 127.0.0.1, 8765

The HTTP MCP server

DEPLOY_HOST

(required for deploy)

SSH target of the VPS, for example root@203.0.113.10

DOMAIN

(required for Part D)

Public domain of the MCP endpoint and the digest page

SITE_URL

https://$DOMAIN

Base URL in the Atom feed

DIGEST_EPOCH

2026-10-17

First day of the first digest window

DIGEST_WRITE_MODEL, DIGEST_WRITE_EFFORT

gpt-6-sol, medium

The model that writes the digest text

DIGEST_FLEX

1

0 turns off the OpenAI Flex tier

Digest page

https://<DOMAIN>/ shows the latest finished 3-day window. The server keeps no old digests, and /digest/... links go to /. /feed.xml is an Atom feed with one entry for each new digest.

  • media.py runs in the hourly sync. It saves the pictures, videos, and GIFs of new posts (full albums too) to /var/www/tg-digest/media/. ffmpeg converts videos to H.264 at 720p or less, with a poster frame. Posts that have only topics from [digest] no_media get no downloads.

  • digest.py runs daily at 01:30 UTC (timer tg-archive-digest). When a new window is finished, it does these steps:

    1. It removes humor, off-topic posts, and empty posts (less than 60 characters and no useful links).

    2. It puts posts about the same thing into one item and gives each item an importance (gpt-5-mini). Then it merges items whose posts link the same GitHub repo.

    3. It writes each item in the language of its sources, and writes again each item that comes back in the wrong language.

    4. It writes the overview as short lists by theme, in Russian. To change the language, edit OVERVIEW_SYSTEM in digest.py.

    5. It saves the result to /var/www/tg-digest/data.json, renders templates/ to index.html and feed.xml, and deletes the media of posts from before the new window.

  • [digest] in channels.toml sets the section order and the sections without media.

  • After a template change, run digest.py --render-only. This makes no AI calls. To write only a new overview, run digest.py --redo-overview.

  • Cost: all calls use the OpenAI Flex tier at half price. If OpenAI refuses a Flex call, the code sends it again on the standard tier. One digest costs about $0.12 with gpt-6-sol. data.json stores the real cost under usage.

  • Media on disk: the current window and the next window, 250 MB at most.

Security model

  • tg.session and .env are readable only by tgarchive (mode 600).

  • The public MCP server runs as tgmcp. systemd blocks its access to tg.session, .env, /root, and /home. Its secrets are in /etc/tg-archive-mcp.env, which only root can read.

  • The database is in /var/lib/tg-archive/. The group tgarchive-db gives both users access.

  • Clients send Authorization: Bearer <MCP_TOKEN>, or ?token= when they cannot send headers.

  • Caddy keeps no access log, so tokens in URLs are not written to disk.

Notes

  • Keyword search uses prefix matching, not stemming. Semantic search (embeddings) finds paraphrases and works across Russian and English.

  • The sync uses about 4 s of CPU per hour. The database is about 73 MB for 17,000 posts.

License

MIT. See LICENSE.

Available Tools

5 tools
get_postA

Full text and metadata of one post (use after search_posts returns a snippet).

Inputs: msg_id (required), channel (required).

list_topicsB

Topics, post kinds and availability values (with descriptions and post counts), plus the author's most used hashtags. Call this to pick filters for search_posts.

Inputs: top_hashtags.

recentB

Posts from the last N days, newest first, optionally filtered by topics/channels. Good for "what happened this week in agents".

Inputs: days, limit, topics, channels. Returns: result

search_postsA

Hybrid keyword + semantic search over all archived posts.

query: natural-language or keyword query (any language; exact names like "Qwen3" or arXiv ids work too).
channels: restrict to these channel usernames.
since / until: ISO dates, e.g. "2026-09-01" (until is exclusive).
topics: any-of filter, values from list_topics().
kind: one post type from list_topics() (e.g. model-release, tool-release, paper, tutorial, humor).
hashtags: any-of filter on the author's own hashtags, e.g. ["text2video", "gaussian"] (very precise).
availability: any-of filter, e.g. ["open-weights", "code-only"] for things you can run locally.
Returns posts with date, channel, summary, topics, links, text snippet and permalink.

Inputs: query (required), kind, limit, since, +5 more. Returns: result

statsB

Per-channel post counts, date range and enrichment progress.

Inputs: no named parameters. Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.6/5.0

    Scored across 5 tools

    Disambiguation4/5

    get_post, list_topics, and stats are clearly distinct, and the descriptions even prescribe a workflow (list_topics → search_posts → get_post). The only mild overlap is between recent (time-windowed browsing) and search_posts (query-based retrieval), both returning filtered post lists, but the time-vs-query distinction is documented well enough to separate them.

    Naming Consistency3/5

    Two tools use the verb_noun pattern (get_post, list_topics) while the other three are bare nouns or adjectives (recent, stats, search_posts), and 'recent' in particular reads more like a parameter than a tool name. The set is still readable and unambiguous, but the convention is mixed.

    Tool Count5/5

    Five tools is a tight, well-scoped surface for a read-only archive server: discovery (list_topics, stats), retrieval (search_posts, recent), and expansion (get_post). Each tool has a clear job and none feels redundant.

    Completeness4/5

    The read-only lifecycle is covered end to end: find filters, search, browse recent, expand full text, and inspect corpus stats. Minor gaps remain — no explicit channel-enumeration tool (channels must be guessed or inferred from stats) and no pagination/offset control on search or recent results for large result sets.

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      B
      quality
      D
      maintenance
      Enables Claude to interact with Telegram channels and groups through API or web scraping for reading posts and searching.
      15
      35 npm
      MIT
    • A
      license
      B
      quality
      D
      maintenance
      Enables users to read, search, and manage Telegram messages in channels, groups, and private chats through MCP tools.
      8
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables AI assistants like Claude to read, search, and send messages through your personal Telegram account via MCP tools.
      MIT