# MarketTwits MCP

Local SQLite archive and MCP search service for the Telegram channel
[`@markettwits`](https://t.me/markettwits) (`-1001203560567`).

## What it stores

Telegram text messages are stored in `data/market_twits.sqlite3` with:

- channel ID
- UTC message datetime
- Telegram message ID
- text only
- views count
- extracted hashtags

The normalized `message_hashtags` table makes hashtag queries efficient. The
`sync_coverage_ranges` table records successfully downloaded intervals, including
empty ones, without concealing gaps between disjoint downloads.

Every Telegram fetch is appended to `data/download_control.csv` with:

```text
channel_id,last_message_id,last_message_datetime,messages_requested,status,input_timestamp
```

## Telegram credentials

1. Sign in at <https://my.telegram.org/apps> and create an application.
2. Copy `.env.example` to `.env`.
3. Set `TELEGRAM_API_ID`, `TELEGRAM_API_HASH`, and `TELEGRAM_PHONE`.

Use a Telegram user account. Telegram bots generally cannot retrieve historical
channel messages. On the first sync, Telethon asks for the login code and creates
`data/telegram_user.session`. Later MCP calls reuse that authorized session.

Never commit `.env` or the session file; both are excluded by `.gitignore`.

## Install and initial 2024-to-now download

```powershell
cd C:\_tools\202607_tradingview_mcp\twits_mcp
Copy-Item .env.example .env
# Edit .env first.
uv sync
uv run twits-sync
```

The initial sync downloads from `telegram.history_start` in `config.yaml` through
the current UTC time. It can be stopped and safely rerun: messages are upserted by
`(channel_id, message_id)`. A failed or interrupted partial range is not marked
covered. Already committed batches and the final partial batch remain in SQLite,
and the audit row reports the number committed during that attempt.

## Instrument mapping

Edit `instrument_hashtags.csv`. It uses one row per relationship:

```csv
instrument,hashtag
WBD,#WBD
PSKY,#PSKY
BRENT,#ормуз
BRENT,#нефть
```

Matching is case-insensitive. One instrument can have several hashtags, and the
same hashtag can be mapped to several instruments. The tool does not guess links:
an unmapped instrument returns an error.

The production mapping accepts either `instrument,hashtag` or
`instrument,hashtag,frequency`; `frequency` is ignored by MCP queries. Mapping
files must be UTF-8, with or without BOM. UTF-8 with BOM is recommended for
reliable Cyrillic editing in Excel.

To regenerate a review file containing every stored hashtag and the number of
distinct messages containing it, run:

```powershell
uv run twits-tags
```

This writes `tags/instrument_hashtags.draft.csv` with the columns
`instrument,hashtag,frequency`. Existing assignments in that output are
preserved on later runs; frequencies are always recalculated from SQLite and
new hashtags are added as `UNMAPPED`.

## Run the MCP server

Stdio:

```powershell
uv run twits-mcp
```

HTTP:

```powershell
uv run twits-mcp streamable-http --host 127.0.0.1 --port 8020
```

The HTTP endpoint is `http://127.0.0.1:8020/mcp`.

Codex registration:

```powershell
codex mcp add market-twits --url http://127.0.0.1:8020/mcp
```

## Tool

```text
asset_twits(
  instrument="WBD",
  start="2026-07-20T19:00:00Z",
  end="2026-07-22T19:00:00Z",
  format="structured"
)
```

`start` and `end` are inclusive. Both accept ISO-8601 timestamps;
values without an offset are interpreted as UTC and values with an offset are
converted to UTC. `end` defaults to the current UTC time. If `start` is omitted,
the legacy `days` parameter calculates `start = end - days` and defaults to five
days. A request with `start > end` returns an error; equal values request that
exact timestamp.

`instrument` accepts either a bare symbol or TradingView-style
`EXCHANGE:SYMBOL`. The exchange prefix is discarded before mapping. Supported
examples include `CME_MINI:NQ1!`, `RUS:MX1!`, `ICEEUR:BRN1!`,
`BITSTAMP:BTCUSD`, and `COMEX:GC1!`. Both `SI1!` and `RUS:SI1!` are normalized
to `USDRUB`. Instrument aliases are configured in `instrument_aliases.json`.

Before querying SQLite, the tool downloads any part of the requested date window
not recorded in `sync_coverage_ranges`. Results are newest first. `format="text"` is
the default and returns timestamp-prefixed strings. `format="structured"`
returns message objects:

```json
{
  "instrument": "WBD",
  "start": "2026-07-20T19:00:00Z",
  "end": "2026-07-22T19:00:00Z",
  "format": "structured",
  "count": 1,
  "truncated": false,
  "messages": [
    {
      "id": "-1001203560567:123456",
      "ts": "2026-07-22T18:27:00Z",
      "text": "ЕС одобрил слияние Warner Bros. и Paramount",
      "hashtags": ["#PSKY", "#WBD"]
    }
  ]
}
```

Structured IDs combine the Telegram channel and message IDs, so they remain
stable across overlapping requests. Message bodies have country flags,
promotional/legal footers, repeated whitespace, and inline hashtags removed;
warning markers are retained and hashtags are returned in their own array.
Structured timestamps and sync ranges use ISO-8601 UTC with `Z` notation.

Every bounded window is returned in full: there is no result cap or silent
truncation, and `truncated` is therefore `false`. The response also reports how
many Telegram messages were fetched during that call.

## Notes

- Telegram history is downloaded newest-first and written to SQLite in batches
  of 500 messages.
- Requests earlier than `history_start` are clamped to that configured boundary.
- Requests later than now fetch only through the current UTC time.
- SQLite uses WAL mode and a 30-second busy timeout.
- Concurrent calls in one MCP process share a synchronization lock, preventing
  duplicate downloads of the same missing interval.