# HKEX News Database

This project downloads HKEX Title Search records into a local SQLite database and
exposes a read-only MCP server for AI assistants.

## Setup and first download

```bash
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
.venv/bin/python hkex_store.py setup-embeddings
.venv/bin/python hkex_store.py sync --from 2026-08-01
```

The initial date determines how much historical data is downloaded. The default
sync includes current securities only, excluding delisted stocks. Later runs
require no dates and re-download the previous sync date to capture late
postings safely:

```bash
.venv/bin/python hkex_store.py sync
```

Only announcements listing one stock code below 10000 are stored. Notices with
multiple stock codes (including ETF listings), or codes for derivative warrants,
CBBCs, debt securities and other securities numbered 10000 or above, are skipped
and their PDFs are not downloaded.

Use `--security-status all` or `--security-status delisted` when historical
delisted announcements are required. The database is stored at
`data/hkex_news.db`.
Sync progress is printed to the terminal and logged to `data/hkex_news.log`;
use `--log-file PATH` to select a different log file.

## MCP server

`.mcp.json` configures the local stdio server for GitHub Copilot-compatible
clients that support workspace MCP configuration. After the virtual environment
and first sync exist, trust and start **hkex-news** in the client's MCP server
manager.

For another local MCP client, add the equivalent server configuration:

```json
{
  "command": "/absolute/path/to/financestreet/.venv/bin/python",
  "args": [
    "/absolute/path/to/financestreet/hkex_mcp.py",
    "--database",
    "/absolute/path/to/financestreet/data/hkex_news.db"
  ]
}
```

### Remote access over HTTP

To let clients on other machines connect, serve the same read-only tools over
streamable HTTP. A bearer token of at least 16 characters is required:

```bash
export HKEX_MCP_TOKEN="$(openssl rand -hex 32)"
.venv/bin/python hkex_mcp.py --transport http --host 0.0.0.0 --port 8000
```

Clients connect to `http://<server>:8000/mcp` with the header
`Authorization: Bearer <token>`:

```json
{
  "type": "http",
  "url": "https://mcp.example.com/mcp",
  "headers": { "Authorization": "Bearer <token>" }
}
```

Use `--allowed-host mcp.example.com` (repeatable) to restrict accepted `Host`
headers. The token travels in plain text over HTTP, so put the server behind a
TLS reverse proxy (Caddy, nginx) or a tunnel (Tailscale, Cloudflare Tunnel)
rather than exposing the port directly to the internet.

During sync, each linked PDF for a current security is downloaded and its
extracted text is stored in the `news.document_text` column. HKEX's `file_type`
field identifies most PDFs; the stored `document_url` also normally ends in
`.pdf` and is used as a fallback when that field is absent. Failed downloads retain an error in `news.document_error` and are retried on the next
overlapping sync. PDFs that exceed the 50 MiB limit, cannot be read, or contain
text that cannot be stored as UTF-8 are registered in `invalid_document_urls`
and marked ignored with a timestamp in `news.document_ignored_at`; the URL is
never retried by later syncs or backfills. Downloads run in randomized batches
of one to five simultaneous requests.
The server provides `database_status`, `search_news`, `get_news`, and
`list_categories`; `search_news` searches document text and supports an exact
sector filter, while `get_news` returns document text and the assigned sector.
`list_sectors` summarizes classified companies and announcements. The server
only opens the SQLite database in read-only mode. To process every stored
current-security PDF from every date without making a new HKEX search, run
`backfill-pdfs`:

```bash
.venv/bin/python hkex_store.py backfill-pdfs
```

## Company sectors and embeddings

`setup-embeddings` downloads **BAAI/bge-small-en-v1.5** via FastEmbed into
`data/fastembed/` and checks its 384-dimensional output. It runs locally on CPU;
no API key or GPU is required. Subsequent classification runs use only cached
files and fail explicitly if the model is missing, rather than substituting
keyword rules or hashing. For an offline installation, set
`HKEX_EMBEDDING_MODEL_PATH` to a directory containing the FastEmbed-compatible
ONNX model and tokenizer files.

Sync and PDF backfill automatically reuse an existing company sector, including
for renamed companies sharing a stock code. For a previously unseen company,
they embed principal-business passages from financial reports and compare
them with separate industry examples for each sector. The examples and sector
names come from `SECTOR_PHRASES` and `EMBEDDING_EXTRA_TERMS` in `hkex_sector.py`;
`sector.csv` is a reference file and is not read by the classifier.
Pure investment-holding descriptions are excluded in favor of operating-business
evidence. When no principal-business passages are found, up to twelve
450-character excerpts sampled across the report are used instead; these
fallback assignments can reflect incidental report content and need extra review.
Every company with usable annual/interim report text receives the closest sector,
even for weak or tied matches. There is no similarity or margin cutoff.
Reports are selected using HKEX annual/interim report and final/interim/quarterly
results categories, with title matching only when the category is missing.
Other notices merely mentioning an annual report are not used as report evidence.
To run this classification independently:

```bash
.venv/bin/python hkex_store.py classify-sectors
```

The `company` table stores every known issuer name, its latest `stock_code`,
nullable `sector`, `sector_confidence`, and provenance (`source_news_id`, `evidence`,
`classified_at`, `classification_model`). `sector_confidence` is the winning raw
cosine similarity (-1 to 1), not a probability or calibrated accuracy estimate;
it does not measure the lead over competing sectors. `stock_name` is the primary
key and the join key to
`news`; historical name aliases have separate rows with the same sector.
Companies without usable report text retain a null sector and confidence.
MCP `search_news` and `get_news` return both sector and confidence from this table;
`news.sector` is maintained as a compatibility copy.

```sql
SELECT n.news_id, n.title, n.stock_code, c.sector, c.sector_confidence
FROM news AS n
LEFT JOIN company AS c ON c.stock_name = n.stock_name;
```

Opening an existing database through `hkex_store.py` automatically migrates the
old `company_sectors` table into `company`, preserving its evidence and marking
its assignments as `legacy`, and adds the nullable `sector_confidence` column.
Normal classification preserves existing assignments, including manual changes.
Existing assignments have null confidence until recomputed. To replace all
assignments and compute confidence using BGE, run (this also replaces manual sectors):

```bash
.venv/bin/python hkex_store.py classify-sectors --reclassify
```

Review weaker assignments first:

```sql
SELECT stock_code, stock_name, sector, sector_confidence, source_news_id, evidence
FROM company
ORDER BY sector_confidence ASC NULLS FIRST;
```