# MetalsMarket Bot

A self-hosted toolkit that collects metal & commodity price data from
[metalsmarket.net](https://www.metalsmarket.net) and serves it to any
MCP-capable LLM client (Claude Desktop, Cursor, or an HTTP MCP host) through a
small [Model Context Protocol](https://modelcontextprotocol.io) server.

> [!WARNING]
> **Responsible use.** This tool automates login and data collection against a
> third-party commercial site. Use **only your own account**, respect
> metalsmarket.net's Terms of Service and rate limits, and do not redistribute
> the scraped price data. It is provided for educational and personal use, with
> no warranty. You are responsible for how you use it.

---

## What it does

There are two independent data paths plus an MCP server that reads their output
(the MCP server never talks to the site directly):

| Component | File | Produces | Runs |
|-----------|------|----------|------|
| **Live collector** | `collector.py` → `metalsmarket_bot.py` | `data/latest_snapshot.json`, `data/history.csv` | Continuous loop, random 40–60 min interval (+ on-demand) |
| **Historical factory** | `metals_master_factory.py` | `Metals_Historical_Full_2026.xlsx` (one sheet per instrument) | One-shot backfill |
| **MCP server** | `mcp_server.py` | 3 tools over stdio **or** Streamable HTTP | Reads the files above |

### MCP tools

- `metalsmarket_canli_fiyat` — live/last price for a metal or instrument
  (Turkish/English aliases: altın→gold, çinko→zinc, bakır→copper, …).
- `metalsmarket_gecmis_fiyat` — historical price for a single date **or** a date
  range/trend, from the backfill workbook.
- `metalsmarket_veri_durumu` — freshness/health of the collected data.

## Key features

- **Single POST per round** pulls the entire dashboard (~580 cells).
- **Randomized polling** (no fixed period) to avoid a predictable request cadence.
- **On-demand refresh** — a live query older than ~2 minutes triggers a single,
  coalesced, throttled refresh through the collector's session.
- **Resilient login** — on a session kick it re-logs in automatically; backs off
  only on HTTP 429 (5/15/30 min escalating) to stay rate-limit friendly.
- **Two MCP transports** — stdio (Claude Desktop) and Streamable HTTP (URL-based
  hosts). Optional JWT auth + localhost-only binding for access control.

## Requirements

- Python 3.11+
- A valid metalsmarket.net account
- `pip install -r requirements.txt`

## Setup

```bash
# 1. install dependencies
python -m pip install -r requirements.txt

# 2. configure credentials
cp .env.example .env        # then edit .env with your username/password

# 3. (first run only) build the cell-name metadata + run the collector
#    Windows:
start.bat                   # installs deps, builds metadata, starts the loop
```

`index.html` (the site's Historical symbol dropdown, included as the instrument
catalogue) is required by the collector and the historical factory.

## Usage

```bash
# Live collector — continuous loop (random 40-60 min, refreshes on demand)
python collector.py
python collector.py --once          # single round then exit

# Historical backfill — one-shot scrape into the .xlsx workbook
python metals_master_factory.py
python metals_master_factory.py --debug-one <symbol_id>   # verify the parser first

# MCP server
python mcp_server.py                 # stdio (Claude Desktop / Cursor)
python mcp_server.py --http --port 3011   # Streamable HTTP at /mcp
```

### Connecting an HTTP MCP host

The HTTP server binds to `127.0.0.1` by default, so only the local machine (and
Docker Desktop containers via `host.docker.internal`) can reach it — the LAN and
the public internet cannot. Point your MCP host at:

```
http://host.docker.internal:3011/mcp     # from a Docker container
http://127.0.0.1:3011/mcp                 # from the host
```

For token-based access control, set `MCP_JWT_SECRET` in `.env`; the endpoint then
requires `Authorization: Bearer <HS256 JWT>` and rejects everything else with 401.

## Project layout

```
collector.py              live collector loop (+ on-demand refresh)
metalsmarket_bot.py       login + single-POST dashboard fetch + resilience
csv_writer.py             snapshot.json / history.csv writers
mcp_server.py             MCP server (stdio + Streamable HTTP, 3 tools)
metals_master_factory.py  one-shot historical backfill -> .xlsx
cell_metadata_factory.py  builds cell-id -> name metadata (setup, HTTP)
cell_metadata_pw.py       Playwright supplement for JS-only widgets (setup)
index.html                instrument catalogue (site symbol dropdown)
start.bat / factory.bat / mcp_http.bat   Windows launchers
```

Scraped data (`data/`, `*.xlsx`, `scraped_ids.txt`, `debug_pages/`) and `.env`
are git-ignored and never published.