Skip to main content
Glama

ForecastMCP

ForecastMCP is an MCP server and agent harness that exposes pretrained time-series foundation models (Chronos T5, with optional TimesFM and Moirai adapters) and classical baselines (seasonal naive, AutoETS, AutoTheta) as typed, budgeted, permissioned, and traced forecasting tools. A client agent inspects a series, requests several candidate forecasters, quantifies their disagreement, reconciles them with a recorded rule, and writes a stakeholder-readable forecast rationale in which every number carries an inline [call:<id>] citation back to the trace entry that produced it. A frozen rolling-origin holdout, DuckDB store, OpenTelemetry JSONL tracing, byte-identical replay, a probabilistic evaluation harness with bootstrap CIs, an agent-level evaluation suite, Docker packaging, and a real benchmark (including the negative results) ship in the same repository.

Architecture

                    ┌──────────────────────────────────────────────┐
                    │ Agent loop (forecastmcp/agent)               │
                    │  planner (LLM or deterministic fallback)     │
                    │  -> MCP client session                       │
                    │  -> context compaction (2500-token budget)   │
                    │  -> reconcile -> report.md with citations    │
                    └──────────────────────┬───────────────────────┘
                                           │ MCP (stdio or Streamable HTTP)
                    ┌──────────────────────▼───────────────────────┐
                    │ MCP server (forecastmcp/mcp/server.py)       │
                    │  list_series describe_series forecast        │
                    │  ensemble_forecast backtest model_card sql   │
                    └──────────────────────┬───────────────────────┘
                                           │
                    ┌──────────────────────▼───────────────────────┐
                    │ ToolRuntime                                  │
                    │  Pydantic input/output schemas               │
                    │  permissions (analyst | admin)               │
                    │  BudgetGuard: calls, wall clock, cost,       │
                    │    per-call timeout -> typed BudgetExceeded  │
                    │  OpenTelemetry spans -> trace.jsonl          │
                    │  call outputs -> calls/<call_id>.json        │
                    └───────┬─────────────────────────┬────────────┘
                            │                         │
              ┌─────────────▼────────────┐  ┌─────────▼─────────────┐
              │ Forecaster registry      │  │ DuckDB                │
              │  Chronos T5 (+ LoRA)     │  │  series runs          │
              │  TimesFM / Moirai (opt)  │  │  forecasts metrics    │
              │  SeasonalNaive AutoETS   │  └───────────────────────┘
              │  AutoTheta promo-naive   │
              └──────────────────────────┘
                            │
              artifacts/runs/<run_id>/{trace.jsonl, calls/*.json, report.md}
  • Typed tools. Every tool has a strict Pydantic input and output model; unknown fields and malformed horizons, quantile levels, weights, or covariate lengths are rejected before any model runs.

  • Budgets. BudgetGuard enforces a per-call timeout, per-run call ceiling, wall-clock ceiling, and cost ceiling. Violations raise BudgetExceeded with a machine-readable reason; the agent retries with a cheaper model and records the downgrade as an agent.downgrade span.

  • Permissions. analyst may inspect, forecast, backtest, ensemble, and run read-only SQL; admin additionally may trigger fine-tuning. Denials raise PermissionDenied and are logged as spans, never silently ignored.

  • Tracing. A custom SpanExporter appends one JSON line per span to artifacts/runs/<run_id>/trace.jsonl with tool name, validated-input SHA-256, model and checkpoint revision, latency, estimated cost, success/error, and the output path/SHA-256. The serialized tool output lives in calls/<call_id>.json.

  • Replay. forecastmcp replay <run_id> re-executes every successful call from trace.jsonl, normalizes wall-clock-dependent fields, and asserts byte-identical output against the cached call files. Failed calls are re-executed and must fail again.

  • Context compaction. The agent's ContextWindow approximates tokens at four characters per token; when older tool outputs push the context past 2500 tokens it replaces the middle with a summary that preserves call IDs and tool names, and emits an agent.compaction span.

  • Freezing. configs/holdout.yaml is written once and committed. Every benchmark, backtest, and agent task reads those exact boundaries; no experiment derives its own splits.

Related MCP server: sentinel-mcp

Installation and usage

Requires Python 3.11. Core install includes PyTorch, the Chronos adapter, the classical baselines, DuckDB, the MCP SDK, and OpenTelemetry. The [fm] extra adds the optional TimesFM and Moirai adapters.

python3.11 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"        # core: baselines + Chronos adapter, tests, ruff
pip install -e ".[fm,dev]"     # optional TimesFM and Moirai adapters too

Chronos weights (amazon/chronos-t5-tiny) download on first use and are cached by Hugging Face. Set FORECASTMCP_OFFLINE=1 to force the deterministic synthetic supply-chain panel and restrict the registry to CPU-cheap models, which is how CI runs without model downloads.

forecastmcp data prepare --dataset synthetic_supply
forecastmcp data prepare --dataset m4_daily
forecastmcp data describe
forecastmcp models list

forecastmcp forecast --model chronos --dataset synthetic_supply --series S001 --horizon 28
forecastmcp forecast --model seasonal_naive --dataset synthetic_supply --series S001 --horizon 28

Run the agent and replay its trace:

forecastmcp ask "What will capacity demand for S001 look like over the next 4 weeks, with an 80% interval?" --run-id demo
forecastmcp replay demo

Run the full benchmark, fine-tune comparison, agent evaluation, plots, and summary:

scripts/run_all.sh              # full run (~20-35 min on CPU)
scripts/run_all.sh --fast       # CI smoke run on synthetic data
make bench                      # benchmark + plots + summary only

Serve the MCP server over stdio (for an MCP client configured as a subprocess):

forecastmcp serve

Or over Streamable HTTP:

forecastmcp serve --transport streamable-http --host 0.0.0.0 --port 8000

An MCP-compatible client configuration for the stdio server:

{
  "mcpServers": {
    "forecastmcp": {
      "command": "/absolute/path/to/ForecastMCP/.venv/bin/forecastmcp",
      "args": ["serve"]
    }
  }
}

For the HTTP container, point the client at http://localhost:8000/mcp.

Docker:

docker build -t forecastmcp:latest .
docker run --rm forecastmcp:latest version
docker run --rm forecastmcp:latest forecast --model chronos --series S001 --horizon 3
docker compose up --build

The image runs as a non-root user and the Compose service exposes port 8000.

Tool surface and roles:

tool

purpose

analyst

admin

list_series

paginated series metadata with prefix/category filters

yes

yes

describe_series

stats, covariates, recent values, frozen windows

yes

yes

forecast

quantile forecast at a frozen origin

yes

yes

ensemble_forecast

reconcile 2+ models by vincentization, inverse error, or best-on-backtest

yes

yes

backtest

WQL/CRPS/MASE/coverage on the frozen windows

yes

yes

model_card

availability, params, context length, training domain

yes

yes

sql

guarded read-only SELECT against metrics/forecasts (max 500 rows)

yes

yes

LoRA training is a CLI operation (forecastmcp finetune), not an MCP tool; the permission map reserves it for the admin role. sql rejects non-SELECT/WITH statements, multiple statements, and write keywords before executing; results are capped at 500 rows. forecastmcp models list is the live availability source of truth on a given machine.

Results

All numbers below were produced by scripts/run_all.sh on this build (macOS arm64, CPU; Python 3.11.14, torch 2.15.0, transformers 5.22.2, chronos-forecasting 2.3.2). They match results/summary.md and results/summary.json. Costs are estimates from configs/costs.yaml (configured per-call price versus device-time price), not cloud bills. Bootstrap intervals are 95% percentile CIs from 1000 series-level resamples (seed 42).

Datasets and holdout

dataset

series evaluated

series available

frequency

horizon

rolling windows

synthetic_supply

50

200

daily (D)

28

6

m4_daily

23

50 loaded (23 cover the full holdout)

daily (D)

28

6

The synthetic panel is a seeded generator of 200 SKU-level demand series with trend, weekly and annual seasonality, promotion spikes, intermittency, and lead-time covariates. m4_daily is a deterministic 50-series head subset from GluonTS; 23 of those series cover all six frozen test windows, and evaluation uses those 23. Windows are non-overlapping and anchored at the end of each panel.

Model comparison

synthetic_supply (50 series, 6 windows):

model

WQL (95% CI)

CRPS (95% CI)

MASE (95% CI)

cov@80 (95% CI)

cov@90 (95% CI)

median latency (ms)

cost/call (USD)

autoets

2.5660 [1.9077, 3.2701]

50.334 [36.530, 65.940]

0.885 [0.825, 0.946]

0.843 [0.818, 0.867]

0.921 [0.906, 0.936]

656.2

0.000600

autotheta

2.6127 [1.9607, 3.3195]

50.818 [37.126, 66.475]

0.912 [0.854, 0.977]

0.816 [0.789, 0.842]

0.907 [0.892, 0.922]

535.8

0.000500

chronos

2.7966 [2.1028, 3.5478]

54.692 [39.942, 72.428]

0.946 [0.889, 1.001]

0.524 [0.480, 0.568]

0.593 [0.544, 0.641]

239.1

0.000400

chronos-lora-v1

2.7915 [2.0780, 3.5540]

54.782 [39.591, 72.507]

0.939 [0.882, 0.996]

0.524 [0.480, 0.568]

0.595 [0.547, 0.644]

285.4

0.000420

ensemble_best_on_backtest

2.5660 [1.9077, 3.2701]

50.334 [36.530, 65.940]

0.885 [0.825, 0.946]

0.843 [0.818, 0.867]

0.921 [0.906, 0.936]

1717.2

0.001970

ensemble_inverse_error

2.6097 [1.9498, 3.3268]

50.532 [36.843, 66.658]

0.896 [0.840, 0.954]

0.830 [0.811, 0.850]

0.914 [0.904, 0.925]

1717.2

0.001970

ensemble_vincentization

2.6508 [1.9810, 3.3733]

51.278 [37.388, 67.651]

0.907 [0.851, 0.964]

0.837 [0.818, 0.855]

0.918 [0.907, 0.928]

1717.2

0.001970

seasonal_naive

3.7609 [2.8908, 4.7105]

72.416 [53.527, 93.840]

1.155 [1.086, 1.234]

0.900 [0.885, 0.915]

0.948 [0.939, 0.956]

0.4

0.000020

seasonal_naive_cov

3.3833 [2.5122, 4.3287]

64.583 [46.607, 85.625]

1.097 [1.028, 1.168]

0.734 [0.703, 0.763]

0.840 [0.817, 0.863]

0.4

0.000030

m4_daily (23 series, 6 windows):

model

WQL (95% CI)

CRPS (95% CI)

MASE (95% CI)

cov@80 (95% CI)

cov@90 (95% CI)

median latency (ms)

cost/call (USD)

autoets

0.5834 [0.3694, 0.8575]

146.735 [97.373, 210.773]

1.702 [1.300, 2.141]

0.823 [0.750, 0.889]

0.889 [0.826, 0.943]

481.0

0.000600

autotheta

0.6529 [0.4144, 0.9560]

161.256 [106.453, 235.144]

1.797 [1.378, 2.291]

0.775 [0.685, 0.856]

0.844 [0.767, 0.913]

160.2

0.000500

chronos

0.4921 [0.3389, 0.6943]

128.033 [88.417, 179.645]

1.889 [1.426, 2.359]

0.579 [0.519, 0.635]

0.675 [0.615, 0.731]

141.3

0.000400

ensemble_best_on_backtest

0.4921 [0.3389, 0.6943]

128.033 [88.417, 179.645]

1.721 [1.325, 2.161]

0.579 [0.519, 0.635]

0.675 [0.615, 0.731]

782.9

0.001520

ensemble_inverse_error

0.5534 [0.3678, 0.7843]

138.992 [96.145, 195.848]

1.741 [1.324, 2.212]

0.784 [0.712, 0.851]

0.855 [0.792, 0.911]

782.9

0.001520

ensemble_vincentization

0.5662 [0.3759, 0.7992]

141.881 [97.904, 199.914]

1.755 [1.331, 2.238]

0.786 [0.710, 0.854]

0.862 [0.798, 0.918]

782.9

0.001520

seasonal_naive

0.7038 [0.4606, 0.9991]

174.450 [119.865, 247.534]

2.123 [1.554, 2.757]

0.772 [0.667, 0.867]

0.839 [0.750, 0.918]

0.5

0.000020

ensemble_best_on_backtest is identical to the best single model by construction: the rule selects the lowest-WQL member on the same holdout, so it is an optimistic selection upper bound, not evidence that ensembling adds information. Latency is measured per series and amortized over the batch; ensemble latency is the sum of its members' median latencies because every member must still be run.

Zero-shot vs LoRA fine-tuned

The adapter is LoRA (r=4, alpha=8, dropout=0.05, target modules q/k/v/o) over amazon/chronos-t5-tiny, trained on 272 windows from 60 series using the train portion of frozen window 1. It has 98,304 trainable parameters out of 8,492,800 (1.1575%). Training ran on CPU for 536.9 s; validation quantile loss improved from 25.6641 to 24.1441 over 3 epochs and did not trigger patience-based early stopping.

Paired series-level bootstrap on the benchmark holdout (50 series, 6 windows; fine-tuned minus zero-shot, negative is better):

metric

delta (fine-tuned - zero-shot)

95% CI

crosses zero

verdict

WQL

-0.0052

[-0.0506, +0.0533]

yes

not significant

CRPS

+0.0901

[-0.8851, +1.3825]

yes

not significant

MASE

-0.0070

[-0.0194, +0.0055]

yes

not significant

coverage_80

+0.0004

[-0.0120, +0.0132]

yes

not significant

A separate 20-series compare run gives the same qualitative picture: WQL -0.0014, CRPS +0.8784, MASE -0.0035, coverage_80 +0.0006, coverage_90 -0.0048 (artifacts/finetune/v1/compare.json). Verdict: LoRA fine-tuning did not produce a statistically significant improvement on this holdout; the CRPS point estimate is worse, and calibration is unchanged.

Covariate and reconciliation ablations

Synthetic dataset, WQL deltas with 95% paired bootstrap CIs (A minus B; negative is better):

| ablation | comparison | delta | 95% CI | crosses zero | |---|---|---|---:|---:|---:| | covariates | seasonal_naive_cov - seasonal_naive | -0.3776 | [-0.4598, -0.3003] | no | | ensemble | ensemble_inverse_error - autoets | +0.0436 | [+0.0058, +0.0830] | no | | ensemble | ensemble_vincentization - autoets | +0.0848 | [+0.0395, +0.1315] | no |

M4 dataset, WQL delta ensemble_inverse_error - chronos: +0.0613 [-0.0047, +0.1423] (crosses zero); ensemble_vincentization - chronos: +0.0741 [+0.0006, +0.1654] (significantly worse).

The promotion covariate helps the weak seasonal-naive baseline, but that baseline remains far behind AutoETS (3.3833 vs 2.5660 WQL). Neither reconciliation method beat the best single model. Multivariate and foundation-model covariate ablations were not possible: TimesFM and Moirai, the adapters that accept exogenous or multivariate inputs, were not installable in this environment (see availability).

Agent harness evaluation

12 verifiable holdout tasks, each run 3 times (36 runs). Ground truth is the holdout actuals; the oracle is the best single model on the same series/window and the same quantile grid. Injected failure tasks: 3 tasks x 3 trials = 9 runs.

metric

value

task success rate

1.0000

tool-call validity rate

1.0000

unsupported-number rate in report.md

0.0000

budget adherence rate

1.0000

recovery rate after injected tool failure

1.0000

selection regret vs oracle (WQL)

0.0911

reconciled-forecast regret vs oracle (WQL)

0.1576

selection-regret run-to-run std

0.0000

mean latency per task

6.2216 s (run-to-run std 1.7948)

mean cost per task

0.003060 USD (run-to-run std 8.8e-19)

mean tool calls per task

9.0

Selection regret is the WQL of the highest-weight member the agent selected minus the oracle best single model, recomputed on the agent's requested quantile grid. The agent's own reconciled forecast was 0.1576 WQL worse than the oracle best single model. Regret is deterministic across seeded trials (std 0.0000); latency varies between 3.767 s and 8.395 s per task depending on which forecasters run.

Model availability

model

available

ran on

reason

seasonal_naive

yes

synthetic_supply, m4_daily

statsforecast importable

autoets

yes

synthetic_supply, m4_daily

statsforecast importable

autotheta

yes

synthetic_supply, m4_daily

statsforecast importable

seasonal_naive_cov

yes

synthetic_supply

ok

chronos

yes

synthetic_supply, m4_daily

chronos importable (cached)

chronos-lora-v1

yes

synthetic_supply

base + adapter available

chronos-small

no

-

disabled in configs/models.yaml

timesfm

no

-

ModuleNotFoundError: No module named 'timesfm'; install the [fm] extra

moirai

no

-

ModuleNotFoundError: No module named 'uni2ts'; install the [fm] extra

Chronos T5 tiny is the foundation model that actually ran here. TimesFM and Moirai are import-guarded adapters that report their exact ImportError without crashing.

Figures

Calibration on synthetic supply chain

Calibration on M4 daily

Accuracy vs cost on synthetic supply chain

Accuracy vs cost on M4 daily

WQL-by-horizon curves are in results/figures/wql_by_horizon_synthetic_supply.png and results/figures/wql_by_horizon_m4_daily.png.

Negative results and limitations

The generated results/NEGATIVE_RESULTS.md records where the methods did not help:

  • Fine-tuning did not help. The LoRA adapter's WQL delta is -0.0052 with a 95% CI crossing zero; CRPS is worse by +0.0901. No statistically detectable gain.

  • No ensemble beat the best single model. Inverse-error and vincentization weighted ensembles were significantly worse on synthetic (by 0.0436 and 0.0848 WQL). Best-on-backtest is identical by construction and is an oracle upper bound.

  • Covariates only helped a weak baseline. The promo-aware seasonal naive beat plain seasonal naive (WQL -0.3776, CI excluding zero) but remained far behind AutoETS, and Chronos T5 cannot consume the covariates at all.

  • Calibration is poor for Chronos. Its 80% intervals covered 0.524 of synthetic actuals and 0.579 of M4 actuals; 90% intervals covered 0.593 and 0.675. AutoETS was much better calibrated (0.843 and 0.921 at 80%; 0.823 and 0.889 at 90%).

  • Routing has a cost. The agent's model selection incurred 0.0911 WQL regret and about 6.22 s / 0.003060 USD per task; the reconciled forecast did not close the gap to the oracle best single model.

  • TimesFM and Moirai were unavailable, so the multivariate and foundation-model covariate ablation is an honest gap, not a result.

Repository map

forecastmcp/
  data/        loaders, synthetic generator, frozen splits, DuckDB store, requests
  forecasters/ protocol, baselines, Chronos, TimesFM, Moirai, finetuned adapter, registry
  mcp/         typed tools, server, budgets, permissions, schemas
  obs/         OpenTelemetry JSONL tracing, replay
  agent/       planner, MCP client loop, reconciliation, citation-checked report
  finetune/    leakage-checked windows, LoRA training, merge
  eval/        metrics, bootstrap, runner, plots, agent evaluation, summary
configs/       models, costs, benchmark, agent tasks, fine-tune profiles, frozen holdout
results/       generated benchmark CSVs/JSON, summary.md, NEGATIVE_RESULTS.md, figures
artifacts/     runs/ traces and call outputs; finetune/v1 adapter + model card
tests/         45 tests, including closed-form metric checks and leakage assertions
scripts/       run_all.sh, reproduce.sh

Reproducing

requirements.lock pins the environment that produced these results. scripts/reproduce.sh reruns data preparation, fine-tuning (optional with SKIP_FINETUNE=1), the full benchmark, fine-tune comparison, 36 agent runs, plots, and the generated summary with FORECASTMCP_SEED=42 and PYTHONHASHSEED=0.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI agents to query governed data warehouses through plain language, returning grounded answers with SQL, confidence grades, and signed receipts while enforcing access, testing, and audit controls.
      72 PyPI
      17
      Apache 2.0
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables governed tool-calling agents with policy decisions, optional human approval, hash-chained audit logging, and deterministic evaluation.
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables evidence-first regression testing for AI agents by turning production traces into reviewed, replayable cases that gate releases. It supports reproducible, auditable agent evaluation with controlled tool execution, evidence-based judging, and versioned quality gates.
      MIT
    • A
      license
      A
      quality
      B
      maintenance
      Enables AI agents to query Trino and Apache Pinot lakehouses under enforced governance, where every SQL statement is AST-validated, table-allowlisted, priced from the engine's own plan before it runs, and blocked or admitted against scan-byte and intermediate-row budgets. It also grounds agents with schema discovery tools, returns verified results with warnings instead of misleading answers, and records every tool call on an audit trail.
      3
      1
      Apache 2.0