evaluate
Compares candidate VARMA forecasts with univariate yardstick models across rolling origins to verify gains in RMSE and MAPE, flagging models that do not improve forecast accuracy.
Instructions
N5 — The yardstick: does the candidate forecast better than the univariates?
Estimates the candidate AND the diagonal system (the univariate models) on
the first estwin observations, holds the parameters fixed, and forecasts
from every origin to the end of the data, horizon steps ahead. Compares
RMSE and MAPE series by series and horizon by horizon. A VARMA that does not
gain here has no reason to exist, whatever its in-sample significance.
estwin counts observations of the FIRST series; leave enough data after it
(at least a few dozen origins) or the comparison says little. links,
zeros and indices: as in estimate, the same candidate.
ON A VEC (the current model is one, estimate_vec): the VEC's rolling-origin forecasts (drvec's -estwin/-f: the fit on the first rows, parameters held, errors in each series' own units) against the univariates on the same origins, and against the VARMA in differences when one was estimated. THE RULE depends on the purpose recorded at N1c: a VETO when it is "forecast", a CAVEAT (always reported) for a "long run" claim. p, q and the VARMA arguments are then ignored.
yardstick (route V): the analyst's univariate models, one fue file per
series (a JSON list or comma-separated paths), e.g. their .pre from art.
Without it the yardstick is route U's light builder, which takes the first
of a tie in art's ranking; the report names the models either way. Once
given it stays for the session (forecast_uncertainty and structure use it).
On route U the yardstick is already the loaded files.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"name": "<name>"
},
"name": "evaluate"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| p | No | ||
| q | No | ||
| name | Yes | ||
| cross | No | additive | |
| links | No | ||
| zeros | No | ||
| estwin | No | ||
| diagcov | No | ||
| horizon | No | ||
| indices | No | ||
| yardstick | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |