inferwatch

Real-time and historical metrics for locally served LLMs — Ollama and vLLM — with a browser dashboard, a settings screen, and an MCP server so an agent can query the same data.

One Python process, one SQLite file. No Docker, no Node, no Prometheus, no external services. It never sits in the request path, so it cannot slow down or break inference.

┌── Ollama ──────────────┐        ┌── vLLM ────────────────┐
│ journald / file /      │        │ GET /metrics           │
│ docker logs            │        │ (native Prometheus)    │
└──────────┬─────────────┘        └──────────┬─────────────┘
           │ per-request rows                │ pre-aggregated
           ▼                                 ▼
        ┌──────────────── SQLite (WAL) ────────────────┐
        │  requests · rollups · vllm_samples/hist      │
        └───────┬──────────────────────────┬───────────┘
                ▼                          ▼
         dashboard :7070            MCP server (stdio)

The two engines are not symmetric, and the tool does not pretend otherwise

This is the central design fact, so it is worth stating plainly.

Both tabs show GPU utilisation, VRAM, temperature and power draw, since those are measured by nvidia-smi rather than by either engine. Temperature and power get separate charts rather than sharing an axis, and each is aggregated the way its unit demands: utilisation averages across cards, VRAM and watts sum, temperature reports the hottest card. Temperature is the one series not plotted from zero — a 33–68 °C range starting at 0 wastes most of the plot.

So they get separate dashboard tabs, separate tables and separate MCP tools. No attempt is made to reconstruct per-request rows for vLLM by differencing counters: you cannot recover which TTFT belonged to which request, and faking it would put invented rows beside real ones.

Ollama: where the numbers come from

Ollama exposes no /metrics endpoint (verified — the route is not in the binary). With OLLAMA_DEBUG=1, the embedded llama.cpp prints a timing block per request, which is combined with the access line and the scheduler line:

slot print_timing: id 0 | task 6763 | prompt eval time = 1254.52 ms /  55 tokens
slot print_timing: id 0 | task 6763 |        eval time = 14591.31 ms / 416 tokens
[GIN] ... | 200 | 16.862061865s | 192.0.2.10 | POST "/v1/chat/completions"
time=... msg="context for request finished" runner.name=.../llama3.2:3b

That yields TTFT, prefill/decode split, token counts, decode rate, status, client, endpoint and model — for every request, from every client, without touching the request path. Two numbers fall out of the combination that neither source has alone:

• queue wait = wall latency − runner time: time spent waiting rather than generating. A proxy cannot separate these.

• prompt cache reuse = full prompt length − tokens actually evaluated.

OLLAMA_DEBUG=1 is required. Without it llama.cpp prints no timing lines: request rates, statuses and GPU metrics still work, but TTFT and token counts stay empty. The dashboard says so in a banner instead of showing zeros.

# /etc/systemd/system/ollama.service.d/override.conf
[Service]
Environment="OLLAMA_DEBUG=1"

vLLM: where the numbers come from

vLLM's native Prometheus endpoint is scraped every collection.scrape_interval_s (default 10s). Cumulative counters are differenced; histogram buckets are differenced per bucket; both are written aggregated per minute, because storing every scrape would add millions of rows a month at resolutions no chart uses.

Three details worth knowing:

• vLLM's own bucket bounds are stored with the counts. Its bounds step 1ms/20ms/250ms/2.5s/40s/640s; Ollama's step 25ms/200ms/1.5s/15s/60s. Neither is a refinement of the other, so re-bucketing one into the other would require interpolating between bounds — inventing numbers. Percentiles are computed against each source's own bounds and reported as bucket upper bounds.

• _sum and _count are exact, so the mean is exact. Since vLLM's buckets are coarse in the seconds range, the dashboard and MCP tools lead with the mean and label percentiles as "at most".

• Restarts are detected via process_start_time_seconds (and by a counter going backwards). The interval spanning a restart is dropped rather than emitted as a bogus delta.

Inter-token latency has been spelled time_per_output_token_seconds, inter_token_latency_seconds and request_time_per_output_token_seconds across vLLM releases. All are collected and whichever has data is used, so this works against old and new servers with no configuration.

Related MCP server: System Monitor MCP Server

Install

Requires Python 3.10 or newer — not because of this code (which is 3.9-clean) but because fastapi, uvicorn, starlette and mcp all require it.

pip install git+https://github.com/floatsmyboat/inferwatch     # or:
git clone https://github.com/floatsmyboat/inferwatch && cd inferwatch
python3 -m venv --upgrade-deps .venv && .venv/bin/pip install -e ".[dev]"

Installing gives you two commands:

Not on PyPI yet; install from git for now.

Backfill from a journal you already have, then look at the database:

inferwatch ingest --since 2d      # or: python -m inferwatch.main ingest
inferwatch stats

Run it:

inferwatch serve                  # http://127.0.0.1:7070

As a service — the unit is rendered from systemd/inferwatch.service.in for the current user, checkout path and interpreter, so nothing is hardcoded:

./scripts/install-systemd.sh          # system service (uses sudo)
sudo systemctl enable --now inferwatch

./scripts/install-systemd.sh --user   # or per-user, no sudo
systemctl --user enable --now inferwatch

Override with HOST=0.0.0.0 PORT=7070 DATADIR=... ./scripts/install-systemd.sh.

Exposing it on a network

There is no authentication. The dashboard is read-only (GET only) and stores no prompt or response text — only counts, timings, model names and client addresses. Restrict it at the firewall:

sudo ufw allow from 192.168.1.0/24 to any port 7070 proto tcp comment "inferwatch"

Configuring what is monitored

Everything is editable from the dashboard's Settings tab, or from the CLI:

python -m inferwatch.main sources                       # list
python -m inferwatch.main sources add --kind vllm --name qwen \
    --set url=http://127.0.0.1:8000
python -m inferwatch.main sources add --kind ollama --name box \
    --set reader=file --set path=~/.ollama/logs/server.log
python -m inferwatch.main sources disable qwen

Ollama log readers. Not everyone runs Ollama under systemd:

The file reader follows like tail -F, surviving rotation (inode change) and truncation, and persists an offset so a restart does not replay. Ollama's Go lines carry time=, but llama.cpp's slot lines — the ones holding the token counts — carry no timestamp, so the most recent one seen is carried forward. Ordering, which the joins depend on, always holds; absolute precision is lower than journald's, and a second-resolution [GIN] line is clamped forward so time never appears to run backwards.

Settings precedence

spec default  <  database (Settings tab)  <  environment  <  command line

A key supplied by the environment or a flag is shown read-only in the Settings tab with its origin, because the process was told to use it and a browser must not override that silently. Saving is all-or-nothing, so a typo in one field cannot leave a half-applied configuration. Changes to sources, intervals and retention apply without a restart; server.host and server.port are marked as needing one, and the API says so after saving.

Configuration reference

Every setting below is editable in the Settings tab, settable as an environment variable, and some are pinnable with a flag. The tables are generated from the code (scripts/gen-config-docs.py), so they cannot drift from what the program actually accepts.

Generated by scripts/gen-config-docs.py — do not edit by hand.

Collection

Retention

Dashboard

Server

Source fields

Set these with --set key=value on sources add, or in the Settings tab.

Ollama (--kind ollama)

vLLM (--kind vllm)

Command-line flags

Flags that pin a setting outrank both the environment and the Settings tab; the tab shows those keys read-only with their origin.

Scope: one Ollama source, many vLLM sources

vLLM rows are keyed by source throughout, so any number of vLLM instances can be monitored side by side. The Ollama tables (requests, events, ps_samples) are not source-partitioned, so exactly one Ollama source runs at a time; enabling a second logs a warning and ignores it rather than silently blending two instances into one set of numbers. Partitioning those tables is a schema change worth doing deliberately.

Dashboard

http://127.0.0.1:7070 — three tabs: Ollama, vLLM, Settings.

Which GPUs belong to which engine

A host often runs more than one engine, so plotting every card on an instance's pane would imply it uses all of them. Each vLLM instance's GPUs are resolved by following processes — the port it serves → the listening pid → its descendants → intersected with nvidia-smi's compute processes → the cards those hold. The instance's cards carry the series colour and its VRAM tile counts only those; the host's other cards stay visible in grey, labelled "other engine".

Attribution needs ss, a local instance, and nvidia-smi process visibility (often absent inside containers). When any of those is missing the pane says so and shows every card without emphasis, rather than guessing.

One filter row scopes everything below it. Every chart has a Table toggle showing the same series as numbers, so no value is reachable only by hovering. A live SSE feed drives the request ticker and the current-rate figure.

URL parameters: ?tab=vllm, ?window=6h, ?model=llama3.2:3b, ?source=name, ?nostream=1 (disables the live feed — useful for kiosk displays and screenshot tools, which otherwise wait forever on an open stream).

API

/api/sources/probe checks a definition before it is saved, so a typo surfaces there rather than as silence in the charts.

MCP server

./scripts/install-mcp.sh      # writes .mcp.json for this checkout (gitignored)

or claude mcp add inferwatch -- /path/to/.venv/bin/python -m inferwatch.mcp_server.

Opens the same SQLite file read-only (mode=ro plus PRAGMA query_only) and answers through the same query layer as the dashboard, so a number it reports always matches the number on screen.

Retention

• Raw per-request rows (Ollama): 7 days (retention.raw_days).

• GPU samples, events, vLLM rows: 30 days (retention.sample_days).

• rollup_1m and rollup_1h: kept indefinitely.

Rollups store fixed-bucket histograms of TTFT and latency, not pre-computed percentiles. Histograms add, so a percentile over any range is computed by summing buckets and walking to the target rank. Percentiles of percentiles would be meaningless; this is not.

Queries inside the raw window return exact percentiles; beyond it they come from histograms and are reported as the upper bound of the containing bucket. Every response carries exact: true|false.

Restarts and reboots

Resume state is per source — a journald cursor, a file inode+offset, or a docker timestamp — and is flushed on SIGTERM. Two things make that safe rather than merely likely:

Writes are idempotent. Every request and event row carries a dedupe_key under a UNIQUE index, and inserts are INSERT OR IGNORE. Re-reading lines that were already stored is a no-op, so ingest can be run repeatedly and a resume can safely overlap.

A cursor that cannot be used is not trusted. If the journal a cursor points into was rotated away, journalctl silently repositions using the timestamp embedded in the cursor and resumes correctly. But a cursor stamped in the future (clock skew, a restored database) makes journalctl wait for entries that will not arrive, stalling collection silently; such a cursor is rejected on startup. A follow attempt yielding nothing twice running does the same.

systemctl stop completes in well under a second. systemd records ExecMainStatus=15 alongside Result=success: uvicorn deliberately re-raises the signal after shutting down, so exiting by SIGTERM is expected, not a crash.

Honest limitations

Attribution under parallelism (Ollama). The task id in llama.cpp's timing lines and the access line's status never appear together, so they are joined by arrival order. With one request in flight that is exact. When two finish before either access line prints, nothing in the log disambiguates them — those rows are stored as attribution='ambiguous' rather than guessed. Values: exact, ambiguous, none (failed before reaching the runner — no model is guessed either), orphan (timings with no access line). A 2-day backfill on the development host scored 174 exact, 9 ambiguous, 4 orphan, with OLLAMA_NUM_PARALLEL=1 in effect; expect a higher ambiguous share the more requests run concurrently.

vLLM's token and request counters are not per-request aligned. generation_tokens_total advances as tokens stream; request_success_total advances only when a request completes. Over a short window they therefore describe overlapping but different sets of requests, and dividing one by the other does not give tokens per request. The API marks this with counters_aligned: false, and the dashboard says so on the vLLM tab.

No per-request anything for vLLM. Covered above. If you need per-request detail from vLLM, its request-level logging is the only source, and it logs prompt text — which this tool deliberately never stores.

Logs are the feed, not the archive. A journal may hold only a day or two depending on journald.conf; the SQLite file is the historian. If logs rotate faster than inferwatch runs, that gap is unrecoverable.

Two byte-identical events in the same microsecond collapse to one. The dedupe key for events is built from their values, so an identical warning logged twice within a microsecond keeps one row. A deliberate trade for guaranteed idempotency — dropping a repeated warning beats duplicating history.

Health-check traffic is separated, not counted. HEAD / and GET /api/ps were 96% of requests on the development host. They are stored with class='health' and excluded from inference rates unless dashboard.include_health is on; requests_all always includes them.

Depends on log formats and metric names. Ollama's timing lines are debug output, not a contract, and vLLM renames metrics between releases. tests/test_parse.py holds verbatim fixture lines and tests/test_vllm.py a real /metrics excerpt; if an upgrade breaks parsing, those tests fail and show what changed.

Tests

python -m unittest discover -s tests -t .

CI runs this on Python 3.10 through 3.14, plus a packaging job that builds the wheel, asserts the dashboard HTML is inside it, and installs it into a clean environment from an empty directory so the source tree cannot mask a packaging mistake. No network, GPU or engine is required. Parser fixtures are verbatim real log lines and a real /metrics excerpt. Coverage includes the correlator's join and its ambiguous/orphan/failed cases, histogram percentiles and rollup idempotency, the schema migration, signal-safe commits, cursor validation, file rotation and truncation, timestamp monotonicity, counter-reset detection, and config precedence and locking. The configuration reference in this file is generated from the spec and a test fails if it drifts. The dashboard's JavaScript is syntax-checked with a pure-Python parser and its formatters executed in a real JS engine (both optional — no Node needed).

.venv/bin/python scripts/gen-config-docs.py --check   # docs match the code?

Layout

inferwatch/parse.py        ollama log line parsers (pure, fixture-tested)
inferwatch/readers.py      journald / file / docker log readers
inferwatch/collect.py      correlator, GPU + model pollers, maintainer
inferwatch/vllm.py         Prometheus scraper, delta and reset handling
inferwatch/gpuproc.py      maps GPUs to the process tree holding them
inferwatch/vllm_metrics.py vLLM query layer
inferwatch/metrics.py      ollama query layer (shared by API and MCP)
inferwatch/store.py        SQLite schema, rollups, histograms, retention
inferwatch/config.py       typed settings spec, precedence, source validation
inferwatch/supervisor.py   builds and rebuilds collectors from the sources table
inferwatch/api.py          FastAPI endpoints + SSE
inferwatch/web/index.html  dashboard (single file, no CDN, no build step)
inferwatch/mcp_server.py   MCP server (read-only)
inferwatch/main.py         serve / ingest / stats / sources

Contributing

Issues and pull requests are welcome. Two things make a change easy to accept:

• python -m unittest discover -s tests -t . passes.

• If you touched inferwatch/config.py, run python scripts/gen-config-docs.py so the README's configuration reference matches the code — a test enforces it.

Parser changes should come with a fixture line copied verbatim from real engine output, the way the existing tests do. Log formats and metric names are not contracts, and a real fixture is what makes a future break obvious.

License

Apache License 2.0 — see LICENSE and NOTICE.