llm-monitor/README.md

llm_monitor — gufo + vLLM real-time dashboard

One page, two engines, live. Watches a gufo (llama.cpp-family) server and a vLLM server side by side: sessions/slots, queueing, KV/GPU cache pressure, throughput, and latency histograms — refreshed every second without a page reload.

• Read-only by construction. The backend only ever calls client.get(). There is no POST/PUT/PATCH/DELETE route anywhere (curl -X POST /api/gufo/slots → 405), nothing is written to either engine, and upstream bodies are parsed and re-emitted as JSON instead of echoed.
• Per-engine health. Each engine is polled and tracked independently. One engine being down never blocks or blanks the other.
• Opt-in engines. A server only appears if its *_BASE_URL env var is set. Send this to someone who runs just vLLM and they see one clean vLLM panel — no gufo half stuck in red.
• No build step, no CDN, no browser dependencies. 2 app files (app.py, static/index.html). Sparklines are hand-rolled inline SVG — the engines live on an internet-disconnected podman network, so a CDN chart library would just be a dead request.

Endpoints and ports (source of truth)

Taken from ~/Deployments/reeseapps/ai/deployments:

Engine	Deployment	Host	Reachable as	Port	Upstream endpoints used
gufo	deployments/gufo	deskwork-ai	deskwork.reeselink.com	8731	GET /slots (alias /v1/slots), GET /metrics
vLLM (radiance)	deployments/vllm-radiance	driveripper-ai	driveripper.reeselink.com	8000	GET /metrics
this dashboard	—	wherever you run it	—	8090	serves the page + /api/*

Note the port: 8000 is already vLLM/juggernaut in that fleet and 8731 is gufo, so the dashboard listens on 8090 by default. The 8080/8000 defaults you may expect for the upstreams are wrong for this fleet — 8731 is gufo here.

Install & run (uv)

This is a uv project: pyproject.toml + uv.lock, no build backend (package = false — it is an app, not a library). uv manages the venv; you never activate it manually.

uv sync                                  # create .venv from uv.lock (incl. dev group)

# Point at the engines you actually run (these are the fleet's real endpoints), then start:
export GUFO_BASE_URL=http://deskwork.reeselink.com:8731      # gufo  — deskwork-ai:8731
export VLLM_BASE_URL=http://driveripper.reeselink.com:8000   # vLLM  — driveripper-ai:8000
uv run uvicorn app:app --host 127.0.0.1 --port 8090

Set only what exists. With just VLLM_BASE_URL the page is a single-engine vLLM dashboard and /api/gufo/* answers 404 {"configured": false} without ever dialling a port; with neither set it shows a "no engines configured" card instead of two permanently red panels.

Open http://127.0.0.1:8090.

uv run python app.py                     # same thing, reads DASHBOARD_HOST/DASHBOARD_PORT
uv run uvicorn app:app --port 8090 --reload   # dev loop (watches app.py + static/)
uv sync --no-dev                         # production install: skips the pytest group

Keep the --port 8090. Bare uvicorn app:app binds :8000, which is where vLLM already lives on driveripper-ai — the dashboard would refuse to start there (and if it started on a machine without vLLM, VLLM_BASE_URL would point at itself).

Can't reach those hosts directly (VPN, firewall, different network)? Tunnel them to loopback and point the env vars at the tunnel:

ssh -N -L 8731:127.0.0.1:8731 -L 8000:127.0.0.1:8000 deskwork-ai
export GUFO_BASE_URL=http://127.0.0.1:8731 VLLM_BASE_URL=http://127.0.0.1:8000

After editing dependencies: uv lock (updates uv.lock), then regenerate the pip shim: uv export --no-dev --no-hashes > requirements.txt.

Without uv

requirements.txt is a pinned export of uv.lock, so a host that has no uv still works:

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
uvicorn app:app --host 127.0.0.1 --port 8090

In a container

The Containerfile builds a slim, non-root image (uv-locked deps, no build tools, no frontend build step — the page is copied verbatim). CI publishes it to gitea.reeseapps.com/Experimental/llm-monitor on every push to main and on release tags (.gitea/workflows/build-and-push.yml).

podman build -t localhost/llm-monitor:main -f Containerfile .
podman run -d --name llm-monitor -p 8090:8090 \
  -e GUFO_BASE_URL=http://deskwork.reeselink.com:8731 \
  -e VLLM_BASE_URL=http://driveripper.reeselink.com:8000 \
  localhost/llm-monitor:main

Nothing is baked in: no host guesses and no API keys live in the image, so the same image serves a two-engine deployment, a vLLM-only deployment, or a gufo-only one — set (or omit) GUFO_BASE_URL / VLLM_API_KEY at run time. Inside the container the dashboard binds 0.0.0.0:8090; reach the engines by whatever name is resolvable from that network (or publish the container onto the engines' podman network and use the quadlet hostnames, deskwork-ai / driveripper-ai). The HEALTHCHECK probes /api/config, which never dials an engine — a dead engine must never mark the dashboard itself unhealthy.

Tests

uv run pytest                      # 114 tests, no network, no real engines
uv run pytest --cov                # with coverage (app.py 98%)
uv run pytest -k parsing -vv       # just the Prometheus parser / field maps
uv run -p 3.10 pytest              # check the requires-python floor

Supported Python: >=3.10 (CI-worthy runs verified on 3.10 / 3.13 / 3.14). There is no .python-version pin on purpose — uv uses whatever interpreter is present, since this gets dropped onto whatever box hosts the engines.

tests/conftest.py swaps app.state.client for a FakeUpstream that records the URLs it was asked for and can raise httpx.ConnectError / TimeoutException on demand — so engine outages are asserted, not hand-tested. Suite covers: parser edge cases (aliases, absent series, label escaping, histogram interpolation), passthrough fidelity, 405 on every non-GET verb, 502/504 translation, per-engine auth, and the no-third-party-assets rule.

Configuration (env vars only)

Var	Default	Meaning
GUFO_BASE_URL	unset → gufo hidden	gufo origin (scheme://host[:port]); unset/blank means this deployment has no gufo
VLLM_BASE_URL	unset → vLLM hidden	vLLM origin; unset/blank means this deployment has no vLLM
GUFO_API_KEY	unset	if set, sent upstream as Authorization: Bearer <key>
VLLM_API_KEY	unset	if set, sent upstream as Authorization: Bearer <key>
UPSTREAM_TIMEOUT_SECONDS	3	per-request upstream timeout (short: the UI polls every 1s)
RATE_WINDOW_SECONDS	10	rolling window for vLLM Δcounter rates (see “Where prompt tok/s comes from”)
POLL_MS	1000	browser poll interval
HISTORY_LEN	60	sparkline samples kept in the browser
DASHBOARD_HOST/DASHBOARD_PORT	127.0.0.1 / 8090	only used by python app.py

There is deliberately no default host for the two engines. A built-in 127.0.0.1:8731 looked convenient until a vLLM-only user got a red gufo panel for a server they never installed; "not configured" and "configured but down" are different statements, so only the env var decides. http://host:port/ is accepted (the trailing slash is trimmed) and a whitespace-only value counts as unset. /api/config exposes gufo_enabled / vllm_enabled, and the page builds its layout from those.

No auth on the dashboard itself by default — bind it to 127.0.0.1 (default) or put it behind your reverse proxy. The keys are never returned by any endpoint; /api/config reports only gufo_auth: true/false.

Upstream start commands

These are the commands the dashboard expects to be able to scrape. Full worked examples are in each deployment's README.md in the deployments repo.

gufo — /slots and /metrics are built in; --sessions N is what sizes the slot grid:

gufo serve llm --sessions 4 \
  --model /models/qwen3.8-flash-next/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
  --context 262144 --prefill-chunk 4096 --served-model-name turbo \
  --host 0.0.0.0 --port 8731
# quadlet form: gufo serve --host 0.0.0.0 --port 8731 llm --sessions 2 ...

vLLM — metrics are on by default at /metrics (no flag needed; --disable-metrics would break the vLLM panel):

vllm serve amd/Qwen3.8-27B-Quark-AWQ-MXFP4 \
  --served-model-name=juggernaut \
  --max-model-len=131072 --max-num-seqs=4 --max-num-batched-tokens=4096 \
  --kv-cache-dtype=fp8 --enable-prefix-caching --gpu-memory-utilization=0.95 \
  --host=0.0.0.0 --port=8000

Backend routes

Route	Upstream	Returns
GET /	—	the dashboard (static/index.html)
GET /api/config	—	base URLs, *_enabled (which engines exist), poll/history settings, *_auth booleans (never the keys)
GET /api/gufo/slots	GET {GUFO_BASE_URL}/slots	upstream JSON passed through byte-for-byte; falls back to /v1/slots
GET /api/gufo/metrics	GET {GUFO_BASE_URL}/metrics	Prometheus text → JSON, gufo field map
GET /api/vllm/metrics	GET {VLLM_BASE_URL}/metrics	Prometheus text → JSON, vLLM field map

Envelope for the JSON routes:

{ "ok": true, "engine": "vllm", "upstream": "http://127.0.0.1:8000/metrics",
  "scraped_at": 1760000000.123, "series": 21, "metrics": { "...": {} } }

If an engine's *_BASE_URL was unset, its two routes answer 404 {"ok": false, "engine": "gufo", "configured": false, "detail": "gufo is not configured — set GUFO_BASE_URL and restart"} and no upstream request is made — a server that is not installed is not an outage, so it must not look like one.

Every metric cell is {"value": <number|null>, "source": "<series or computed:...>|null", "derived": <bool>}. A missing series is value: null, which the page renders as n/a — it never 500s because a metric name changed. Upstream failures return 502 (unreachable / HTTP error) or 504 (timeout) with {"ok": false, "detail": "..."}, and that detail is what the panel shows.

One Prometheus parser, two field maps

app.py contains a single Prometheus text parser (class Prom: tolerant line matcher, label unescaping, sum/max/avg aggregation, by-label grouping, histogram _sum/_count/_bucket → mean + interpolated p95). The engines differ only in their field maps:

The tables below are what the deployed engines actually export (captured live; see tests/fixtures/*_real.txt). Each field is an alias list, so the names from the original spec and the legacy llama: prefix still work — but the deployed name comes first, and the live fixtures are the regression test that keeps them honest.

gufo — note tokens_predicted_* (not generated), and the speed gauges are named *_tokens_seconds while their unit is tokens/second:

UI field	Series (deployed name first)	Agg
Running	llamacpp:requests_processing	sum
Waiting	llamacpp:requests_deferred	sum
KV cache	llamacpp:kv_cache_usage_ratio	max
prompt tok·s⁻¹ (all slots)	computed:Δllamacpp:prompt_tokens_total/Δt (RateEstimator), else llamacpp:prompt_tokens_seconds	Δ / avg
prompt tok·s⁻¹ (last pass)	llamacpp:prompt_tokens_seconds — one request's last prefill pass; exposed as prompt_tokens_per_second_native	avg
decode tok·s⁻¹ (all slots)	computed:Δllamacpp:tokens_predicted_total/Δt (RateEstimator), else llamacpp:predicted_tokens_seconds	Δ / avg
decode tok·s⁻¹ (last pass)	llamacpp:predicted_tokens_seconds — one request's last decode pass, holds that value while the box idles; exposed as decode_tokens_per_second_native	avg
prompt tokens	llamacpp:prompt_tokens_total — excludes cache hits	sum
cache-hit tokens	llamacpp:prompt_tokens_cached_total	sum
generated tokens	llamacpp:tokens_predicted_total	sum
total tokens	computed:prompt+cached+generated	—
prompt cache	computed:cached/(prompt+cached)	—
peak ctx	llamacpp:n_tokens_max	max
gpu losses	gufo_device_lost_total (non-zero = incident, shown red)	sum
spec accept	computed:accepted/draft_tokens from llamacpp:spec_decode_num_*	—

vLLM — kv_cache_usage_perc (not gpu_cache_usage), generation_tokens_total (not generation_tokens_total), TPOT as inter_token_latency_seconds, and every series tagged engine= / model_name=:

UI field	Series (deployed name first)	Notes
Running	vllm:num_requests_running	sum across engine= label sets
Waiting	vllm:num_requests_waiting	sum
Waiting · capacity	vllm:num_requests_waiting_by_reason{reason="capacity"}	live reasons: capacity, deferred
GPU cache	vllm:kv_cache_usage_perc, vllm:gpu_cache_usage	max (never sums past 1.0)
TTFT	vllm:time_to_first_token_seconds	histogram → mean, p95
TPOT	vllm:inter_token_latency_seconds, vllm:time_per_output_token_seconds, vllm:request_time_per_output_token_seconds	histogram → mean, p95
Queue time	vllm:request_queue_time_seconds	histogram → mean, p95
prompt / generation tokens	vllm:prompt_tokens_total, vllm:generation_tokens_total	sum
prefix cache hit	computed:hits/queries from vllm:prefix_cache_{hits,queries}_total	—
preemptions	vllm:num_preemptions_total (non-zero = KV pressure, shown red)	sum
prompt tok·s⁻¹	vllm:prompt_throughput, vllm:avg_prompt_throughput_toks_per_s — neither exists on this build — else computed:Δprompt_tokens_total/Δt, else computed:mean(request_prompt_tokens)/mean(request_prefill_time_seconds)	see below
prefill avg	computed:mean(request_prompt_tokens)/mean(request_prefill_time_seconds)	lifetime average, ~4.9k tok/s live
decode tok·s⁻¹	vllm:generation_throughput if present, else computed:Δgeneration_tokens_total/Δt, else computed:running/tpot	derived: true; the UI tags the source
per-seq tok·s⁻¹	computed:1/tpot	from TPOT mean

Where vLLM's prompt tok/s comes from

That vLLM exports no rate gauge at all — no *_throughput, no *_toks_per_s, nothing with a rate-ish name (grep its /metrics and you get zero hits). Counters only. So the number is computed, in this order, and the UI labels which one you are looking at:

1. vllm:prompt_throughput if a future build provides it — the only real scraped gauge.
2. Measured Δ — Δprompt_tokens_total / Δt over a rolling RATE_WINDOW_SECONDS window (default 10s). This is actual current throughput, tagged Δ10.0s. A bare 1-second delta jitters between 0 and spikes because prompts arrive in bursts, hence the window. Counter resets (engine restart) yield n/a for one scrape rather than a huge negative rate.
3. Lifetime average prefill speed — mean(request_prompt_tokens) / mean(request_prefill_time_seconds), i.e. tokens-per-second this engine has achieved on prefill since it started (~4.9k tok/s live). Used only until there are two scrapes to diff, and shown on its own as prefill avg.

Decode tok/s follows the same precedence, and the measured Δ replaced the old running/tpot estimate as step 2 — the Δ counts real output, the estimate assumed every running request was decoding at that instant. The per-request figure is still 1/TPOT.

An n/a now means the counters themselves are gone, not that a gauge name changed.

Where gufo's decode tok/s comes from

gufo's decode tok/s is the aggregate over every slot generating at once, and it is measured, not scraped. llamacpp:predicted_tokens_seconds is a single unlabeled gauge holding the last decode pass of one request: with two slots decoding at 35 tok/s it still reads 35, and when the box goes idle it keeps serving that stale number indefinitely (live: stuck at 38.0 with requests_processing == 0). There are no label sets to sum, so no aggregation rule could ever fix it.

llamacpp:tokens_predicted_total advances once per token from every slot, so the same RateEstimator used for vLLM answers the question directly — two slots at 35 tok/s advance the counter 70 tokens per second and the cell reads ~70:

1. Measured Δ — Δtokens_predicted_total / Δt over the RATE_WINDOW_SECONDS window, tagged Δ10.0s. Goes to a genuine 0.0 when nothing is generating.
2. The scraped gauge — only until there are two scrapes to diff, or on a build missing the counter; rendered with no tag (it is a real scrape) and still exposed separately as decode_tokens_per_second_native.

This is the one place where measured outranks a native gauge, deliberately inverting the usual precedence: vLLM's native throughput gauge really is an aggregate, gufo's is not. Both engines share the single RATES estimator and cannot cross-read, because it keys off the resolved series name — which also means a counter rename starts a fresh window instead of diffing two different series and inventing throughput.

gufo's /slots is passed through untouched and returns a bare JSON array (older builds wrap it as {"slots": [...]}; the page accepts both). Slot fields read: id, is_processing (authoritative; state 0/1 is the fallback), id_task/task_id, n_prompt_tokens, n_prompt_tokens_cache, n_prompt_tokens_processed, n_ctx, model, speculative, and next_token[0].{n_decoded,n_remain} (-1 means "nothing pending" → n/a).

Dashboard layout

• Fleet total (very top): one number per metric summed across both engines — running, waiting, prompt tok·s⁻¹, decode tok·s⁻¹, total tokens. Hidden entirely when fewer than two engines are configured (a cross-engine sum of one engine just duplicates its own card). Rules:
  • Only engines answering now are counted. A dead engine keeps its last good sample in its own panel (labelled stale, correct there), but adding a frozen number to a fleet total would present hours-old data as current — so it is dropped, the header drops to 1 of 2 engines, and the accent/dot go amber. Zero answering → every cell n/a, dot red. Liveness is keyed on each engine's /metrics job only: gufo's /slots failing must not remove it from a total that came from /metrics.
  • Summed in the browser, not on the backend. The page already holds both parsed payloads, so the API stays one-scrape-pure with no second polling path hammering the engines.
  • Hover any cell for the itemised breakdown (gufo 45 + vLLM 37 · excluded: …), with each part marked (measured) when it is a derived rate.
  • Sums are computed on the raw values then rounded once, so a cell can differ from the two rounded panel numbers above it by 1 (308 + 177 shown, 486 total).
  • The prompt sum is the one imperfect addition: gufo's prompt counter excludes cache hits (so it measures newly-computed prefill work) while vLLM's includes cached prefix tokens. Both are per-engine aggregates and additive within an engine, but the fleet number leans slightly low for gufo. The prompt tok/s label carries the caveat on hover.
• Header line names only the engines that exist (vLLM http://host:8000 · poll 1000ms · read-only), and the tab title follows (LLM Monitor · vLLM).
• Shared stats strip (top): per engine — cache-usage bar (blue → amber ≥75% → red ≥90%), prompt tok·s⁻¹, decode tok·s⁻¹, total tokens, plus slots-busy (gufo) / per-seq tok·s⁻¹ (vLLM). Strip labels stay terse (they share one dense grid), so gufo's cell just reads decode tok/s with its all-slots meaning on hover; the gufo panel's sparkline caption spells it out as decode tok/s · all slots. Every derived rate carries its provenance tag (Δ10.0s, from TPOT, lifetime avg) with the exact computed: source on hover.
• gufo panel: vLLM-style banner Running: N reqs, Waiting: M, sparklines for the all-slots decode tok·s⁻¹ and KV usage, then a grid with one card per /slots entry: slot id + served model (+spec when speculative decoding is on), status pill (green processing / grey idle), request id, prompt tokens, cache hits (with % of prompt), generated so far (next_token[0].n_decoded), remaining budget, and slot context size. Queued requests are not in /slots — that is what Waiting (deferred) covers.
• Alarm colours: gpu losses (gufo gufo_device_lost_total) and preemptions (vllm:num_preemptions_total) turn red above zero — both are cumulative counters, so a moving number there means something is wrong, not busy.
• vLLM panel: banner Running: N reqs, Waiting: M, sparklines for the all-slots decode tok·s⁻¹ and GPU cache usage — deliberately the same pair of charts as the gufo panel, both fed by decode_tokens_per_second (Δvllm:generation_tokens_total/Δt here) so the two engines read alike. The running count is no longer charted; it lives in the banner and the fleet total. Then metric cards for waiting-by-reason, GPU cache bar, TTFT / TPOT / queue time (mean + p95), prompt + generation token totals, and series scraped.
• Sparkline y-axis: each chart labels the top, middle and bottom of the scale it actually plotted against (49 / 43 / 37, or 100% / 50% / 0% for the cache charts, which are pinned to 0–1 rather than autoscaled), with faint rules at those levels. The plots autoscale, so without the labels a 1% drift and a 2× spike are the same squiggle. Labels are rendered as HTML beside the SVG, never inside it — preserveAspectRatio="none" stretches the viewBox to the card width and would smear the text. Precision adapts so the three never collide (a 1–2 request axis reads 2.0 / 1.5 / 1.0, not 2 / 2 / 1), and a flat window labels all three levels with the value itself — an idle box says 0 / 0 / 0, not the ±0.001 the plot's zero-range padding happens to produce. Fewer than two points → axis n/a.
• Every second: all four /api/* endpoints are polled independently (setInterval + fire-and-forget, with an in-flight guard so a slow upstream can't pile up requests). DOM is updated in place — slot cards are keyed by slot id and reused, so the poll is invisible. Per-engine green/red dot, "updated HH:MM:SS" stamp, and a stale Ns counter that only appears when that engine stopped answering. Space or the pause button freezes polling for reading a snapshot.

Failure behaviour

Situation	Result
Engine's *_BASE_URL unset	that engine is not on the page at all (no strip card, no panel, no polling, no fleet share) and its routes answer 404 configured:false; neither configured → a setup card naming the env vars
Engine stopped / unreachable	that panel's dot goes red, red banner with the upstream error, last good values kept and greyed, stale Ns grows; the other engine keeps polling normally
Engine comes back	dot goes green and the banner clears on the next successful poll — no reload needed
A metric series is absent	that one cell shows n/a (and series absent → n/a in its foot), everything else renders
Upstream returns non-Prometheus garbage	parsed as 0 series → all cells n/a, page still healthy
Sparkline has < 2 points	chart stays faded and its y-axis reads n/a rather than inventing a scale
Upstream slower than UPSTREAM_TIMEOUT_SECONDS	504, dot red, and the request is abandoned rather than queued

Smoke test without live engines

Start the app first (uv run uvicorn app:app --host 127.0.0.1 --port 8090), then:

curl -s localhost:8090/api/config | python3 -m json.tool   # what it is pointed at
curl -s localhost:8090/api/gufo/slots | head -c 200        # proxy passthrough
curl -s localhost:8090/api/gufo/metrics | python3 -m json.tool | head -30
curl -s localhost:8090/api/vllm/metrics | python3 -m json.tool | head -30
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8090/api/gufo/slots   # -> 405 (read-only)

One-engine deployment (what a vLLM-only recipient sees) — gufo answers 404 configured:false and the page has no gufo section at all:

env -u GUFO_BASE_URL VLLM_BASE_URL=http://127.0.0.1:8000 \
  uv run uvicorn app:app --host 127.0.0.1 --port 8090
curl -s localhost:8090/api/config | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["gufo_enabled"], d["vllm_enabled"])'   # False True
curl -s -o /dev/null -w '%{http_code}\n' localhost:8090/api/gufo/metrics   # -> 404, nothing dialled

n/a + red-dot path described above.

To see what an engine really exposes (do this before trusting any metric name, including the ones written in this README):

curl -s "$GUFO_BASE_URL/metrics"  | grep -v '^#' | sed -E 's/[{ ].*//' | sort -u
curl -s "$VLLM_BASE_URL/metrics" | grep -v '^#' | sed -E 's/[{ ].*//' | sort -u
curl -s "$GUFO_BASE_URL/slots"   | python3 -m json.tool | head    # a bare array

Files

pyproject.toml       uv project: deps, dev group, package=false, pytest/coverage config
uv.lock              pinned resolution (commit it)
app.py               FastAPI app: config, Prometheus parser, 2 field maps, 5 GET routes
static/index.html    the dashboard: CSS + vanilla JS + inline-SVG sparklines
tests/               conftest.py (FakeUpstream), test_parsing.py, test_proxy.py,
                     test_real_contracts.py + fixtures/ (verbatim captures from the live engines)
requirements.txt     pip shim, generated from uv.lock
AGENTS.md            working rules for humans and agents on this repo
README.md

There are no JS dependencies and nothing to build: static/index.html is served verbatim.