Experimental/llm-monitor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial import: read-only gufo + vLLM dashboard
Build and Push Container / build-and-push (push) Successful in 38s
Details

Browse Source 
Single-file FastAPI proxy plus a no-CDN vanilla-JS page for watching a gufo
server and a vLLM server side by side: one Prometheus parser (class Prom) with
per-engine field maps, measured Δcounter/Δt rates for engines that export no
rate gauge, and strict read-only GET-only routing. Engines are opt-in via
*_BASE_URL — unset means no panel, no poll, and a 404 {"configured": false}
without dialling.

Includes the container image and CI so the dashboard can be deployed as an
image rather than a checkout:

- Containerfile: uv-locked two-stage build on python:3.14-slim, non-root
  (uid 10001), binds 8090 (8000 is vLLM's port in this fleet), healthcheck on
  /api/config so a dead engine never marks the dashboard unhealthy. No frontend
  build stage — static/index.html is bundler-free by design and is copied as-is.
- .gitea/workflows/build-and-push.yml: publishes
  gitea.reeseapps.com/Experimental/llm-monitor on main + release tags.

114 tests, no network and no real engines required.

LLM-Generated: true
This commit is contained in:
ducoterra
2026-10-16 12:49:45 -04:00
commit 48bc92b9af
19 changed files with 6069 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+432
View File
@@ -0,0 +1,432 @@
# llm_monitor — gufo + vLLM real-time dashboard
One page, two engines, live. Watches a **gufo** (llama.cpp-family) server and a **vLLM**
server side by side: sessions/slots, queueing, KV/GPU cache pressure, throughput, and
latency histograms — refreshed every second without a page reload.
- **Read-only by construction.** The backend only ever calls `client.get()`. There is no
POST/PUT/PATCH/DELETE route anywhere (`curl -X POST /api/gufo/slots` → `405`), nothing is
written to either engine, and upstream bodies are parsed and re-emitted as JSON instead of
echoed.
- **Per-engine health.** Each engine is polled and tracked independently. One engine being
down never blocks or blanks the other.
- **Opt-in engines.** A server only appears if its `*_BASE_URL` env var is set. Send this to
someone who runs just vLLM and they see one clean vLLM panel — no gufo half stuck in red.
- **No build step, no CDN, no browser dependencies.** 2 app files (`app.py`,
`static/index.html`). Sparklines are hand-rolled inline SVG — the engines live on an
internet-disconnected podman network, so a CDN chart library would just be a dead request.
## Endpoints and ports (source of truth)
Taken from `~/Deployments/reeseapps/ai/deployments`:
| Engine | Deployment | Host | Reachable as | Port | Upstream endpoints used |
| -------------------- | ---------------------------- | ---------------- | -------------------------------- | ---- | ------------------------------------------------ |
| gufo | `deployments/gufo` | `deskwork-ai` | `deskwork.reeselink.com` | 8731 | `GET /slots` (alias `/v1/slots`), `GET /metrics` |
| vLLM (radiance) | `deployments/vllm-radiance` | `driveripper-ai` | `driveripper.reeselink.com` | 8000 | `GET /metrics` |
| **this dashboard** | — | wherever you run it | — | **8090** | serves the page + `/api/*` |
> Note the port: `8000` is already vLLM/juggernaut in that fleet and `8731` is gufo, so the
> dashboard listens on **8090** by default. The `8080`/`8000` defaults you may expect for the
> *upstreams* are wrong for this fleet — `8731` is gufo here.
## Install & run (uv)
This is a [uv](https://docs.astral.sh/uv/) project: `pyproject.toml` + `uv.lock`, no build
backend (`package = false` — it is an app, not a library). uv manages the venv; you never
activate it manually.
```bash
uv sync # create .venv from uv.lock (incl. dev group)
# Point at the engines you actually run (these are the fleet's real endpoints), then start:
export GUFO_BASE_URL=http://deskwork.reeselink.com:8731 # gufo — deskwork-ai:8731
export VLLM_BASE_URL=http://driveripper.reeselink.com:8000 # vLLM — driveripper-ai:8000
uv run uvicorn app:app --host 127.0.0.1 --port 8090
```
Set only what exists. With just `VLLM_BASE_URL` the page is a single-engine vLLM dashboard
and `/api/gufo/*` answers `404 {"configured": false}` without ever dialling a port; with
neither set it shows a "no engines configured" card instead of two permanently red panels.
Open <http://127.0.0.1:8090>.
```bash
uv run python app.py # same thing, reads DASHBOARD_HOST/DASHBOARD_PORT
uv run uvicorn app:app --port 8090 --reload # dev loop (watches app.py + static/)
uv sync --no-dev # production install: skips the pytest group
```
> Keep the `--port 8090`. Bare `uvicorn app:app` binds `:8000`, which is where vLLM already
> lives on `driveripper-ai` — the dashboard would refuse to start there (and if it started on
> a machine without vLLM, `VLLM_BASE_URL` would point at itself).
Can't reach those hosts directly (VPN, firewall, different network)? Tunnel them to loopback
and point the env vars at the tunnel:
```bash
ssh -N -L 8731:127.0.0.1:8731 -L 8000:127.0.0.1:8000 deskwork-ai
export GUFO_BASE_URL=http://127.0.0.1:8731 VLLM_BASE_URL=http://127.0.0.1:8000
```
After editing dependencies: `uv lock` (updates `uv.lock`), then regenerate the pip shim:
`uv export --no-dev --no-hashes > requirements.txt`.
### Without uv
`requirements.txt` is a pinned export of `uv.lock`, so a host that has no uv still works:
```bash
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
uvicorn app:app --host 127.0.0.1 --port 8090
```
### In a container
The `Containerfile` builds a slim, non-root image (uv-locked deps, no build tools, no
frontend build step — the page is copied verbatim). CI publishes it to
`gitea.reeseapps.com/Experimental/llm-monitor` on every push to `main` and on release tags
(`.gitea/workflows/build-and-push.yml`).
```bash
podman build -t localhost/llm-monitor:main -f Containerfile .
podman run -d --name llm-monitor -p 8090:8090 \
-e GUFO_BASE_URL=http://deskwork.reeselink.com:8731 \
-e VLLM_BASE_URL=http://driveripper.reeselink.com:8000 \
localhost/llm-monitor:main
```
Nothing is baked in: no host guesses and no API keys live in the image, so the same image
serves a two-engine deployment, a vLLM-only deployment, or a gufo-only one — set (or omit)
`GUFO_BASE_URL` / `VLLM_API_KEY` at run time. Inside the container the dashboard binds
`0.0.0.0:8090`; reach the engines by whatever name is resolvable from that network (or
publish the container onto the engines' podman network and use the quadlet hostnames,
`deskwork-ai` / `driveripper-ai`). The `HEALTHCHECK` probes `/api/config`, which never dials
an engine — a dead engine must never mark the dashboard itself unhealthy.
## Tests
```bash
uv run pytest # 114 tests, no network, no real engines
uv run pytest --cov # with coverage (app.py 98%)
uv run pytest -k parsing -vv # just the Prometheus parser / field maps
uv run -p 3.10 pytest # check the requires-python floor
```
Supported Python: **>=3.10** (CI-worthy runs verified on 3.10 / 3.13 / 3.14). There is no
`.python-version` pin on purpose — uv uses whatever interpreter is present, since this gets
dropped onto whatever box hosts the engines.
`tests/conftest.py` swaps `app.state.client` for a `FakeUpstream` that records the URLs it
was asked for and can raise `httpx.ConnectError` / `TimeoutException` on demand — so engine
outages are asserted, not hand-tested. Suite covers: parser edge cases (aliases, absent
series, label escaping, histogram interpolation), passthrough fidelity, 405 on every
non-GET verb, 502/504 translation, per-engine auth, and the no-third-party-assets rule.
## Configuration (env vars only)
| Var | Default | Meaning |
| ------------------------- | ------------------------ | -------------------------------------------------------------- |
| `GUFO_BASE_URL` | **_unset_ → gufo hidden** | gufo origin (scheme://host[:port]); unset/blank means this deployment has no gufo |
| `VLLM_BASE_URL` | **_unset_ → vLLM hidden** | vLLM origin; unset/blank means this deployment has no vLLM |
| `GUFO_API_KEY` | _unset_ | if set, sent upstream as `Authorization: Bearer <key>` |
| `VLLM_API_KEY` | _unset_ | if set, sent upstream as `Authorization: Bearer <key>` |
| `UPSTREAM_TIMEOUT_SECONDS`| `3` | per-request upstream timeout (short: the UI polls every 1s) |
| `RATE_WINDOW_SECONDS` | `10` | rolling window for vLLM Δcounter rates (see “Where prompt tok/s comes from”) |
| `POLL_MS` | `1000` | browser poll interval |
| `HISTORY_LEN` | `60` | sparkline samples kept in the browser |
| `DASHBOARD_HOST`/`DASHBOARD_PORT` | `127.0.0.1` / `8090` | only used by `python app.py` |
There is deliberately **no default host** for the two engines. A built-in `127.0.0.1:8731`
looked convenient until a vLLM-only user got a red gufo panel for a server they never
installed; "not configured" and "configured but down" are different statements, so only the
env var decides. `http://host:port/` is accepted (the trailing slash is trimmed) and a
whitespace-only value counts as unset. `/api/config` exposes `gufo_enabled` / `vllm_enabled`,
and the page builds its layout from those.
No auth on the dashboard itself by default — bind it to `127.0.0.1` (default) or put it
behind your reverse proxy. The keys are never returned by any endpoint; `/api/config`
reports only `gufo_auth: true/false`.
## Upstream start commands
These are the commands the dashboard expects to be able to scrape. Full worked examples are
in each deployment's `README.md` in the deployments repo.
**gufo** — `/slots` and `/metrics` are built in; `--sessions N` is what sizes the slot grid:
```bash
gufo serve llm --sessions 4 \
--model /models/qwen3.8-flash-next/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
--context 262144 --prefill-chunk 4096 --served-model-name turbo \
--host 0.0.0.0 --port 8731
# quadlet form: gufo serve --host 0.0.0.0 --port 8731 llm --sessions 2 ...
```
**vLLM** — metrics are on by default at `/metrics` (no flag needed; `--disable-metrics` would
break the vLLM panel):
```bash
vllm serve amd/Qwen3.8-27B-Quark-AWQ-MXFP4 \
--served-model-name=juggernaut \
--max-model-len=131072 --max-num-seqs=4 --max-num-batched-tokens=4096 \
--kv-cache-dtype=fp8 --enable-prefix-caching --gpu-memory-utilization=0.95 \
--host=0.0.0.0 --port=8000
```
## Backend routes
| Route | Upstream | Returns |
| -------------------- | --------------------------- | ----------------------------------------------------------------------- |
| `GET /` | — | the dashboard (`static/index.html`) |
| `GET /api/config` | — | base URLs, `*_enabled` (which engines exist), poll/history settings, `*_auth` booleans (never the keys) |
| `GET /api/gufo/slots`| `GET {GUFO_BASE_URL}/slots` | upstream JSON **passed through byte-for-byte**; falls back to `/v1/slots` |
| `GET /api/gufo/metrics` | `GET {GUFO_BASE_URL}/metrics` | Prometheus text → JSON, gufo field map |
| `GET /api/vllm/metrics` | `GET {VLLM_BASE_URL}/metrics` | Prometheus text → JSON, vLLM field map |
Envelope for the JSON routes:
```jsonc
{ "ok": true, "engine": "vllm", "upstream": "http://127.0.0.1:8000/metrics",
"scraped_at": 1760000000.123, "series": 21, "metrics": { "...": {} } }
```
If an engine's `*_BASE_URL` was unset, its two routes answer
`404 {"ok": false, "engine": "gufo", "configured": false, "detail": "gufo is not configured —
set GUFO_BASE_URL and restart"}` and **no upstream request is made** — a server that is not
installed is not an outage, so it must not look like one.
Every metric cell is `{"value": <number|null>, "source": "<series or computed:...>|null",
"derived": <bool>}`. A missing series is `value: null`, which the page renders as **`n/a`** —
it never 500s because a metric name changed. Upstream failures return `502` (unreachable /
HTTP error) or `504` (timeout) with `{"ok": false, "detail": "..."}`, and that `detail` is
what the panel shows.
## One Prometheus parser, two field maps
`app.py` contains a single Prometheus text parser (`class Prom`: tolerant line matcher,
label unescaping, `sum`/`max`/`avg` aggregation, by-label grouping, histogram
`_sum`/`_count`/`_bucket` → mean + interpolated p95). The engines differ only in their field
maps:
The tables below are **what the deployed engines actually export** (captured live; see
`tests/fixtures/*_real.txt`). Each field is an alias list, so the names from the original
spec and the legacy `llama:` prefix still work — but the deployed name comes first, and the
live fixtures are the regression test that keeps them honest.
**gufo** — note `tokens_predicted_*` (not `generated`), and the speed gauges are named
`*_tokens_seconds` while their unit is tokens/second:
| UI field | Series (deployed name first) | Agg |
| ------------------- | ------------------------------------------------------------------- | ---- |
| Running | `llamacpp:requests_processing` | sum |
| Waiting | `llamacpp:requests_deferred` | sum |
| KV cache | `llamacpp:kv_cache_usage_ratio` | max |
| prompt tok·s⁻¹ (all slots) | **`computed:Δllamacpp:prompt_tokens_total/Δt`** (`RateEstimator`), else `llamacpp:prompt_tokens_seconds` | Δ / avg |
| prompt tok·s⁻¹ (last pass) | `llamacpp:prompt_tokens_seconds` — **one request's last prefill pass**; exposed as `prompt_tokens_per_second_native` | avg |
| decode tok·s⁻¹ (all slots) | **`computed:Δllamacpp:tokens_predicted_total/Δt`** (`RateEstimator`), else `llamacpp:predicted_tokens_seconds` | Δ / avg |
| decode tok·s⁻¹ (last pass) | `llamacpp:predicted_tokens_seconds` — **one request's last decode pass**, holds that value while the box idles; exposed as `decode_tokens_per_second_native` | avg |
| prompt tokens | `llamacpp:prompt_tokens_total` — **excludes cache hits** | sum |
| cache-hit tokens | `llamacpp:prompt_tokens_cached_total` | sum |
| generated tokens | `llamacpp:tokens_predicted_total` | sum |
| total tokens | `computed:prompt+cached+generated` | — |
| prompt cache | `computed:cached/(prompt+cached)` | — |
| peak ctx | `llamacpp:n_tokens_max` | max |
| gpu losses | `gufo_device_lost_total` (non-zero = incident, shown red) | sum |
| spec accept | `computed:accepted/draft_tokens` from `llamacpp:spec_decode_num_*` | — |
**vLLM** — `kv_cache_usage_perc` (not `gpu_cache_usage`), `generation_tokens_total` (not
`generation_tokens_total`), TPOT as `inter_token_latency_seconds`, and every series tagged
`engine=` / `model_name=`:
| UI field | Series (deployed name first) | Notes |
| --------------------- | ---------------------------------------------------------------- | ------------------------------------- |
| Running | `vllm:num_requests_running` | sum across `engine=` label sets |
| Waiting | `vllm:num_requests_waiting` | sum |
| Waiting · capacity | `vllm:num_requests_waiting_by_reason{reason="capacity"}` | live reasons: `capacity`, `deferred` |
| GPU cache | `vllm:kv_cache_usage_perc`, `vllm:gpu_cache_usage` | max (never sums past 1.0) |
| TTFT | `vllm:time_to_first_token_seconds` | histogram → mean, p95 |
| TPOT | `vllm:inter_token_latency_seconds`, `vllm:time_per_output_token_seconds`, `vllm:request_time_per_output_token_seconds` | histogram → mean, p95 |
| Queue time | `vllm:request_queue_time_seconds` | histogram → mean, p95 |
| prompt / generation tokens | `vllm:prompt_tokens_total`, `vllm:generation_tokens_total` | sum |
| prefix cache hit | `computed:hits/queries` from `vllm:prefix_cache_{hits,queries}_total` | — |
| preemptions | `vllm:num_preemptions_total` (non-zero = KV pressure, shown red) | sum |
| prompt tok·s⁻¹ | `vllm:prompt_throughput`, `vllm:avg_prompt_throughput_toks_per_s` — **neither exists on this build** — else **`computed:Δprompt_tokens_total/Δt`**, else **`computed:mean(request_prompt_tokens)/mean(request_prefill_time_seconds)`** | see below |
| prefill avg | `computed:mean(request_prompt_tokens)/mean(request_prefill_time_seconds)` | lifetime average, ~4.9k tok/s live |
| decode tok·s⁻¹ | `vllm:generation_throughput` if present, else **`computed:Δgeneration_tokens_total/Δt`**, else **`computed:running/tpot`** | `derived: true`; the UI tags the source |
| per-seq tok·s⁻¹ | `computed:1/tpot` | from TPOT mean |
### Where vLLM's prompt tok/s comes from
That vLLM exports **no rate gauge at all** — no `*_throughput`, no `*_toks_per_s`, nothing
with a rate-ish name (grep its `/metrics` and you get zero hits). Counters only. So the
number is computed, in this order, and the UI labels which one you are looking at:
1. **`vllm:prompt_throughput`** if a future build provides it — the only real scraped gauge.
2. **Measured Δ** — `Δprompt_tokens_total / Δt` over a rolling `RATE_WINDOW_SECONDS` window
(default 10s). This is *actual* current throughput, tagged `Δ10.0s`. A bare 1-second delta
jitters between 0 and spikes because prompts arrive in bursts, hence the window. Counter
resets (engine restart) yield `n/a` for one scrape rather than a huge negative rate.
3. **Lifetime average prefill speed** — `mean(request_prompt_tokens) / mean(request_prefill_time_seconds)`,
i.e. tokens-per-second this engine has achieved on prefill since it started (~4.9k tok/s
live). Used only until there are two scrapes to diff, and shown on its own as **prefill avg**.
Decode tok/s follows the same precedence, and the measured Δ replaced the old
`running/tpot` estimate as step 2 — the Δ counts real output, the estimate assumed every
running request was decoding at that instant. The per-request figure is still `1/TPOT`.
An `n/a` now means the counters themselves are gone, not that a gauge name changed.
### Where gufo's decode tok/s comes from
gufo's **decode tok/s is the aggregate over every slot generating at once**, and it is
measured, not scraped. `llamacpp:predicted_tokens_seconds` is a **single unlabeled gauge**
holding the last decode pass of **one** request: with two slots decoding at 35 tok/s it still
reads 35, and when the box goes idle it keeps serving that stale number indefinitely (live:
stuck at 38.0 with `requests_processing == 0`). There are no label sets to `sum`, so no
aggregation rule could ever fix it.
`llamacpp:tokens_predicted_total` advances once per token from every slot, so the same
`RateEstimator` used for vLLM answers the question directly — two slots at 35 tok/s advance
the counter 70 tokens per second and the cell reads ~70:
1. **Measured Δ** — `Δtokens_predicted_total / Δt` over the `RATE_WINDOW_SECONDS` window,
tagged `Δ10.0s`. Goes to a genuine `0.0` when nothing is generating.
2. **The scraped gauge** — only until there are two scrapes to diff, or on a build missing
the counter; rendered with no tag (it is a real scrape) and still exposed separately as
`decode_tokens_per_second_native`.
This is the one place where measured **outranks** a native gauge, deliberately inverting the
usual precedence: vLLM's native throughput gauge really is an aggregate, gufo's is not. Both
engines share the single `RATES` estimator and cannot cross-read, because it keys off the
*resolved* series name — which also means a counter rename starts a fresh window instead of
diffing two different series and inventing throughput.
gufo's `/slots` is passed through untouched and returns a **bare JSON array** (older builds
wrap it as `{"slots": [...]}`; the page accepts both). Slot fields read: `id`,
`is_processing` (authoritative; `state` 0/1 is the fallback), `id_task`/`task_id`,
`n_prompt_tokens`, `n_prompt_tokens_cache`, `n_prompt_tokens_processed`, `n_ctx`, `model`,
`speculative`, and `next_token[0].{n_decoded,n_remain}` (`-1` means "nothing pending" → `n/a`).
## Dashboard layout
- **Fleet total** (very top): one number per metric summed across both engines — running,
waiting, prompt tok·s⁻¹, decode tok·s⁻¹, total tokens. Hidden entirely when fewer than two
engines are configured (a cross-engine sum of one engine just duplicates its own card).
Rules:
- **Only engines answering now are counted.** A dead engine keeps its last good sample in
its own panel (labelled stale, correct there), but adding a frozen number to a fleet total
would present hours-old data as current — so it is dropped, the header drops to `1 of 2
engines`, and the accent/dot go amber. Zero answering → every cell `n/a`, dot red.
Liveness is keyed on each engine's **`/metrics` job only**: gufo's `/slots` failing must
not remove it from a total that came from `/metrics`.
- **Summed in the browser**, not on the backend. The page already holds both parsed payloads,
so the API stays one-scrape-pure with no second polling path hammering the engines.
- Hover any cell for the itemised breakdown (`gufo 45 + vLLM 37 · excluded: …`), with each
part marked `(measured)` when it is a derived rate.
- Sums are computed on the raw values then rounded once, so a cell can differ from the two
rounded panel numbers above it by 1 (`308 + 177` shown, `486` total).
- **The prompt sum is the one imperfect addition**: gufo's prompt counter *excludes* cache
hits (so it measures newly-computed prefill work) while vLLM's *includes* cached prefix
tokens. Both are per-engine aggregates and additive within an engine, but the fleet number
leans slightly low for gufo. The `prompt tok/s` label carries the caveat on hover.
- **Header line** names only the engines that exist (`vLLM http://host:8000 · poll 1000ms ·
read-only`), and the tab title follows (`LLM Monitor · vLLM`).
- **Shared stats strip** (top): per engine — cache-usage bar (blue → amber ≥75% → red ≥90%),
prompt tok·s⁻¹, decode tok·s⁻¹, total tokens, plus slots-busy (gufo) / per-seq tok·s⁻¹
(vLLM). Strip labels stay terse (they share one dense grid), so gufo's cell just reads
**decode tok/s** with its all-slots meaning on hover; the gufo panel's sparkline caption
spells it out as **decode tok/s · all slots**. Every derived rate carries its provenance tag
(`Δ10.0s`, `from TPOT`, `lifetime avg`) with the exact `computed:` source on hover.
- **gufo panel**: vLLM-style banner `Running: N reqs, Waiting: M`, sparklines for the
all-slots decode tok·s⁻¹ and KV usage, then a grid with **one card per `/slots` entry**: slot id + served
model (`+spec` when speculative decoding is on), status pill (green `processing` / grey
`idle`), request id, prompt tokens, cache hits (with % of prompt), generated so far
(`next_token[0].n_decoded`), remaining budget, and slot context size. Queued requests are
not in `/slots` — that is what `Waiting` (deferred) covers.
- **Alarm colours**: `gpu losses` (gufo `gufo_device_lost_total`) and `preemptions`
(`vllm:num_preemptions_total`) turn red above zero — both are cumulative counters, so a
moving number there means something is wrong, not busy.
- **vLLM panel**: banner `Running: N reqs, Waiting: M`, sparklines for the all-slots decode
tok·s⁻¹ and GPU cache usage — deliberately the **same pair of charts as the gufo panel**, both
fed by `decode_tokens_per_second` (`Δvllm:generation_tokens_total/Δt` here) so the two engines
read alike. The running count is no longer charted; it lives in the banner and the fleet
total. Then metric cards for waiting-by-reason, GPU cache bar, TTFT / TPOT / queue
time (mean + p95), prompt + generation token totals, and series scraped.
- **Sparkline y-axis**: each chart labels the top, middle and bottom of the scale it actually
plotted against (`49 / 43 / 37`, or `100% / 50% / 0%` for the cache charts, which are pinned
to 0–1 rather than autoscaled), with faint rules at those levels. The plots autoscale, so
without the labels a 1% drift and a 2× spike are the same squiggle. Labels are rendered as
HTML beside the SVG, never inside it — `preserveAspectRatio="none"` stretches the viewBox to
the card width and would smear the text. Precision adapts so the three never collide (a 1–2
request axis reads `2.0 / 1.5 / 1.0`, not `2 / 2 / 1`), and a **flat** window labels all three
levels with the value itself — an idle box says `0 / 0 / 0`, not the `±0.001` the plot's
zero-range padding happens to produce. Fewer than two points → axis `n/a`.
- **Every second**: all four `/api/*` endpoints are polled independently
(`setInterval` + fire-and-forget, with an in-flight guard so a slow upstream can't pile up
requests). DOM is updated in place — slot cards are keyed by slot id and reused, so the
poll is invisible. Per-engine green/red dot, "updated HH:MM:SS" stamp, and a
`stale Ns` counter that only appears when that engine stopped answering. `Space` or the
**pause** button freezes polling for reading a snapshot.
## Failure behaviour
| Situation | Result |
| ---------------------------------------- | ----------------------------------------------------------------- |
| Engine's `*_BASE_URL` unset | that engine is **not on the page at all** (no strip card, no panel, no polling, no fleet share) and its routes answer `404 configured:false`; neither configured → a setup card naming the env vars |
| Engine stopped / unreachable | that panel's dot goes red, red banner with the upstream error, last good values kept and greyed, `stale Ns` grows; **the other engine keeps polling normally** |
| Engine comes back | dot goes green and the banner clears on the next successful poll — no reload needed |
| A metric series is absent | that one cell shows `n/a` (and `series absent → n/a` in its foot), everything else renders |
| Upstream returns non-Prometheus garbage | parsed as 0 series → all cells `n/a`, page still healthy |
| Sparkline has < 2 points | chart stays faded and its y-axis reads `n/a` rather than inventing a scale |
| Upstream slower than `UPSTREAM_TIMEOUT_SECONDS` | `504`, dot red, and the request is abandoned rather than queued |
## Smoke test without live engines
Start the app first (`uv run uvicorn app:app --host 127.0.0.1 --port 8090`), then:
```bash
curl -s localhost:8090/api/config | python3 -m json.tool # what it is pointed at
curl -s localhost:8090/api/gufo/slots | head -c 200 # proxy passthrough
curl -s localhost:8090/api/gufo/metrics | python3 -m json.tool | head -30
curl -s localhost:8090/api/vllm/metrics | python3 -m json.tool | head -30
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8090/api/gufo/slots # -> 405 (read-only)
```
One-engine deployment (what a vLLM-only recipient sees) — gufo answers `404 configured:false`
and the page has no gufo section at all:
```bash
env -u GUFO_BASE_URL VLLM_BASE_URL=http://127.0.0.1:8000 \
uv run uvicorn app:app --host 127.0.0.1 --port 8090
curl -s localhost:8090/api/config | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["gufo_enabled"], d["vllm_enabled"])' # False True
curl -s -o /dev/null -w '%{http_code}\n' localhost:8090/api/gufo/metrics # -> 404, nothing dialled
```
`n/a` + red-dot path described above.
To see what an engine really exposes (do this before trusting any metric name, including the
ones written in this README):
```bash
curl -s "$GUFO_BASE_URL/metrics" | grep -v '^#' | sed -E 's/[{ ].*//' | sort -u
curl -s "$VLLM_BASE_URL/metrics" | grep -v '^#' | sed -E 's/[{ ].*//' | sort -u
curl -s "$GUFO_BASE_URL/slots" | python3 -m json.tool | head # a bare array
```
## Files
```
pyproject.toml uv project: deps, dev group, package=false, pytest/coverage config
uv.lock pinned resolution (commit it)
app.py FastAPI app: config, Prometheus parser, 2 field maps, 5 GET routes
static/index.html the dashboard: CSS + vanilla JS + inline-SVG sparklines
tests/ conftest.py (FakeUpstream), test_parsing.py, test_proxy.py,
test_real_contracts.py + fixtures/ (verbatim captures from the live engines)
requirements.txt pip shim, generated from uv.lock
AGENTS.md working rules for humans and agents on this repo
README.md
```
There are no JS dependencies and nothing to build: `static/index.html` is served verbatim.