phase: 07_llm_monitor_style_pages
**Phase 07 final verification pass — all tasks already in `complete/`; every criterion re-verified live.**
- **Old slot table gone**: no `#slots-card`/`#slots-table`/`renderSlots` in HTML/JS/CSS (only the new grid's `.slots-empty` *class*); pinned by `test_the_broken_slot_table_is_gone_from_html_js_and_css`.
- **`tests/fixtures/slots_gufo.json`**: present; live `GET ${GUFO_BASE_URL}/slots` this pass matches it key-for-key (row + `next_token[0]`), so the capture is reproducible; drift proven to fail (mutation tests + live `test_gufo_slots_answer_the_pinned_shape`, 9 live tests pass).
- **Aggregate block**: live page shows `2 of 2 engines`, running/waiting/prompt/decode/total-tokens straight from `fleet_totals()`, dot `ok`, no verdict colour; on a throwaway single-engine instance (:8091, private temp DB) `#fleet` is `hidden`, one full-width strip card + one panel, zero banners → "one clean half".
- **Strip + slot grid on live data**: 2 `.card.stat` (KV bars `bar ok` 13 %/0 %), gufo grid rendered real cards incl. a busy slot (`#2189`, 67849 prompt tok, 99 % cache share, `processing`), vLLM note-only with no grid; no `NaN`/`undefined` anywhere. Screenshots: `data/logs/llm-phase07-final-live-{two-engines,single-engine}.png`, `ups-charts-aria-labels.png`.
- **Defect found in earlier work and fixed**: `/ups`' four `LineChart`s shipped no `ariaLabel`, violating AGENTS.md's "every chart names itself for assistive tech" (`/system`+`/llm` were pinned, `/ups` wasn't) → added labels to `page-ups.js` and `test_every_ups_chart_names_itself_for_assistive_tech`; browser-verified `role="img"` + label on all four.
- **Gate**: `bash .agents/validate.sh` → **exit 0**; `uv run pytest --cov --cov-fail-under=94` → **604 passed, 9 skipped, coverage 97.91 %**; `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → **0 errors**; `node --check` every page script ok; `uv run pytest -m live --live` → **9 passed**; `uv pip list --outdated` empty.
- **Docs**: README + AGENTS.md reviewed end-to-end; fixed test count (582→604), `/llm` source column + both architecture diagrams now name gufo `/slots`, AGENTS.md agent line gained `GET /schema` + status page, ariaLabel convention now covers all three chart pages, README Testing block now lists `bash .agents/validate.sh`. Env-var table, agent/install flags, Containerfile facts, `/api/*` shapes, range chips, remote URL all verified accurate — no other changes needed.
- **Live-infra**: no fleet/host/config change; the temp :8091 instance, its temp DB and the fake processes are removed; dev server restarted and `/api/health` `ok` (all 6 sources ok). One thing I deliberately did **not** touch: the gitignored local `.env.dev` still carries a comment claiming the NUT `[dashboards]` account has only battery-test `instcmds`, whereas AGENTS.md/README record `instcmds = all` (verified on hardware 2026-10-17) — operator-owned file, flag for the operator to reword.
- **Next pending phase**: `08_events_paging_and_copy`.
This commit is contained in:
@@ -6,7 +6,7 @@ Three dashboards, one domain, one process, one SQLite file:
|
||||
| ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- |
|
||||
| `/` | Is anything wrong, and which dashboard do I need? | fleet + alerts in one request (`/api/summary`) |
|
||||
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) + vLLM `/metrics`, read-only |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics` — read-only |
|
||||
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
|
||||
|
||||
No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
|
||||
@@ -120,6 +120,7 @@ unreachable host writes its failure message into the event table and `/api/event
|
||||
```
|
||||
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
|
||||
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD
|
||||
│ │ + /slots (gufo) │
|
||||
▼ ▼ ▼
|
||||
src/dashboards/collectors.py ────────────────────────────────────────────┐
|
||||
│ parse + Δcounter/Δt (rates.py) │
|
||||
@@ -193,16 +194,45 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
|
||||
processes — folds into a collapsed
|
||||
"More detail" disclosure. Each panel pulls its series in **one** `/api/system/history` request
|
||||
(`hosts=<that host>&metrics=…&range=…`) every refresh, stack open or not, and a host that stops
|
||||
reporting takes its panel, its Charts toggle and its charts off the page. The LLM page (`/llm`) is per engine too: two big
|
||||
reporting takes its panel, its Charts toggle and its charts off the page. `/llm` opens with
|
||||
the **fleet**: one block above the panels — `ALL ENGINES`, an `N of M engines` tag, the note
|
||||
*sums only engines answering now*, then **running / waiting / prompt tok/s / decode tok/s /
|
||||
total tokens** — and every number on it is `fleet_totals()`, so the page sums nothing itself.
|
||||
A dead engine keeps its last-good numbers in its own (stale-labelled) panel but is dropped
|
||||
from a total presented as current: the tag counts what answered, names the missing engine in
|
||||
its tooltip and dims while the sums are partial. Nothing on the block is coloured —
|
||||
`severity.py` rates no throughput and no queue depth, because 0 tok/s on an idle fleet is
|
||||
healthy — the dot is reachability, and `fleet.seams` (how the two engines count prompt
|
||||
tokens differently) is a tooltip on the prompt sum rather than another line of ink. Below
|
||||
two configured engines the block is hidden: a cross-engine sum of one engine duplicates that
|
||||
engine's panel. Below the fleet block sits the **stats strip**: one card per configured
|
||||
engine — reachability dot, name, age stamp, the **KV-cache bar** (the one rated cell: the
|
||||
fill and a breach's text take their colour from the payload's `severity` and nothing else),
|
||||
then `prompt tok/s` and `decode tok/s` with their provenance tag (`Δ10.0s`, scraped,
|
||||
estimated), `total tokens`, the sessions this engine reports (`slots busy` counted from
|
||||
gufo's `/slots`, or `running`/`waiting` for vLLM, plus `waiting`), `prefix cache hit`,
|
||||
`spec accept` (gufo), `queue time` with its `p95 …` foot (vLLM) — and red alarm chips for
|
||||
preemptions / GPU-context losses that exist only while non-zero. Then the engine panels
|
||||
hold the detail the strip does not print: two big
|
||||
throughput numbers (decode, prompt — uncoloured, because 0 tok/s on an idle server is
|
||||
healthy) with their provenance tag (`Δ10.0s`, scraped, estimated), then a compact list of
|
||||
metric rows — sessions, queue, KV pressure (the only rated cell), prefix/spec percentages,
|
||||
and TTFT/TPOT/queue-time means with a `p95 …` foot line — each carrying its own sparkline
|
||||
over the selected window. Preemptions and GPU-context losses render only as red alarm chips
|
||||
while non-zero; the token counter is a footnote. The four trend charts sit below the panels
|
||||
healthy) with their provenance tag, and the latency rows (TTFT, TPOT) each with a `p95 …`
|
||||
foot line and its own sparkline over the selected window. Under the gufo panel's
|
||||
**`sessions (GET /slots)`** label sits the **slot grid**: one card per parallel slot, keyed by
|
||||
the slot's own id, created once and patched in place so a busy slot never flickers — request
|
||||
id, prompt tokens (with `processed …` while prefill runs), cache hit and its share of the
|
||||
prompt, generated tokens, remaining, context, and the model beside the slot number. A busy
|
||||
card is a border, a glyph and the word `processing`, never a green/amber/red verdict (a slot
|
||||
generating tokens is the job). Every cell reads exactly one key of the captured `/slots`
|
||||
shape — no `a ?? b` fallback chains — so a field rename fails a contract test instead of
|
||||
printing dashes, and `-1` sentinels print `idle`/`n/a`, never a negative countdown. When
|
||||
`/slots` does not answer, the panel says so with the API's own sentence rather than rendering
|
||||
a card of nothing, and vLLM's panel carries one line (*exposes no per-slot JSON*) and no grid.
|
||||
No row appears in both places —
|
||||
one value, one home — and there is no peak-context cell, because this page prints a number
|
||||
the upstream published or `n/a`. The four trend charts sit below the panels
|
||||
at half height, and one `/api/llm/history` request per refresh feeds all of it. With a
|
||||
single engine configured the totals card hides and the page is one clean half — never a red
|
||||
panel for a server you do not run. Each UPS card (`/ups`) opens with its battery glyph and
|
||||
single engine configured the page is one clean card in the strip and one clean panel —
|
||||
never a red half for a server you do not run. Each UPS card (`/ups`) opens with its battery glyph and
|
||||
three big numbers — **charge %**, **load %**, **runtime left** — coloured from the card's
|
||||
`severity` dict, with the electrical readings (line in, output, battery volts and amps,
|
||||
battery °C) kept below them in smaller muted type because they are what you read *while* the
|
||||
@@ -217,8 +247,10 @@ uv run pytest --cov # coverage gate
|
||||
set -a; . ./.env.dev; set +a # the live suite reads the same env as the app
|
||||
uv run pytest -m live --live # optional: real fleet / engines / upsd
|
||||
uv run ruff check src tests agent && uv run pyright # lint + types
|
||||
./scripts/capture-fixtures.sh # re-record ground truth: host agents + UPS replies
|
||||
./scripts/capture-fixtures.sh # re-record ground truth: host agents, UPS replies, gufo /slots
|
||||
./scripts/capture-fixtures.sh --with-llm # ... and the engines' /metrics while they are up
|
||||
bash .agents/validate.sh # all of the above + `node --check` on every page
|
||||
# script + repo hygiene, with the 94% coverage floor
|
||||
```
|
||||
|
||||
Test philosophy: **real captures as fixtures, mocks for behaviour.** Anything that must
|
||||
@@ -242,7 +274,17 @@ verbatim as `tests/fixtures/nut_<ups>.vars` by `./scripts/capture-fixtures.sh`,
|
||||
device serials scrubbed — and again against live upsd. The captures earn their place: the
|
||||
CyberPower answers `battery.mfr.date: CPS`, the APC's `battery.date` is a factory
|
||||
placeholder, and **neither** publishes `battery.temperature` or `battery.charger.status`, so
|
||||
a hand-written variable table would have tested nobody's memory.
|
||||
a hand-written variable table would have tested nobody's memory. `tests/slots_contract.py`
|
||||
pins gufo's `/slots` rows the same way — hermetically from
|
||||
`tests/fixtures/slots_gufo.json` (the verbatim `/slots` reply, captured whenever
|
||||
`GUFO_BASE_URL` is set; the idle-slot rows are the shape the grid must handle) and again
|
||||
against the running engine. It exists because that table was once guessed: the old slot
|
||||
table read field names the server never published and rendered dashes forever. It also picks
|
||||
the **canonical name of each fact** — the live server answers `id_task` *and* `task_id` for
|
||||
one request — and the page renders only the canonical key, so a rename fails a test instead of
|
||||
quietly becoming a fallback chain. A busy
|
||||
slot's `prompt` is the user's in-flight request text, so the capture script blanks it and
|
||||
the page never reads it.
|
||||
|
||||
## Battery-test buttons
|
||||
|
||||
|
||||
Reference in New Issue
Block a user