services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 07_llm_monitor_style_pages

Browse Source 
**Phase 07 final verification pass — all tasks already in `complete/`; every criterion re-verified live.**

- **Old slot table gone**: no `#slots-card`/`#slots-table`/`renderSlots` in HTML/JS/CSS (only the new grid's `.slots-empty` *class*); pinned by `test_the_broken_slot_table_is_gone_from_html_js_and_css`.
- **`tests/fixtures/slots_gufo.json`**: present; live `GET ${GUFO_BASE_URL}/slots` this pass matches it key-for-key (row + `next_token[0]`), so the capture is reproducible; drift proven to fail (mutation tests + live `test_gufo_slots_answer_the_pinned_shape`, 9 live tests pass).
- **Aggregate block**: live page shows `2 of 2 engines`, running/waiting/prompt/decode/total-tokens straight from `fleet_totals()`, dot `ok`, no verdict colour; on a throwaway single-engine instance (:8091, private temp DB) `#fleet` is `hidden`, one full-width strip card + one panel, zero banners → "one clean half".
- **Strip + slot grid on live data**: 2 `.card.stat` (KV bars `bar ok` 13 %/0 %), gufo grid rendered real cards incl. a busy slot (`#2189`, 67849 prompt tok, 99 % cache share, `processing`), vLLM note-only with no grid; no `NaN`/`undefined` anywhere. Screenshots: `data/logs/llm-phase07-final-live-{two-engines,single-engine}.png`, `ups-charts-aria-labels.png`.
- **Defect found in earlier work and fixed**: `/ups`' four `LineChart`s shipped no `ariaLabel`, violating AGENTS.md's "every chart names itself for assistive tech" (`/system`+`/llm` were pinned, `/ups` wasn't) → added labels to `page-ups.js` and `test_every_ups_chart_names_itself_for_assistive_tech`; browser-verified `role="img"` + label on all four.
- **Gate**: `bash .agents/validate.sh` → **exit 0**; `uv run pytest --cov --cov-fail-under=94` → **604 passed, 9 skipped, coverage 97.91 %**; `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → **0 errors**; `node --check` every page script ok; `uv run pytest -m live --live` → **9 passed**; `uv pip list --outdated` empty.
- **Docs**: README + AGENTS.md reviewed end-to-end; fixed test count (582→604), `/llm` source column + both architecture diagrams now name gufo `/slots`, AGENTS.md agent line gained `GET /schema` + status page, ariaLabel convention now covers all three chart pages, README Testing block now lists `bash .agents/validate.sh`. Env-var table, agent/install flags, Containerfile facts, `/api/*` shapes, range chips, remote URL all verified accurate — no other changes needed.
- **Live-infra**: no fleet/host/config change; the temp :8091 instance, its temp DB and the fake processes are removed; dev server restarted and `/api/health` `ok` (all 6 sources ok). One thing I deliberately did **not** touch: the gitignored local `.env.dev` still carries a comment claiming the NUT `[dashboards]` account has only battery-test `instcmds`, whereas AGENTS.md/README record `instcmds = all` (verified on hardware 2026-10-17) — operator-owned file, flag for the operator to reword.
- **Next pending phase**: `08_events_paging_and_copy`.
This commit is contained in:
ducoterra
2026-10-17 17:50:19 -04:00
parent de325caa49
commit 3d5ccd7a39
43 changed files with 2544 additions and 210 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+108 -29
View File
@@ -26,7 +26,7 @@ file — plus an overview page that says which of the three you need:
| --------- | ------------------------------------------------------------------------------ | ------------------------------------------------- |
| `/` | Is anything wrong, and where do I look first? | all three, in one request |
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent/host |
| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo (llama.cpp family) + vLLM `/metrics` |
| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo `/metrics` + `/slots`, vLLM `/metrics` |
| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) |
History is a first-class feature, not a sparkline afterthought: every number worth graphing
@@ -96,6 +96,7 @@ page still renders.
```
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD
│ │ + /slots (gufo) │
▼ ▼ ▼
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command ──┐
│ parse + Δcounter/Δt │
@@ -109,8 +110,9 @@ store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ─
Responsibilities, one line each:
- `agent/host_agent.py` — stdlib-only (3.10+), reads `/proc`, `/sys/block`, `/sys/class/drm`,
`statvfs`; serves `GET /snapshot` + `GET /info` + `GET /health`. Refuses to bind
off-loopback without a token (exit 2). Counters stay counters: the dashboard computes
`statvfs`; serves `GET /snapshot` + `GET /info` + `GET /schema` + `GET /health`, plus an
HTML status page at `GET /`. Refuses to bind off-loopback without a token (exit 2).
Counters stay counters: the dashboard computes
rates so the agent never remembers who polled it last.
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
env var always wins. Defaults are safe except the sources, which default to unset.
@@ -254,11 +256,11 @@ cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets
./scripts/dev-server.sh status # per-source ok/DOWN
./scripts/dev-server.sh logs
uv run pytest # hermetic: 550 passed + 8 live-skipped, no network
uv run pytest # hermetic: 604 passed + 9 live-skipped, no network
uv run pytest --cov # coverage gate (97.9%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh # re-record ground truth (agents + UPS captures)
./scripts/capture-fixtures.sh # re-record ground truth (agents + UPS + gufo /slots)
bash .agents/validate.sh # the phase gate: the four checks above (pytest
# --cov --cov-fail-under=94, ruff, pyright, plus a
@@ -300,7 +302,21 @@ branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only
`test_ups_hero_fields_are_present_on_the_real_batteries` drives it against live upsd. The
captures are what make the driver disagreements testable: one of them answers
`battery.mfr.date: CPS`, neither exports `battery.temperature` or
`battery.charger.status`.
`battery.charger.status`. `tests/slots_contract.py` is the same device for gufo's `/slots`
rows, which the `/llm` slot grid (phase 07) reads (`assert_slot_shape`, required/optional
field tables, duplicate slot ids, the `next_token[0]` entries): hermetically
from `tests/fixtures/slots_gufo.json` — the verbatim `/slots` reply, idle slots and all,
captured whenever `GUFO_BASE_URL` is set — and live via
`test_gufo_slots_answer_the_pinned_shape`. That fixture exists because its absence is how
the deleted slot table rendered dashes forever: its field names (`slot.state`,
`slot.n_prompt`, `slot.decode_per_second`) were a memory of an upstream, not a capture.
**The contract names one canonical key per fact and the grid renders that one**: the capture
answers `id_task` *and* `task_id` for the same request, so `CANONICAL_TASK_ID_FIELD` decides
and the alias is captured but rendered by nothing — a rename fails a test instead of becoming
a silent `a ?? b` chain in the page. `tests/test_api.py` reads the served `SLOT_FIELDS` table
and resolves every key it names against the capture, in both directions.
A busy slot's `prompt` is the user's in-flight request text, so the capture script blanks
it and the page never reads it.
- `live` (`tests/test_live_fleet.py`) re-runs the production code paths against the real
fleet, engines, and upsd. Skipped unless `--live`; each test skips when its source is
unset, so partial fleet availability is normal. **Live tests must not write** — a battery
@@ -320,11 +336,12 @@ Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ<window>` l
measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status
dots in the header. Every chart names itself for assistive tech: pages pass `ariaLabel` to
`LineChart`, which puts `role="img"` + `aria-label` on the `<svg>` (an unlabelled plot is
silence, and the legend is a sibling that labels nothing) — `/system` does this for all three
slots, pinned by `tests/test_api.py`. Window chips (5m…30d on `/system` and `/llm`, 1h…90d on
`/ups`; the API accepts 5m…90d). A panel that cannot get data says
why in the panel — the operator should never have to open devtools to learn that a source is
unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
silence, and the legend is a sibling that labels nothing). Every `LineChart` on every page has
one — `/system`'s three per-host slots, `/llm`'s four trend plots, `/ups`'s four battery
plots — each pinned by `tests/test_api.py`, so a chart that ships anonymous fails a test.
Window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups`; the API accepts 5m…90d). A
panel that cannot get data says why in the panel — the operator should never have to open
devtools to learn that a source is unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
The overview is exactly two panels — **Warnings & errors** (`ul.alerts`, one row per
`/api/summary` alert: verdict class, source linked to its dashboard via the alert's own
@@ -362,8 +379,10 @@ point, `labels: {current, max}` renders the `now / max` foot through the caller'
there is no animation at all (`prefers-reduced-motion` holds by construction). `minSpan` is the
other half of the same honesty for count metrics: history arrives as rollups, so eight seconds
of one queued request averages to 0.14 and an axis fitted to that draws a full-height spike for
a queue nobody was in — `/llm` passes `minSpan: 1` on sessions and queued so its axis is never
finer than one request.
a queue nobody was in. `/llm`'s count rows moved to the phase-07 stats strip (numbers, not
sparks), so no page passes `minSpan` today and `drawSparks()` still plumbs it through — a count
sparkline that comes back must set it, and the component's own contract test keeps the option
honest (`test_sparkline_ships_the_fixed_scale_and_verdict_options`).
A battery is `DashCharts.battery(container, {charge, severity, state, label})` — one inline SVG
whose **fill *is* the charge**: 1 viewBox unit per percent, clamped to 0–100 and never rounded up
@@ -441,24 +460,84 @@ always say whether it was scraped, measured `Δcounter/Δt`, or estimated (`llm_
`setThroughput` idiom; dropping the tag to look cleaner is a regression, not a simplification).
**The `/llm` page reads per engine** (phase 04, decided in
`.agents/phases/complete/04_llm_chart_readability/01_chart_set_decision.md`): a row of two
`Dash.hero()` throughput numbers (decode, prompt — deliberately **uncoloured**, because 0 tok/s
on an idle server is healthy and `severity.py` has no throughput row), then a compact `.cells`
list — label left, tabular value right, a `sparkline()` fixed at 160px so rows line up, and a
mono `p95 …` foot under every latency mean (no hover; p95 needs no stored series). Rows are
engine-scoped by the `only` flag: gufo never grows a TTFT mystery, vLLM never a spec-accept
hole. `preemptions` / `device_lost` are **alarm chips that exist only while non-zero** (`.chip.fail`,
`.agents/phases/complete/04_llm_chart_readability/01_chart_set_decision.md`, laid out by phase
07): a row of two `Dash.hero()` throughput numbers (decode, prompt — deliberately **uncoloured**,
because 0 tok/s on an idle server is healthy and `severity.py` has no throughput row) and the
latency rows, each a mean with a mono `p95 …` foot (no hover; p95 needs no stored series) and a
`sparkline()` fixed at 160px so rows line up. Rows are engine-scoped by the `only` flag on both
components: gufo never grows a TTFT mystery, vLLM never a spec-accept hole.
**The stats strip (`#strip`, phase 07 task 03) is the summary; the panels are the detail, and no
row is printed in both** — the UPS hero row's "one value drawn twice" rule, widened to two
components. One `.card.stat` per *configured* engine between the fleet block and the panels: the
reachability dot, the engine name and the age stamp in the head (same distinction every card
draws — the dot speaks reachability, the numbers never do), then the **KV-cache bar** — the one
cell on the page with a verdict, so the fill and a breach's text take their colour from the
payload's `severity` through `.bar.ok/.warn/.fail` and the ratio→percent conversion the panel's
cell used to make, with `ok` staying quiet ink and a missing gauge an empty bar plus `n/a` — then
`prompt tok/s` / `decode tok/s` with their `Δ<window>` tags, `total tokens` (lifetime, so its
source is the tooltip), the sessions this engine reports, `prefix cache hit`, `spec accept`
(gufo) and `queue time` with its `p95 …` foot (vLLM). `STRIP_ROWS` is the whole table and its
`key` **is** the `cellValue()` key, so every number on it is a cell of `/api/llm/snapshot`: no
field was invented in `sources/llm.py` for this card, and there is deliberately **no peak-ctx
row** (phase assumption). The session row is engine-shaped — vLLM publishes `running`/`waiting`
counts, gufo's is `slots busy`, counted from the `sessions.list` rows `/slots` answered
(`is_processing`, the shape pinned by `tests/slots_contract.py`); when `/slots` did not answer
there are no rows to count, so it falls back to the published `sessions.active` carrying the
API's own `sessions.detail` sentence rather than reporting 0 busy slots from no data (rule 2).
`preemptions` / `device_lost` are **alarm chips that exist only while non-zero** (`.chip.fail`,
container emptied and `hidden` at zero — the rule is the presence of an incident, never a graded
threshold); lifetime token counters are a quiet footnote line. **One `/api/llm/history` request
per refresh** carries every chart *and* spark series (`HISTORY_SERIES`; the endpoint answers
unknown names as empty series on purpose), and the two series that addition required
(`llm.tpot_seconds` vLLM-only, `llm.spec_accept_rate` gufo-only) are stored additively in
`sources/llm.py` HISTORY_METRICS. The engine-totals card kept only engines-live/active/queued
and **hides below two engines** — a cross-engine sum of one engine duplicates that engine's own
panel, and the summed-KV bar went because `fleet_totals` carries no `severity` to render. The
threshold) and they live on this card, not in a permanent cell; the lifetime token counter left
the panel's footnote line for the strip's row. **One `/api/llm/history` request per refresh**
carries every chart *and* spark series (`HISTORY_SERIES` = the four chart series +
`SPARK_CELLS`; the endpoint answers unknown names as empty series on purpose), and the two series
phase 04 added (`llm.tpot_seconds` vLLM-only, `llm.spec_accept_rate` gufo-only) stay stored
additively in `sources/llm.py` HISTORY_METRICS. Above the strip sits **the fleet block**
(`#fleet`, phase 07: "I'm missing the aggregate throughput at the top") — `ALL ENGINES`, a
reachability dot, an
`N of M engines` tag, the note *sums only engines answering now*, then running / waiting /
prompt tok/s / decode tok/s / total tokens. `FLEET_FIELDS` in `page-llm.js` is a list of
`fleet_totals()` keys and nothing else: the page sums no number of its own, a dead engine is
dropped from every total presented as current (its own stale-labelled panel still shows its
last good values), and the tag counts the answering engines, names the missing one in its
tooltip and takes the shared `.stale` dim while the sums are partial. It **hides below two
engines** — a cross-engine sum of one engine duplicates that engine's own panel — and it
carries **no severity colour anywhere**: `fleet_totals` has no `severity` to render (the
summed-KV bar is gone for that reason) and `severity.py` rates no throughput or queue depth,
so the numbers stay ink and only the dot speaks reachability. `fleet.seams` — the API's own
note on how the two engines count prompt tokens differently — is the prompt sum's tooltip
rather than another line of ink. The
four trend charts stay but at height 110 *below* the panels, each with an `ariaLabel`. Panels
sit at their own height (`#engines` carries `.items-start`): a 5-row gufo panel next to a 7-row
vLLM one otherwise grows a blank bottom, and a blank panel reads as "no data".
sit at their own height (`#engines` carries `.items-start`): a two-hero gufo card next to a
latency-row vLLM one otherwise grows a blank bottom, and a blank panel reads as "no data". An
engine that leaves the payload leaves both components (`pruneEngines`), never a stale card in
either grid.
**The slot grid (`llm_monitor`'s, phase 07 task 04) belongs to the gufo panel, under its
`sessions (GET /slots)` label.** One card per row of `sessions.list`, **keyed by the row's own
`id`** in `state.slots`: built once by `slotCard()`, patched in place afterwards, and reordered
by `appendChild` of the existing node so DOM order stays slot order — rebuilding a busy slot's
card every 2 s poll re-announces the whole card to assistive tech for a difference of a few
tokens. `SLOT_FIELDS` is the whole table and each cell names **exactly one** `/slots` key
(`path: 'next_token'` reads `next_token[0]`, where the capture puts `n_remain`); `slotValue()`
is the only place a row is indexed, so `tests/test_api.py` can resolve every printed field
against `tests/fixtures/slots_gufo.json` and a rename fails there rather than printing dashes
or a plausible wrong number. No per-slot speed cell: the capture publishes no per-slot rate key
and a page-side Δn_decoded/Δt would be a measurement with no estimator, no `Δ<window>` label and
no `source` string (rule 2) — the engine's rate lives on its strip card and panel. llama.cpp's
`-1` sentinels print a word or nothing: `id_task: -1` is **`idle`**, `n_remain: -1` is **`n/a`**,
never a negative countdown and never the `0` that reads as one token from done. The busy marker
is a **border, a glyph (`○`/`●`) and the word `processing`** in the brand `--accent`, never an
`ok|warn|fail` token — a slot generating tokens is the job (phase 06's GPU reasoning), so
`severity.py` rates no slot and the stylesheet section contains no verdict token. The empty
state is a sentence, never a card of nothing, and it distinguishes the two cases: no `/slots`
answer prints the API's own `sessions.detail`, an answer with no rows prints *engine reports 0
slots*; `sessions.source` is the gate. The note under the grid stays because a **queued
(deferred) request is not in `/slots`** — the waiting row counts those. vLLM's panel gets one
line (*exposes no per-slot JSON*) with `sessions.detail` as its tooltip and **no grid**: a
component of permanent `n/a`s for a server that cannot answer is the dead-cell anti-pattern.
The gate is the engine name (`SLOT_ENGINES`), not row count, so an engine that has not answered
yet still shows its label and its honest empty sentence.
**Charts belong to the host they chart.** `/system` has two sections — the host panels and the
transition events — and nothing else: the fleet-wide chart stack, the nine-figure fleet-totals