phase: 34_radiance_llm_engine
Everything is verified and green. Final report: **Phase 34 final verification pass — complete** - Verified all 6 completed tasks' work against the live fleet and hermetic suite: `llm:radiance` job registered, `llm_engines()` registry drives 404 gate / history defaults / `/api/config` (additive, byte-identical gufo/vllm keys), radiance field map + `/stats` sidecar + 7-position spec curve, `/llm` strip/curve/rows. - **Defect found & fixed (pre-existing, task 05):** the vLLM-only sentence "vLLM exposes no per-slot JSON…" rendered on the **radiance** panel (`slotSection`'s non-gufo branch). Fix: vLLM line gated to `engine === 'vllm'`; radiance's panel slot section renders nothing (its session home is the strip) — `page-llm.js` + the served-byte pin in `tests/test_api.py` (now asserts the radiance branch is empty and the sentence can't leak). Browser-verified; re-shot `data/logs/34-llm-radiance-rest.png` (the 4.5-min wait found no in-flight request, so the busy shot remains the earlier pass's — its strip rows still stand; only the panel text predates the fix). - Tests: `uv run pytest` → **2031 passed, 19 skipped** (100%); `uv run pytest --cov` → **98.73%** (floor 94%; config 100%, llm 99%, collectors 98%, api 99%); `uv run ruff check src tests agent && uv run pyright` → clean; `node --check` OK; `uv pip list --outdated` → empty. - `uv run pytest -m live --live` → **13 passed, 2 skipped** (vLLM unset — correct; live-write opt-in — correct); the read-only radiance test passed. - Completion criteria: (1) engines `gufo`+`radiance`, `fleet.engines_live` 2, `config.llm.radiance.enabled` true — **pass (live curl)**; (2) unset → 404 no-dial + no job — **pass (spy-pinned, green)**; (3) fixtures present, both families (79 `radiance:` / 198 `vllm:` lines), drift pin — **pass**; (4) panel/curve/rows — **pass (re-shot rest screenshot)**; (5) live radiance test — **pass**; (6) suite/coverage/lint/`bash .agents/validate.sh` (exit 0) — **pass**; (7) docs — below; (8) gufo/vLLM behaviour unchanged — **pass (existing pins untouched)**. - Documentation: **README.md** full review — env table, LLM-engines section, architecture diagram, capture-script and `/llm` section all name radiance correctly; no inconsistencies found, no changes needed. **AGENTS.md** — the three task-06 fixes (fleet line, route row, module bullets) are in place and all other statements match reality; no changes needed. - Dev server healthy (`./scripts/dev-server.sh status`: all sources ok, `llm:radiance` answering); all changes left in the working tree for the harness. - Next pending phase: none — 34 was the last phase in `todo/`.
This commit is contained in:
@@ -8,7 +8,7 @@ file:
|
||||
| `/` | Is anything wrong, and which dashboard do I need? | all five, in one request (`/api/summary`) |
|
||||
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
|
||||
| `/disks` | Are any of my **physical** disks failing, and which one — health, temperature, power-on hours, wear, sector alarms | the same agents, running `smartctl -i -A -H` per whole drive (never a partition or a mount); a host whose agent has no `smartctl` is not a host on this page |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics` — read-only |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics`, radiance `/metrics` plus its `/stats` as a best-effort sidecar (its answer to `/slots`) — read-only |
|
||||
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
|
||||
| `/gateway` | Which domains are coming in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`): a trunk per bound port with its configured routes branching off it, each line reading the config's own **inbound** word (`server_name`, `map` key, `backend` name) on line 1 over the counterpart it routes to on line 2 (`→ host:port`), plus its live connection count — and one filter box above the map that hides the routes whose own labels do not contain what you typed, asking nothing of the server | the agent on **one** host, `GATEWAY_HOST` (default `gateway`), reading its `haproxy.cfg` + nginx tree and `/proc/net/tcp{,6}` — so it needs **agent 1.2.0**, plus **1.3.0**'s `resolver` section to put a number on a route whose target the config writes as a *name* (without it those lines are drawn and dashed), plus **1.4.0**'s `#off` marker read to show a route the switch has turned off **or back on** (without it an off route is drawn on, because the marker is a comment a 1.3.0 agent does not read) — an off route stays drawn and counted, its row simply reads `enabled: false` (its id gains a `#off` suffix), and the config-integrity warnings the marker *caused* (a backend emptied of its last `server`, a map entry pointed at the switch's sink) are read as a chosen state while the same shape from a hand-deleted line is still a warning — and the whole page is *unconfigured* (never red) unless `GATEWAY_HOST` names a `HOST_TARGETS` entry |
|
||||
|
||||
@@ -770,7 +770,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
|
||||
| Variable | Meaning |
|
||||
| ------------------------------------- | -------------------------------------------------------------- |
|
||||
| `HOST_TARGETS` | `name=url` per host running the agent. The shape a new host gets is `https://host:9120` with **no userinfo** — a certificate-authenticated agent, verified against this app's own root CA and presented with its client certificate, with a revoked host refused before the dial (`certs.py`, `collectors.AgentTls`). The compatibility shape is a token-mode agent's URL userinfo (`http://TOKEN@host:9120`), moved into an `Authorization` header by `parse_targets` so it never reaches `/api/config` or an error message |
|
||||
| `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
|
||||
| `GUFO_BASE_URL` / `VLLM_BASE_URL` / `RADIANCE_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
|
||||
| `RATE_WINDOW_SECONDS` | the **host's** Δ window (default 10) — bytes/sec and jiffies→CPU %, measured over what a 10 s poll can fill; each value carries the window it actually got, and `/system`'s footnote prints that, not this default. A window the poll loop cannot honour — the ~0.2 s catch-up after a fetch slower than the interval, or a base older than 2× the window — answers one `n/a` instead of a spike, and a rate 8× above the last one is held until a window measured from a **different** sample confirms it (phase 13). The LLM rates do **not** use it: they are measured over `LLM_INTERVAL_SECONDS` (phase 23) |
|
||||
| `LLM_INTERVAL_SECONDS` | the `/llm` poll cadence (default 2 s) **and** the Δ window behind every token rate — one estimator per measuring cadence, so a slot that starts reads steady within two polls instead of ramping over five. Slowing the poll slows the window with it; there is no separate LLM window knob, and a window narrower than the interval would answer `n/a` every poll |
|
||||
| `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) |
|
||||
@@ -779,7 +779,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
|
||||
| `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others |
|
||||
| `DISKS_TIMEOUT_SECONDS` | the disks fetch's own budget (default 30 s), because the agent runs `smartctl` inside its snapshot handler and a wedged SMART pass must not read as a dead host |
|
||||
| `GATEWAY_HOST` | which `HOST_TARGETS` name is the ingress the `/gateway` page watches (default `gateway`). It has no URL of its own — the source reads the same agent `/snapshot` the system page reads, projected onto agent 1.2.0's `haproxy`/`nginx`/`sockets` sections — so a name that is not in `HOST_TARGETS` means "no ingress to watch", and the source is unconfigured rather than red. The page refetches itself every 15 s in the browser and has no `POLL_MS_*` knob, because the payload only moves when that one job re-reads the ingress |
|
||||
| `GUFO_API_KEY` / `VLLM_API_KEY` | only if an engine is behind a bearer token; they never leave the server |
|
||||
| `GUFO_API_KEY` / `VLLM_API_KEY` / `RADIANCE_API_KEY` | only if an engine is behind a bearer token; they never leave the server |
|
||||
| `NUT_HOST` / `NUT_PORT` | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT |
|
||||
| `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) |
|
||||
| `NUT_ALLOW_ACTIONS` | `1` enables the UPS write endpoint that dials **anything** — the `/api/auth/*` and `/api/certs/*` writes touch only this process and its `meta` table |
|
||||
@@ -804,10 +804,10 @@ installed.
|
||||
## Architecture
|
||||
|
||||
```
|
||||
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
|
||||
agent/host_agent.py (each host) gufo / vLLM / radiance upsd (NUT)
|
||||
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD
|
||||
│ (three readers: system 10s, │ + /slots (gufo) │
|
||||
│ disks 5m, gateway 30s) │ │
|
||||
│ disks 5m, gateway 30s) │ + /stats (radiance) │
|
||||
▼ ▼ ▼
|
||||
src/dashboards/collectors.py ────────────────────────────────────────────┐
|
||||
│ parse + Δcounter/Δt (rates.py) │
|
||||
@@ -1132,16 +1132,28 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
|
||||
duplicate its own cards), and each host's **Charts** disclosure behaves exactly like `/system`'s:
|
||||
collapsed, built only while open, and fed by one `/api/disks/history` request per host per
|
||||
refresh naming `disk.temp_c@dev` and `disk.percentage_used@dev` for every drive it has.
|
||||
`/llm` opens with
|
||||
the **fleet**: one block above the panels — `ALL ENGINES`, an `N of M engines` tag, the note
|
||||
*sums only engines answering now*, then **running / waiting / prompt tok/s / decode tok/s /
|
||||
total tokens** — and every number on it is `fleet_totals()`, so the page sums nothing itself.
|
||||
`/llm` watches up to three engines, each configured by its own `*_BASE_URL` (unset means
|
||||
*not deployed here*, and its card and panel disappear — never a dead half): **gufo**
|
||||
(a llama.cpp-family server — `/metrics` plus the per-slot `/slots` JSON), **vLLM**
|
||||
(`/metrics` only, which is why its panel says *exposes no per-slot JSON*), and
|
||||
**radiance** — a vLLM-derived engine (this fleet's `driveripper:8000`) that answers
|
||||
`/metrics` (its `radiance:` native windowed gauges **win** the throughput cells when
|
||||
present, falling back to the measured `Δ<window>` of its vLLM-shaped counters when
|
||||
absent), `/stats` — its answer to `/slots`: fetched best-effort beside the `/metrics`
|
||||
poll, so a dead sidecar degrades the session arm and the queue cap to `n/a`, never `0`,
|
||||
and never fails the poll — and `/v1/models`. `/llm` opens with
|
||||
the **fleet**: one block above the panels — `ALL ENGINES`, an `N of M engines` tag, a note
|
||||
that is the API's own `fleet.seams` sentence (the shipped line, *sums only engines
|
||||
answering now*, is the fallback for a payload that predates it), then **running / waiting /
|
||||
prompt tok/s / decode tok/s / total tokens** — and every number on it is `fleet_totals()`,
|
||||
so the page sums nothing itself.
|
||||
A dead engine keeps its last-good numbers in its own (stale-labelled) panel but is dropped
|
||||
from a total presented as current: the tag counts what answered, names the missing engine in
|
||||
its tooltip and dims while the sums are partial. Nothing on the block is coloured —
|
||||
`severity.py` rates no throughput and no queue depth, because 0 tok/s on an idle fleet is
|
||||
healthy — the dot is reachability, and `fleet.seams` (how the two engines count prompt
|
||||
tokens differently) is a tooltip on the prompt sum rather than another line of ink. Below
|
||||
healthy — the dot is reachability, and `fleet.seams` (how the engines count prompt tokens
|
||||
and where their throughputs come from) is the note under the count and a tooltip on the
|
||||
prompt sum. Below
|
||||
two configured engines the block is hidden: a cross-engine sum of one engine duplicates that
|
||||
engine's panel. Below the fleet block — with the page's shared `.85rem` gap, as everywhere else
|
||||
in the section stack — sits the **stats strip**: one card per configured
|
||||
@@ -1149,12 +1161,23 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
|
||||
fill and a breach's text take their colour from the payload's `severity` and nothing else),
|
||||
then `prompt tok/s` and `decode tok/s` with their provenance tag (`Δ2.0s` at the default cadence —
|
||||
the LLM window is one poll interval, phase 23 — scraped, estimated), `total tokens`, the sessions this engine reports (`slots busy` counted from
|
||||
gufo's `/slots`, or `running`/`waiting` for vLLM, plus `waiting`), `prefix cache hit`,
|
||||
`spec accept` (gufo), `queue time` with its `p95 …` foot (vLLM) — and red alarm chips for
|
||||
preemptions / GPU-context losses that exist only while non-zero. Then the engine panels
|
||||
gufo's `/slots`, or `running`/`waiting` for vLLM and radiance, plus `waiting`), `prefix cache hit`,
|
||||
`spec accept` (gufo and radiance — radiance's gauge reuses gufo's key, the same 0-1
|
||||
scale), `queue time` with its `p95 …` foot (vLLM; the dev radiance build publishes no
|
||||
queue-time histogram, so its cell is `n/a` by absence, never back-filled) — and red alarm chips for
|
||||
preemptions / GPU-context losses that exist only while non-zero. The radiance card adds
|
||||
its own four cells — `queue cap` (the admission ceiling, `n/a` while `/stats` is not
|
||||
answering, never `0`), `build` (the engine's own version string), `uptime`, and `VRAM
|
||||
(card 0)` (the engine's own view of the card in GiB = 2³⁰; `/system`'s GPU row is the
|
||||
agent's view of the same card) — and two more things: a **7-bar spec-acceptance curve**
|
||||
(one bar per position the engine published, in order, height ∝ the 0-1 ratio, drawn in
|
||||
neutral ink because an efficiency number is not a verdict) and its **per-request rows**
|
||||
beside the counts (the engine's own fields, as-is — at rest the table is hidden and the
|
||||
API's own count-otherwise sentence takes their place). Then the engine panels
|
||||
hold the detail the strip does not print: two big
|
||||
throughput numbers (decode, prompt — uncoloured, because 0 tok/s on an idle server is
|
||||
healthy) with their provenance tag, and the latency rows (TTFT, TPOT) each with a `p95 …`
|
||||
healthy) with their provenance tag, and the latency rows (TTFT, TPOT — vLLM and radiance;
|
||||
gufo's build does not publish them) each with a `p95 …`
|
||||
foot line and its own sparkline over the selected window. Under the gufo panel's
|
||||
**`sessions (GET /slots)`** label sits the **slot grid**: one card per parallel slot, keyed by
|
||||
the slot's own id, created once and patched in place so a busy slot never flickers — request
|
||||
@@ -1312,6 +1335,7 @@ DASH_LLM_WATCH_REPLAY=data/logs/llm-window-watch.tsv \
|
||||
uv run ruff check src tests agent && uv run pyright # lint + types
|
||||
./scripts/capture-fixtures.sh # re-record ground truth: host agents, UPS replies, gufo /slots
|
||||
./scripts/capture-fixtures.sh --with-llm # ... and the engines' /metrics while they are up
|
||||
# (radiance also records /stats + /v1/models in the same pass)
|
||||
SMART_HOSTS="host-a-ops host-b-ops" ./scripts/capture-fixtures.sh
|
||||
# ... and smartctl per physical disk: plain smartctl,
|
||||
# `sudo -n` if that is denied; scrubs serials/WWNs
|
||||
|
||||
Reference in New Issue
Block a user