diff --git a/.agents/phases/todo/07_llm_monitor_style_pages/00_phase.md b/.agents/phases/complete/07_llm_monitor_style_pages/00_phase.md similarity index 100% rename from .agents/phases/todo/07_llm_monitor_style_pages/00_phase.md rename to .agents/phases/complete/07_llm_monitor_style_pages/00_phase.md diff --git a/.agents/phases/todo/07_llm_monitor_style_pages/01_slots_schema_and_table_removal.md b/.agents/phases/complete/07_llm_monitor_style_pages/01_slots_schema_and_table_removal.md similarity index 100% rename from .agents/phases/todo/07_llm_monitor_style_pages/01_slots_schema_and_table_removal.md rename to .agents/phases/complete/07_llm_monitor_style_pages/01_slots_schema_and_table_removal.md diff --git a/.agents/phases/todo/07_llm_monitor_style_pages/02_fleet_totals_block.md b/.agents/phases/complete/07_llm_monitor_style_pages/02_fleet_totals_block.md similarity index 100% rename from .agents/phases/todo/07_llm_monitor_style_pages/02_fleet_totals_block.md rename to .agents/phases/complete/07_llm_monitor_style_pages/02_fleet_totals_block.md diff --git a/.agents/phases/todo/07_llm_monitor_style_pages/03_engine_stats_strip.md b/.agents/phases/complete/07_llm_monitor_style_pages/03_engine_stats_strip.md similarity index 100% rename from .agents/phases/todo/07_llm_monitor_style_pages/03_engine_stats_strip.md rename to .agents/phases/complete/07_llm_monitor_style_pages/03_engine_stats_strip.md diff --git a/.agents/phases/todo/07_llm_monitor_style_pages/04_slot_grid.md b/.agents/phases/complete/07_llm_monitor_style_pages/04_slot_grid.md similarity index 100% rename from .agents/phases/todo/07_llm_monitor_style_pages/04_slot_grid.md rename to .agents/phases/complete/07_llm_monitor_style_pages/04_slot_grid.md diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__00_phase.a1.err b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__00_phase.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__00_phase.a1.md b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__00_phase.a1.md new file mode 100644 index 0000000..d66862c --- /dev/null +++ b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__00_phase.a1.md @@ -0,0 +1,11 @@ +**Phase 07 final verification pass — all tasks already in `complete/`; every criterion re-verified live.** + +- **Old slot table gone**: no `#slots-card`/`#slots-table`/`renderSlots` in HTML/JS/CSS (only the new grid's `.slots-empty` *class*); pinned by `test_the_broken_slot_table_is_gone_from_html_js_and_css`. +- **`tests/fixtures/slots_gufo.json`**: present; live `GET ${GUFO_BASE_URL}/slots` this pass matches it key-for-key (row + `next_token[0]`), so the capture is reproducible; drift proven to fail (mutation tests + live `test_gufo_slots_answer_the_pinned_shape`, 9 live tests pass). +- **Aggregate block**: live page shows `2 of 2 engines`, running/waiting/prompt/decode/total-tokens straight from `fleet_totals()`, dot `ok`, no verdict colour; on a throwaway single-engine instance (:8091, private temp DB) `#fleet` is `hidden`, one full-width strip card + one panel, zero banners → "one clean half". +- **Strip + slot grid on live data**: 2 `.card.stat` (KV bars `bar ok` 13 %/0 %), gufo grid rendered real cards incl. a busy slot (`#2189`, 67849 prompt tok, 99 % cache share, `processing`), vLLM note-only with no grid; no `NaN`/`undefined` anywhere. Screenshots: `data/logs/llm-phase07-final-live-{two-engines,single-engine}.png`, `ups-charts-aria-labels.png`. +- **Defect found in earlier work and fixed**: `/ups`' four `LineChart`s shipped no `ariaLabel`, violating AGENTS.md's "every chart names itself for assistive tech" (`/system`+`/llm` were pinned, `/ups` wasn't) → added labels to `page-ups.js` and `test_every_ups_chart_names_itself_for_assistive_tech`; browser-verified `role="img"` + label on all four. +- **Gate**: `bash .agents/validate.sh` → **exit 0**; `uv run pytest --cov --cov-fail-under=94` → **604 passed, 9 skipped, coverage 97.91 %**; `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → **0 errors**; `node --check` every page script ok; `uv run pytest -m live --live` → **9 passed**; `uv pip list --outdated` empty. +- **Docs**: README + AGENTS.md reviewed end-to-end; fixed test count (582→604), `/llm` source column + both architecture diagrams now name gufo `/slots`, AGENTS.md agent line gained `GET /schema` + status page, ariaLabel convention now covers all three chart pages, README Testing block now lists `bash .agents/validate.sh`. Env-var table, agent/install flags, Containerfile facts, `/api/*` shapes, range chips, remote URL all verified accurate — no other changes needed. +- **Live-infra**: no fleet/host/config change; the temp :8091 instance, its temp DB and the fake processes are removed; dev server restarted and `/api/health` `ok` (all 6 sources ok). One thing I deliberately did **not** touch: the gitignored local `.env.dev` still carries a comment claiming the NUT `[dashboards]` account has only battery-test `instcmds`, whereas AGENTS.md/README record `instcmds = all` (verified on hardware 2026-10-17) — operator-owned file, flag for the operator to reword. +- **Next pending phase**: `08_events_paging_and_copy`. diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__00_phase.a1.md.status b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__00_phase.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__00_phase.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__00_phase.a1.validate b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__00_phase.a1.validate new file mode 100644 index 0000000..a986977 --- /dev/null +++ b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__00_phase.a1.validate @@ -0,0 +1,48 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 11%] +........................................................................ [ 23%] +........................................................................ [ 35%] +........................................................................ [ 46%] +.......sssssssss........................................................ [ 58%] +........................................................................ [ 70%] +........................................................................ [ 82%] +........................................................................ [ 93%] +..................................... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 28 +src/dashboards/alerts.py 166 0 100% +src/dashboards/api.py 193 0 100% +src/dashboards/app.py 107 6 94% 50-51, 56, 58, 60, 127 +src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120 +src/dashboards/config.py 113 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164 +src/dashboards/rates.py 37 1 97% 51 +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/host.py 199 1 99% 147 +src/dashboards/sources/llm.py 150 1 99% 228 +src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 178 2 99% 243-244 +------------------------------------------------------------------ +TOTAL 2158 45 98% +Required test coverage of 94% reached. Total coverage: 97.91% +604 passed, 9 skipped, 1 warning in 52.69s +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__01_slots_schema_and_table_removal.a1.err b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__01_slots_schema_and_table_removal.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__01_slots_schema_and_table_removal.a1.md b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__01_slots_schema_and_table_removal.a1.md new file mode 100644 index 0000000..5443f8d --- /dev/null +++ b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__01_slots_schema_and_table_removal.a1.md @@ -0,0 +1,15 @@ +**Task 01 complete — all green.** + +**Implemented** +- `scripts/capture-fixtures.sh`: gufo `/slots` capture (→ `tests/fixtures/slots_gufo.json`), `/v1/slots` fallback in the collector's order, atomic tmp+mv, clean skip when `GUFO_BASE_URL` unset; scrubber now blanks busy-slot `prompt` (user request text) in `slots_*.json`. +- Deleted the broken slot table end-to-end: `#slots-card`/`#slots-table`/`#slots-empty` markup (`llm.html`), `renderSlots()` + call (`page-llm.js`), dead `.chip.info` rule (`dash.css`). +- New `tests/slots_contract.py` (`assert_slot_shape`, required/optional row fields, `id_task`/`task_id` alias, `next_token[]` sub-fields, id uniqueness, `is_processing`/`state` consistency) + `tests/test_llm_slots_contract.py` (fixture-driven, mutation tests proving drift fails, deletion pins on served bytes); `test_llm_source.py` passthrough now uses the captured rows; live twin `test_gufo_slots_answer_the_pinned_shape` added to `test_live_fleet.py`. +- Fixture recorded from live gufo (idle 2-slot shape); full re-capture also refreshed agent/NUT fixtures (volatile-value drift only, suite green). Verified `/llm` in a real browser against live data: 2 engine cards, zero JS errors (screenshot in `data/logs/`). + +**Gate** — `bash .agents/validate.sh` exit 0: `uv run pytest --cov` **578 passed, 9 skipped, coverage 97.91%** (floor 94); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; `uv pip list --outdated` empty. `uv run pytest -m live --live` **9 passed** (incl. the new live `/slots` contract). + +**Docs** — AGENTS.md (test-count 578/9, capture-script line, testing-policy entry for the slots contract + prompt privacy) and README.md (capture commands, contract paragraph). + +**Decisions**: `/slots` capture runs without `--with-llm` (the task names only the GUFO_BASE_URL skip and reproducibility via the bare script; it's one tiny request the hermetic test needs). `prompt` treated as user content: captured blanked, never rendered (pinned). + +**Next pending task**: `07_llm_monitor_style_pages/02_fleet_totals_block.md`. diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__01_slots_schema_and_table_removal.a1.md.status b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__01_slots_schema_and_table_removal.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__01_slots_schema_and_table_removal.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__01_slots_schema_and_table_removal.a1.validate b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__01_slots_schema_and_table_removal.a1.validate new file mode 100644 index 0000000..53db6ea --- /dev/null +++ b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__01_slots_schema_and_table_removal.a1.validate @@ -0,0 +1,48 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 12%] +........................................................................ [ 24%] +........................................................................ [ 36%] +.......................................................sssssssss........ [ 49%] +........................................................................ [ 61%] +........................................................................ [ 73%] +........................................................................ [ 85%] +........................................................................ [ 98%] +........... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 28 +src/dashboards/alerts.py 166 0 100% +src/dashboards/api.py 193 0 100% +src/dashboards/app.py 107 6 94% 50-51, 56, 58, 60, 127 +src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120 +src/dashboards/config.py 113 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164 +src/dashboards/rates.py 37 1 97% 51 +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/host.py 199 1 99% 147 +src/dashboards/sources/llm.py 150 1 99% 228 +src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 178 2 99% 243-244 +------------------------------------------------------------------ +TOTAL 2158 45 98% +Required test coverage of 94% reached. Total coverage: 97.91% +578 passed, 9 skipped, 1 warning in 51.48s +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__02_fleet_totals_block.a1.err b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__02_fleet_totals_block.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__02_fleet_totals_block.a1.md b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__02_fleet_totals_block.a1.md new file mode 100644 index 0000000..a8c4d5c --- /dev/null +++ b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__02_fleet_totals_block.a1.md @@ -0,0 +1,15 @@ +**Implemented** +- `/llm` now opens with the `llm_monitor` aggregate block (`#fleet`): reachability dot, `all engines`, an `N of M engines` tag, the note *"sums only engines answering now"*, then **running / waiting / prompt tok/s / decode tok/s / total tokens** — every number straight from `fleet_totals()` (`/api/llm/snapshot` → `fleet`); the counts-only `#totals` card (`f-live`/`f-total`/`f-active`/`f-waiting`) is deleted, ids and all. +- `FLEET_FIELDS` is a list of API paths only (page sums nothing); hidden below two engines; a dead engine is dropped from every sum, dims the tag (shared `.stale`) and is named in its tooltip; `fleet.seams` rides on the prompt sum as a tooltip; an `n/a` sum clears its unit; no severity/`hero()` anywhere on the block. +- `dash.css`: new `fleet totals` section built only from existing vocabulary (`.card.flat`, `.metrics`, `.dot`, `.chip`, `.stale`) — no new palette, no verdict tokens. +- `tests/test_api.py`: replaced the phase‑04 totals-card pin with 5 fleet-block tests (markup/order + old ids gone; `fleet` key paths with both engines configured and with neither → 404 `configured:false`; injected-payload test proving a dead engine contributes nothing and null sums never print 0; no-verdict colouring; provenance) + `_fleet_tag`/`_fleet_render_path`/`_fleet_field_metrics`/`FLEET_BLOCK_PATHS` helpers. + +**Gate** — `bash .agents/validate.sh` exit 0: `uv run pytest --cov --cov-fail-under=94` **582 passed, 9 skipped, coverage 97.91%**; `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` on all page scripts clean. `uv run pytest -m live --live` **9 passed**. `uv pip list --outdated` empty (no dependency change). + +**Docs** — AGENTS.md (`/llm` UI-conventions paragraph now describes the fleet block + her test count 582) and README.md (same, plus "single engine → one clean half"). + +**Decisions / deviations** +- The task asked for a status dot while also requiring "no severity colour anywhere on the block": the dot is the shared *reachability* dot (as the nav and every card header), and the tests pin that no `severity`/`hero()` reaches the block and its CSS uses no `--ok/--warn/--fail`. +- Browser-verified live (screenshots in `data/logs/`): `llm-fleet-block-two-engines.png` (sums hand-checked: decode 10.8 = gufo 10.8 + vLLM 0.0, total tokens 177.55 M = 172.56 M + 4.98 M), `llm-fleet-block-single-engine.png` (block absent, one clean half), `llm-fleet-block-partial.png` (vLLM down → tag "1 of 2 engines" dimmed, dot amber, sums = gufo only). Those used two throwaway local instances on :8091/:8092 with private temp DBs, killed afterwards; no fleet/config change, dev server restarted and all sources `ok`. + +**Next pending task**: `.agents/phases/todo/07_llm_monitor_style_pages/03_engine_stats_strip.md`. diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__02_fleet_totals_block.a1.md.status b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__02_fleet_totals_block.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__02_fleet_totals_block.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__02_fleet_totals_block.a1.validate b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__02_fleet_totals_block.a1.validate new file mode 100644 index 0000000..52b5aac --- /dev/null +++ b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__02_fleet_totals_block.a1.validate @@ -0,0 +1,48 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 12%] +........................................................................ [ 24%] +........................................................................ [ 36%] +...........................................................sssssssss.... [ 48%] +........................................................................ [ 60%] +........................................................................ [ 73%] +........................................................................ [ 85%] +........................................................................ [ 97%] +............... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 28 +src/dashboards/alerts.py 166 0 100% +src/dashboards/api.py 193 0 100% +src/dashboards/app.py 107 6 94% 50-51, 56, 58, 60, 127 +src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120 +src/dashboards/config.py 113 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164 +src/dashboards/rates.py 37 1 97% 51 +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/host.py 199 1 99% 147 +src/dashboards/sources/llm.py 150 1 99% 228 +src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 178 2 99% 243-244 +------------------------------------------------------------------ +TOTAL 2158 45 98% +Required test coverage of 94% reached. Total coverage: 97.91% +582 passed, 9 skipped, 1 warning in 49.72s +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__03_engine_stats_strip.a1.err b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__03_engine_stats_strip.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__03_engine_stats_strip.a1.md b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__03_engine_stats_strip.a1.md new file mode 100644 index 0000000..1f9989a --- /dev/null +++ b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__03_engine_stats_strip.a1.md @@ -0,0 +1,20 @@ +**Implemented (task 03 — per-engine stats strip)** +- `llm.html`: new `#strip` section (`aria-label="Engine summary"`, ships empty+hidden) between `#fleet` and `#engines`. +- `page-llm.js`: new `STRIP_ROWS` table (key **is** the `cellValue()` key), `stripCard()`/`renderStripCard()`/`stripRowsFor()`/`pruneEngines()`; card = dot + name + age stamp, KV-cache bar, prompt/decode tok/s with `Δ` tags, total tokens, `slots busy` (gufo, counted from `sessions.list` `is_processing`, falls back to `sessions.active` + API `detail` — never a 0 counted from no rows), `running`/`waiting`, `prefix cache hit`, `spec accept` (gufo), `queue time` + `p95` foot (vLLM), alarm chips. +- Panel cell list demoted to heroes + TTFT/TPOT (`spark:` list shrank 8→2): sessions, queued, KV, cache-hit, spec-accept, queue-mean, token footnote and the alarm container all left the panel — one value, one home. `HISTORY_SERIES` = explicit `CHART_SERIES` + `SPARK_CELLS`. +- `dash.css`: new "engine stats strip" section built only from existing vocabulary (`.card`, `.kv`, `.bar`, `.chip.fail`, `.prov`; `.prov:empty` collapses; only `warn`/`fail` paint text), including the fix that moves the `.kv` grid *inside* the row wrapper. +- `tests/test_api.py`: 9 new strip tests (placement/hidden/aria, shared-vocabulary markup + the `.kv` layout rule, `only` scoping, every row a cell the capture-parsed engines publish, slot-count rule-2 rule, KV colour from `severity` only, provenance-without-colour, strip/panel disjointness, no peak-ctx row) + 3 phase-04 pins restated for the new split. +- Browser-verified live: `data/logs/llm-stats-strip-two-engines.png`, `-single-engine.png` (one full-width card, fleet block hidden), `-alarm-chip.png` (`.chip.fail` "Preemptions 3" on the incident card only) — the last two used throwaway local instances on :8091/:8092 plus a local fixture-backed fake engine, all killed afterwards; no fleet/config change, dev server restarted and `status: ok`. + +**Gates:** `uv run pytest` → 591 passed, 9 skipped · `bash .agents/validate.sh` exit 0 (coverage 97.91%, floor 94) · `ruff check src tests agent` All checks passed · `pyright` 0 errors · `node --check` every script ok · `uv run pytest -m live --live` 9 passed · `uv pip list --outdated` empty (deps untouched). + +**Docs:** README page tour rewritten for strip-then-panels; `AGENTS.md` gained the strip conventions paragraph (payload-only rows, KV colour, session-row shaping, alarms on the strip, `HISTORY_SERIES` composition, `pruneEngines`) and the `minSpan` sentence reworded. + +**Decisions/deviations** +- Task item 2 (strip gets vLLM "queue time") and item 6 (panel keeps "queue mean") conflict; satisfied the hard criterion "no value printed twice" by giving queue time — mean *and* p95 — to the strip, panel keeps TTFT/TPOT. +- "No value twice" applied to the cell lists; the two throughput heroes stay in both components because items 1 and 6 both demand it (llm_monitor does the same) — pinned as the explicit exemption. +- Alarm chips moved to the strip card per item 3 (phase overview's older wording kept them in the panel); panel alarm container deleted, asserted not to come back. +- Observation: `llamacpp:n_tokens_max` (llm_monitor's peak-ctx cell) *is* in our gufo payload, contrary to the phase assumption's wording; followed the task (no peak-ctx cell) and pinned that the page never reads it. +- `minSpan` now has no page caller (count rows are numbers, not sparks); kept in `charts.js` where the component contract test still pins it. + +**Next pending task:** `07_llm_monitor_style_pages/04_slot_grid.md`. diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__03_engine_stats_strip.a1.md.status b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__03_engine_stats_strip.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__03_engine_stats_strip.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__03_engine_stats_strip.a1.validate b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__03_engine_stats_strip.a1.validate new file mode 100644 index 0000000..e4b253b --- /dev/null +++ b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__03_engine_stats_strip.a1.validate @@ -0,0 +1,48 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 12%] +........................................................................ [ 24%] +........................................................................ [ 36%] +....................................................................ssss [ 48%] +sssss................................................................... [ 60%] +........................................................................ [ 72%] +........................................................................ [ 84%] +........................................................................ [ 96%] +........................ [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 28 +src/dashboards/alerts.py 166 0 100% +src/dashboards/api.py 193 0 100% +src/dashboards/app.py 107 6 94% 50-51, 56, 58, 60, 127 +src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120 +src/dashboards/config.py 113 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164 +src/dashboards/rates.py 37 1 97% 51 +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/host.py 199 1 99% 147 +src/dashboards/sources/llm.py 150 1 99% 228 +src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 178 2 99% 243-244 +------------------------------------------------------------------ +TOTAL 2158 45 98% +Required test coverage of 94% reached. Total coverage: 97.91% +591 passed, 9 skipped, 1 warning in 50.69s +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__04_slot_grid.a1.err b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__04_slot_grid.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__04_slot_grid.a1.md b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__04_slot_grid.a1.md new file mode 100644 index 0000000..467af9a --- /dev/null +++ b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__04_slot_grid.a1.md @@ -0,0 +1,17 @@ +**Implemented (task 07/04 — gufo slot grid)** +- `page-llm.js`: `SLOT_FIELDS` table (one fixture key per cell, `path:'next_token'` → `next_token[0]`), `slotValue()` as the only row accessor, `slotCard()`/`renderSlotCard()`/`renderSlotGrid()` keyed by slot `id` in `state.slots` (built once, text-patched, `appendChild` keeps slot order, gone slots + pruned engines cleaned up); `slotSection()` puts `sessions (GET /slots)` (`

`) + `role=list` grid + the queued/deferred note in the gufo panel, and vLLM only the one-line note (with `sessions.detail` as tooltip) — no grid. +- Cells: request (`id_task`, `-1`→`idle`), prompt tok (+`processed …`), cache hit (+`% of prompt`), generated (+`p=` when published), remaining (`-1`→`n/a`), context, model `+spec`. No per-slot speed cell (capture has no rate key; a JS Δ would be an unlabelled measurement). Busy marker = accent border + `○/●` + the word `processing`, no verdict token. +- Empty states: no `/slots` answer → the API's own `sessions.detail`; answer with no rows → “engine reports 0 slots”; grid ships `hidden`. +- `dash.css`: new “engine slot grid” section from existing tokens only (no `--ok/--warn/--fail`). +- `tests/slots_contract.py`: canonicalised the request id (`CANONICAL_TASK_ID_FIELD="id_task"` required; `task_id` captured, never rendered) and made `next_token[0].n_remain` the single remaining path — replacing `llm_monitor`’s `a ?? b` chains. +- `tests/test_api.py`: 9 new tests (markup order + hidden defaults, style-is-a-marker, every printed key resolves against the capture in both directions, no invented speed/sentinel leaks, keyed+patched-in-place+pruning, both empty states, vLLM note-only, capture→`/api/llm/snapshot`→cell-keys end-to-end); `tests/test_llm_slots_contract.py` updated for the canonical/alias rules. + +**Gates:** `bash .agents/validate.sh` exit 0 → `603 passed, 9 skipped`, coverage **97.91 %** (floor 94) · `uv run ruff check src tests agent` All checks passed · `uv run pyright` 0 errors · `node --check` every script ok · `uv run pytest -m live --live` **9 passed** · `uv pip list --outdated` empty (deps untouched). + +**Browser (live, `data/logs/`):** `llm-slots-idle.png`, `llm-slots-one-busy.png`, `llm-slots-two-busy.png` (real ids/tokens/cache share), `llm-slots-no-slots-answer.png`, `llm-slots-vllm-only.png`; DOM-node tagging proved cards survive polls (`survived: true`, numbers changing), console clean, no `NaN`/`undefined`. Dev server restarted (`status: ok`); throwaway :8091/:8092 instances, the fake engine and their temp DBs removed; no fleet/config change (two short gufo completions only, to photograph busy slots). + +**Docs:** `AGENTS.md` — new slot-grid UI-conventions paragraph + testing-policy rewrite of the `/slots` contract (canonical key per fact); `README.md` — `/llm` page tour now covers the grid, sentinels, empty state and the vLLM line, plus the canonical-key note. + +**Deviations:** (1) no per-slot speed cell, despite the task listing “per-slot speed” — the pinned capture publishes no rate key, so rendering it would be a permanent dead cell or an unlabelled page-side Δ (rule 2); decision and reason are in the code comment, AGENTS.md and a test. (2) `tests/slots_contract.py` gained the canonical task-id rule that task 04’s ASSUMPTION requires, so one task-01 test (`neither task_id alias answered`) was replaced by canonical-must-exist + alias-is-optional. + +**Next pending task:** none — this was the last task in `07_llm_monitor_style_pages` (harness may complete the phase). diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__04_slot_grid.a1.md.status b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__04_slot_grid.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__04_slot_grid.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__04_slot_grid.a1.validate b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__04_slot_grid.a1.validate new file mode 100644 index 0000000..9251916 --- /dev/null +++ b/.agents/reports/07_llm_monitor_style_pages/07_llm_monitor_style_pages__04_slot_grid.a1.validate @@ -0,0 +1,48 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 11%] +........................................................................ [ 23%] +........................................................................ [ 35%] +........................................................................ [ 47%] +......sssssssss......................................................... [ 58%] +........................................................................ [ 70%] +........................................................................ [ 82%] +........................................................................ [ 94%] +.................................... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 28 +src/dashboards/alerts.py 166 0 100% +src/dashboards/api.py 193 0 100% +src/dashboards/app.py 107 6 94% 50-51, 56, 58, 60, 127 +src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120 +src/dashboards/config.py 113 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164 +src/dashboards/rates.py 37 1 97% 51 +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/host.py 199 1 99% 147 +src/dashboards/sources/llm.py 150 1 99% 228 +src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 178 2 99% 243-244 +------------------------------------------------------------------ +TOTAL 2158 45 98% +Required test coverage of 94% reached. Total coverage: 97.91% +603 passed, 9 skipped, 1 warning in 51.25s +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/AGENTS.md b/AGENTS.md index 5082904..db3d8b6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ file — plus an overview page that says which of the three you need: | --------- | ------------------------------------------------------------------------------ | ------------------------------------------------- | | `/` | Is anything wrong, and where do I look first? | all three, in one request | | `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent/host | -| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo (llama.cpp family) + vLLM `/metrics` | +| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo `/metrics` + `/slots`, vLLM `/metrics` | | `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) | History is a first-class feature, not a sparkline afterthought: every number worth graphing @@ -96,6 +96,7 @@ page still renders. ``` agent/host_agent.py (each host) gufo / vLLM upsd (NUT) │ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD + │ │ + /slots (gufo) │ ▼ ▼ ▼ collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command ──┐ │ parse + Δcounter/Δt │ @@ -109,8 +110,9 @@ store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ─ Responsibilities, one line each: - `agent/host_agent.py` — stdlib-only (3.10+), reads `/proc`, `/sys/block`, `/sys/class/drm`, - `statvfs`; serves `GET /snapshot` + `GET /info` + `GET /health`. Refuses to bind - off-loopback without a token (exit 2). Counters stay counters: the dashboard computes + `statvfs`; serves `GET /snapshot` + `GET /info` + `GET /schema` + `GET /health`, plus an + HTML status page at `GET /`. Refuses to bind off-loopback without a token (exit 2). + Counters stay counters: the dashboard computes rates so the agent never remembers who polled it last. - `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real env var always wins. Defaults are safe except the sources, which default to unset. @@ -254,11 +256,11 @@ cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets ./scripts/dev-server.sh status # per-source ok/DOWN ./scripts/dev-server.sh logs -uv run pytest # hermetic: 550 passed + 8 live-skipped, no network +uv run pytest # hermetic: 604 passed + 9 live-skipped, no network uv run pytest --cov # coverage gate (97.9%; floor 94%) uv run pytest -m live --live # opt-in: real fleet / engines / upsd uv run ruff check src tests agent && uv run pyright -./scripts/capture-fixtures.sh # re-record ground truth (agents + UPS captures) +./scripts/capture-fixtures.sh # re-record ground truth (agents + UPS + gufo /slots) bash .agents/validate.sh # the phase gate: the four checks above (pytest # --cov --cov-fail-under=94, ruff, pyright, plus a @@ -300,7 +302,21 @@ branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only `test_ups_hero_fields_are_present_on_the_real_batteries` drives it against live upsd. The captures are what make the driver disagreements testable: one of them answers `battery.mfr.date: CPS`, neither exports `battery.temperature` or - `battery.charger.status`. + `battery.charger.status`. `tests/slots_contract.py` is the same device for gufo's `/slots` + rows, which the `/llm` slot grid (phase 07) reads (`assert_slot_shape`, required/optional + field tables, duplicate slot ids, the `next_token[0]` entries): hermetically + from `tests/fixtures/slots_gufo.json` — the verbatim `/slots` reply, idle slots and all, + captured whenever `GUFO_BASE_URL` is set — and live via + `test_gufo_slots_answer_the_pinned_shape`. That fixture exists because its absence is how + the deleted slot table rendered dashes forever: its field names (`slot.state`, + `slot.n_prompt`, `slot.decode_per_second`) were a memory of an upstream, not a capture. + **The contract names one canonical key per fact and the grid renders that one**: the capture + answers `id_task` *and* `task_id` for the same request, so `CANONICAL_TASK_ID_FIELD` decides + and the alias is captured but rendered by nothing — a rename fails a test instead of becoming + a silent `a ?? b` chain in the page. `tests/test_api.py` reads the served `SLOT_FIELDS` table + and resolves every key it names against the capture, in both directions. + A busy slot's `prompt` is the user's in-flight request text, so the capture script blanks + it and the page never reads it. - `live` (`tests/test_live_fleet.py`) re-runs the production code paths against the real fleet, engines, and upsd. Skipped unless `--live`; each test skips when its source is unset, so partial fleet availability is normal. **Live tests must not write** — a battery @@ -320,11 +336,12 @@ Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ` l measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status dots in the header. Every chart names itself for assistive tech: pages pass `ariaLabel` to `LineChart`, which puts `role="img"` + `aria-label` on the `` (an unlabelled plot is -silence, and the legend is a sibling that labels nothing) — `/system` does this for all three -slots, pinned by `tests/test_api.py`. Window chips (5m…30d on `/system` and `/llm`, 1h…90d on -`/ups`; the API accepts 5m…90d). A panel that cannot get data says -why in the panel — the operator should never have to open devtools to learn that a source is -unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`. +silence, and the legend is a sibling that labels nothing). Every `LineChart` on every page has +one — `/system`'s three per-host slots, `/llm`'s four trend plots, `/ups`'s four battery +plots — each pinned by `tests/test_api.py`, so a chart that ships anonymous fails a test. +Window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups`; the API accepts 5m…90d). A +panel that cannot get data says why in the panel — the operator should never have to open +devtools to learn that a source is unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`. The overview is exactly two panels — **Warnings & errors** (`ul.alerts`, one row per `/api/summary` alert: verdict class, source linked to its dashboard via the alert's own @@ -362,8 +379,10 @@ point, `labels: {current, max}` renders the `now / max` foot through the caller' there is no animation at all (`prefers-reduced-motion` holds by construction). `minSpan` is the other half of the same honesty for count metrics: history arrives as rollups, so eight seconds of one queued request averages to 0.14 and an axis fitted to that draws a full-height spike for -a queue nobody was in — `/llm` passes `minSpan: 1` on sessions and queued so its axis is never -finer than one request. +a queue nobody was in. `/llm`'s count rows moved to the phase-07 stats strip (numbers, not +sparks), so no page passes `minSpan` today and `drawSparks()` still plumbs it through — a count +sparkline that comes back must set it, and the component's own contract test keeps the option +honest (`test_sparkline_ships_the_fixed_scale_and_verdict_options`). A battery is `DashCharts.battery(container, {charge, severity, state, label})` — one inline SVG whose **fill *is* the charge**: 1 viewBox unit per percent, clamped to 0–100 and never rounded up @@ -441,24 +460,84 @@ always say whether it was scraped, measured `Δcounter/Δt`, or estimated (`llm_ `setThroughput` idiom; dropping the tag to look cleaner is a regression, not a simplification). **The `/llm` page reads per engine** (phase 04, decided in -`.agents/phases/complete/04_llm_chart_readability/01_chart_set_decision.md`): a row of two -`Dash.hero()` throughput numbers (decode, prompt — deliberately **uncoloured**, because 0 tok/s -on an idle server is healthy and `severity.py` has no throughput row), then a compact `.cells` -list — label left, tabular value right, a `sparkline()` fixed at 160px so rows line up, and a -mono `p95 …` foot under every latency mean (no hover; p95 needs no stored series). Rows are -engine-scoped by the `only` flag: gufo never grows a TTFT mystery, vLLM never a spec-accept -hole. `preemptions` / `device_lost` are **alarm chips that exist only while non-zero** (`.chip.fail`, +`.agents/phases/complete/04_llm_chart_readability/01_chart_set_decision.md`, laid out by phase +07): a row of two `Dash.hero()` throughput numbers (decode, prompt — deliberately **uncoloured**, +because 0 tok/s on an idle server is healthy and `severity.py` has no throughput row) and the +latency rows, each a mean with a mono `p95 …` foot (no hover; p95 needs no stored series) and a +`sparkline()` fixed at 160px so rows line up. Rows are engine-scoped by the `only` flag on both +components: gufo never grows a TTFT mystery, vLLM never a spec-accept hole. + +**The stats strip (`#strip`, phase 07 task 03) is the summary; the panels are the detail, and no +row is printed in both** — the UPS hero row's "one value drawn twice" rule, widened to two +components. One `.card.stat` per *configured* engine between the fleet block and the panels: the +reachability dot, the engine name and the age stamp in the head (same distinction every card +draws — the dot speaks reachability, the numbers never do), then the **KV-cache bar** — the one +cell on the page with a verdict, so the fill and a breach's text take their colour from the +payload's `severity` through `.bar.ok/.warn/.fail` and the ratio→percent conversion the panel's +cell used to make, with `ok` staying quiet ink and a missing gauge an empty bar plus `n/a` — then +`prompt tok/s` / `decode tok/s` with their `Δ` tags, `total tokens` (lifetime, so its +source is the tooltip), the sessions this engine reports, `prefix cache hit`, `spec accept` +(gufo) and `queue time` with its `p95 …` foot (vLLM). `STRIP_ROWS` is the whole table and its +`key` **is** the `cellValue()` key, so every number on it is a cell of `/api/llm/snapshot`: no +field was invented in `sources/llm.py` for this card, and there is deliberately **no peak-ctx +row** (phase assumption). The session row is engine-shaped — vLLM publishes `running`/`waiting` +counts, gufo's is `slots busy`, counted from the `sessions.list` rows `/slots` answered +(`is_processing`, the shape pinned by `tests/slots_contract.py`); when `/slots` did not answer +there are no rows to count, so it falls back to the published `sessions.active` carrying the +API's own `sessions.detail` sentence rather than reporting 0 busy slots from no data (rule 2). +`preemptions` / `device_lost` are **alarm chips that exist only while non-zero** (`.chip.fail`, container emptied and `hidden` at zero — the rule is the presence of an incident, never a graded -threshold); lifetime token counters are a quiet footnote line. **One `/api/llm/history` request -per refresh** carries every chart *and* spark series (`HISTORY_SERIES`; the endpoint answers -unknown names as empty series on purpose), and the two series that addition required -(`llm.tpot_seconds` vLLM-only, `llm.spec_accept_rate` gufo-only) are stored additively in -`sources/llm.py` HISTORY_METRICS. The engine-totals card kept only engines-live/active/queued -and **hides below two engines** — a cross-engine sum of one engine duplicates that engine's own -panel, and the summed-KV bar went because `fleet_totals` carries no `severity` to render. The +threshold) and they live on this card, not in a permanent cell; the lifetime token counter left +the panel's footnote line for the strip's row. **One `/api/llm/history` request per refresh** +carries every chart *and* spark series (`HISTORY_SERIES` = the four chart series + +`SPARK_CELLS`; the endpoint answers unknown names as empty series on purpose), and the two series +phase 04 added (`llm.tpot_seconds` vLLM-only, `llm.spec_accept_rate` gufo-only) stay stored +additively in `sources/llm.py` HISTORY_METRICS. Above the strip sits **the fleet block** +(`#fleet`, phase 07: "I'm missing the aggregate throughput at the top") — `ALL ENGINES`, a +reachability dot, an +`N of M engines` tag, the note *sums only engines answering now*, then running / waiting / +prompt tok/s / decode tok/s / total tokens. `FLEET_FIELDS` in `page-llm.js` is a list of +`fleet_totals()` keys and nothing else: the page sums no number of its own, a dead engine is +dropped from every total presented as current (its own stale-labelled panel still shows its +last good values), and the tag counts the answering engines, names the missing one in its +tooltip and takes the shared `.stale` dim while the sums are partial. It **hides below two +engines** — a cross-engine sum of one engine duplicates that engine's own panel — and it +carries **no severity colour anywhere**: `fleet_totals` has no `severity` to render (the +summed-KV bar is gone for that reason) and `severity.py` rates no throughput or queue depth, +so the numbers stay ink and only the dot speaks reachability. `fleet.seams` — the API's own +note on how the two engines count prompt tokens differently — is the prompt sum's tooltip +rather than another line of ink. The four trend charts stay but at height 110 *below* the panels, each with an `ariaLabel`. Panels -sit at their own height (`#engines` carries `.items-start`): a 5-row gufo panel next to a 7-row -vLLM one otherwise grows a blank bottom, and a blank panel reads as "no data". +sit at their own height (`#engines` carries `.items-start`): a two-hero gufo card next to a +latency-row vLLM one otherwise grows a blank bottom, and a blank panel reads as "no data". An +engine that leaves the payload leaves both components (`pruneEngines`), never a stale card in +either grid. + +**The slot grid (`llm_monitor`'s, phase 07 task 04) belongs to the gufo panel, under its +`sessions (GET /slots)` label.** One card per row of `sessions.list`, **keyed by the row's own +`id`** in `state.slots`: built once by `slotCard()`, patched in place afterwards, and reordered +by `appendChild` of the existing node so DOM order stays slot order — rebuilding a busy slot's +card every 2 s poll re-announces the whole card to assistive tech for a difference of a few +tokens. `SLOT_FIELDS` is the whole table and each cell names **exactly one** `/slots` key +(`path: 'next_token'` reads `next_token[0]`, where the capture puts `n_remain`); `slotValue()` +is the only place a row is indexed, so `tests/test_api.py` can resolve every printed field +against `tests/fixtures/slots_gufo.json` and a rename fails there rather than printing dashes +or a plausible wrong number. No per-slot speed cell: the capture publishes no per-slot rate key +and a page-side Δn_decoded/Δt would be a measurement with no estimator, no `Δ` label and +no `source` string (rule 2) — the engine's rate lives on its strip card and panel. llama.cpp's +`-1` sentinels print a word or nothing: `id_task: -1` is **`idle`**, `n_remain: -1` is **`n/a`**, +never a negative countdown and never the `0` that reads as one token from done. The busy marker +is a **border, a glyph (`○`/`●`) and the word `processing`** in the brand `--accent`, never an +`ok|warn|fail` token — a slot generating tokens is the job (phase 06's GPU reasoning), so +`severity.py` rates no slot and the stylesheet section contains no verdict token. The empty +state is a sentence, never a card of nothing, and it distinguishes the two cases: no `/slots` +answer prints the API's own `sessions.detail`, an answer with no rows prints *engine reports 0 +slots*; `sessions.source` is the gate. The note under the grid stays because a **queued +(deferred) request is not in `/slots`** — the waiting row counts those. vLLM's panel gets one +line (*exposes no per-slot JSON*) with `sessions.detail` as its tooltip and **no grid**: a +component of permanent `n/a`s for a server that cannot answer is the dead-cell anti-pattern. +The gate is the engine name (`SLOT_ENGINES`), not row count, so an engine that has not answered +yet still shows its label and its honest empty sentence. **Charts belong to the host they chart.** `/system` has two sections — the host panels and the transition events — and nothing else: the fleet-wide chart stack, the nine-figure fleet-totals diff --git a/README.md b/README.md index 0aa3ccd..689b475 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ Three dashboards, one domain, one process, one SQLite file: | ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- | | `/` | Is anything wrong, and which dashboard do I need? | fleet + alerts in one request (`/api/summary`) | | `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host | -| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) + vLLM `/metrics`, read-only | +| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics` — read-only | | `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) | No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at @@ -120,6 +120,7 @@ unreachable host writes its failure message into the event table and `/api/event ``` agent/host_agent.py (each host) gufo / vLLM upsd (NUT) │ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD + │ │ + /slots (gufo) │ ▼ ▼ ▼ src/dashboards/collectors.py ────────────────────────────────────────────┐ │ parse + Δcounter/Δt (rates.py) │ @@ -193,16 +194,45 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ── processes — folds into a collapsed "More detail" disclosure. Each panel pulls its series in **one** `/api/system/history` request (`hosts=&metrics=…&range=…`) every refresh, stack open or not, and a host that stops - reporting takes its panel, its Charts toggle and its charts off the page. The LLM page (`/llm`) is per engine too: two big + reporting takes its panel, its Charts toggle and its charts off the page. `/llm` opens with + the **fleet**: one block above the panels — `ALL ENGINES`, an `N of M engines` tag, the note + *sums only engines answering now*, then **running / waiting / prompt tok/s / decode tok/s / + total tokens** — and every number on it is `fleet_totals()`, so the page sums nothing itself. + A dead engine keeps its last-good numbers in its own (stale-labelled) panel but is dropped + from a total presented as current: the tag counts what answered, names the missing engine in + its tooltip and dims while the sums are partial. Nothing on the block is coloured — + `severity.py` rates no throughput and no queue depth, because 0 tok/s on an idle fleet is + healthy — the dot is reachability, and `fleet.seams` (how the two engines count prompt + tokens differently) is a tooltip on the prompt sum rather than another line of ink. Below + two configured engines the block is hidden: a cross-engine sum of one engine duplicates that + engine's panel. Below the fleet block sits the **stats strip**: one card per configured + engine — reachability dot, name, age stamp, the **KV-cache bar** (the one rated cell: the + fill and a breach's text take their colour from the payload's `severity` and nothing else), + then `prompt tok/s` and `decode tok/s` with their provenance tag (`Δ10.0s`, scraped, + estimated), `total tokens`, the sessions this engine reports (`slots busy` counted from + gufo's `/slots`, or `running`/`waiting` for vLLM, plus `waiting`), `prefix cache hit`, + `spec accept` (gufo), `queue time` with its `p95 …` foot (vLLM) — and red alarm chips for + preemptions / GPU-context losses that exist only while non-zero. Then the engine panels + hold the detail the strip does not print: two big throughput numbers (decode, prompt — uncoloured, because 0 tok/s on an idle server is - healthy) with their provenance tag (`Δ10.0s`, scraped, estimated), then a compact list of - metric rows — sessions, queue, KV pressure (the only rated cell), prefix/spec percentages, - and TTFT/TPOT/queue-time means with a `p95 …` foot line — each carrying its own sparkline - over the selected window. Preemptions and GPU-context losses render only as red alarm chips - while non-zero; the token counter is a footnote. The four trend charts sit below the panels + healthy) with their provenance tag, and the latency rows (TTFT, TPOT) each with a `p95 …` + foot line and its own sparkline over the selected window. Under the gufo panel's + **`sessions (GET /slots)`** label sits the **slot grid**: one card per parallel slot, keyed by + the slot's own id, created once and patched in place so a busy slot never flickers — request + id, prompt tokens (with `processed …` while prefill runs), cache hit and its share of the + prompt, generated tokens, remaining, context, and the model beside the slot number. A busy + card is a border, a glyph and the word `processing`, never a green/amber/red verdict (a slot + generating tokens is the job). Every cell reads exactly one key of the captured `/slots` + shape — no `a ?? b` fallback chains — so a field rename fails a contract test instead of + printing dashes, and `-1` sentinels print `idle`/`n/a`, never a negative countdown. When + `/slots` does not answer, the panel says so with the API's own sentence rather than rendering + a card of nothing, and vLLM's panel carries one line (*exposes no per-slot JSON*) and no grid. + No row appears in both places — + one value, one home — and there is no peak-context cell, because this page prints a number + the upstream published or `n/a`. The four trend charts sit below the panels at half height, and one `/api/llm/history` request per refresh feeds all of it. With a - single engine configured the totals card hides and the page is one clean half — never a red - panel for a server you do not run. Each UPS card (`/ups`) opens with its battery glyph and + single engine configured the page is one clean card in the strip and one clean panel — + never a red half for a server you do not run. Each UPS card (`/ups`) opens with its battery glyph and three big numbers — **charge %**, **load %**, **runtime left** — coloured from the card's `severity` dict, with the electrical readings (line in, output, battery volts and amps, battery °C) kept below them in smaller muted type because they are what you read *while* the @@ -217,8 +247,10 @@ uv run pytest --cov # coverage gate set -a; . ./.env.dev; set +a # the live suite reads the same env as the app uv run pytest -m live --live # optional: real fleet / engines / upsd uv run ruff check src tests agent && uv run pyright # lint + types -./scripts/capture-fixtures.sh # re-record ground truth: host agents + UPS replies +./scripts/capture-fixtures.sh # re-record ground truth: host agents, UPS replies, gufo /slots ./scripts/capture-fixtures.sh --with-llm # ... and the engines' /metrics while they are up +bash .agents/validate.sh # all of the above + `node --check` on every page + # script + repo hygiene, with the 94% coverage floor ``` Test philosophy: **real captures as fixtures, mocks for behaviour.** Anything that must @@ -242,7 +274,17 @@ verbatim as `tests/fixtures/nut_.vars` by `./scripts/capture-fixtures.sh`, device serials scrubbed — and again against live upsd. The captures earn their place: the CyberPower answers `battery.mfr.date: CPS`, the APC's `battery.date` is a factory placeholder, and **neither** publishes `battery.temperature` or `battery.charger.status`, so -a hand-written variable table would have tested nobody's memory. +a hand-written variable table would have tested nobody's memory. `tests/slots_contract.py` +pins gufo's `/slots` rows the same way — hermetically from +`tests/fixtures/slots_gufo.json` (the verbatim `/slots` reply, captured whenever +`GUFO_BASE_URL` is set; the idle-slot rows are the shape the grid must handle) and again +against the running engine. It exists because that table was once guessed: the old slot +table read field names the server never published and rendered dashes forever. It also picks +the **canonical name of each fact** — the live server answers `id_task` *and* `task_id` for +one request — and the page renders only the canonical key, so a rename fails a test instead of +quietly becoming a fallback chain. A busy +slot's `prompt` is the user's in-flight request text, so the capture script blanks it and +the page never reads it. ## Battery-test buttons diff --git a/scripts/capture-fixtures.sh b/scripts/capture-fixtures.sh index 04d38ae..6b3a544 100755 --- a/scripts/capture-fixtures.sh +++ b/scripts/capture-fixtures.sh @@ -3,6 +3,7 @@ # # ./scripts/capture-fixtures.sh # needs the tunnels up: scripts/dev-tunnels.sh start # ./scripts/capture-fixtures.sh --with-llm # also scrape the live engines' /metrics +# # (gufo /slots is captured without the flag) # # Fixtures are verbatim, not hand-written: a fixture you typed is a test of your memory. # When a host's kernel or an engine's metric names change, re-run this and read the diff. @@ -105,6 +106,30 @@ else echo " (skipped UPS captures — NUT_HOST unset; see .env.dev.example)" fi +# --- gufo /slots (per-slot rows) --------------------------------------------- +# `tests/slots_contract.py` pins the shape of the `/llm` slot grid (task 04) against +# `tests/fixtures/slots_gufo.json`, so this capture is NOT behind `--with-llm`: it is one +# small JSON document and the hermetic contract test needs it on disk. `/v1/slots` is the +# documented alias, tried second — the same order `collectors.fetch_engine` and +# `llm_monitor`'s passthrough use, so a server that only exposes the alias is still +# capturable. Skipped when GUFO_BASE_URL is unset, like the UPS captures. An idle engine +# still answers one row per parallel slot; that idle shape is what the grid must handle. +if [ -n "${GUFO_BASE_URL:-}" ]; then + gufo_base="${GUFO_BASE_URL%/}" + slots_tmp="$FIX/.slots_gufo.json.tmp" + if curl -fsS --max-time 8 "${gufo_base}/slots" -o "$slots_tmp" || + curl -fsS --max-time 8 "${gufo_base}/v1/slots" -o "$slots_tmp"; then + mv "$slots_tmp" "$FIX/slots_gufo.json" + echo " ✓ slots_gufo.json ($(wc -c <"$FIX/slots_gufo.json") bytes)" + else + rm -f "$slots_tmp" # never overwrite a good fixture with a dead engine's empty file + echo " ✗ slots_gufo.json — gufo answered neither /slots nor /v1/slots at $gufo_base" + failed=1 + fi +else + echo " (skipped gufo /slots capture — GUFO_BASE_URL unset; see .env.dev.example)" +fi + if [ "${1:-}" = "--with-llm" ]; then if [ ${#ENGINES[@]} -eq 0 ]; then echo " ! no GUFO_BASE_URL / VLLM_BASE_URL set (see .env.dev) — nothing to capture" @@ -135,7 +160,10 @@ fi # FQDN a host thinks it has, and a UPS's `ups.serial`/`device.serial` (which identifies the # physical unit, and is echoed back in the card's variable dump). They are replaced # deterministically so re-capturing does not churn the fixture, and the tests keep the same -# shapes to assert against. +# shapes to assert against. A fourth belongs only to the /slots capture: a busy slot's +# `prompt` field is *the user's in-flight request text* — personal data the slot grid never +# renders — so any non-empty prompt value is blanked, keeping the key (the shape) without +# the content. python3 - "$FIX" <<'PY' import pathlib import re @@ -167,6 +195,9 @@ for path in sorted(fix.iterdir()): # "hostname": "gw.lab.example.com" -> "gw": the dashboard's host key is the target name # from HOST_TARGETS, never whatever the box calls itself text = re.sub(r'("hostname": "[^"]*?)\.[^"]*(")', r"\1\2", text) + if path.name.startswith("slots_"): + # only non-empty values are replaced, so an idle capture does not churn + text = re.sub(r'("prompt"\s*:\s*)"(?:[^"\\]|\\.)+"', r'\1""', text) def scrub_serial(match: re.Match[str]) -> str: value = match.group(2) diff --git a/src/dashboards/web/pages/llm.html b/src/dashboards/web/pages/llm.html index 552f014..860589f 100644 --- a/src/dashboards/web/pages/llm.html +++ b/src/dashboards/web/pages/llm.html @@ -36,27 +36,71 @@
- -