diff --git a/.agents/PLAN.md b/.agents/PLAN.md new file mode 100644 index 0000000..9a2ec1e --- /dev/null +++ b/.agents/PLAN.md @@ -0,0 +1,129 @@ +# Homelab Dashboards — Master Plan + +> Master design document for a phased-execution project. The phase roadmap below was +> derived from `TODO.md` on 2026-10-16; each phase overview and task file cites its source +> lines in that file. The **LOCKED DECISIONS** below are binding: no phase may introduce +> technology outside them without explicit user permission. +> +> `AGENTS.md` (repo root) is the sibling document: the invariants, the fleet facts, and the +> known gaps. It is not superseded by this file — read both. + +## Assumptions & Design Principles + +- **The dashboard's job is a fast, honest glance.** Every TODO item is a legibility + complaint, not a feature request: the same truth with less ink and clearer colour. +- **One colour language.** Green/amber/red means exactly one thing across all four pages, + computed server-side in Python so it is unit-tested and cannot drift per page. +- **Fewer numbers, larger.** Removing a metric from a page is a feature, but only when the + history stays in the DB. Pages stop plotting what the operator stopped asking for; the + stored series are kept so the decision is reversible. +- **The existing pages are payload-driven and stay that way.** No page scrapes HTML, no page + grows its own business logic. If a card needs a judgement ("is 5.3 years old bad?"), the + API supplies it. +- **Charts stay hand-rolled SVG.** `llm_monitor`'s per-cell sparklines are the reference for + "useful and readable"; they are inline SVG with no library, which is exactly the house + style, so they port rather than get replaced. +- **Every UI change is verified in a browser against live data** (`AGENTS.md` definition of + done) — a screenshot in `data/logs/`, not a claim. + +## Architectural Anchors + +| COMPONENT | DECISION | RATIONALE | STATUS | +|-----------|----------|-----------|--------| +| Backend | FastAPI, single process, one domain, four routes (`/`, `/system`, `/llm`, `/ups`) | Inherited from the existing code | LOCKED | +| Frontend | Vanilla JS + one stylesheet. **No framework, no CDN, no build step, nothing fetched at runtime** | The engines live on a network with no outbound access; a CDN reference is a dead request | LOCKED | +| Charts | Hand-rolled SVG in `web/static/charts.js` | Same reason as above; `llm_monitor` proved it works for a year | LOCKED | +| History | SQLite (`sample(source,host,metric,res,ts)` + `event`), raw → 1m → 1h rollups | Inherited; bounded, no TSDB to operate | LOCKED | +| Writes | Read-only except `POST /api/ups/command`, gated four times | Invariant 1 of `AGENTS.md` | LOCKED | +| API shape | **Additive only**: existing `/api/*` fields keep their names and meaning | The four pages, the live tests, and future consumers depend on them | LOCKED (ratified 2026-10-16) | +| Severity | New `src/dashboards/severity.py` is the only place thresholds live; API responses carry `severity` per value; CSS/JS only render the class | Testable once, identical on every page; JS business logic is untested by definition | LOCKED (ratified 2026-10-16) | +| JS tests | **None added.** Coverage gate (>90%) applies to new/modified Python; UI is proven by payload-contract tests plus browser screenshots | Adding a JS toolchain would violate the no-build-step anchor for no extra confidence in the colour logic | LOCKED (ratified 2026-10-16) | +| Dropped visuals | Per-core CPU charts and shared multi-host charts are removed from `/system`; the underlying series stay in SQLite | Explicit user request; reversible because nothing is deleted from history | LOCKED (ratified 2026-10-16) | +| Toolchain | Python 3.14, pytest, ruff, pyright, `uv` | Inherited; the lock resolves for 3.14 only | LOCKED | +| Planning tree | `.agents/` is tracked and committed; `.agents/phase-sessions/` and `.agents/pipeline.log` are ignored | The roadmap is project history, not scratch | LOCKED (ratified 2026-10-16) | + +## High-Level Architecture + +``` +agent/host_agent.py (each host) gufo / vLLM upsd (NUT, port 3051) + │ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD + ▼ ▼ ▼ +collectors.py ─── fetch, one UpstreamError type ────────────────────────────────┐ + │ parse │ + ▼ ▼ +sources/host.py · sources/llm.py · sources/nut.py poller.py (task per source) + │ Result(payload, samples, events) ◄── NEW: severity.py classifies values + ▼ ▼ +store.py ── SQLite (WAL): sample + event ──────────────────────────► api.py ──► web/ + /api/summary is the + overview's one call +``` + +Pages and their scripts (`src/dashboards/web/`): + +| Page | HTML | Script | Charts it builds today | +|------|------|--------|------------------------| +| `/` | `pages/index.html` | `page-index.js` | none (three summary cards + fleet table + events) | +| `/system` | `pages/system.html` | `page-system.js` | `#chart-cpu`, `#chart-mem`, `#chart-disk`, `#chart-net`, `#chart-gpu`, `#chart-temp`, `#chart-detail` — all shared, multi-host | +| `/llm` | `pages/llm.html` | `page-llm.js` | `#chart-decode`, `#chart-prompt`, `#chart-kv`, `#chart-queue` | +| `/ups` | `pages/ups.html` | `page-ups.js` | battery charge, runtime estimate, voltages, load | + +Shared helpers: `app.js` (nav, poll loop, `fetchJSON`, formatting), `charts.js` +(`DashCharts.LineChart`, escape/axis helpers), `dash.css` (tokens, cards, bars). + +## Data Model + +- `sample(source, host, metric, res, ts)` — `source` ∈ {host, llm, ups}; `res` ∈ + {raw, 1m, 1h}; `metric` is dotted with `@stream` for per-device series + (`cpu.usage_pct@core7`, `disk.used_pct@/srv`, `gpu.busy_pct@0`). **The metric namespace is + a public interface**: phases may stop plotting a series, they may not rename or stop + storing it. +- `event(ts, source, kind, severity, message)` — transitions only, which is what the + overview's alert list will reuse. +- Host payload (`/api/system/snapshot`, per host): `cpu.{usage_pct,temp_c,load1}`, + `memory.used_pct`, `disks.{worst_used_pct,worst_mount,util_pct}`, + `network.{rx_bytes_per_s,tx_bytes_per_s,interfaces[]}`, + `gpus[].{busy_pct,vram_used_bytes,vram_total_bytes,vram_used_pct,mem_used_pct,temp_c}`, + `temperatures.{cpu_c,max_c}`, `processes`, `agent.errors`. Everything the per-host cards + ask for **already exists** — phase 03 tests the contract rather than inventing fields. +- UPS payload (`/api/ups/snapshot`): per card `status`, `states[]`, `on_battery`, `numbers` + (`charge_pct`, `load_pct`, `battery_runtime`, `battery_voltage`, `line_voltage`, + `battery_temperature`), `health.{overall,severity,checks[],counts}`, `commands[]` + (each with `enabled`, `reason`, `confirm_text`, `risk`), `simulated` at the top level. + Real fleet today: `apc` (APC Back-UPS RS 1500MS) and `cyberpower` (CyberPower + PR1500LCDRT2U) on driveripper's upsd at port **3051**, forwarded to `127.0.0.1:3494` in + dev. Neither driver reports a battery temperature; `battery.mfr.date` is the only date + that means anything. +- `/api/summary` returns `{ts, system:{enabled,fleet,hosts[],states}, llm:{...}, ups:{...}, + recent_events[]}` — phase 02 adds `alerts` and the overview page shrinks to fleet+alerts. + +## Validation / Verification Workflow + +Per task, in order: + +1. `uv run pytest` — hermetic, no network; must stay green (baseline at conversion: + **258 passed, 6 skipped, 94% coverage**). +2. `uv run pytest --cov` — coverage on new/modified Python **>90%**; overall must not drop + below 94%. +3. `uv run ruff check src tests agent && uv run pyright` — clean. +4. `uv pip list --outdated` — empty whenever dependencies moved. +5. `uv run pytest -m live --live` (with `.env.dev` sourced) — the opt-in suite that drives + the production code paths against the real fleet, both engines, and real upsd. Baseline: + **6 passed**. UI phases must not break it. +6. Browser verification of every touched page with live data, screenshot saved to + `data/logs/` (`./scripts/dev-server.sh restart` keeps http://127.0.0.1:8090 up). +7. Commit per task with `--no-gpg-sign` and the `LLM-Generated: true` trailer. + +## Phase Roadmap + +| # | Phase | TODO source | Objective (one line) | +|---|-------|-------------|----------------------| +| 01 | `01_severity_language` | `TODO.md` L5, L7 (colour-coding clauses) | One server-side threshold table + severity on every value the pages colour, plus the CSS/JS vocabulary to render it | +| 02 | `02_overview_fleet_and_alerts` | `TODO.md` L3 | Overview is fleet + warnings/errors and nothing else | +| 03 | `03_system_per_host_panels` | `TODO.md` L4, L5 | `/system` is per-host hero cards (cpu/mem/temp/disk/gpu/net, no per-core) with per-host charts, no shared charts | +| 04 | `04_llm_chart_readability` | `TODO.md` L6 | LLM page readable again: per-cell sparklines and percentiles in the `llm_monitor` idiom, big colour-coded rates | +| 05 | `05_ups_hero_battery_glyph` | `TODO.md` L7 | Charge / load / runtime big and colour-coded, with an SVG battery that shows charge | + +There is deliberately **no foundation phase**: pytest, ruff and pyright are configured, the +suite is green, and the project launches, which are the only conditions that would have +required one. diff --git a/.agents/phases/complete/.gitkeep b/.agents/phases/complete/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/.agents/phases/todo/01_severity_language/00_phase.md b/.agents/phases/todo/01_severity_language/00_phase.md new file mode 100644 index 0000000..8cced0c --- /dev/null +++ b/.agents/phases/todo/01_severity_language/00_phase.md @@ -0,0 +1,45 @@ +# Phase 01 — Severity language + +**Source:** `TODO.md` L5, L7 — "proper green (good), yellow (warn), red (danger) color coding" / "charge, load, and runtime left to be big, obvious, and color coded" +**Story:** n/a +**Context:** `.agents/PLAN.md` → *Architectural Anchors* (severity computed in Python, JS renders classes), `src/dashboards/sources/host.py`, `src/dashboards/sources/nut.py`, `src/dashboards/api.py`, `src/dashboards/web/static/dash.css`, `src/dashboards/web/static/app.js` + +## Objective + +Introduce one place that decides whether a number is fine, worrying, or bad, and put that +verdict on the values the API already returns. Phases 02–05 all want the same colour +language; doing it here means they render a class instead of each inventing its own +thresholds. + +Deliverable: `severity` next to the value in the payloads the pages colour, and CSS/JS +helpers that make a "hero number" one call. + +## Dependencies + +— (none) + +## Tasks + +1. `01_severity_module.md` — `src/dashboards/severity.py`: the threshold table and the classifier, unit-tested. +2. `02_severity_in_payloads.md` — attach `severity` to host and UPS values and to the `/api/summary` rows; API tests. +3. `03_hero_number_style.md` — a `.hero` big-number style on the existing `.ok/.warn/.fail` colours, plus `Dash.hero()` and `Dash.severityFrom(payload)` so pages read the verdict instead of passing thresholds. + +## Testing & Quality + +- Unit: every threshold boundary (below warn, exactly at warn, between, exactly at danger, + above), `None` input, and the "unknown is not ok" rule — a value the upstream does not + publish must get `severity: null`, never a green pass. +- API: `/api/system/snapshot`, `/api/ups/snapshot`, `/api/summary` carry `severity` where + the spec says, and **no existing field changes name or meaning** (additive-only anchor). +- Coverage: **>90%** on new/modified code. +- Live: `uv run pytest -m live --live` must stay at 6 passed. + +## Completion Criteria + +- [ ] `uv run python -c "from dashboards import severity; print(severity.classify('cpu.usage_pct', 95))"` → `fail`; `60` → `ok`; `None` → `None` +- [ ] `curl -s localhost:8090/api/ups/snapshot | python3 -m json.tool` shows `severity` beside `charge_pct`, `load_pct`, `battery_runtime` +- [ ] no new colour vocabulary: classes stay `.ok` / `.warn` / `.fail` as in `dash.css` +- [ ] `uv run pytest` green; coverage ≥94% overall, >90% on `severity.py` +- [ ] `uv run ruff check src tests agent && uv run pyright` clean +- [ ] Existing pages render unchanged (severity is additive; nothing consumes it yet) — one screenshot in `data/logs/` +- [ ] no behavior change in completed phases diff --git a/.agents/phases/todo/01_severity_language/01_severity_module.md b/.agents/phases/todo/01_severity_language/01_severity_module.md new file mode 100644 index 0000000..37f7225 --- /dev/null +++ b/.agents/phases/todo/01_severity_language/01_severity_module.md @@ -0,0 +1,61 @@ +# Task 01 — Severity module + +**Phase:** `01_severity_language` · **Source:** `TODO.md:5` — "proper green (good), yellow (warn), red (danger) color coding" and `TODO.md:7` — "charge, load, and runtime left to be big, obvious, and color coded" +**Story:** n/a + +## Objective + +Move the good/worrying/bad decision out of the browser and into one Python module, so the +four pages cannot drift into four different opinions about what red means. + +Today the thresholds live in call sites: `page-system.js:129` passes `75, 90`, +`page-system.js:139` and `:144` pass `80, 92`, `page-index.js:89` passes `80, 92`, +`page-llm.js:195` and `:307` pass `75, 90`, and `Dash.severityFor` (`app.js:105`) defaults +to `75/90`. That is the drift this task ends. + +## Work + +1. `src/dashboards/severity.py` (new) — + - `THRESHOLDS: dict[str, tuple[float, float]]` keyed by metric name, with the ratified + defaults: `cpu.usage_pct` 75/90, `mem.used_pct` 80/92, `disk.used_pct` 80/90, + `cpu.temp_c` 75/90, `gpu.busy_pct` 85/95, `gpu.mem_used_pct` 85/95, + `llm.kv_cache_usage_pct` 75/90, `ups.charge_pct` 50/20, `ups.battery_runtime` 600/300, + `ups.load_pct` 70/90. + - `INVERTED: set[str]` — metrics where **lower is worse** (`ups.charge_pct`, + `ups.battery_runtime`), so callers cannot get the direction wrong. + - `classify(metric: str, value: float | None) -> str | None` returning `ok`, `warn`, + `fail`, or `None`. `None` in → `None` out, always. + - `attach(metric: str, value: float | None) -> dict` returning + `{"value": value, "severity": classify(...)}` for the payload shapes that are plain + numbers, and `with_severity(metric, cell)` for payloads that already are + `{"value":…, "source":…, "derived":…}` (LLM metrics) — additive, never a rename. + - Module docstring stating the two rules: unknown is **not** ok (it is `None`, the page + renders `n/a` with no colour), and a threshold exists nowhere else in the app. Ties to + the `AGENTS.md` "never invent a number" rule. +2. `tests/test_severity.py` (new) — + - boundaries inclusive on the warn edge (`75 → warn`, `74.9 → ok`), danger edge + (`90 → fail`), every metric in the table, both directions for the inverted ones, + unknown metric name → `None` (no exception), `None` and `float("nan")` → `None`. + - a table sanity test asserting each entry's bounds are ordered correctly for its + direction, so a future edit cannot silently invert a metric. + - ASSUMPTION: the numbers above were confirmed in chat on 2026-10-16; they match what + the call sites already pass, except the two UPS metrics which are new. + - ASSUMPTION: the vocabulary is **`ok|warn|fail`**, not `ok|warn|danger` — it is what + `dash.css` (`.bar.ok`, `.dot.warn`, `.chip.fail`, `ul.checks li.fail`) and + `sources/nut.py`'s `health.checks[].status` already use. The TODO's word "danger" maps + to `fail`; do not introduce a synonym for the same colour. + - ASSUMPTION: `sources/nut.py`'s existing `health.checks[].status` values are left + untouched (additive-only anchor). `severity` is a new field on *values*, computed by + this module; `nut.assess()` keeps its own judgement about *conditions*. + +## Testing & Quality + +- Unit only, no I/O, no fixtures needed. +- Coverage: **>90%** on `severity.py`. + +## Completion Criteria + +- [ ] `uv run pytest tests/test_severity.py -v` green +- [ ] `uv run python -c "from dashboards.severity import classify as c; print(c('cpu.usage_pct',95), c('cpu.usage_pct',60), c('cpu.usage_pct',None), c('ups.charge_pct',15))"` → `fail ok None fail` +- [ ] `uv run pytest` green overall; `ruff` + `pyright` clean +- [ ] Nothing else changed yet — this task adds a leaf module and its tests diff --git a/.agents/phases/todo/01_severity_language/02_severity_in_payloads.md b/.agents/phases/todo/01_severity_language/02_severity_in_payloads.md new file mode 100644 index 0000000..1f53901 --- /dev/null +++ b/.agents/phases/todo/01_severity_language/02_severity_in_payloads.md @@ -0,0 +1,62 @@ +# Task 02 — Severity in the payloads + +**Phase:** `01_severity_language` · **Source:** `TODO.md:5` — "proper green (good), yellow (warn), red (danger) color coding" and `TODO.md:7` — "big, obvious, and color coded" +**Story:** n/a + +## Objective + +Put the verdict where the pages can read it: every value that will be colour-coded travels +with a `severity`, computed server-side. + +## Work + +1. `src/dashboards/sources/host.py` — in `parse_host_snapshot()`'s payload, add a + `"severity"` sub-dict to the sections that have rated values, leaving every existing key + exactly as it is: + - `cpu.severity = {"usage_pct": …, "temp_c": …}` + - `memory.severity = {"used_pct": …}` + - `disks.severity = {"worst_used_pct": …}` + - each row of `gpus[]` gets `"severity": {"busy_pct": …, "mem_used_pct": …}` + Use `severity.classify(metric, value)`; import it as `from ..severity import classify` + (or `from .. import severity` if that reads better against the existing imports). +2. `src/dashboards/sources/nut.py` — in `payload()`, add + `"severity": {"charge_pct": …, "load_pct": …, "battery_runtime": …}` to each UPS card. + Do not touch `health`, `checks[].status`, or `numbers` — the checks answer "is this + condition satisfied", severity answers "how bad is this number", and both stay. +3. `src/dashboards/sources/llm.py` — for the metrics that have thresholds + (`kv_cache_usage`-family and any other entry in `THRESHOLDS`), attach `severity` inside + the existing `{"value":…, "source":…, "derived":…}` cell via `severity.with_severity`, + which must leave `value`/`source`/`derived` untouched. +4. `src/dashboards/api.py` — in `/api/summary`, add a `severity` key to each host row + (`cpu_usage_pct`, `mem_used_pct`, `disk_worst_used_pct`, `temp_c`), each UPS row + (`charge_pct`, `load_pct`, `runtime_seconds`), and reuse the section severities for the + fleet table. `/api/health` and `/api/config` are unchanged. +5. Tests — extend `tests/test_host_source.py`, `tests/test_nut.py`, `tests/test_llm_source.py` + and `tests/test_api.py`: + - a fixture host whose values straddle thresholds yields the expected severity per field + (drive it from the real `tests/fixtures/agent_driveripper.json`, do not hand-copy a + snapshot); + - the APC fixture/UPS reading with `battery.charge=100` → `ok`, and the same reading with + charge forced to 15 → `fail`; + - a value the upstream omits (`battery_temperature` is missing on both real UPSes) gets + `severity: null`, and the existing `unknown` health check still says `unknown`; + - `/api/summary` still contains every key it had before — assert the shape did not shrink + (additive-only anchor), then assert the new `severity` keys are present. + - ASSUMPTION: severity is attached as a sibling `"severity"` dict rather than by + restructuring `cpu.usage_pct` into `cpu.usage_pct.value`, because the existing + pages, `tests/test_api.py`, and the live tests read the flat form. + +## Testing & Quality + +- Unit + API tests above; run the full suite (`uv run pytest`). +- Coverage: **>90%** on the changed functions in `sources/host.py`, `sources/nut.py`, + `sources/llm.py`, `api.py`. +- `uv run pytest -m live --live` against the real fleet + real upsd must stay green — this + is the first task whose output can be checked against actual numbers. + +## Completion Criteria + +- [ ] `curl -s localhost:8090/api/system/snapshot | python3 -c "import json,sys; d=json.load(sys.stdin); print(d['hosts'][0]['cpu']['severity'], d['hosts'][0]['memory']['severity'])"` prints a verdict per rated field +- [ ] `curl -s localhost:8090/api/ups/snapshot` shows `severity` for charge / load / runtime on both `apc` and `cyberpower`, and `battery_temperature` is `null` +- [ ] no existing key was renamed or removed in any payload (diff the JSON keys before/after) +- [ ] `uv run pytest` green, coverage not below 94%, `ruff` + `pyright` clean diff --git a/.agents/phases/todo/01_severity_language/03_hero_number_style.md b/.agents/phases/todo/01_severity_language/03_hero_number_style.md new file mode 100644 index 0000000..bcfe2af --- /dev/null +++ b/.agents/phases/todo/01_severity_language/03_hero_number_style.md @@ -0,0 +1,54 @@ +# Task 03 — Hero-number style and the JS side of severity + +**Phase:** `01_severity_language` · **Source:** `TODO.md:5,7` — "Larger numbers" / "big, obvious, and color coded" +**Story:** n/a + +## Objective + +Give the pages one way to render a big, colour-coded number, reading the verdict from the +payload instead of comparing thresholds in JavaScript. + +## Work + +1. `src/dashboards/web/static/dash.css` — add a `.hero` block built entirely from the + existing tokens (`--ok`, `--ok-soft`, `--warn`, `--warn-soft`, `--fail`, `--fail-soft`, + `--text`, `--text-muted`, `--text-faint`): + - `.hero` (label + value + unit), `.hero .value` at a display size (roughly 2.4rem, tabular + numerals so digits do not jitter between polls), `.hero .unit` small and muted, + `.hero .sub` for the `Δ10.0s` / "at current load" footnotes; + - `.hero.ok .value { color: var(--ok); }` and the `.warn` / `.fail` equivalents — **reuse + the `ok|warn|fail` classes the stylesheet already uses** (`.bar.ok`, `.dot.warn`, + `.chip.fail`), do not add a parallel `.sev-*` family; + - a `.hero-bar` variant that pairs the number with the existing `.bar` element, since the + per-host cards (phase 03) and the UPS hero row (phase 05) both want number + bar. +2. `src/dashboards/web/static/app.js` — + - `hero(el, {label, value, unit, digits, severity, sub})`: renders/updates a hero, writes + `naText` when `value` is null **and applies no severity class** (an unknown number must + not look green); + - `severityFrom(cellOrSeverity)`: accepts a payload cell (`{value, severity}`), a bare + severity string, or `null`, and returns the class name to apply — this is what lets a + page stop knowing thresholds; + - keep `severityFor(value, warnAt, failAt)` working (pages still call it in + `page-index.js`, `page-system.js`, `page-llm.js`, `page-ups.js`) but add a comment that + it is the fallback for values the API does not rate, and that new call sites should read + `severity` from the payload instead. +3. Do **not** convert the existing pages in this task — phases 02–05 do that page by page so + each page's change is reviewable and revertible on its own. + - ASSUMPTION: leaving `severityFor` in place during this phase keeps every page rendering + exactly as it does today, which is what "no behaviour change until your phase" requires. + +## Testing & Quality + +- No JS test framework exists (locked anchor). The contract that matters here is that the + helpers are the only path to a colour class, which the next phases' diffs make obvious. +- If `tests/test_api.py` or a new test can assert the static assets are served and the CSS + contains the `.hero` rules (a string check on `GET /static/dash.css`), add it — that is + the cheap, honest end-to-end signal available without a browser test runner. +- Coverage: >90% on any Python touched (ideally none beyond that static-asset assertion). + +## Completion Criteria + +- [ ] `curl -s localhost:8090/static/dash.css | grep -c "\.hero"` > 0 +- [ ] Every page still renders as before, unchanged, verified by a screenshot of `/`, `/system`, `/llm`, `/ups` in `data/logs/` +- [ ] `uv run pytest` green; `ruff` + `pyright` clean +- [ ] no `.sev-*` classes were introduced; colour still comes only from `--ok/--warn/--fail` diff --git a/.agents/phases/todo/02_overview_fleet_and_alerts/00_phase.md b/.agents/phases/todo/02_overview_fleet_and_alerts/00_phase.md new file mode 100644 index 0000000..e8e186d --- /dev/null +++ b/.agents/phases/todo/02_overview_fleet_and_alerts/00_phase.md @@ -0,0 +1,47 @@ +# Phase 02 — Overview is fleet and alerts + +**Source:** `TODO.md` L3 — "The overview page should only show Fleet and warnings/errors. Should be very simple." +**Story:** n/a +**Context:** `.agents/PLAN.md` → *Data Model* (`/api/summary`), `01_severity_language` (alerts reuse the verdicts), `src/dashboards/api.py:153` (`/api/summary`), `src/dashboards/web/pages/index.html`, `src/dashboards/web/static/page-index.js` + +## Objective + +Turn the landing page into one question answered instantly: *is anything wrong, and which +page do I need?* Fleet table plus a warnings/errors panel — the three summary cards and the +raw event feed go away. + +The event list is what makes the current overview busy. Its useful content is not "vLLM is +answering again" (that is history) but "something is wrong right now", which is a derived +state and belongs server-side. + +## Dependencies + +- `01_severity_language` (todo) — alerts are built from the `severity` verdicts and the UPS + `health.checks[]` results, not from new thresholds invented in the browser. + +## Tasks + +1. `01_alerts_builder.md` — `src/dashboards/alerts.py`: pure function from hub states + payloads to an alert list; `/api/summary` returns it. +2. `02_rewrite_overview_page.md` — `index.html` + `page-index.js` reduced to fleet table + alerts panel. +3. `03_retire_unused_overview_code.md` — delete the dead cards/events code and confirm the API contract other pages depend on is intact. + +## Testing & Quality + +- Unit: `tests/test_alerts.py` — a source down, a source stale-but-serving, a UPS `fail` + check, a threshold breach, an agent reporting `degraded`, an unconfigured source (must + **not** alert — unconfigured is not broken, `AGENTS.md` invariant 3), a simulated UPS + (warn, labelled), and the all-clear case producing `[]`. +- API: `/api/summary` gains `alerts` and keeps every existing key. +- Coverage: **>90%** on `alerts.py` and the changed part of `api.py`. +- Browser: screenshot of `/` with a real problem present (stop the driveripper agent tunnel + and re-shoot) — an overview that has never been seen in its alarm state has not been + tested. + +## Completion Criteria + +- [ ] `/` shows exactly two panels: Fleet and Warnings/Errors +- [ ] `curl -s localhost:8090/api/summary | python3 -c "import json,sys; print(json.load(sys.stdin)['alerts'])"` is `[]` when everything is healthy +- [ ] Killing one host tunnel produces exactly one alert naming that host, with a link to `/system` +- [ ] `uv run pytest` green (plus `-m live --live` unchanged), coverage ≥94%, `ruff` + `pyright` clean +- [ ] no behaviour change on `/system`, `/llm`, `/ups` +- [ ] no behavior change in completed phases diff --git a/.agents/phases/todo/02_overview_fleet_and_alerts/01_alerts_builder.md b/.agents/phases/todo/02_overview_fleet_and_alerts/01_alerts_builder.md new file mode 100644 index 0000000..0fc4ccd --- /dev/null +++ b/.agents/phases/todo/02_overview_fleet_and_alerts/01_alerts_builder.md @@ -0,0 +1,56 @@ +# Task 01 — Alerts builder + +**Phase:** `02_overview_fleet_and_alerts` · **Source:** `TODO.md:3` — "The overview page should only show Fleet and warnings/errors." +**Story:** n/a + +## Objective + +Derive "what is wrong right now" on the server, so the overview is a list of verdicts and +every other consumer (future mobile view, notification later) gets the same answer. + +## Work + +1. `src/dashboards/alerts.py` (new) — `build(settings, hub_states, host_payloads, engine_payloads, ups_payload, recent_events) -> list[dict]`, + each alert `{"id", "severity": "ok|warn|fail", "source", "message", "href", "at"}`: + - **source down** — a configured job whose `Job.state.ok` is false → `fail`, message from + the recorded error, `href` to its dashboard. Stale-but-serving (`age_seconds` beyond + ~3× its poll interval) is `warn`, not `fail`: the last good value is still on screen and + saying "dead" would be a lie. + - **host threshold breach** — from the `severity` sub-dicts added by phase 01: any + `fail` → an alert naming host + metric + value; `warn` → `warn`. + - **agent degraded** — `payload["degraded"]` non-empty (a `/proc` read failed on the box) + → `warn`, naming the sections. + - **UPS** — any `health.checks[]` entry with `status: "fail"` → `fail`, `"warn"` → `warn`, + carrying the check name and its detail (`"Battery age: 5.3 years — past the 4-year + replacement window"` is exactly the kind of thing the landing page should say); + `simulated: true` → one `warn` alert saying the numbers are modelled, not batteries. + - **bad target config** — `settings.host_target_problems` → `warn` per problem. + - **never** for an unset/unconfigured source. `AGENTS.md` invariant 3: a source nobody + installed must not raise a red flag. + - **dedupe and bound** — one alert per (source, metric); cap the list (say 12) with the + count of suppressed ones so a 40-disk fleet cannot flood the page. + - sort fail → warn, then by source name. +2. `src/dashboards/api.py` — `/api/summary` calls it and adds `"alerts": [...]`. Keep + `system`, `llm`, `ups`, `recent_events` exactly as they are (additive-only anchor), + because `/api/summary` is also the cheapest whole-fleet probe anyone has. +3. `tests/test_alerts.py` (new) — every bullet above, using the existing fixtures + (`tests/fixtures/agent_*.json`) and the fakes in `tests/conftest.py`; the all-clear case + asserts `[]`, and the unconfigured-source case asserts no alert is produced even though + the state map has nothing for it. + - ASSUMPTION: `recent_events` is passed in only so a *currently failing* source can show + when it started failing; the event list itself is not re-rendered as alerts. + - ASSUMPTION: staleness threshold is 3× the source's own poll interval rather than a + single global number, because system polls at 10s and UPS at 5s. + +## Testing & Quality + +- Unit tests for all rules; no I/O in `alerts.py` (it is a pure projection, which is what + makes it worth its own module). +- Coverage: **>90%**. + +## Completion Criteria + +- [ ] `uv run pytest tests/test_alerts.py -v` green +- [ ] `curl -s localhost:8090/api/summary | python3 -m json.tool | head -40` shows `alerts` alongside the untouched `system`/`llm`/`ups` +- [ ] with a healthy fleet the list is `[]`; with `NUT_SIMULATE=1` it contains exactly the simulated-data warning +- [ ] `uv run pytest` green; `ruff` + `pyright` clean diff --git a/.agents/phases/todo/02_overview_fleet_and_alerts/02_rewrite_overview_page.md b/.agents/phases/todo/02_overview_fleet_and_alerts/02_rewrite_overview_page.md new file mode 100644 index 0000000..d913b42 --- /dev/null +++ b/.agents/phases/todo/02_overview_fleet_and_alerts/02_rewrite_overview_page.md @@ -0,0 +1,59 @@ +# Task 02 — Rewrite the overview as fleet + alerts + +**Phase:** `02_overview_fleet_and_alerts` · **Source:** `TODO.md:3` — "The overview page should only show Fleet and warnings/errors. Should be very simple." +**Story:** n/a + +## Objective + +The landing page becomes two things: the fleet table, and a list of what needs attention. + +## Work + +1. `src/dashboards/web/pages/index.html` — + - delete the three dashboard summary cards: the whole + `
` block, i.e. `#sys-hosts-live`, + `#sys-cpu`, `#sys-mem`, `#sys-mem-unit`, `#sys-disk`, `#sys-gpus`, `#llm-live`, + `#llm-decode`, `#llm-prompt`, `#llm-active`, `#llm-waiting`, `#llm-kv`, `#ups-list` + and the `#dot-system` / `#dot-llm` / `#dot-ups` dots they own; + - delete the events panel (`#events` and its card) — alerts replace it; + - keep the masthead, `#nav-status` (the header source dots already live in `app.js` + chrome), `#clock`, `#theme-toggle`, `#config-chip`, `#banners`, `#fleet-table`, + `#fleet-empty`, `#foot-versions`; + - add one panel **Warnings & errors** (`#alerts`, `aria-live="polite"`) with an empty + state that says the fleet is clear — an empty list is a message, not a blank box. + - keep `#fleet-table` columns as they are (host, CPU, load 1, mem, worst disk, GPU busy, + temp, up). It is already the glanceable fleet view; simplifying it further loses the + comparison that makes it worth having. +2. `src/dashboards/web/static/page-index.js` — + - delete `renderEvents()` and the `#sys-*` / `#llm-*` / `#ups-list` population code in + `renderSummary()`; + - add `renderAlerts(alerts)`: one row per alert with the severity class from phase 01 + (`.ok`/`.warn`/`.fail`), the source name as a link to its dashboard (`href` from the + API), the message, and the age via `Dash.age()`; empty → the "all clear" state; + - `refresh()` still calls `/api/summary` once and updates `#summary-line` + ("3 hosts · 2 engines · 2 UPS · 1 warning") — that one line is the only summary left; + - keep `Dash.bindPause`, `Dash.rangeControl` (if the page used it) and the poll loop. +3. Layout: alerts panel sits above the fleet table when something is wrong and below it when + the list is empty, so a clean morning looks calm and a bad one puts the problem first. + - ASSUMPTION: the UPS/LLM summary content the cards showed is *available* on their own + pages, so removing it here loses nothing that is not one click away. The header status + dots stay, because "which of the three is down" is the overview's job. + - ASSUMPTION: `#banners` (config/setup hints, simulated-data notice) stays — it is a + warning surface, which is exactly what the TODO keeps. + +## Testing & Quality + +- Update `tests/test_api.py` only if it asserts page structure; the page is not unit-tested + (locked anchor: no JS tests). +- Browser check with live data, both states: + healthy (`alerts: []` → "all clear") and broken (stop one host agent: + `ssh "pkill -f '[h]ost_agent.py'"` or stop its tunnel) → one red row naming it. + Screenshots to `data/logs/overview-clear.png` and `data/logs/overview-alert.png`. + +## Completion Criteria + +- [ ] `/` renders exactly two panels; `document.querySelectorAll('section').length` is 2 (check via the browser console or an `evaluate`) +- [ ] `#events`, `#sys-cpu`, `#llm-kv`, `#ups-list` no longer exist in the served HTML (`curl -s localhost:8090/ | grep -c 'id="sys-cpu"'` → 0) +- [ ] alert rows link to `/system`, `/llm`, `/ups` correctly +- [ ] both screenshots exist in `data/logs/` and show what they claim +- [ ] `uv run pytest` green; `ruff` + `pyright` clean diff --git a/.agents/phases/todo/02_overview_fleet_and_alerts/03_retire_unused_overview_code.md b/.agents/phases/todo/02_overview_fleet_and_alerts/03_retire_unused_overview_code.md new file mode 100644 index 0000000..1840515 --- /dev/null +++ b/.agents/phases/todo/02_overview_fleet_and_alerts/03_retire_unused_overview_code.md @@ -0,0 +1,43 @@ +# Task 03 — Retire the dead overview code, keep the API honest + +**Phase:** `02_overview_fleet_and_alerts` · **Source:** `TODO.md:3` — "Should be very simple." (simplicity includes not carrying code nothing renders) +**Story:** n/a + +## Objective + +Leave no orphaned code behind, and prove the `/api/summary` fields the *other* consumers +rely on were not quietly dropped along with the cards. + +## Work + +1. `src/dashboards/web/static/page-index.js` — remove helpers that only the deleted cards + used; run a grep for each symbol before deleting so nothing still referenced goes: + `grep -n "sys-cpu\|llm-kv\|ups-list\|renderEvents" src/dashboards/web/static/*`. +2. `src/dashboards/web/static/dash.css` — drop rules that only the deleted cards matched + (check with `grep -n "metric\b" dash.css` against the remaining pages before removing; + `.metric` is used by `/system` and `/llm` too, so most of it stays). +3. `src/dashboards/api.py` — **do not remove** the `system`/`llm`/`ups` sections from + `/api/summary` even though the page no longer renders them: it is the single cheapest + whole-fleet probe, the live tests and future alerting want it, and the API anchor is + additive-only. Add a short comment on `/api/summary` saying the page renders + `alerts` + `system.fleet`/`hosts` and that the rest is for programmatic use. +4. `tests/test_api.py` — add an explicit regression test: `/api/summary` contains the keys + `system.enabled`, `system.fleet`, `system.hosts`, `llm.engines`, `ups.ups`, + `recent_events`, `alerts`, so a later cleanup cannot silently shrink the contract. +5. `README.md` — the overview row in the top table still says "one request, all of it"; + update the wording to "fleet + alerts in one request". + +## Testing & Quality + +- Full suite green, including the new contract test. +- Coverage: >90% on changed Python (`api.py` only). +- `grep` evidence that no dangling selector survives (paste the empty grep result in the + commit body or task notes). + +## Completion Criteria + +- [ ] `grep -rn "sys-cpu\|llm-kv\|ups-list\|renderEvents" src/dashboards/web/` returns nothing +- [ ] `curl -s localhost:8090/api/summary` still returns every key listed in step 4 +- [ ] `uv run pytest` green (contract test included), coverage ≥94%, `ruff` + `pyright` clean +- [ ] `uv run pytest -m live --live` still 6 passed +- [ ] no behavior change in completed phases (01's severity fields still present) diff --git a/.agents/phases/todo/03_system_per_host_panels/00_phase.md b/.agents/phases/todo/03_system_per_host_panels/00_phase.md new file mode 100644 index 0000000..c6edb1b --- /dev/null +++ b/.agents/phases/todo/03_system_per_host_panels/00_phase.md @@ -0,0 +1,50 @@ +# Phase 03 — System page is per-host panels + +**Source:** `TODO.md` L4 — "The system page doesn't need shared charts for anything. It should only have per-host charts." · L5 — "Per host charts need to be simpler. Larger numbers, proper green/yellow/red color coding. Don't need per-core. Just need cpu%, mem%, cpu temp, disk usage %, graphics card details (usage and vram), and network up and down." +**Story:** n/a +**Context:** `.agents/PLAN.md` → *Anchors* (dropped visuals are intentional, series stay in SQLite), `01_severity_language`, `src/dashboards/web/pages/system.html`, `src/dashboards/web/static/page-system.js`, `src/dashboards/sources/host.py` + +## Objective + +`/system` stops drawing seven fleet-wide charts and a "pick a host, pick a metric" detail +chart. It becomes one panel per host: six big colour-coded numbers — cpu %, mem %, cpu temp, +disk usage %, GPU usage + VRAM, network up/down — and that host's own chart underneath. + +Nothing per-core anywhere on the page, and nothing that requires a selection to make sense. + +## Dependencies + +- `01_severity_language` (todo) — the cards render `severity` from the payload; they do not + get to keep their own thresholds. +- `02_overview_fleet_and_alerts` (todo) — the fleet-wide "what stands out" role moved to the + overview's alerts list, which is what makes removing the shared charts safe. + +## Tasks + +1. `01_host_card_payload_contract.md` — prove the six card metrics already exist in `/api/system/snapshot` and pin them with tests (add fields only if genuinely missing). +2. `02_hero_cards_without_per_core.md` — rewrite `hostCard()`/`renderHostCard()` into big colour-coded numbers, no `renderCores`. +3. `03_per_host_charts_replace_shared.md` — delete the fleet-chart and series-detail sections; give each host its own compact charts. +4. `04_verify_every_host_shape.md` — browser verification across a discrete GPU, an APU, and a box with no GPU. + +## Testing & Quality + +- Python: contract tests for the payload fields the cards read; coverage **>90%** on any + changed Python (ideally none — this phase is frontend plus tests). +- `uv run pytest -m live --live` must stay green: `test_every_host_agent_answers_and_parses` + is what proves the fields exist on all three real hosts, including the ones that report + `n/a`. +- Browser, live data, every host shape: driveripper (discrete R9700), deskwork (Strix Halo + APU — VRAM *and* GTT, so "3% VRAM" and "78% memory" are both true), gateway (no GPU at all + → the honest "no GPU reported by this agent" state). +- The `cpu.usage_pct@coreN` series must still be stored and still be queryable from + `/api/system/history` — removing the plot is not removing the data. + +## Completion Criteria + +- [ ] `/system` contains no chart whose legend lists more than one host +- [ ] no per-core visualisation anywhere on `/system` (`grep -rn "renderCores\|core_usage" src/dashboards/web/` → nothing, or charts.js-only if kept unused on purpose) +- [ ] each host panel shows exactly: cpu %, mem %, cpu temp, disk %, GPU busy + VRAM (or the no-GPU state), network ↓/↑ +- [ ] `curl -s "localhost:8090/api/system/history?host=driveripper&metric=cpu.usage_pct@core7&range=1h"` still returns points +- [ ] three screenshots in `data/logs/`, one per host shape +- [ ] `uv run pytest` green, coverage ≥94%, `ruff` + `pyright` clean +- [ ] no behavior change in completed phases diff --git a/.agents/phases/todo/03_system_per_host_panels/01_host_card_payload_contract.md b/.agents/phases/todo/03_system_per_host_panels/01_host_card_payload_contract.md new file mode 100644 index 0000000..8f88be2 --- /dev/null +++ b/.agents/phases/todo/03_system_per_host_panels/01_host_card_payload_contract.md @@ -0,0 +1,50 @@ +# Task 01 — Host card payload contract + +**Phase:** `03_system_per_host_panels` · **Source:** `TODO.md:5` — "Just need cpu%, mem%, cpu temp, disk usage %, graphics card details (usage and vram), and network up and down." +**Story:** n/a + +## Objective + +Establish, with tests, that everything the new cards need is already served — so the card +rewrite is a rendering change and nobody "fixes" a field that was never missing. + +## Work + +1. Read the current payload in `parse_host_snapshot()` (`src/dashboards/sources/host.py`) + and confirm the six card metrics map to: + - cpu % → `cpu.usage_pct` + - mem % → `memory.used_pct` + - cpu temp → `temperatures.cpu_c` **and** `cpu.temp_c` (both exist; pick one and say which + in the card code comment — `cpu.temp_c` is the one `/api/summary` already uses) + - disk usage % → `disks.worst_used_pct` + `disks.worst_mount` (the mount is what makes the + number actionable) + - GPU → `gpus[].busy_pct`, `gpus[].vram_used_bytes`, `gpus[].vram_total_bytes`, + `gpus[].vram_used_pct`, `gpus[].mem_used_pct` (the honest APU view), `gpus[].model` + - network ↓/↑ → `network.rx_bytes_per_s`, `network.tx_bytes_per_s` + plus the phase-01 `severity` sub-dicts (`cpu.severity`, `memory.severity`, + `disks.severity`, `gpus[].severity`). +2. `tests/test_system_contract.py` (new) — one test that runs a real fixture snapshot + (`tests/fixtures/agent_driveripper.json`) through `parse_host_snapshot()` and asserts each + of the paths above exists (keys present, values may be `None`), and one that runs the + `agent_gateway.json` fixture (no GPU) asserting `gpus == []` rather than a missing key. +3. Add the **only** payload change this phase is allowed, if the card needs it: + `disks.worst.severity` must be present (it comes from phase 01) and + `network` gets no severity (throughput has no good/bad — say so in a comment so nobody + adds a meaningless green later). +4. Note in the test docstring what is *deliberately* not on the card: per-core usage + (still stored as `cpu.usage_pct@coreN`), swap, processes, sensors, per-interface rows. + - ASSUMPTION: the card reads `cpu.temp_c`; if the executor finds it absent where + `temperatures.cpu_c` is present on a real host, use `temperatures.cpu_c` and record the + choice in the commit message rather than changing the payload. + +## Testing & Quality + +- New tests green; they must fail if a field is renamed, which is the point. +- `uv run pytest -m live --live` for the same assertions against the real fleet. + +## Completion Criteria + +- [ ] `uv run pytest tests/test_system_contract.py -v` green +- [ ] no payload field renamed or removed (`git diff src/dashboards/sources/host.py` is empty or adds only a comment) +- [ ] `uv run pytest`, `ruff`, `pyright` clean +- [ ] no visual change yet — this task changes no page code diff --git a/.agents/phases/todo/03_system_per_host_panels/02_hero_cards_without_per_core.md b/.agents/phases/todo/03_system_per_host_panels/02_hero_cards_without_per_core.md new file mode 100644 index 0000000..d50165e --- /dev/null +++ b/.agents/phases/todo/03_system_per_host_panels/02_hero_cards_without_per_core.md @@ -0,0 +1,60 @@ +# Task 02 — Hero cards, no per-core + +**Phase:** `03_system_per_host_panels` · **Source:** `TODO.md:5` — "Per host charts need to be simpler. Larger numbers, proper green (good), yellow (warn), red (danger) color coding. Don't need per-core. Just need cpu%, mem%, cpu temp, disk usage %, graphics card details (usage and vram), and network up and down." +**Story:** n/a + +## Objective + +Each host panel leads with six big colour-coded numbers and nothing else in the way. + +## Work + +1. `src/dashboards/web/static/page-system.js` — `hostCard()`: + - replace the two `.metrics` blocks with a single hero row built from the phase-01 `.hero` + markup: CPU %, MEM %, TEMP °C, DISK %, GPU %, NET ↓/↑ (network is one hero with two + lines — in and out — rather than two competing numbers); + - **delete the per-core block** (`
Per-core
` and + `data-k="cores"`); + - drop from the top row: Load 1/15, Clock GHz, Disk read/write/busy, Up — into the + existing `
` (see below), not deleted from the page's reach; + - keep `data-k` names for whatever survives so `renderHostCard()` keeps working, and give + each hero a `data-sev` hook for the severity class. +2. `renderHostCard()`: + - drive colour from `host.cpu.severity.usage_pct`, `host.memory.severity.used_pct`, + `host.disks.severity.worst_used_pct`, `gpu.severity.busy_pct` — via + `Dash.severityFrom(...)`; delete the `severityFor(value, 75, 90)` calls on + `page-system.js:129/139/144`; + - value `null` → `n/a` with **no** severity class; + - memory hero shows `%` big, `200 / 253 GiB` as the `.sub` line (the percentage is the + verdict, the bytes are the detail); + - disk hero shows `%` big, the mount (`disks.worst_mount`) as `.sub` — "80 % of what?" + must be answerable without a click; + - GPU block: one line per card — model, `busy %` with severity, `VRAM used / total` and + the `.sub` pool view (`mem_used_pct`) for APUs. No GPU → the existing + "No GPU reported by this agent" line; + - network hero uses `Dash.rate()` on `network.rx_bytes_per_s` / `tx_bytes_per_s`, and + renders `n/a` when `any_net` produced nulls rather than a confident `0 B/s`; + - the existing `
` becomes "More detail": load averages, clock, disk read/write/ + busy, uptime, filesystems, top processes. Collapsed by default. +3. `src/dashboards/web/static/dash.css` — add only what the hero row needs if phase 01 did + not already (grid of 6 heroes, wraps to 2×3 and 3×2 at narrow widths; `.cores` rule can + stay until task 03 confirms nothing else uses it). + - ASSUMPTION: `temp` uses `cpu.temp_c` (see task 01). Hosts with no CPU sensor show `n/a` + uncoloured — `gateway` in this fleet. + - ASSUMPTION: the `.details`/`
` disclosure keeps the information that is not + glance-worthy. The TODO asks for a simpler surface, not a less complete page. + +## Testing & Quality + +- No JS tests (locked anchor). Contract is covered by `tests/test_system_contract.py`. +- Browser: verify against live data on driveripper (GPU), and confirm the severity colours + change when a value crosses a threshold — e.g. temporarily override with + `THRESHOLDS` in a REPL or by pointing at a host under load. One screenshot per state. + +## Completion Criteria + +- [ ] each host card shows exactly six heroes in this order: CPU, MEM, TEMP, DISK, GPU, NET +- [ ] `grep -n "cores\|renderCores" src/dashboards/web/static/page-system.js` → no matches in the card code +- [ ] no `severityFor(` call remains in `page-system.js` +- [ ] a host with no GPU and no CPU temp renders `n/a` with no colour (screenshot) +- [ ] `uv run pytest` green; `ruff` + `pyright` clean diff --git a/.agents/phases/todo/03_system_per_host_panels/03_per_host_charts_replace_shared.md b/.agents/phases/todo/03_system_per_host_panels/03_per_host_charts_replace_shared.md new file mode 100644 index 0000000..ad47b4f --- /dev/null +++ b/.agents/phases/todo/03_system_per_host_panels/03_per_host_charts_replace_shared.md @@ -0,0 +1,66 @@ +# Task 03 — Per-host charts replace the shared ones + +**Phase:** `03_system_per_host_panels` · **Source:** `TODO.md:4` — "The system page doesn't need shared charts for anything. It should only have per-host charts." and `TODO.md:5` — "Per host charts need to be simpler" +**Story:** n/a + +## Objective + +Delete the six fleet charts and the "pick a host / pick a metric" detail chart. Every chart +that survives belongs to exactly one host and is inside that host's panel. + +## Work + +1. `src/dashboards/web/pages/system.html` — delete: + - the whole `
` (L65–91): + `#chart-cpu`, `#legend-cpu`, `#chart-mem`, `#legend-mem`, `#chart-disk`, `#legend-disk`, + `#chart-net`, `#legend-net`, `#chart-gpu`, `#legend-gpu`, `#chart-temp`, `#legend-temp`; + - the whole "Series detail" `
` (L93–107): `#detail-host`, `#detail-metric`, + `#chart-detail`; + - the `
` (L40–63) — its job moved to + the overview's fleet table + alerts in phase 02; + - inside each host panel, add a chart slot: `
` + and `
` (plus `data-chart="gpu"` rendered + only when the host reports GPUs). +2. `src/dashboards/web/static/page-system.js` — + - delete `makeCharts()`'s seven `state.charts.*` constructions, `refreshCharts()`, + `refreshGpuChart()`, and the `#detail-*` select wiring; + - add `renderHostCharts(host)`: per host, **one** utilisation chart (cpu %, mem %, + disk-worst %, shared 0–100 y-axis, three lines, one legend) and **one** network chart + (↓ and ↑ on the same axis), plus a GPU utilisation chart when `host.gpus.length`; + - source them from `/api/system/history` — the endpoint already returns per-host series + (`{"hosts": [...], "series": [{"host", "metric", "points"}]}`), so this is a query per + host-metric set, not a new endpoint; keep one request per host per refresh window, not + one per metric, if the endpoint supports a metric list (it accepts + `host=`/`metric=`/`range=` — reuse `seriesFor()`); + - respect the existing range control (`state.range`) and `Dash.bindPause`; + - height ~120px: these are supporting evidence under a number, not the main event. +3. `src/dashboards/web/static/charts.js` — if `renderCores` now has no callers + (`grep -rn "renderCores" src/dashboards/web/`), delete it and its CSS. Do not otherwise + touch `LineChart`; phase 04 depends on it. +4. `src/dashboards/web/static/dash.css` — remove `.cores` rules and any now-unused chart + grid classes, verified by grep before deletion. + - ASSUMPTION: three charts per host (utilisation, network, GPU-if-present) rather than + one chart per metric. Six metrics × three hosts = eighteen charts on screen is the + unreadable version of "per-host charts". + - ASSUMPTION: no cross-host chart stays anywhere. Comparing hosts is the overview fleet + table's job; `/system` answers "what is this machine doing". + - ASSUMPTION: history retention and the stored series are untouched — + `cpu.usage_pct@coreN` keeps being polled and stored, and remains queryable. + +## Testing & Quality + +- `tests/test_api.py`: if it asserts the removed chart endpoints, update it; the endpoints + themselves stay (`/api/system/history` is used by the per-host charts and the live tests). +- Browser, live data: chart per host, no legend with two host names in it, and the range + chips (1h/6h/24h/7d/30d/90d) still re-fetch. Watch for the old bug class: a chart that + keeps drawing after its host disappears from the payload. +- Performance sanity: with three hosts the page must still poll smoothly — count the XHRs per + refresh in devtools and keep it under ~6. + +## Completion Criteria + +- [ ] `curl -s localhost:8090/system | grep -c 'id="chart-cpu"\|id="chart-detail"'` → 0 +- [ ] every `LineChart` on `/system` is inside a `[data-host]` panel +- [ ] `curl -s "localhost:8090/api/system/history?host=driveripper&metric=cpu.usage_pct@core7&range=1h"` still returns points +- [ ] `uv run pytest` green (including `-m live --live`), coverage ≥94%, `ruff` + `pyright` clean +- [ ] screenshot of `/system` with all three hosts in `data/logs/` diff --git a/.agents/phases/todo/03_system_per_host_panels/04_verify_every_host_shape.md b/.agents/phases/todo/03_system_per_host_panels/04_verify_every_host_shape.md new file mode 100644 index 0000000..a881e4f --- /dev/null +++ b/.agents/phases/todo/03_system_per_host_panels/04_verify_every_host_shape.md @@ -0,0 +1,49 @@ +# Task 04 — Verify every host shape in a browser + +**Phase:** `03_system_per_host_panels` · **Source:** `TODO.md:4–5` — the page must be right for *every* host, not just the one that was convenient to develop against +**Story:** n/a + +## Objective + +Prove the new `/system` page on all three real hosts — including the two shapes that usually +get missed: an APU with two memory pools, and a box with no GPU and no CPU temperature. + +## Work + +1. Confirm the dev loop is live: `./scripts/dev-tunnels.sh status` (three agents + the upsd + forward OK), `./scripts/dev-server.sh restart`, `/api/health` all `ok`. +2. Verify and screenshot each host panel on `http://127.0.0.1:8090/system`: + - **driveripper** — discrete Radeon AI PRO R9700: busy %, VRAM used/total (~32 GiB), + power/temp where reported; + - **deskwork** — Strix Halo APU: `vram_used_pct` low while `mem_used_pct` is high is + *correct*, and the `.sub` pool line must make that legible rather than contradictory; + - **gateway** — no GPU: the "No GPU reported by this agent" state, GPU hero `n/a` with + **no** colour; CPU temp `n/a` likewise. +3. Check the failure path, not just the happy path: stop one agent + (`./scripts/dev-tunnels.sh stop` for a single host, or `ssh "pkill -f '[h]ost_agent.py'"`) + and confirm the panel keeps its last good numbers, marks them stale + (`stalenessText`), turns its dot amber/red, and does **not** blank or zero the heroes. + Screenshot it. Restore afterwards (`./scripts/dev-tunnels.sh start`). +4. Accessibility pass on the new markup: hero values are real text (selectable, not SVG + paths), each chart container has a label or `aria-label`, colour is never the *only* + signal — the number itself must carry the meaning. +5. Light and dark theme both checked (the header toggle), since severity colours come from + theme tokens. +6. Write a short verification note under this phase's directory as + `NOTES-verify.md`: date, what was checked, what looked wrong (if anything), screenshots + referenced by filename. + - ASSUMPTION: the note lives next to the phase rather than in git history alone, so the + next reader can tell what was actually seen on screen. + +## Testing & Quality + +- This task produces evidence, not code. If verification finds a defect, fix it in this task + and re-shoot; do not leave it as a note. + +## Completion Criteria + +- [ ] `data/logs/system-driveripper.png`, `data/logs/system-deskwork.png`, `data/logs/system-gateway.png` exist and were opened/read before declaring success +- [ ] a stale-host screenshot shows last-good values marked stale, not zeros +- [ ] both themes checked; no colour-only signalling +- [ ] `NOTES-verify.md` written in this phase directory +- [ ] `uv run pytest` still green after any fix made during verification diff --git a/.agents/phases/todo/04_llm_chart_readability/00_phase.md b/.agents/phases/todo/04_llm_chart_readability/00_phase.md new file mode 100644 index 0000000..8295c67 --- /dev/null +++ b/.agents/phases/todo/04_llm_chart_readability/00_phase.md @@ -0,0 +1,51 @@ +# Phase 04 — LLM page is readable again + +**Source:** `TODO.md` L6 — "LLM usage drifed from initial request to copy Experimental/llm_monitor. The charts got less useful. Need simplified and more readable." +**Story:** n/a +**Context:** `.agents/PLAN.md` → *Anchors* (hand-rolled SVG only), `~/Projects/Experimental/llm_monitor/static/index.html` (the reference implementation), `src/dashboards/web/static/page-llm.js` (`CELLS`, `makeCharts`), `src/dashboards/web/static/charts.js` (`sparkline`, `LineChart`), `src/dashboards/sources/llm.py` + +## Objective + +The LLM page drifted into a 14-cell grid where everything is the same size and nothing shows +recent shape. Bring back what made `llm_monitor` legible — big throughput numbers, a sparkline +in each metric cell, percentiles as foot lines — without losing what this project added since: +per-engine provenance tags, history windows, and severity colour. + +`llm_monitor` is the design reference, not a dependency. It is a separate app on `:8095` and +stays that way; nothing here imports from it. + +## Dependencies + +- `01_severity_language` (todo) — KV-cache pressure and queue depth get coloured from the + API verdict, not a per-page threshold. +- `03_system_per_host_panels` (todo) — touches `charts.js` first; this phase builds on the + same shared helpers and must not fight it for them. + +## Tasks + +1. `01_chart_set_decision.md` — read both implementations side by side and write the decision note (which cells live, which get sparklines, what is demoted). +2. `02_sparkline_upgrade.md` — bring `DashCharts.sparkline()` up to `llm_monitor`'s `drawSpark` behaviour: right-aligned growth, fixed max, current/max labels, severity colour. +3. `03_llm_layout_simplified.md` — restructure `llm.html` + `page-llm.js`: hero throughput, compact sparkline cells, percentile foot lines, conditional alarm cells. +4. `04_side_by_side_verification.md` — verify against `llm_monitor` on `:8095` with the same live engines. + +## Testing & Quality + +- Python: none expected beyond the phase-01 severity fields; `/api/llm/*` stays as-is + (additive-only anchor). +- `uv run pytest -m live --live` must stay green — `test_engine_still_publishes_the_series_we_parse` + is the assertion that the numbers on screen are backed by resolved upstream series. +- Browser: both engines live (gufo on `deskwork:8731`, vLLM on `driveripper:8000`), plus the + single-engine case: with one `*_BASE_URL` unset the page must show one clean half rather + than a red panel. +- Coverage: ≥94% overall maintained. + +## Completion Criteria + +- [ ] decode and prompt throughput are the two largest numbers per engine, colour-coded +- [ ] every remaining metric cell carries a sparkline showing the selected window +- [ ] latency cells show mean **and** p95 without needing a hover +- [ ] a source-provenance tag is still visible on every throughput number (`AGENTS.md`: never invent a number) +- [ ] `/llm` with only one engine configured still renders cleanly +- [ ] screenshots of this page and `llm_monitor` side by side in `data/logs/`, plus `NOTES-chart-set.md` +- [ ] `uv run pytest` green, coverage ≥94%, `ruff` + `pyright` clean +- [ ] no behavior change in completed phases diff --git a/.agents/phases/todo/04_llm_chart_readability/01_chart_set_decision.md b/.agents/phases/todo/04_llm_chart_readability/01_chart_set_decision.md new file mode 100644 index 0000000..e65d149 --- /dev/null +++ b/.agents/phases/todo/04_llm_chart_readability/01_chart_set_decision.md @@ -0,0 +1,52 @@ +# Task 01 — Chart-set decision note + +**Phase:** `04_llm_chart_readability` · **Source:** `TODO.md:6` — "LLM usage drifed from initial request to copy Experimental/llm_monitor. The charts got less useful. Need simplified and more readable." +**Story:** n/a + +## Objective + +Decide, in writing, what "simplified and more readable" means for this page before touching +it — the drift happened precisely because nobody wrote down what the page was for. + +## Work + +1. Read the reference implementation: + `~/Projects/Experimental/llm_monitor/static/index.html` — specifically + `drawSpark(name, values, format, fixedMax)` (~L1092, with the right-aligned growth and the + top/middle/bottom scale marks ~L1134), `setThroughput(el, cell)` (~L1303), + `UI.historyLen = 60` (~L1003), the per-engine cell markup (`#gufo-ptps`, `#gufo-dtps`, + `#vllm-ptps`, `#vllm-dtps`), the percentile foot lines (`#vllm-ttft-p95`, + `#vllm-tpot-p95`, `#vllm-queue-p95`), and `buildSlotCard`/`renderSlots` (~L1179/L1209). +2. Read the current page: `src/dashboards/web/pages/llm.html` (engine-totals card L39–60, + four charts L62–81, `#engines` grid L83–91) and `page-llm.js` (`CELLS`, 14 entries; + `cellValue()`; `severityFor(value, 75, 90)` calls at L195/L307). +3. Write `.agents/phases/todo/04_llm_chart_readability/NOTES-chart-set.md`: + - **keep**: provenance tags per throughput cell, history window chips, per-engine panels, + the gufo slot/session list; + - **hero**: `decode_tokens_per_second`, `prompt_tokens_per_second` per engine; + - **sparkline cells** (compact, one per row): sessions active, queued, KV cache %, + TTFT (mean + p95 as foot), TPOT (mean + p95 as foot), queue mean, prefix-cache hit %, + spec-accept %; + - **conditional / demoted**: tokens-lifetime counter (quiet footnote), `preemptions` and + `device_lost` — shown as alarm chips **only when non-zero**, otherwise absent. A permanent + cell for something that is always 0 trains the eye to ignore the row. + - **charts**: keep decode/prompt/KV/queue `LineChart`s, one engine per series, with the + existing legend and a units-bearing y-axis; the four stay but drop to half-height so the + cells read first. + - the colour rules for KV pressure and queue depth (from `severity.py`), and the explicit + statement that throughput has *no* severity — 0 tok/s on an idle server is not red. +4. No code changes. + - ASSUMPTION: the cell set above is the judgment call; if the operator disagrees it is + cheaper to change this note now than the page later. + - ASSUMPTION: `llm_monitor` stays a separate app. Importing its files would break the + no-build, single-app structure and it is not ours to own. + +## Testing & Quality + +- The note is the deliverable; it must name real line numbers in both codebases so the next + task does not re-derive them. + +## Completion Criteria + +- [ ] `NOTES-chart-set.md` exists in this phase directory, with keep/hero/demote lists and the two source files cited by path and line +- [ ] no application file changed in this task (`git status` clean except the note) diff --git a/.agents/phases/todo/04_llm_chart_readability/02_sparkline_upgrade.md b/.agents/phases/todo/04_llm_chart_readability/02_sparkline_upgrade.md new file mode 100644 index 0000000..a500c2a --- /dev/null +++ b/.agents/phases/todo/04_llm_chart_readability/02_sparkline_upgrade.md @@ -0,0 +1,51 @@ +# Task 02 — Sparkline upgrade + +**Phase:** `04_llm_chart_readability` · **Source:** `TODO.md:6` — "The charts got less useful." +**Story:** n/a + +## Objective + +Make the shared `sparkline()` as informative as `llm_monitor`'s `drawSpark`, because every +page (LLM cells now, per-host cards later) draws on it. + +`charts.js:352` already has `sparkline(container, points, opts)` and it is exported at +`charts.js:377` — **no page uses it today**, which is exactly why the LLM page reads flat. + +## Work + +1. `src/dashboards/web/static/charts.js` — extend `sparkline()` with these opts, defaulting + to today's behaviour so no existing caller breaks: + - `fixedMax` — plot against a known ceiling (percent metrics must not rescale to their own + data: 8 % KV cache next to a 92 % spike is the lie this fixes); + - `align: 'right'` — grow from the left, newest pinned right, as in `drawSpark`; + - `labels: {current, max}` — the small `now / max` foot `llm_monitor` prints, using the + `format` callback; + - `severity` — stroke colour from the theme token for `ok|warn|fail` (`cssVar('--ok')` etc.), + no colour and a faint grey stroke when severity is `null`; + - `minPoints` (default 2) — keep the existing "no history" text for a series that has no + shape yet; never draw a confident line through one point. +2. Keep it dependency-free SVG (`` + ``/`` + optional baseline), and keep + the output small enough to sit in a 160×34 box — that is the whole point of a sparkline. +3. Add scale marks (top/bottom values only) when `fixedMax` is set, so a reader can tell that + "the line is at a third" means a third *of what*. +4. If `renderCores` survived phase 03's cleanup, leave it alone — do not widen this task. + - ASSUMPTION: sparkline stays in `charts.js` rather than a new file; one shared chart + module is the current architecture. + - ASSUMPTION: no animation, but respect `prefers-reduced-motion` by not adding any + transition at all. + +## Testing & Quality + +- `tests/test_api.py` already asserts static assets are served; extend it to assert the new + option names are present in `/static/charts.js` (cheap, catches a bad merge). +- Browser: a page that uses it (any LLM cell after task 03) plus a synthetic check in devtools + — draw into an empty container (0, 1, 2, 60 points) and confirm each renders the right thing + rather than throwing. +- Coverage: ≥94% overall maintained; no new Python expected. + +## Completion Criteria + +- [ ] `curl -s localhost:8090/static/charts.js | grep -c "fixedMax\|align\|severity"` ≥ 3 +- [ ] a sparkline with `fixedMax: 100` does not rescale when its data ranges 5–10 (verified in devtools, screenshot) +- [ ] 0- and 1-point series render "no history", not a spike +- [ ] `uv run pytest` green; `ruff` + `pyright` clean; `LineChart` untouched (`git diff` shows no changes to its code) diff --git a/.agents/phases/todo/04_llm_chart_readability/03_llm_layout_simplified.md b/.agents/phases/todo/04_llm_chart_readability/03_llm_layout_simplified.md new file mode 100644 index 0000000..cdd1aaf --- /dev/null +++ b/.agents/phases/todo/04_llm_chart_readability/03_llm_layout_simplified.md @@ -0,0 +1,62 @@ +# Task 03 — Simplify the LLM page layout + +**Phase:** `04_llm_chart_readability` · **Source:** `TODO.md:6` — "LLM usage drifed from initial request to copy Experimental/llm_monitor. The charts got less useful. Need simplified and more readable." +**Story:** n/a + +## Objective + +Per engine: two big throughput numbers you can read across the room, then a compact list of +metric cells that each carry their own sparkline and percentile foot line. + +## Work + +1. `src/dashboards/web/pages/llm.html` — + - keep the engine-totals card (L39–60) but shrink it to what the overview no longer shows: + engines answering, active sessions, queued. Drop the summed decode/prompt/KV numbers — + per-engine is the honest view and the overview owns the aggregate; + - reduce the four `LineChart` cards (L62–81) to half height, and drop `#chart-queue` if + `NOTES-chart-set.md` says queue reads better as a cell; + - per-engine panel (`#engines` grid, `page-llm.js:140+`): hero row (decode, prompt) using + the phase-01 `.hero` markup, then the cell list from `NOTES-chart-set.md`, then the gufo + slot list (unchanged behaviour). +2. `src/dashboards/web/static/page-llm.js` — + - `CELLS`: mark each entry with `hero: true` / `spark: true` / `alarm: true` and drive the + markup from those flags instead of 14 identical cells; + - render each `spark: true` cell through `DashCharts.sparkline()` with data from + `/api/llm/history` (same endpoint, one request per engine per window — batch the metrics + the endpoint supports rather than one XHR per cell); + - percentile foot lines: TTFT/TPOT/queue cells print `mean` big and `p95 …` beneath, as + `llm_monitor` does (`#vllm-ttft-p95` pattern); + - `alarm: true` cells (`preemptions`, `device_lost`) render only when non-zero, as a + `.chip.fail`; the row must not exist when they are 0; + - replace `severityFor(value, 75, 90)` (L195, L307) with the payload `severity` via + `Dash.severityFrom()`; + - **keep the provenance tag** on every throughput cell — the file's header comment explains + why (`78 tok/s` means different things when scraped, measured Δ/Δt, or estimated from + TPOT). Removing it to look cleaner is a regression, not a simplification. +3. `src/dashboards/web/static/dash.css` — cell-list styling: label left, value right-aligned + tabular, sparkline on the third column at a fixed width so rows line up; the alarm chip + reuses `.chip.fail`. + - ASSUMPTION: throughput gets no severity colour. A server doing 0 tok/s while idle is + healthy; a red zero every night destroys the signal. + - ASSUMPTION: cells are a vertical list per engine, not a 4×4 grid — grids make 14 numbers + look equally important, which is the complaint. + +## Testing & Quality + +- `uv run pytest -m live --live` against the real gufo and vLLM: `test_engine_still_publishes_the_series_we_parse` + guards the metric names the new layout renders. +- Browser, both engines live, and with one engine disabled (`VLLM_BASE_URL=` unset, restart) + to prove the page degrades to one clean half instead of a dead panel + (`AGENTS.md` invariant 3). +- Idle vs loaded engine if possible (`llm_monitor` on `:8095` is watching the same upstreams, + so compare like-for-like). + +## Completion Criteria + +- [ ] decode + prompt are the largest elements per engine, and the rest of the cells are visibly secondary +- [ ] every cell has a sparkline for the selected window; TTFT/TPOT show mean + p95 +- [ ] `preemptions` / `device_lost` are absent at zero and appear as red chips when non-zero (verified in devtools with a mocked payload) +- [ ] provenance tags still present on throughput cells (`grep -n "source" page-llm.js` shows the tag path intact) +- [ ] `curl -s localhost:8090/llm` renders with one engine configured and shows no dead panel (screenshot) +- [ ] `uv run pytest` green, `ruff` + `pyright` clean diff --git a/.agents/phases/todo/04_llm_chart_readability/04_side_by_side_verification.md b/.agents/phases/todo/04_llm_chart_readability/04_side_by_side_verification.md new file mode 100644 index 0000000..480942b --- /dev/null +++ b/.agents/phases/todo/04_llm_chart_readability/04_side_by_side_verification.md @@ -0,0 +1,44 @@ +# Task 04 — Verify side by side with llm_monitor + +**Phase:** `04_llm_chart_readability` · **Source:** `TODO.md:6` — "LLM usage drifed from initial request to copy Experimental/llm_monitor." +**Story:** n/a + +## Objective + +Prove the new page is at least as readable as the app it drifted from, using the same live +upstreams. + +## Work + +1. Both apps watch the same engines, so compare honestly: + - `llm_monitor` on `http://127.0.0.1:8095` (it is already running in this fleet's dev + loop; if not, `cd ~/Projects/Experimental/llm_monitor && uv run uvicorn app:app --port 8095`); + - this app on `http://127.0.0.1:8090/llm`. +2. Screenshot both, same moment, same window length. Read the two images and answer in + writing: which one tells you *right now* whether the engines are busy, whether anyone is + queued, and whether the KV cache is about to bite? The answer must be "ours" or the phase + is not done. +3. Exercise the states, not just the happy path: + - one engine down (stop gufo or point `GUFO_BASE_URL` at a dead port, restart the dev + server) → one clean half, no red panel for the missing engine; + - both engines idle → throughput `0 tok/s` with **no** severity colour, and the sparklines + still show the shape of the last window rather than a flat lie; + - a real request in flight if the operator is willing (`curl` a chat completion against + gufo) so decode/prompt are visibly non-zero and the provenance tag is readable. +4. Confirm the history window chips (1h…90d) still drive the four `LineChart`s and that the + sparklines use the same window — a cell that silently ignores the selected range is worse + than no sparkline. +5. Append the comparison and the answers to `NOTES-chart-set.md`. + - ASSUMPTION: `llm_monitor` is treated as the design reference and left running; this phase + neither modifies nor deletes it. + +## Testing & Quality + +- Evidence task. Any defect found is fixed here, then re-screenshotted. + +## Completion Criteria + +- [ ] `data/logs/llm-before-monitor.png` (the reference) and `data/logs/llm-after.png` (ours) exist and were both read +- [ ] the single-engine and idle states are screenshotted too +- [ ] `NOTES-chart-set.md` has a short written verdict + what was fixed after looking +- [ ] `uv run pytest` and `uv run pytest -m live --live` green diff --git a/.agents/phases/todo/05_ups_hero_battery_glyph/00_phase.md b/.agents/phases/todo/05_ups_hero_battery_glyph/00_phase.md new file mode 100644 index 0000000..93be599 --- /dev/null +++ b/.agents/phases/todo/05_ups_hero_battery_glyph/00_phase.md @@ -0,0 +1,48 @@ +# Phase 05 — UPS hero numbers and a battery glyph + +**Source:** `TODO.md` L7 — "UPS looks good but need charge, load, and runtime left to be big, obvious, and color coded. Also an SVG battery symbol that shows current charge would be a nice touch." +**Story:** n/a +**Context:** `.agents/PLAN.md` → *Architectural Anchors* (severity from the API, hand-rolled SVG), `01_severity_language`, `src/dashboards/web/pages/ups.html`, `src/dashboards/web/static/page-ups.js`, `src/dashboards/web/static/charts.js`, `src/dashboards/sources/nut.py` + +## Objective + +Each UPS card opens with a battery glyph that visibly holds its charge, next to three big, +colour-coded numbers: **charge %**, **load %**, **runtime left**. The health checks, diagnostics +buttons and history stay below them, working exactly as they do now. + +## Dependencies + +- `01_severity_language` (todo) — charge/load/runtime verdicts come from the API, and the + `.hero` style comes from `dash.css`. +- `03_system_per_host_panels` (todo) — proves the hero pattern on `/system` first, so the UPS + page is not the place that pattern gets invented. + +## Tasks + +1. `01_ups_hero_data.md` — confirm the API gives what the hero row and glyph need (`numbers.charge_pct`, `numbers.load_pct`, `numbers.battery_runtime`, `numbers.battery_voltage`, `charger_status`, `on_battery`) with severity attached; contract tests against real upsd. +2. `02_battery_glyph.md` — `DashCharts.battery(container, {charge, severity, state, label})`: inline SVG with a fill proportional to charge, severity colour, charging/on-battery treatment, accessible label. +3. `03_ups_hero_row.md` — restructure the card in `page-ups.js`/`ups.html`: hero row + glyph on top, checks and diagnostics below, `n/a` never coloured. +4. `04_verify_real_batteries.md` — verify against both real UPSes plus the low-charge and on-battery states reachable only through the simulator. + +## Testing & Quality + +- Python: the only expected change is in `sources/nut.py`'s `payload()` (severity + any + missing glyph field); coverage **>90%** there. +- Contract: `tests/test_ups_contract.py` asserts the hero fields and their severities exist for + the shapes the real drivers produce — including `battery_temperature` being `None`, which is + true of both the APC and the CyberPower here. +- Glyph states are exercised in devtools with synthetic values (`100 ok`, `60 warn`, + `12 fail`, `null`) **and** in the live simulator (`NUT_SIMULATE=1`), which can be driven to + low charge and on-battery — real hardware will not do that on demand, and `nut_sim.py`'s + quick test already flips `OB` for ~12s. +- Live: `uv run pytest -m live --live` covers the real UPSes' numbers. + +## Completion Criteria + +- [ ] each UPS card shows charge / load / runtime as its three largest elements, coloured from API `severity` +- [ ] the glyph's fill visibly tracks charge: `data/logs/ups-glyph-*.png` at 100 %, mid, low, and the unknown state +- [ ] on battery the glyph/card state changes visibly (simulator is acceptable proof — say which in `NOTES-verify.md`) +- [ ] `SIMULATED` still badges the card when `NUT_SIMULATE=1`, and never when reading real upsd +- [ ] battery-test buttons unchanged: `Quick battery test` → confirm dialog → **Cancel** works, and `DANGER` is still required for the power-cutting ones +- [ ] `uv run pytest` green, coverage ≥94%, `ruff` + `pyright` clean, `-m live --live` green +- [ ] no behavior change in completed phases diff --git a/.agents/phases/todo/05_ups_hero_battery_glyph/01_ups_hero_data.md b/.agents/phases/todo/05_ups_hero_battery_glyph/01_ups_hero_data.md new file mode 100644 index 0000000..840e209 --- /dev/null +++ b/.agents/phases/todo/05_ups_hero_battery_glyph/01_ups_hero_data.md @@ -0,0 +1,51 @@ +# Task 01 — UPS hero data contract + +**Phase:** `05_ups_hero_battery_glyph` · **Source:** `TODO.md:7` — "charge, load, and runtime left to be big, obvious, and color coded" +**Story:** n/a + +## Objective + +Know exactly what the hero row reads, and prove it — with real batteries, since the +CyberPower and the APC disagree in ways the fixtures do not cover. + +## Work + +1. Inspect what `/api/ups/snapshot` returns today for the two real UPSes: + `curl -s localhost:8090/api/ups/snapshot | python3 -m json.tool` + (dev forwards `127.0.0.1:3494` → driveripper's upsd on `3051`). +2. Confirm `payload()` in `src/dashboards/sources/nut.py` supplies, per card: + - `numbers.charge_pct`, `numbers.load_pct`, `numbers.battery_runtime`, + `numbers.battery_voltage`, `numbers.battery_temperature` + - `charger_status` (`full` / `ol` / `dchg`) and `on_battery` — the glyph's state comes from + these, not from guessing + - `severity` for charge / load / runtime (from phase 01; `sources/nut.py`'s + `payload()` is where to attach it if it is not there yet) + - `health.overall` and `health.counts` (unchanged — the card header already uses them). +3. `tests/test_ups_contract.py` (new) — + - a `UpsReading` built from captured-style vars asserts each of the paths above exists, and + that a missing `battery.temperature` yields `None` rather than 0 (both real drivers omit + it — this is the exact real-world shape); + - charge 100 → `ok`; charge 45 → `warn`; charge 15 → `fail` (inverted metric); + - runtime 3600 s → `ok`; 400 s → `warn`; 200 s → `fail`; `None` → `None` severity, and the + page-side formatting of `None` must be `n/a` (assert whatever helper is used returns the + app's `naText`, not `0s`); + - `charger_status == "full"` with `on_battery == False` renders "charging" nowhere, and + `on_battery == True` makes the glyph's on-battery state available. + - ASSUMPTION: the glyph derives its state from `charger_status` + `on_battery` rather than + NUT status flags being re-parsed in JS — the flags are already parsed server-side into + `states[]` (`UPS_STATE_LABELS`) and re-parsing them client-side duplicates that map. + - ASSUMPTION: runtime prose stays `Dash.duration()`-formatted (`10m 21s`, `1h 32m`) — no new + formatter, because the UPS page already prints runtime that way. + +## Testing & Quality + +- Coverage **>90%** on `sources/nut.py` changes. +- Live: `uv run pytest -m live --live` — `test_upsd_reports_both_upses_with_sane_units` is the + real-hardware version of the same assertions. + +## Completion Criteria + +- [ ] `uv run pytest tests/test_ups_contract.py -v` green +- [ ] `curl -s localhost:8090/api/ups/snapshot` shows `severity` for charge/load/runtime on **both** `apc` and `cyberpower` +- [ ] `battery_temperature` is `null` on both, and nothing in the API turned it into 0 +- [ ] `uv run pytest` green; `ruff` + `pyright` clean diff --git a/.agents/phases/todo/05_ups_hero_battery_glyph/02_battery_glyph.md b/.agents/phases/todo/05_ups_hero_battery_glyph/02_battery_glyph.md new file mode 100644 index 0000000..b0377b8 --- /dev/null +++ b/.agents/phases/todo/05_ups_hero_battery_glyph/02_battery_glyph.md @@ -0,0 +1,58 @@ +# Task 02 — Battery glyph + +**Phase:** `05_ups_hero_battery_glyph` · **Source:** `TODO.md:7` — "an SVG battery symbol that shows current charge would be a nice touch" +**Story:** n/a + +## Objective + +An inline SVG battery whose fill *is* the charge, whose colour *is* the severity, and whose +outline says "unknown" when there is no number — in the same hand-rolled-SVG house style as +the charts. + +## Work + +1. `src/dashboards/web/static/charts.js` — `battery(container, opts)` with + `{charge: number|null, severity: 'ok'|'warn'|'fail'|null, state: 'online'|'onbattery'|'charging'|'unknown', label?: string}`: + - horizontal cell: rounded body + terminal nub on the right, drawn as one `` with a + `` fill sized `charge/100` of the interior, clipped so a 0 % charge leaves an + empty shell rather than a stray pixel; + - colour from the theme tokens via the existing `cssVar()`: `--ok` / `--warn` / `--fail`; + `severity: null` → outline only, no fill, muted stroke; + - `charge` clamped to 0–100 but never *rounded up* to visible: 2 % must look nearly empty, + because a reassuring full-looking battery at 2 % is a lie; + - state affordances drawn in SVG, no icon font: a bolt overlay when + `state === 'charging'`, a subtle hatch or dashed outline when on battery, and a dash + pattern through the body when charge is unknown; + - accessible by construction: `role="img"`, `` built from the inputs + ("Charge 100 percent, on line"), and `aria-hidden` only when the adjacent hero number + already carries the same text — pick one, and comment which; + - scales by container width (`viewBox` + `preserveAspectRatio`) so the hero row can make it + big on desktop and small on a phone. +2. No animation. If a fill transition is added at all, it must sit behind + `@media (prefers-reduced-motion: no-preference)`. +3. Export it from the `window.DashCharts = { … }` object at `charts.js:377`. +4. `src/dashboards/web/static/dash.css` — only what the glyph needs (sizing, `.battery` + wrapper, the on-battery hatch if it is CSS rather than SVG). + - ASSUMPTION: horizontal cell, not vertical — runtime/charge read left-to-right next to the + hero numbers and it matches how NUT prints battery bars in `upsc`. + - ASSUMPTION: no emoji/Unicode battery character anywhere; it would render differently per + platform and cannot be coloured by severity. + - ASSUMPTION: `state` is derived in the page from `status`/`charger_status`/`on_battery` + (`page-ups.js`), because that mapping is presentation, not data. + +## Testing & Quality + +- Static-asset assertion in `tests/test_api.py` that `battery` is exported from + `/static/charts.js`. +- Browser/devtools matrix — call it directly with synthetic inputs and screenshot each: + `{charge:100, severity:'ok', state:'online'}`, `{charge:60, severity:'ok', state:'charging'}`, + `{charge:35, severity:'warn', state:'onbattery'}`, `{charge:12, severity:'fail', state:'onbattery'}`, + `{charge:2, severity:'fail', state:'onbattery'}`, `{charge:null, severity:null, state:'unknown'}`, + and `charge:0`. Also zoom to 200 % and check both themes. + +## Completion Criteria + +- [ ] `curl -s localhost:8090/static/charts.js | grep -c "function battery"` → 1 +- [ ] the six-state devtools matrix is screenshotted into `data/logs/` and each state is distinguishable without colour (shape/hatch/label) +- [ ] `` present and correct for at least the online and on-battery cases +- [ ] `uv run pytest` green; `ruff` + `pyright` clean diff --git a/.agents/phases/todo/05_ups_hero_battery_glyph/03_ups_hero_row.md b/.agents/phases/todo/05_ups_hero_battery_glyph/03_ups_hero_row.md new file mode 100644 index 0000000..c3e7f28 --- /dev/null +++ b/.agents/phases/todo/05_ups_hero_battery_glyph/03_ups_hero_row.md @@ -0,0 +1,64 @@ +# Task 03 — The hero row + +**Phase:** `05_ups_hero_battery_glyph` · **Source:** `TODO.md:7` — "charge, load, and runtime left to be big, obvious, and color coded. Also an SVG battery symbol that shows current charge" +**Story:** n/a + +## Objective + +Each UPS card opens with the battery glyph and three big numbers. Everything else on the card +moves below them and changes nothing about how it works. + +## Work + +1. `src/dashboards/web/static/page-ups.js` — `upsCard()` (L34): + - replace the top of the `.metrics` block (L50–68) with a hero row: the `data-k="glyph"` + container plus three `.hero` blocks — **Charge %**, **Load %**, **Runtime left** — with + `.sub` lines for the context that matters (charge: `12 V nominal · 27.3 V`; load: + `128 W of 900 W` when `realpower`/`realpower_nominal` allow it, otherwise just the %; + runtime: `at the current load`); + - demote to a secondary, visibly smaller row or the existing disclosure: line in, output, + battery volts, battery amps, battery °C. Do not delete them — they are the numbers you + want the moment the UPS is on battery; + - keep unchanged: the state chips (`OL`/`OB`/`LB`), the `SIMULATED` badge, the + `degraded`/`healthy` chip, health checks (`data-k="checks"`), diagnostics + (`data-k="commands"`), the NUT variable dump (`data-k="raw"`), and the whole + `#confirm-dialog` flow including typing `DANGER`. +2. `renderCard()` (L84): + - hero values via `Dash.hero()`, colours via `Dash.severityFrom(item.severity.charge_pct)` + etc. — **delete the `severityFor(...)` call in this file**; + - `runtimeText()` (L24) stays the single place runtime prose is built (`1h 32m`, + `10m 21s`); `null` → `n/a` with no colour — never `0m`; + - derive the glyph state: `on_battery` → `'onbattery'`; else `charger_status` in + ('dchg',) → `'charging'`; `charger_status === 'full'` or `status` containing `OL` → + `'online'`; nothing known → `'unknown'`. Comment the mapping — NUT's flags are subtle + (`OL` and `CHRG` can both be present, and a fully-charged-on-line unit reports + `charger.status: full`, which is not "charging"); + - draw the glyph through `DashCharts.battery()`; re-draw only when charge/state changes so + the poll loop does not rebuild DOM every 5s. +3. `src/dashboards/web/pages/ups.html` — the cards are built in JS into `#ups-cards`; only + the four history charts (`#chart-charge`, `#chart-runtime`, `#chart-voltage`, + `#chart-load`), the events card and the dialog live here. Leave them; if a hero row needs a + wrapper class, put it in `dash.css`. +4. `src/dashboards/web/static/dash.css` — hero row layout (glyph + three heroes, wraps to a + stack on narrow screens), secondary row visibly secondary (muted, smaller), and the card + must not grow so tall that two cards stop fitting side by side on a laptop. + - ASSUMPTION: the four history charts and the event log stay where they are. The TODO says + the page "looks good"; only the top of each card changes. + - ASSUMPTION: charge wins the space next to the glyph, runtime is the number people + actually act on. Both are heroes; neither is buried. + +## Testing & Quality + +- Contract already covered by `tests/test_ups_contract.py` (task 01). +- Browser: real fleet (`apc`, `cyberpower`), both themes, plus the synthetic low/on-battery + states from task 04's simulator pass. +- Watch the poll: numbers update every `NUT_INTERVAL_SECONDS` (5s) without layout shift — the + tabular numerals from phase 01 make that true; verify rather than assume. + +## Completion Criteria + +- [ ] each card's three largest elements are Charge / Load / Runtime, colour-coded from API severity +- [ ] the glyph sits with them and visibly matches charge (both real UPSes are at 100 %, so verify the match with the synthetic states too) +- [ ] battery °C renders `n/a` on both real UPSes without a failure colour +- [ ] battery-test buttons + `DANGER` confirmation still work (click one through to the dialog, then **Cancel** — do not send a command) +- [ ] `uv run pytest` green, `ruff` + `pyright` clean diff --git a/.agents/phases/todo/05_ups_hero_battery_glyph/04_verify_real_batteries.md b/.agents/phases/todo/05_ups_hero_battery_glyph/04_verify_real_batteries.md new file mode 100644 index 0000000..d0d1f34 --- /dev/null +++ b/.agents/phases/todo/05_ups_hero_battery_glyph/04_verify_real_batteries.md @@ -0,0 +1,57 @@ +# Task 04 — Verify on real batteries, and the states they will not give you + +**Phase:** `05_ups_hero_battery_glyph` · **Source:** `TODO.md:7` — the whole item; the colour coding only means something if the alarming states have been seen +**Story:** n/a + +## Objective + +See the hero row in every state that matters — including the two that only the simulator can +produce, because unplugging a live UPS to test a dashboard is not a test. + +## Work + +1. Real fleet (dev env points at `127.0.0.1:3494` → upsd on driveripper:3051): + - `apc` — Back-UPS RS 1500MS: charge 100 %, load ~47 %, runtime ~10m, battery age + **5.3 years → warn**, no battery temp; + - `cyberpower` — PR1500LCDRT2U: charge 100 %, load ~14 %, runtime ~56m, battery age + **unknown** (`battery.mfr.date: CPS` is not a date and must not become an age), no + battery temp. + Screenshot both cards. The two "unknown/warn" differences are the point: they prove the + page is not rendering a template. +2. States only `nut_sim.py` can produce — start the app with `NUT_SIMULATE=1` and + `NUT_ALLOW_ACTIONS=0` and reach them the honest way (drive `SimUps.tick()`/`step()` in a + unit test or a REPL and read `NutSimulator.read()` output; the quick battery test puts a + unit on `OB` for ~12s): + - 100 % online → green, no bolt; + - ~60 % charging → bolt visible; + - ~35 % on battery → amber, on-battery treatment; + - 12 % on battery (`OB LB`) → red; + - charge absent → outline only, `n/a`, no colour, no fake runtime. + Screenshot each. `SIMULATED` must be visible in every one of these shots — if it is not, + that is a bug in this phase, not a cosmetic miss. +3. Failure path: kill the upsd forward (`./scripts/dev-tunnels.sh stop`/`start`, or stop + `nut-server` remotely **only** if the operator agrees) and confirm the cards keep the last + good numbers and mark them stale rather than showing `n/a` everywhere + (`AGENTS.md` invariant 4). +4. Commands: click **Quick battery test** → dialog appears → type `DANGER` → then **Cancel**. + Verifying the gate without sending the command is the correct amount of clever here; if + the operator explicitly asks for a live quick test, run it and watch `ups.test.result` + change to `PASSED` (and the health check go green), which is the payoff for the whole + phase. +5. Record what was seen in `NOTES-verify.md` in this phase directory, listing each screenshot + and which state it proves. + - ASSUMPTION: no instant command is sent as part of automated verification. A deep test + discharges a battery that something real is plugged into. + +## Testing & Quality + +- Evidence task; fix on discovery, then re-shoot. + +## Completion Criteria + +- [ ] `data/logs/ups-real-apc.png` and `data/logs/ups-real-cyberpower.png` exist and were read +- [ ] five synthetic states screenshotted, each with the `SIMULATED` badge visible +- [ ] stale-source screenshot shows last-good values, not zeros +- [ ] the `DANGER` dialog path exercised and cancelled; no command was sent without the operator asking +- [ ] `NOTES-verify.md` written in this phase directory +- [ ] `uv run pytest` and `uv run pytest -m live --live` green diff --git a/.gitignore b/.gitignore index cf3b263..01e245c 100644 --- a/.gitignore +++ b/.gitignore @@ -31,9 +31,10 @@ agent/*.tar.gz .idea/ .DS_Store -# Agent/pipeline scratch. This project's plan lives in AGENTS.md (tracked) — nothing under -# .agents/ is anything another checkout needs, so all of it stays out. -.agents/ +# Agent/pipeline scratch ONLY. The planning tree (.agents/PLAN.md, .agents/phases/) is +# tracked on purpose — it is the project's roadmap and history, not throwaway state. +.agents/phase-sessions/ +.agents/pipeline.log # The original brief for this project. Kept in the checkout as local context, not history: # README.md and AGENTS.md carry the decisions that came out of it. diff --git a/AGENTS.md b/AGENTS.md index ab58f43..86002fc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,6 +4,19 @@ The plan, the rules, and the environment facts an agent needs to work on this re re-deriving them. `README.md` is the operator-facing how-to; this file is the why, the invariants, and the state of the fleet. +## Working protocol + +This project runs the `.agents/phases/` phased-execution pipeline. + +1. Always read `.agents/PLAN.md` first to understand the project context and goals. +2. Follow the phased execution protocol in `.agents/phases/`. +3. Never modify `.agents/PLAN.md` or any files in `.agents/phases/complete/`. +4. If you need to update any file in `.agents/phases/todo/`, you must ask the user for permission first. +5. Strictly adhere to the **LOCKED DECISIONS** listed in `.agents/PLAN.md`. + +Everything below this section is the project's own content: what the app is, the rules that +keep its numbers honest, and the state of the fleet it watches. + ## What this is Three dashboards for one homelab, behind one domain, in one process, backed by one SQLite diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..4640904 --- /dev/null +++ b/TODO.md @@ -0,0 +1 @@ +# TODO diff --git a/deploy/nut-sudoers b/deploy/nut-sudoers index 1cd08c0..7e6f52b 100644 --- a/deploy/nut-sudoers +++ b/deploy/nut-sudoers @@ -1,55 +1,54 @@ -# driveripper-ops sudo rules for the UPS dashboard work +# driveripper-ops sudo rules for the homelab dashboards UPS work. # -# Install as /etc/sudoers.d/dashboards-nut (visudo -cf it first): +# Install (from the repo checkout, as a user with sudo): # -# sudo visudo -cf - <<'EOF' ... EOF # syntax check -# install -m 0440 deploy/nut-sudoers /tmp/dashboards-nut -# sudo mv /tmp/dashboards-nut /etc/sudoers.d/dashboards-nut +# sudo visudo -cf deploy/nut-sudoers # syntax check FIRST +# sudo install -m 0440 -o root -g root deploy/nut-sudoers /etc/sudoers.d/dashboards-nut +# sudo visudo -cf /etc/sudoers.d/dashboards-nut # and check the installed copy # -# Keep it scoped: paths are absolute (sudoers uses `secure_path`), arguments are wildcards -# only where the command is read-only or idempotent, and nothing here allows a shell, -# `EDITOR=`, `dnf install` of arbitrary code, or `upsd -c` with an attacker-chosen file. +# Scope: absolute paths only (`secure_path` is in effect), wildcards only where the command +# is read-only or idempotent, and nothing here grants a shell, `sh -c`, an editor, or a +# client that can send instant commands. +# +# WHY THESE ARE NEEDED +# Reads already work with zero privileges — upsd on driveripper serves `LIST UPS` / +# `LIST VAR` unauthenticated (on port 3051, not 3493). What is missing is a upsd.users +# entry with `instcmds`, which is the only thing standing between the dashboard and its +# battery-test buttons. `tee /etc/ups/*` is already granted, but writing upsd.users blind +# would drop the accounts upsmon authenticates as — upsmon must run as a user with +# `shutdown`, so an overwritten file means no orderly power-down. So: read first. -# ---- already present on driveripper (this is what the dashboard setup used) ---------- # -# (root) NOPASSWD: /usr/bin/tee /etc/ups/* # write upsd.conf / upsd.users -# (root) NOPASSWD: /usr/bin/cp /etc/ups/* /home/ops/* # back up a config before editing +# ---- already present on driveripper (used during setup) ------------------------------ # +# (root) NOPASSWD: /usr/bin/tee /etc/ups/* # write a config file +# (root) NOPASSWD: /usr/bin/cp /etc/ups/* /home/ops/* # copy one out (root-owned) # (root) NOPASSWD: /usr/bin/systemctl start|restart|reload|enable * -# (root) NOPASSWD: /usr/bin/udevadm * # USB device discovery/replug +# (root) NOPASSWD: /usr/bin/udevadm * # USB device discovery # (root) NOPASSWD: /usr/bin/firewall-cmd * # only if upsd must serve the LAN -# (root) NOPASSWD: /usr/bin/dnf install * # nut, nut-client, nut-dev +# (root) NOPASSWD: /usr/bin/dnf install * # nut, nut-client -# ---- still needed: diagnose why upsd runs but binds nothing -------------------------- # -# Reading is the blocker. `upsd` is active, both drivers are up, and 3493 has no listener, -# but /etc/ups/* is 0640 root:nut and the journal needs group membership: -# $ sudo cat /etc/ups/upsd.conf -> Permission denied -# $ journalctl -u nut-server -n 50 -> No entries (not in systemd-journal/wheel) -# $ upsc -l -> Error: Connection failure: Connection refused - -Cmnd_Alias NUT_DIAGNOSE = \ +# ---- read: what is actually configured, before anything is rewritten ----------------- # +Cmnd_Alias NUT_READ = \ /usr/bin/cat /etc/ups/*, \ - /usr/bin/systemctl status *, \ - /usr/bin/journalctl -u nut-server *, /usr/bin/journalctl -u nut-driver@*, \ /usr/bin/journalctl -u nut*, \ - /usr/sbin/ss -ltnp, \ - /usr/sbin/getenforce, \ - /usr/bin/ausearch *, /usr/sbin/sealert -l *, \ - /usr/sbin/semanage port -l, \ - /usr/bin/upsdrvctl status, /usr/bin/upsdrvctl status *, \ + /usr/bin/systemctl status nut*, \ + /usr/bin/upsdrvctl status, \ /usr/bin/upsc *, \ - /usr/bin/lsusb + /usr/sbin/ss -ltnp, \ + /usr/bin/lsusb, \ + /usr/sbin/getenforce, \ + /usr/bin/ausearch -m avc *, /usr/sbin/sealert -l * -# ---- still needed: apply the fix and verify ------------------------------------------ # -# `tee /etc/ups/*` already writes configs; these cover the reload path, SELinux booleans, -# and a bounded cleanup of the .bak files `tee`-based edits leave behind. -Cmnd_Alias NUT_APPLY = \ - /usr/bin/tee /etc/ups/upsd.conf, /usr/bin/tee /etc/ups/upsd.users, \ - /usr/bin/tee /etc/ups/ups.conf, /usr/bin/tee /etc/ups/nut.conf, \ - /usr/bin/rm -f /etc/ups/*.bak-*, \ - /usr/bin/upsdrvctl restart, /usr/bin/upsdrvctl restart *, \ - /usr/sbin/setsebool -P allow_upsd_tcp_server 1, \ - /usr/sbin/setsebool -P nis_enabled 0 +# ---- write: add/replace a scoped user, then reload ----------------------------------- # +Cmnd_Alias NUT_WRITE = \ + /usr/bin/tee /etc/ups/upsd.users, \ + /usr/bin/tee /etc/ups/upsd.conf, \ + /usr/bin/systemctl restart nut-server, \ + /usr/bin/systemctl reload nut-server -# NOTE: deliberately NOT allowed, so an agent cannot power-cycle anything by accident: -# /usr/bin/upscmd (any form) — the dashboard's own gated POST is the only command path -# systemctl stop * — stopping nut-server would blind the UPS view -# editors, shells, `sh -c` — no escape hatches +# ---- deliberately NOT granted -------------------------------------------------------- # +# /usr/bin/upscmd, /usr/bin/upsrw — the dashboard's gated POST /api/ups/command is the +# ONLY command path in this project, by design +# systemctl stop * — stopping nut-server blinds the UPS view +# tee /etc/ups/ups.conf, upsmon.conf — driver instances and shutdown policy are the +# operator's, not the dashboard's +# sh, bash, vim, tee to anything else — no escape hatches