phase: 01_severity_language
**Gate defect found & fixed (the "does not lint" bug).** `.agents/validate.sh` — the phase gate — ran `python3 -m pytest` on `/usr/bin/python3` + `~/.local` site-packages (not the uv venv), ran `ruff check .` *only if* a ruff happened to be on PATH, and **never ran pyright or the coverage floor**. Rewrote it to the project's real gate: cd to repo root → `uv run pytest --cov --cov-fail-under=94` → `uv run ruff check src tests agent` → `uv run pyright` → `node --check web/static/*.js` → hygiene gates; a missing toolchain now fails loudly. Provoked it to confirm it bites (floor + a seeded F401 both fail).
**Verification (all green)**
- `bash .agents/validate.sh` → **exit 0**: 356 passed, 6 skipped · coverage **94.75%** (floor 94) · ruff `All checks passed!` · pyright `0 errors` · node syntax clean
- `uv run pytest --cov` → `severity.py` **100%**; `uv run pytest -m live --live` → **6 passed**; `uv pip list --outdated` → empty
- Criteria: `classify('cpu.usage_pct',…)` → `fail`/`ok`/`None` ✓ · `/api/ups/snapshot` carries `severity` beside `charge_pct`/`load_pct`/`battery_runtime` (apc now live-`warn` at 498 s — threshold working) ✓ · no `.sev-*`/synonym vocabulary (only a comment saying so) ✓ · pytest/coverage/lint ✓ · dev server restarted, `/api/health` ok 6/6 sources, all pages+assets 200 ✓
- Chromium against live fleet: `severityFrom` cell→`fail`, bare→`warn`, null→`''`, unrated→`''`; `hero` fail → `rgb(207,34,46)`, null → `n/a` with **no** verdict class, warn+bar 42%; all four pages render unchanged → `data/logs/phase01-final-{index,system,llm,ups}.png`
**Docs (full review, incl. pre-existing drift).** README: `/ups` row claimed "upsd on 3493" (fleet is 3051, stated elsewhere in the same file); config table omitted `DASH_HOST/PORT`, llm/ups `*_INTERVAL_SECONDS`, `*_TIMEOUT_SECONDS`, `GUFO/VLLM_API_KEY`, `POLL_MS_*`, `DEBUGPY` (+ dedup); severity bullet omitted `disks`/`gpus[]`/rated LLM cells and the ratio rescale; agent section listed only `/snapshot` though `/info`+`/health` exist. AGENTS.md: same 3493 claim; `install-agent.sh --token-file` (its flags are `--bind/--port/--token/--rotate-token/--status/--uninstall` — it hands *the agent* `--token-file`); gap 3 "Two commits" (now 6, still no `origin`); gap 1 given the sudoers install command + verified live gating reason; coverage line now 80% repo floor vs 94% gate floor; cyberpower `identity.model: GX1500U` vs `ups.description` quirk. `.env.dev.example`: `DEV_FORWARDS` remote port 3493→3051 and engine URLs that pointed at `127.0.0.1` with nothing forwarding them.
**Deviations.** No `ruff format` (30 files would reformat; the project gate is `ruff check` + pyright — reformatting completed phases is out of scope). **No live-infra changes**: enabling the battery-test buttons needs `deploy/nut-sudoers` installed on driveripper + a nut-server restart and a scoped `[dashboards]` `instcmds` user — prod write with a documented shutdown risk (AGENTS gap 1), so repo-side only. 36 pre-existing `ResourceWarning: unclosed database` from test fixtures are noise, not failures; left rather than churn ~15 fixtures. `severity.attach()` is unit-tested but has no production call site until phases 02–05.
**Next pending phase:** `02_overview_fleet_and_alerts` (`01_alerts_builder.md`).
This commit is contained in:
@@ -0,0 +1,45 @@
|
||||
# Phase 01 — Severity language
|
||||
|
||||
**Source:** `TODO.md` L5, L7 — "proper green (good), yellow (warn), red (danger) color coding" / "charge, load, and runtime left to be big, obvious, and color coded"
|
||||
**Story:** n/a
|
||||
**Context:** `.agents/PLAN.md` → *Architectural Anchors* (severity computed in Python, JS renders classes), `src/dashboards/sources/host.py`, `src/dashboards/sources/nut.py`, `src/dashboards/api.py`, `src/dashboards/web/static/dash.css`, `src/dashboards/web/static/app.js`
|
||||
|
||||
## Objective
|
||||
|
||||
Introduce one place that decides whether a number is fine, worrying, or bad, and put that
|
||||
verdict on the values the API already returns. Phases 02–05 all want the same colour
|
||||
language; doing it here means they render a class instead of each inventing its own
|
||||
thresholds.
|
||||
|
||||
Deliverable: `severity` next to the value in the payloads the pages colour, and CSS/JS
|
||||
helpers that make a "hero number" one call.
|
||||
|
||||
## Dependencies
|
||||
|
||||
— (none)
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_severity_module.md` — `src/dashboards/severity.py`: the threshold table and the classifier, unit-tested.
|
||||
2. `02_severity_in_payloads.md` — attach `severity` to host and UPS values and to the `/api/summary` rows; API tests.
|
||||
3. `03_hero_number_style.md` — a `.hero` big-number style on the existing `.ok/.warn/.fail` colours, plus `Dash.hero()` and `Dash.severityFrom(payload)` so pages read the verdict instead of passing thresholds.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: every threshold boundary (below warn, exactly at warn, between, exactly at danger,
|
||||
above), `None` input, and the "unknown is not ok" rule — a value the upstream does not
|
||||
publish must get `severity: null`, never a green pass.
|
||||
- API: `/api/system/snapshot`, `/api/ups/snapshot`, `/api/summary` carry `severity` where
|
||||
the spec says, and **no existing field changes name or meaning** (additive-only anchor).
|
||||
- Coverage: **>90%** on new/modified code.
|
||||
- Live: `uv run pytest -m live --live` must stay at 6 passed.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run python -c "from dashboards import severity; print(severity.classify('cpu.usage_pct', 95))"` → `fail`; `60` → `ok`; `None` → `None`
|
||||
- [ ] `curl -s localhost:8090/api/ups/snapshot | python3 -m json.tool` shows `severity` beside `charge_pct`, `load_pct`, `battery_runtime`
|
||||
- [ ] no new colour vocabulary: classes stay `.ok` / `.warn` / `.fail` as in `dash.css`
|
||||
- [ ] `uv run pytest` green; coverage ≥94% overall, >90% on `severity.py`
|
||||
- [ ] `uv run ruff check src tests agent && uv run pyright` clean
|
||||
- [ ] Existing pages render unchanged (severity is additive; nothing consumes it yet) — one screenshot in `data/logs/`
|
||||
- [ ] no behavior change in completed phases
|
||||
@@ -0,0 +1,61 @@
|
||||
# Task 01 — Severity module
|
||||
|
||||
**Phase:** `01_severity_language` · **Source:** `TODO.md:5` — "proper green (good), yellow (warn), red (danger) color coding" and `TODO.md:7` — "charge, load, and runtime left to be big, obvious, and color coded"
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Move the good/worrying/bad decision out of the browser and into one Python module, so the
|
||||
four pages cannot drift into four different opinions about what red means.
|
||||
|
||||
Today the thresholds live in call sites: `page-system.js:129` passes `75, 90`,
|
||||
`page-system.js:139` and `:144` pass `80, 92`, `page-index.js:89` passes `80, 92`,
|
||||
`page-llm.js:195` and `:307` pass `75, 90`, and `Dash.severityFor` (`app.js:105`) defaults
|
||||
to `75/90`. That is the drift this task ends.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/severity.py` (new) —
|
||||
- `THRESHOLDS: dict[str, tuple[float, float]]` keyed by metric name, with the ratified
|
||||
defaults: `cpu.usage_pct` 75/90, `mem.used_pct` 80/92, `disk.used_pct` 80/90,
|
||||
`cpu.temp_c` 75/90, `gpu.busy_pct` 85/95, `gpu.mem_used_pct` 85/95,
|
||||
`llm.kv_cache_usage_pct` 75/90, `ups.charge_pct` 50/20, `ups.battery_runtime` 600/300,
|
||||
`ups.load_pct` 70/90.
|
||||
- `INVERTED: set[str]` — metrics where **lower is worse** (`ups.charge_pct`,
|
||||
`ups.battery_runtime`), so callers cannot get the direction wrong.
|
||||
- `classify(metric: str, value: float | None) -> str | None` returning `ok`, `warn`,
|
||||
`fail`, or `None`. `None` in → `None` out, always.
|
||||
- `attach(metric: str, value: float | None) -> dict` returning
|
||||
`{"value": value, "severity": classify(...)}` for the payload shapes that are plain
|
||||
numbers, and `with_severity(metric, cell)` for payloads that already are
|
||||
`{"value":…, "source":…, "derived":…}` (LLM metrics) — additive, never a rename.
|
||||
- Module docstring stating the two rules: unknown is **not** ok (it is `None`, the page
|
||||
renders `n/a` with no colour), and a threshold exists nowhere else in the app. Ties to
|
||||
the `AGENTS.md` "never invent a number" rule.
|
||||
2. `tests/test_severity.py` (new) —
|
||||
- boundaries inclusive on the warn edge (`75 → warn`, `74.9 → ok`), danger edge
|
||||
(`90 → fail`), every metric in the table, both directions for the inverted ones,
|
||||
unknown metric name → `None` (no exception), `None` and `float("nan")` → `None`.
|
||||
- a table sanity test asserting each entry's bounds are ordered correctly for its
|
||||
direction, so a future edit cannot silently invert a metric.
|
||||
- ASSUMPTION: the numbers above were confirmed in chat on 2026-10-16; they match what
|
||||
the call sites already pass, except the two UPS metrics which are new.
|
||||
- ASSUMPTION: the vocabulary is **`ok|warn|fail`**, not `ok|warn|danger` — it is what
|
||||
`dash.css` (`.bar.ok`, `.dot.warn`, `.chip.fail`, `ul.checks li.fail`) and
|
||||
`sources/nut.py`'s `health.checks[].status` already use. The TODO's word "danger" maps
|
||||
to `fail`; do not introduce a synonym for the same colour.
|
||||
- ASSUMPTION: `sources/nut.py`'s existing `health.checks[].status` values are left
|
||||
untouched (additive-only anchor). `severity` is a new field on *values*, computed by
|
||||
this module; `nut.assess()` keeps its own judgement about *conditions*.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit only, no I/O, no fixtures needed.
|
||||
- Coverage: **>90%** on `severity.py`.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest tests/test_severity.py -v` green
|
||||
- [ ] `uv run python -c "from dashboards.severity import classify as c; print(c('cpu.usage_pct',95), c('cpu.usage_pct',60), c('cpu.usage_pct',None), c('ups.charge_pct',15))"` → `fail ok None fail`
|
||||
- [ ] `uv run pytest` green overall; `ruff` + `pyright` clean
|
||||
- [ ] Nothing else changed yet — this task adds a leaf module and its tests
|
||||
@@ -0,0 +1,62 @@
|
||||
# Task 02 — Severity in the payloads
|
||||
|
||||
**Phase:** `01_severity_language` · **Source:** `TODO.md:5` — "proper green (good), yellow (warn), red (danger) color coding" and `TODO.md:7` — "big, obvious, and color coded"
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Put the verdict where the pages can read it: every value that will be colour-coded travels
|
||||
with a `severity`, computed server-side.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/sources/host.py` — in `parse_host_snapshot()`'s payload, add a
|
||||
`"severity"` sub-dict to the sections that have rated values, leaving every existing key
|
||||
exactly as it is:
|
||||
- `cpu.severity = {"usage_pct": …, "temp_c": …}`
|
||||
- `memory.severity = {"used_pct": …}`
|
||||
- `disks.severity = {"worst_used_pct": …}`
|
||||
- each row of `gpus[]` gets `"severity": {"busy_pct": …, "mem_used_pct": …}`
|
||||
Use `severity.classify(metric, value)`; import it as `from ..severity import classify`
|
||||
(or `from .. import severity` if that reads better against the existing imports).
|
||||
2. `src/dashboards/sources/nut.py` — in `payload()`, add
|
||||
`"severity": {"charge_pct": …, "load_pct": …, "battery_runtime": …}` to each UPS card.
|
||||
Do not touch `health`, `checks[].status`, or `numbers` — the checks answer "is this
|
||||
condition satisfied", severity answers "how bad is this number", and both stay.
|
||||
3. `src/dashboards/sources/llm.py` — for the metrics that have thresholds
|
||||
(`kv_cache_usage`-family and any other entry in `THRESHOLDS`), attach `severity` inside
|
||||
the existing `{"value":…, "source":…, "derived":…}` cell via `severity.with_severity`,
|
||||
which must leave `value`/`source`/`derived` untouched.
|
||||
4. `src/dashboards/api.py` — in `/api/summary`, add a `severity` key to each host row
|
||||
(`cpu_usage_pct`, `mem_used_pct`, `disk_worst_used_pct`, `temp_c`), each UPS row
|
||||
(`charge_pct`, `load_pct`, `runtime_seconds`), and reuse the section severities for the
|
||||
fleet table. `/api/health` and `/api/config` are unchanged.
|
||||
5. Tests — extend `tests/test_host_source.py`, `tests/test_nut.py`, `tests/test_llm_source.py`
|
||||
and `tests/test_api.py`:
|
||||
- a fixture host whose values straddle thresholds yields the expected severity per field
|
||||
(drive it from the real `tests/fixtures/agent_driveripper.json`, do not hand-copy a
|
||||
snapshot);
|
||||
- the APC fixture/UPS reading with `battery.charge=100` → `ok`, and the same reading with
|
||||
charge forced to 15 → `fail`;
|
||||
- a value the upstream omits (`battery_temperature` is missing on both real UPSes) gets
|
||||
`severity: null`, and the existing `unknown` health check still says `unknown`;
|
||||
- `/api/summary` still contains every key it had before — assert the shape did not shrink
|
||||
(additive-only anchor), then assert the new `severity` keys are present.
|
||||
- ASSUMPTION: severity is attached as a sibling `"severity"` dict rather than by
|
||||
restructuring `cpu.usage_pct` into `cpu.usage_pct.value`, because the existing
|
||||
pages, `tests/test_api.py`, and the live tests read the flat form.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit + API tests above; run the full suite (`uv run pytest`).
|
||||
- Coverage: **>90%** on the changed functions in `sources/host.py`, `sources/nut.py`,
|
||||
`sources/llm.py`, `api.py`.
|
||||
- `uv run pytest -m live --live` against the real fleet + real upsd must stay green — this
|
||||
is the first task whose output can be checked against actual numbers.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `curl -s localhost:8090/api/system/snapshot | python3 -c "import json,sys; d=json.load(sys.stdin); print(d['hosts'][0]['cpu']['severity'], d['hosts'][0]['memory']['severity'])"` prints a verdict per rated field
|
||||
- [ ] `curl -s localhost:8090/api/ups/snapshot` shows `severity` for charge / load / runtime on both `apc` and `cyberpower`, and `battery_temperature` is `null`
|
||||
- [ ] no existing key was renamed or removed in any payload (diff the JSON keys before/after)
|
||||
- [ ] `uv run pytest` green, coverage not below 94%, `ruff` + `pyright` clean
|
||||
@@ -0,0 +1,54 @@
|
||||
# Task 03 — Hero-number style and the JS side of severity
|
||||
|
||||
**Phase:** `01_severity_language` · **Source:** `TODO.md:5,7` — "Larger numbers" / "big, obvious, and color coded"
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Give the pages one way to render a big, colour-coded number, reading the verdict from the
|
||||
payload instead of comparing thresholds in JavaScript.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/web/static/dash.css` — add a `.hero` block built entirely from the
|
||||
existing tokens (`--ok`, `--ok-soft`, `--warn`, `--warn-soft`, `--fail`, `--fail-soft`,
|
||||
`--text`, `--text-muted`, `--text-faint`):
|
||||
- `.hero` (label + value + unit), `.hero .value` at a display size (roughly 2.4rem, tabular
|
||||
numerals so digits do not jitter between polls), `.hero .unit` small and muted,
|
||||
`.hero .sub` for the `Δ10.0s` / "at current load" footnotes;
|
||||
- `.hero.ok .value { color: var(--ok); }` and the `.warn` / `.fail` equivalents — **reuse
|
||||
the `ok|warn|fail` classes the stylesheet already uses** (`.bar.ok`, `.dot.warn`,
|
||||
`.chip.fail`), do not add a parallel `.sev-*` family;
|
||||
- a `.hero-bar` variant that pairs the number with the existing `.bar` element, since the
|
||||
per-host cards (phase 03) and the UPS hero row (phase 05) both want number + bar.
|
||||
2. `src/dashboards/web/static/app.js` —
|
||||
- `hero(el, {label, value, unit, digits, severity, sub})`: renders/updates a hero, writes
|
||||
`naText` when `value` is null **and applies no severity class** (an unknown number must
|
||||
not look green);
|
||||
- `severityFrom(cellOrSeverity)`: accepts a payload cell (`{value, severity}`), a bare
|
||||
severity string, or `null`, and returns the class name to apply — this is what lets a
|
||||
page stop knowing thresholds;
|
||||
- keep `severityFor(value, warnAt, failAt)` working (pages still call it in
|
||||
`page-index.js`, `page-system.js`, `page-llm.js`, `page-ups.js`) but add a comment that
|
||||
it is the fallback for values the API does not rate, and that new call sites should read
|
||||
`severity` from the payload instead.
|
||||
3. Do **not** convert the existing pages in this task — phases 02–05 do that page by page so
|
||||
each page's change is reviewable and revertible on its own.
|
||||
- ASSUMPTION: leaving `severityFor` in place during this phase keeps every page rendering
|
||||
exactly as it does today, which is what "no behaviour change until your phase" requires.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- No JS test framework exists (locked anchor). The contract that matters here is that the
|
||||
helpers are the only path to a colour class, which the next phases' diffs make obvious.
|
||||
- If `tests/test_api.py` or a new test can assert the static assets are served and the CSS
|
||||
contains the `.hero` rules (a string check on `GET /static/dash.css`), add it — that is
|
||||
the cheap, honest end-to-end signal available without a browser test runner.
|
||||
- Coverage: >90% on any Python touched (ideally none beyond that static-asset assertion).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `curl -s localhost:8090/static/dash.css | grep -c "\.hero"` > 0
|
||||
- [ ] Every page still renders as before, unchanged, verified by a screenshot of `/`, `/system`, `/llm`, `/ups` in `data/logs/`
|
||||
- [ ] `uv run pytest` green; `ruff` + `pyright` clean
|
||||
- [ ] no `.sev-*` classes were introduced; colour still comes only from `--ok/--warn/--fail`
|
||||
Reference in New Issue
Block a user