services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 01_severity_language

Browse Source 
**Gate defect found & fixed (the "does not lint" bug).** `.agents/validate.sh` — the phase gate — ran `python3 -m pytest` on `/usr/bin/python3` + `~/.local` site-packages (not the uv venv), ran `ruff check .` *only if* a ruff happened to be on PATH, and **never ran pyright or the coverage floor**. Rewrote it to the project's real gate: cd to repo root → `uv run pytest --cov --cov-fail-under=94` → `uv run ruff check src tests agent` → `uv run pyright` → `node --check web/static/*.js` → hygiene gates; a missing toolchain now fails loudly. Provoked it to confirm it bites (floor + a seeded F401 both fail).

**Verification (all green)**
- `bash .agents/validate.sh` → **exit 0**: 356 passed, 6 skipped · coverage **94.75%** (floor 94) · ruff `All checks passed!` · pyright `0 errors` · node syntax clean
- `uv run pytest --cov` → `severity.py` **100%**; `uv run pytest -m live --live` → **6 passed**; `uv pip list --outdated` → empty
- Criteria: `classify('cpu.usage_pct',…)` → `fail`/`ok`/`None` ✓ · `/api/ups/snapshot` carries `severity` beside `charge_pct`/`load_pct`/`battery_runtime` (apc now live-`warn` at 498 s — threshold working) ✓ · no `.sev-*`/synonym vocabulary (only a comment saying so) ✓ · pytest/coverage/lint ✓ · dev server restarted, `/api/health` ok 6/6 sources, all pages+assets 200 ✓
- Chromium against live fleet: `severityFrom` cell→`fail`, bare→`warn`, null→`''`, unrated→`''`; `hero` fail → `rgb(207,34,46)`, null → `n/a` with **no** verdict class, warn+bar 42%; all four pages render unchanged → `data/logs/phase01-final-{index,system,llm,ups}.png`

**Docs (full review, incl. pre-existing drift).** README: `/ups` row claimed "upsd on 3493" (fleet is 3051, stated elsewhere in the same file); config table omitted `DASH_HOST/PORT`, llm/ups `*_INTERVAL_SECONDS`, `*_TIMEOUT_SECONDS`, `GUFO/VLLM_API_KEY`, `POLL_MS_*`, `DEBUGPY` (+ dedup); severity bullet omitted `disks`/`gpus[]`/rated LLM cells and the ratio rescale; agent section listed only `/snapshot` though `/info`+`/health` exist. AGENTS.md: same 3493 claim; `install-agent.sh --token-file` (its flags are `--bind/--port/--token/--rotate-token/--status/--uninstall` — it hands *the agent* `--token-file`); gap 3 "Two commits" (now 6, still no `origin`); gap 1 given the sudoers install command + verified live gating reason; coverage line now 80% repo floor vs 94% gate floor; cyberpower `identity.model: GX1500U` vs `ups.description` quirk. `.env.dev.example`: `DEV_FORWARDS` remote port 3493→3051 and engine URLs that pointed at `127.0.0.1` with nothing forwarding them.

**Deviations.** No `ruff format` (30 files would reformat; the project gate is `ruff check` + pyright — reformatting completed phases is out of scope). **No live-infra changes**: enabling the battery-test buttons needs `deploy/nut-sudoers` installed on driveripper + a nut-server restart and a scoped `[dashboards]` `instcmds` user — prod write with a documented shutdown risk (AGENTS gap 1), so repo-side only. 36 pre-existing `ResourceWarning: unclosed database` from test fixtures are noise, not failures; left rather than churn ~15 fixtures. `severity.attach()` is unit-tested but has no production call site until phases 02–05.

**Next pending phase:** `02_overview_fleet_and_alerts` (`01_alerts_builder.md`).
This commit is contained in:
ducoterra
2026-10-16 22:22:26 -04:00
parent 4b4962d7fb
commit ad993c7b03
40 changed files with 1384 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+55 -13
View File
@@ -27,7 +27,7 @@ file — plus an overview page that says which of the three you need:
| `/` | Is anything wrong, and where do I look first? | all three, in one request |
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent/host |
| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo (llama.cpp family) + vLLM `/metrics` |
| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` on 3493 (our own wire-protocol client) |
| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) |
History is a first-class feature, not a sparkline afterthought: every number worth graphing
is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`.
@@ -115,11 +115,25 @@ Responsibilities, one line each:
battery. It must never be the default, and every payload it produces says `simulated`.
- `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own).
- `rates.py` — Δ/window estimator, keyed by resolved series name.
- `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric,
`classify()` → `ok`/`warn`/`fail`/`None`, plus `attach()`/`with_severity()` to put the verdict
beside a value. No measurement, no verdict: `None` in → `None` out, never a green pass for
an unknown. The verdicts travel **additively** in payloads — `cpu.severity`,
`memory.severity`, `disks.severity`, `gpus[].severity`, each UPS card's `severity`
(charge/load/runtime), `severity` inside rated LLM cells (KV pressure, classified on a
percent scale because the engines publish a 0-1 ratio), and `severity` per host/fleet/UPS
row in `/api/summary`. Pages render the class; they never compare a number to a threshold.
- `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning.
- `api.py` — GETs everywhere plus the one POST. `/api/config` is redaction boundary.
- `app.py` — assembly; `create_app()` takes injectable settings/store so tests build an app
with an in-memory DB and no targets.
- `web/` — four static pages, one stylesheet, `charts.js`. No framework.
- `web/` — four static pages, one stylesheet, `charts.js`. No framework. `app.js` owns the
colour vocabulary: `Dash.severityFrom(cellOrSeverity)` turns a payload verdict into an
`ok|warn|fail` class (`''` for anything unrated, so unknown never looks green) and
`Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
`.hero` big number (plus an optional `.bar`) from that class — it never compares a number
to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for
values the API does not rate; converting its remaining call sites is phases 02–05's job.
Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
`gpu.busy_pct@0`, `llm.decode_tps`, `ups.charge_pct`), `@stream` for per-device series.
@@ -141,11 +155,13 @@ That scheme is the history API's key — treat it as a public interface.
*Connection refused* and it looks like a dead service — `ss -ltnH` plus `printf 'LIST UPS\n'
| nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down. Two UPSes are
exported over USB and read live through `DEV_FORWARDS="3494:driveripper-ops:3051"`:
`apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U.
`apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U (that is
`ups.description`, the operator's label — the driver's own `identity.model` answers
`GX1500U`, so the description is what identifies the box).
Real-hardware quirks the parsers already handle: `battery.mfr.date` is the useful date
(`battery.date` is often an unset placeholder like `2001-09-25`), CyberPower reports
`battery.mfr.date: CPS` (not a date → `unknown`, never a fake age), neither driver reports
a battery temperature, and `ups.test.result: No test initiated` is the factory state.
a battery temperature, and `ups.test.result: No test initiated` is the factory state on both.
- Host access is via ssh keys as `ops`. Passwordless sudo exists but is command-scoped; the
set we rely on is documented in `deploy/nut-sudoers` (additions are listed there for the
work that is still blocked).
@@ -162,16 +178,24 @@ cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets
./scripts/dev-server.sh status # per-source ok/DOWN
./scripts/dev-server.sh logs
uv run pytest # hermetic: 258 tests, no network
uv run pytest --cov # coverage gate (~94%)
uv run pytest # hermetic: 356 tests, no network
uv run pytest --cov # coverage gate (94.75%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh # re-record ground truth
bash .agents/validate.sh # the phase gate: the four checks above (pytest
# --cov --cov-fail-under=94, ruff, pyright, plus a
# node --check of web/static) + repo hygiene.
# Always `uv run` — a bare `python3`/`ruff` on PATH
# tests an interpreter nobody deploys.
```
Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on `0.0.0.0:8090`,
SQLite under `/var/lib/dashboards` — put a volume there) and `deploy/install-agent.sh`
per monitored host (systemd unit, own user, `--token-file`, `ProtectSystem=strict`).
per monitored host (systemd unit, own user, `--bind`/`--port`/`--token`/`--rotate-token`/
`--status`/`--uninstall`; it stores the token in a root-owned 0600 file and hands the agent
`--token-file`, never argv, and runs under `ProtectSystem=strict`).
The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other
service here.
@@ -190,8 +214,10 @@ service here.
test on hardware something is plugged into is not a test.
- The agent's real-`/proc` test (`tests/test_agent.py`) touches only the box running pytest,
so it stays in the default run and is *not* marked `live`.
- Coverage gate: 80% is the floor, ~94% is where we are; keep it there. API tests use
`create_app()` with an in-memory DB, never a real target.
- Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces
94% (`--cov-fail-under`), and ~95% is where we are; keep it there. New/modified Python
still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB,
never a real target.
## UI conventions
@@ -201,6 +227,12 @@ dots in the header, window chips (1h…90d) on history pages. A panel that canno
why in the panel — the operator should never have to open devtools to learn that a source is
unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
A big number is `Dash.hero()` on `.hero` / `.hero-bar` (`dash.css`): 2.4rem tabular numerals,
muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the payload's
`severity` through the existing `.ok` / `.warn` / `.fail` classes — there is no `.sev-*`
family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS
rule reaches for a token that `:root` does not declare.
## Known gaps / next work
1. **UPS instant commands are read-only in practice.** Reads work against real upsd (see
@@ -208,15 +240,22 @@ unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
`actions` in upsd.users" because no such user exists yet. Creating one safely needs a
*read* of `/etc/ups/upsd.users` first — `sudo tee` can write the file but overwriting it
blind would delete the users upsmon already authenticates as, which can take the box down
at the wrong moment. `deploy/nut-sudoers` lists the read rules; once they are in, add
`[dashboards]` with `instcmds` limited to the battery tests, put the credentials in
`.env.dev`, and the buttons come alive. Never grant `actions = -`.
at the wrong moment. `deploy/nut-sudoers` is the installable ruleset that makes that
read (and only a scoped write + reload) possible — it carries the
`ops ALL=(root) NOPASSWD: NUT_READ, NUT_WRITE` grant and is installed with
`sudo install -m 0440 -o root -g root deploy/nut-sudoers /etc/sudoers.d/dashboards-nut`,
but it is **repo-side only so far**: the live UPS cards still answer "NUT_USER /
NUT_PASSWORD must name a user with `actions` in upsd.users" (verified through
`/api/ups/snapshot`). Once it is installed: read `upsd.users`, add `[dashboards]` with
`instcmds` limited to the battery tests, put the credentials in `.env.dev`, and the
buttons come alive. Never grant `actions = -`, and never write the file blind.
2. **GPU coverage is AMD only.** `host_agent.gpus()` reads amdgpu DRM sysfs
(`gpu_busy_percent`, `mem_info_vram_*`, hwmon temps, clocks). That covers the current
fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at
all — needs a `nvidia-smi`/NVML or DCGM path, and a fixture from a real card before the
field map is written.
3. **No remote yet.** Two commits, no `origin` — the user is providing it.
3. **No remote yet.** No `origin` is configured (`git remote -v` is empty) — the user is
providing it, so every phase so far is local-only history.
4. **UPS history depth.** Events and the numeric series are stored; long-window battery
health (capacity fade over months) has the rollups but no dedicated view.
@@ -240,6 +279,9 @@ unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
## Definition of done (per change)
The first three are exactly what `bash .agents/validate.sh` (the phase gate) checks, so no
phase may be called done on a subset of them.
`uv run pytest` green · coverage not down · `ruff` + `pyright` clean · `uv pip list --outdated`
empty if deps moved · dev server restarted and `/api/health` sane · UI change screenshotted
in a browser against **live** data · fixtures re-captured if an upstream contract changed ·