phase: 01_severity_language
**Gate defect found & fixed (the "does not lint" bug).** `.agents/validate.sh` — the phase gate — ran `python3 -m pytest` on `/usr/bin/python3` + `~/.local` site-packages (not the uv venv), ran `ruff check .` *only if* a ruff happened to be on PATH, and **never ran pyright or the coverage floor**. Rewrote it to the project's real gate: cd to repo root → `uv run pytest --cov --cov-fail-under=94` → `uv run ruff check src tests agent` → `uv run pyright` → `node --check web/static/*.js` → hygiene gates; a missing toolchain now fails loudly. Provoked it to confirm it bites (floor + a seeded F401 both fail).
**Verification (all green)**
- `bash .agents/validate.sh` → **exit 0**: 356 passed, 6 skipped · coverage **94.75%** (floor 94) · ruff `All checks passed!` · pyright `0 errors` · node syntax clean
- `uv run pytest --cov` → `severity.py` **100%**; `uv run pytest -m live --live` → **6 passed**; `uv pip list --outdated` → empty
- Criteria: `classify('cpu.usage_pct',…)` → `fail`/`ok`/`None` ✓ · `/api/ups/snapshot` carries `severity` beside `charge_pct`/`load_pct`/`battery_runtime` (apc now live-`warn` at 498 s — threshold working) ✓ · no `.sev-*`/synonym vocabulary (only a comment saying so) ✓ · pytest/coverage/lint ✓ · dev server restarted, `/api/health` ok 6/6 sources, all pages+assets 200 ✓
- Chromium against live fleet: `severityFrom` cell→`fail`, bare→`warn`, null→`''`, unrated→`''`; `hero` fail → `rgb(207,34,46)`, null → `n/a` with **no** verdict class, warn+bar 42%; all four pages render unchanged → `data/logs/phase01-final-{index,system,llm,ups}.png`
**Docs (full review, incl. pre-existing drift).** README: `/ups` row claimed "upsd on 3493" (fleet is 3051, stated elsewhere in the same file); config table omitted `DASH_HOST/PORT`, llm/ups `*_INTERVAL_SECONDS`, `*_TIMEOUT_SECONDS`, `GUFO/VLLM_API_KEY`, `POLL_MS_*`, `DEBUGPY` (+ dedup); severity bullet omitted `disks`/`gpus[]`/rated LLM cells and the ratio rescale; agent section listed only `/snapshot` though `/info`+`/health` exist. AGENTS.md: same 3493 claim; `install-agent.sh --token-file` (its flags are `--bind/--port/--token/--rotate-token/--status/--uninstall` — it hands *the agent* `--token-file`); gap 3 "Two commits" (now 6, still no `origin`); gap 1 given the sudoers install command + verified live gating reason; coverage line now 80% repo floor vs 94% gate floor; cyberpower `identity.model: GX1500U` vs `ups.description` quirk. `.env.dev.example`: `DEV_FORWARDS` remote port 3493→3051 and engine URLs that pointed at `127.0.0.1` with nothing forwarding them.
**Deviations.** No `ruff format` (30 files would reformat; the project gate is `ruff check` + pyright — reformatting completed phases is out of scope). **No live-infra changes**: enabling the battery-test buttons needs `deploy/nut-sudoers` installed on driveripper + a nut-server restart and a scoped `[dashboards]` `instcmds` user — prod write with a documented shutdown risk (AGENTS gap 1), so repo-side only. 36 pre-existing `ResourceWarning: unclosed database` from test fixtures are noise, not failures; left rather than churn ~15 fixtures. `severity.attach()` is unit-tested but has no production call site until phases 02–05.
**Next pending phase:** `02_overview_fleet_and_alerts` (`01_alerts_builder.md`).
This commit is contained in:
@@ -7,7 +7,7 @@ Three dashboards, one domain, one process, one SQLite file:
|
||||
| `/` | Is anything wrong, and which dashboard do I need? | all of the below, one request |
|
||||
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) + vLLM `/metrics`, read-only |
|
||||
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd` on 3493 (its own wire protocol client) |
|
||||
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
|
||||
|
||||
No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
|
||||
runtime; every asset is served by this app.
|
||||
@@ -39,6 +39,8 @@ for host in ("driveripper-ops", "deskwork-ops", "gateway-ops"):
|
||||
PY
|
||||
./scripts/dev-tunnels.sh start # 9121/9122/9123 -> each host's agent,
|
||||
# + DEV_FORWARDS -> upsd on 127.0.0.1:3494
|
||||
# (the remote port is upsd's real one — 3051
|
||||
# on driveripper, not the 3493 default)
|
||||
set -a; . ./.env.dev; set +a
|
||||
uv run dashboards # foreground
|
||||
./scripts/dev-server.sh restart # or backgrounded, with a pidfile + logs
|
||||
@@ -51,7 +53,9 @@ that already listens there (upsd) — that is how the UPS page reads a real batt
|
||||
## The host agent
|
||||
|
||||
`agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`,
|
||||
`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` as JSON.
|
||||
`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` (everything, one
|
||||
JSON document), `GET /info` (slow-changing identity: OS, kernel, CPU model, GPU list), and
|
||||
`GET /health` (the one route that never needs the token) as JSON.
|
||||
|
||||
```bash
|
||||
python3 host_agent.py --once # print one snapshot, exit
|
||||
@@ -84,15 +88,20 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
|
||||
| Variable | Meaning |
|
||||
| ------------------------------------- | -------------------------------------------------------------- |
|
||||
| `HOST_TARGETS` | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`) |
|
||||
| `HOST_INTERVAL_SECONDS` | poll interval for hosts (default 10) |
|
||||
| `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
|
||||
| `RATE_WINDOW_SECONDS` | the Δ window behind every measured token rate (default 10) |
|
||||
| `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) |
|
||||
| `HOST_INTERVAL_SECONDS` / `LLM_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often each source is polled (10 / 2 / 5 s) |
|
||||
| `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others |
|
||||
| `GUFO_API_KEY` / `VLLM_API_KEY` | only if an engine is behind a bearer token; they never leave the server |
|
||||
| `NUT_HOST` / `NUT_PORT` | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT |
|
||||
| `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) |
|
||||
| `NUT_ALLOW_ACTIONS` | `1` enables the one write endpoint in the app |
|
||||
| `NUT_SIMULATE` | `1` replaces upsd with a **labelled** simulator (development only) |
|
||||
| `DASH_READ_ONLY` | `1` disables every mutating endpoint regardless of other flags |
|
||||
| `DASH_DB` | SQLite path (default `data/dashboards.sqlite3`) |
|
||||
| `POLL_MS_SYSTEM` / `POLL_MS_LLM` / `POLL_MS_UPS` | how often the *browser* refetches each page (10 s / 2 s / 5 s) — shown in the page footer |
|
||||
| `DEBUGPY` / `DEBUGPY_PORT` | `DEBUGPY=1` opens a debugpy listener (default 5678) without blocking startup |
|
||||
| `RETAIN_RAW_HOURS` / `RETAIN_MINUTE_DAYS` / `RETAIN_HOUR_DAYS` | history rollup and pruning |
|
||||
|
||||
A source that is not configured answers `404 {"configured": false}` **without dialling**,
|
||||
@@ -119,8 +128,19 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
|
||||
blanks or blocks another source.
|
||||
* `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are
|
||||
pruned, so a year of history is a few hundred MB rather than a disk emergency.
|
||||
* `severity.py` — the single green/amber/red table (`THRESHOLDS`, plus `INVERTED` for the
|
||||
metrics where lower is worse). Rated values in the API payloads travel with a sibling
|
||||
`severity` (`cpu.severity.usage_pct`, `memory.severity`, `disks.severity`,
|
||||
`gpus[].severity`, a UPS card's `severity.charge_pct`, `severity` inside the rated LLM
|
||||
cells such as KV pressure, `/api/summary` rows), computed server-side, so every page
|
||||
colours the same way and a value nobody reported renders `n/a` instead of green. A metric
|
||||
published as a 0-1 ratio is rescaled *for the verdict only* — the number printed stays
|
||||
what the upstream published.
|
||||
* `web/` — four static pages, one stylesheet, hand-rolled SVG charts (`charts.js`). No
|
||||
framework, no CDN, no bundler.
|
||||
framework, no CDN, no bundler. A big colour-coded number is `Dash.hero(el, {...})` on the
|
||||
`.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via
|
||||
`Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value
|
||||
with no verdict renders `n/a` with no colour at all.
|
||||
|
||||
## Testing
|
||||
|
||||
|
||||
Reference in New Issue
Block a user