services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 11_admin_login

Browse Source 
**Verified (final pass, phase 11 — all 7 tasks already in `complete/`)**
- Gate from the wire on a restarted dev server: `/system`→`303 /login?next=/system` (`/llm?range=1h`→`/login?next=/llm%3Frange%3D1h`), `/api/summary`+`/api/config`→`401 {"ok":false,…}`, `/api/health`→`200`; signed-in `GET` of all six pages `200`, `/login`→`303 /`.
- Auth flows on a throwaway server (`:8199`, `/tmp` DB — dev `auth.verifier` untouched): wrong pw `401`, `testpass` `200`, password change `200`, **old cookie `401` / new cookie `200`**, old pw `401` / new pw `200`, short new `422`, wrong current `403`, cookie write without CSRF `403`, logout `200`; 5 failures then `429 retry-after: 60`; `login-denied`/`password-changed`/`password-denied` events; `meta` = the two auth keys.
- Fail-closed boot (no seed, empty DB): **exit 3**, message names `DASH_ADMIN_PASSWORD`, `auth.verifier`, and the `DELETE FROM meta …` reset. `dev-server.sh status` + the Containerfile `HEALTHCHECK` python one-liner both `200`/exit 0 unauthenticated.
- Live payloads unchanged (phases 01–10 walk): `/api/summary` keys + 8 alerts + hosts 3/2 live; UPS `simulated:false` with charge/load/runtime severity; deskwork `nvme0n1` 51 °C/19 159 h/11 % wear; gateway `smartctl_available:false`; driveripper `unreachable` stub.

**Tests/lint (all green)**: `bash .agents/validate.sh` → exit 0 — `1140 passed, 11 skipped`, coverage `98.62%` (floor 94), ruff `All checks passed!`, pyright `0 errors`, `node --check` clean, repo hygiene clean. `uv run pytest -m live --live` → `8 passed, 3 failed`, all `driveripper: 10.1.0.126:9120 … ConnectError` (pre-existing, auth-free; probe confirms nothing listening).

**Completion criteria**: all met except the live-suite caveat above — installing the driveripper agent needs root on that host (no `ops` grant), so per the live-infra rule I made **no live change**; task 07's five screenshots (`auth-login/-rejected/-locked-out/-signed-in-overview/-settings.png`) remain the browser evidence, and the login/rotation/lockout flows were re-proved over HTTP this pass.

**Docs**: README — fixed "only place the CSRF token is published" (login/password responses mint one), added the undocumented `DASH_RELOAD` row, stated that Sign out ends only this browser's cookie. `AGENTS.md` — new known gap 8 (stateless logout cannot revoke a copied token), gap-4 dev-store sizes dated, fleet section notes `data/` is gitignored. `api.py` — same `status` docstring imprecision fixed. Verified the 4 `DASH_*` rows, reset command, seed-wins rule and invariant 12 against `app.py`/`config.py`.

**Deviations**: no dependency bump (`uv pip list --outdated` advisory: certifi/fastapi/starlette — phase added none); verification notes appended to `data/logs/NOTES-auth.md` §10.

**Next pending phase**: `12_nut_self_test_verdict`.
This commit is contained in:
ducoterra
2026-10-19 07:45:44 -04:00
parent eb4788cba1
commit 1734d81e48
75 changed files with 6941 additions and 121 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+354 -66
View File
@@ -33,6 +33,11 @@ file — plus an overview page that says which of the four you need:
History is a first-class feature, not a sparkline afterthought: every number worth graphing
is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`.
None of it is public. One admin signs in at `/login` (the only page an anonymous visitor
reaches), and `/settings` — `⚙ Account` in the masthead — is where that session is read, ended,
and re-credentialled. Every page above, `/settings`, and every `/api` route but the four named
in invariant 12 require the session.
## Non-negotiable rules
These are the things that make the numbers trustworthy. Breaking one is a bug even if the
@@ -43,7 +48,10 @@ page still renders.
(`NUT_ALLOW_ACTIONS`, not `DASH_READ_ONLY`, the upsd.users account actually having the
command, and a UI confirmation — `DANGER` for anything that can cut power) and the server
re-checks every gate on every request. Hiding a button is not a safety control.
Commands that would cut output are refused outright while the UPS is already on battery.
Commands that would cut output are refused outright while the UPS is already on battery. Since
phase 11 there are two more in front of it — a session, and this session's CSRF token for a
cookie write (invariant 12) — and they are additive: `tests/test_auth_api.py` proves the
`DANGER` requirement and the wire send still behave exactly as they did without them.
2. **Never invent a number.** A value the upstream does not publish renders as `n/a`, never
as `0`. Every derived value carries the `source` string it came from and `derived: true`
so the UI can print *how* it was obtained (`computed:Δtokens/Δt`, `Δ10.0s`). Aggregating
@@ -75,7 +83,14 @@ page still renders.
`config.strip_userinfo`-redacted engine URLs, and every URL inside an `UpstreamError` message
is redacted too — those strings go into the health record *and* the `event` table, which
`/api/events` serves to the browser and `page-system.js` prints. No token, password, or
personal hostname goes into tracked files, fixtures, or the browser.
personal hostname goes into tracked files, fixtures, or the browser. **The admin's own
credential is the same case** (phase 11): `auth.py` records nothing and answers
`False`/`None` to every rejection, because the *reason* a login was refused is a verifier, a
salt, or a session token — and this app turns reasons into health records and `event` rows
that a browser reads. Its one `logging` call and its two `raise` sites (the fail-closed
startup and the empty-new-password refusal) carry **fixed** operator text with nothing
interpolated, not even a prefix of a hash; `tests/test_auth.py` pins that structurally, so
the day someone adds `logger.warning(f"bad verifier {v}")` a test fails instead of a log line.
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
`sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
@@ -91,6 +106,49 @@ page still renders.
10. **The dev server stays up while work happens** (`./scripts/dev-server.sh restart`,
http://127.0.0.1:8090), and UI changes are verified in a real browser with a screenshot
before they are called done. `data/logs/*.png` is where those land.
11. **The dashboard never starts open, and the stored password beats the environment**
(phase 11). The admin verifier and the session-signing key are durable rows in the SQLite
`meta` table (`auth.verifier`, `auth.session_secret`) and are the only place either secret
lives; `DASH_ADMIN_PASSWORD` is a **seed** that hashes into the row on the first boot of a
database and is ignored thereafter (one warning line, value never quoted), so a password is
changeable without a redeploy and no container env can roll it back. A server with neither a
stored verifier nor the seed **refuses to start** — a `RuntimeError` out of the lifespan, a
non-zero exit, and a message naming the variable, the meta key, and the
`sqlite3 <db> "DELETE FROM meta WHERE key IN ('auth.verifier','auth.session_secret')"`
reset command. "Start open with a warning" is not an option: the domain is public and the
fleet's hostnames and UPS controls are behind that login. `testpass` is the local-testing
password (dev config and the hermetic suite) and never a production fallback. Invariant 12 is
the request-time half; this one is why there is always a credential to check against.
12. **The gate is default-deny and its public surface is four entries** (phase 11). `app.py`'s
`PUBLIC_PATHS` (`/login`, `/api/auth/login`, `/api/health`) plus `PUBLIC_PREFIXES`
(`/static/`) are the only things an anonymous caller can reach — every page, every `/api`
data route, and `/api/auth/status`/`logout` require a session read from the
`dash_session` cookie or an `Authorization: Bearer` token. `/api/health` is the one data
route that stays open on purpose (the Containerfile `HEALTHCHECK` and
`./scripts/dev-server.sh status` poll it unsigned, and it leaks nothing); its shape is now a
contract with deployment, which means **nothing may be added to it without re-deciding that**
— a new field there is a field the whole internet reads. Adding a path to that list is a
security decision, and
`tests/test_auth_api.py::test_every_route_that_is_not_public_requires_a_session` walks the
**served** route table so a route that forgets the gate fails a test instead of shipping —
the middleware guards by default, which is why no new route has to remember to. Rules the
gate keeps and each has a test: a page `GET` gets `303` to `/login?next=…` (query string
included) and everything else the one JSON refusal, because a redirect a script cannot render
is not an answer; every answer carries `no-store`; `?next=` passes `auth.safe_next` and is
never followed raw (an open redirect on a login page is a phishing kit with a homelab theme);
a write additionally needs a matching `Origin`/`Sec-Fetch-Site` and — when it came in on the
**cookie** — this session's `X-CSRF-Token`, because a bearer caller is something no other site
can forge. A process whose lifespan never produced `ctx.auth` answers `503` for **everything,
public paths included**: "healthy" from a server that cannot say who its admin is is the
invariant-4 failure mode, and a login form that cannot verify a password is a worse promise
than no form. `POST /api/ups/command` keeps all four of its own gates behind this one (the
`DANGER` word is still what cuts power), and `POST /api/auth/logout` is gated like any other
write because a sign-you-out link is a CSRF. `POST /api/auth/password` (task 04) is the third
gated write and the app's only revocation: guesses at the current password spend their own
`password:<address>` bucket, and a right one replaces `auth.verifier` **and**
`auth.session_secret` so every browser that is not in front of the form dies on its next
request — which is why the response is minted from the new state and never the one being
replaced.
## Architecture
@@ -133,7 +191,14 @@ Responsibilities, one line each:
is single-flight and keeps the old cache on failure, so a bad pass retries on the next
expiry instead of hammering the disk.
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
env var always wins. Defaults are safe except the sources, which default to unset.
env var always wins. Defaults are safe except the sources, which default to unset. The auth
block (phase 11) is `DASH_ADMIN_PASSWORD` (the **seed**, and `_env()` strips, so `VAR=` means
*unset* rather than "the password is empty"), `DASH_SESSION_HOURS` (720 = 30 days: a dashboard
that logs the operator out mid-outage gets bypassed), `DASH_COOKIE_SECURE` (off, because dev is
`http://127.0.0.1`; task 03 also treats `X-Forwarded-Proto: https` as secure), and
`DASH_PBKDF2_ITERATIONS` (default `auth.PBKDF2_ITERATIONS`, imported so the primitive and the
setting cannot drift) — that knob exists so the hermetic suite hashes in microseconds instead
of 0.3 s per client and is not a production tuning parameter.
- `poller.py` — job loop, health records, stale/last-good semantics, transition events.
`/disks` is a **source of its own** (`disks:<host>`, `DISKS_INTERVAL_SECONDS` 300 s,
`DISKS_TIMEOUT_SECONDS` 30 s, events under the dashboard name `disks`) reading the same
@@ -208,6 +273,44 @@ Responsibilities, one line each:
monitor through it, so the assembly and the model cannot disagree about what is enabled.
- `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own).
- `rates.py` — Δ/window estimator, keyed by resolved series name.
- `auth.py` — every authentication decision as a pure function, stdlib only (phase 11):
`hash_password`/`verify_password`/`needs_rehash` over a self-describing
`pbkdf2_sha256$<iterations>$<salt_hex>$<hash_hex>` verifier (so the cost can rise without a
migration), `mint_session`/`read_session` for an HMAC-signed stateless cookie (no session
table and nothing to prune; rotating the stored secret signs every cookie ever issued out,
which is how changing the password evicts every other browser), `mint_csrf`/`verify_csrf`
bound to the session token instead of stored, `LoginThrottle` (in-process exponential
backoff), and `origin_matches`/`sec_fetch_site_allowed` as string functions so the gate is
testable without a `Request`. The primitives **never raise and record nothing**: a malformed
verifier, a tampered cookie, and a rotated secret all answer `False`/`None`, because the
reason an input was rejected is the credential (invariant 6). No HTTP, no SQLite driver, no
new dependency — `app.py`'s middleware and `api.py`'s `/api/auth/*` are the callers, so a
decision made here is made once and is unit-testable without a `Request`. It also carries the
two strings the gate and the pages share (`LOGIN_PATH`, `LOGIN_NEXT`) and `safe_next()`, the
only judge of a `?next=` value: kept verbatim (query string and all, because the range chips
live there) when it stays on this app, dropped otherwise — scheme-prefixed, scheme-relative,
the percent-encoded spellings of both, anything with a control character (not a path but a
second `Location` header), and anything without a leading `/`.
The **verifier bootstrap** (task 02) is the one deliberate exception to "never raises":
`load_auth_state(store, admin_password=…, iterations=…, session_hours=…, db_path=…)` reads the
`meta` rows `auth.verifier` / `auth.session_secret` through a three-method `MetaStore`
`Protocol` (so this module still imports nothing in the package, and its tests use a dict)
and returns a frozen `AuthState` — the environment **seeds** a password once and the stored
verifier wins thereafter (`env_ignored` and one fixed-line warning when both are present, so
`/settings` is the real answer and no redeploy can roll a changed password back); no row and
no seed raises a `RuntimeError` naming `DASH_ADMIN_PASSWORD`, the meta key, and the
`sqlite3 … DELETE FROM meta …` reset command, because "start open with a warning" is the
failure this phase exists to prevent; a stored secret that is not 256 bits of hex is
regenerated and persisted (the effect is one login prompt, never a lockout). Nothing here
reads a clock, so the state carries no expiry — a session's is inside its signed payload.
`update_verifier(store, state, new_password)` writes the new verifier **and** a fresh secret
(secret first, so a crash cannot leave sessions outliving the password they were issued
under) and refuses an empty/whitespace password; it is what `POST /api/auth/password` calls,
because rotating the secret **is** this app's revocation list — a stateless cookie has no row to
delete, so without it a stolen session would outlive the password that "revoked" it for another
`DASH_SESSION_HOURS`. Its two raises, like the loader's, carry
fixed text — `tests/test_auth.py` pins the file's raise sites and its one logging call
structurally, so no future edit can interpolate a credential into either.
- `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric,
`classify()` → `ok`/`warn`/`fail`/`None`, plus `attach()`/`with_severity()` to put the verdict
beside a value. No measurement, no verdict: `None` in → `None` out, never a green pass for
@@ -248,8 +351,45 @@ Responsibilities, one line each:
grading a drive here would be the bug, not the fix (phase 09). Its `HREF` table does carry
`/disks`, so the one alert a disk job *can* raise — SMART failing while its host answers —
links to the page that names the drive.
- `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning.
- `api.py` — GETs everywhere plus the one POST. `/api/config` is redaction boundary.
- `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning. The
`meta` table is the only durable key/value state in the app and is reached through three
named helpers — `get_meta`/`set_meta`/`delete_meta` (the last returns its rowcount, because
the password-reset procedure has to be able to say whether a row was there, and one key per
call, because a reset that deletes a row it did not name deletes the operator's credential).
`stats()` reads `last_maintain` through `get_meta()` too — outside its own lock, which
`get_meta` takes itself and `threading.Lock` is not reentrant — while `maintain()`'s write
stays inline inside that method's transaction.
- `api.py` — GETs everywhere plus the one POST that dials an upstream, and the `auth_router()`
that creates, describes, **re-credentials**, and ends a session (`POST /api/auth/login` public;
`GET /api/auth/status`, `POST /api/auth/logout` and `POST /api/auth/password` protected). Those
three are the app's only other writes and they touch this process and its `meta` table and dial
nothing — invariant 1 is about the fleet, and `tests/test_api.py` pins the two lists separately
rather than widening the one that names `/api/ups/command`.
The login route is where a *sign-in* password is compared: it
throttles before it hashes, refuses an absent/empty/non-object body with `422` so `""` never
reaches `verify_password`, answers every failure with the one sentence `wrong password`, and
answers `429` + `Retry-After` while the per-address backoff runs (the only extra fact it adds,
because a wait is something the caller has to act on). After a good sign-in it also upgrades a
verifier minted at a lower `DASH_PBKDF2_ITERATIONS` (`auth.needs_rehash`), keeping the signing
secret — that knob has to actually buy something, and a login is the one moment the plaintext is
in hand. A refused upgrade write is swallowed with one fixed-text `logging` line (this module's
only one, carrying no credential: invariant 6) because `database is locked` on a bookkeeping row
must not turn a correct password into no login at all. `POST /api/auth/password` is task 04: it
validates `new` first (`MIN_PASSWORD_LENGTH = 8`, non-empty once stripped, different from
`current` — length is the whole policy, because composition mandates produce sticky notes),
throttles the *current* password on its own key `password:<address>`, refuses a wrong one with
`403` + a `password-denied` event, and on the right one calls `auth.update_verifier` and answers
from the **new** state with a cookie and CSRF token minted from the rotated secret. Building that
response from the old `ctx.auth` is the bug the order exists to prevent: the operator would be
signed out by their own form.
`AppContext` carries `auth` and the one
`LoginThrottle` per process — a throttle built per request is a throttle that never trips. A
refused sign-in writes a `login-denied` **event** and nothing else, and so do the two password
outcomes (`password-denied`, `password-changed`): `alerts.py` turns only
`kind: "unreachable"` into an overview row and every feed filters by `source`, so
`/api/events?source=auth` is the audit trail and the overview stays about outages (do not
"fix" that by hiding the event). Neither fact appears on `/api/health` — which is public — or
`/api/config`. `/api/config` is a redaction boundary.
`/api/disks/*` (phase 09) is the drive-health pair: `/snapshot` answers one entry per
configured host (disk rows + `smart` + `smartctl_available` + per-row `severity` + a
`fleet` block summed by the API, with a configured-but-never-answered host as an
@@ -275,10 +415,61 @@ Responsibilities, one line each:
job with no payload yet is listed as a stub (`unreachable: true`, every value `null`) in
`system.hosts`/`llm.engines` exactly as `/api/system/snapshot` does, so `hosts_total`
counts the machines there are and `hosts_live` the ones answering.
- `app.py` — assembly; `create_app()` takes injectable settings/store so tests build an app
with an in-memory DB and no targets.
- `web/` — five static pages (`/`, `/system`, `/disks`, `/llm`, `/ups`), one stylesheet,
`charts.js`. No framework. `app.js`'s masthead status dots are one per dashboard job —
- `app.py` — assembly, and the **session gate** (`session_gate` middleware, invariant 12) whose
`PUBLIC_PATHS`/`PUBLIC_PREFIXES` are the whole anonymous surface. It answers in this order and
the order is the design: no `AuthState` → `503` for everything; no session and no such route →
the router's `404` (a typo is not a sign-in problem, and "sign in required" would otherwise be
the only answer anyone outside ever gets for a wrong URL); public → through, except that the
origin check still applies to a public write (a cross-site login post spends the throttle of
whoever is typing) and `GET /login` for someone already signed in goes to the dashboard, not
back to the form; no session → `303` to `/login?next=…` for a page and the JSON refusal for
everything else; a write → origin, then this session's CSRF token when it arrived on the
cookie. `_own_origin` reads `X-Forwarded-Proto` for the same reason the cookie's `Secure` flag
does, so behind nginx a real same-origin write is not a cross-site refusal — the bug that only
appears in production, which is why it has a test. `create_app()` takes injectable settings/store so tests build an app
with an in-memory DB and no targets. The lifespan builds `ctx.auth`
(`auth.load_auth_state`) after the context exists and **before** the first poll, for the
owned-store and injected-store branches alike — one call site, because two copies drift — and
a `RuntimeError` from it leaves the process non-zero with the operator's fix in the output.
That failure is never caught here: a dashboard that cannot say who its admin is does not start
the loops either (`__main__.py` passes `SETTINGS` to `create_app()` explicitly so the served
app is configured by the same object the bind address came from).
- `web/` — five dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`) plus the login page
(`/login`, `pages/login.html` + `static/page-login.js`), one stylesheet, `charts.js`. No
framework. `/login` renders the masthead's **brand link and theme toggle** and nothing else
from it: no tabs (every tab bounces straight back here, and a nav of links that all redirect is
a lie about what is available), no status dots and no clock (there is nothing signed in for them
to say). It posts through `Dash.fetchJSON` like every other script, which is only safe because
`bounceToLogin()` does nothing when the browser is already on `LOGIN_PATH` — without that guard
a `401` would reload the form instead of printing `Wrong password.`, and the page that exists to
end a bounce loop would be one. So the form branches on `err.status`: `401` is the inline
sentence plus `field.select()` (fixing a typo without retyping is the whole UX here), `429`
shows the server's `detail` and **counts down its `Retry-After`** with the button disabled until
the wait is over — `fetchJSON` therefore carries `response.headers` on the `Error` it throws,
because a wait this page made up would be an invented number (rule 2) — and anything else prints
the server's `detail` verbatim. Every refusal goes out through `textContent`, never
`innerHTML`: this is the one page a stranger reaches first. Its own copy of `auth.safe_next`
guards the one redirect followed on client strength (already-signed-in `?next=`), and
`Dash.ensureAuth()` up front stops a flash of the form for a browser that is already in. The
login page is audited for orphan `id`s and CDN references too (`tests/test_auth_api.py`), fetched
unsigned because a signed-in `GET /login` is a `303` and would audit `/`'s markup under the wrong
name. `/settings` (`pages/settings.html` + `static/page-settings.js`) is a **page and not a
sixth dashboard**: same `<head>` and masthead, the five tabs with **no** `aria-current` (it is
not one of them), no status dots, and it is reached from the `⚙ Account` link in
`.masthead-right` of every dashboard — outside `.tabs`, so the pinned tab structure is untouched
(`tests/test_api.py::test_every_dashboard_page_links_to_settings`). It prints
`/api/auth/status` (name, expiry as a local timestamp *and* as time left), signs out through
`POST /api/auth/logout`, and changes the password through `POST /api/auth/password` after
checking locally what it can judge itself (non-empty current, ≥ 8 new characters —
`tests/test_api.py` pins that number against `api.MIN_PASSWORD_LENGTH` so the two cannot drift —
new matching confirm), printing `403`/`422`/`429` through `textContent` as the server words them,
then re-reads the status so the panel cannot show a session the rotation ended. Its static
sub-line is the precedence rule (env seed → stored verifier wins → reset by deleting
`auth.verifier`), because that sentence is what stops the "I edited the env file and nothing
happened" call. So `app.js` exports **`resetAuth()`**: it clears the memoised `authStatus` and its
in-flight promise, and must be called after a password change (the secret rotated, so the cached
`csrf` is now invalid — every later write, the UPS buttons included, 403s until a reload) and on
any `401` bounce. `app.js`'s masthead status dots are one per dashboard job —
including `disks`, because the same agent can answer the 10-second utilisation poll and
have a wedged SMART pass, and one dot for both would be right about neither. `app.js` owns the
colour vocabulary: `Dash.severityFrom(cellOrSeverity)` turns a payload verdict into an
@@ -320,27 +511,33 @@ That scheme is the history API's key — treat it as a public interface.
neither `.env.dev` nor `.env.dev.example` declares a forwarded port — dev dials the same
addresses the deployed dashboard does, and `scripts/capture-fixtures.sh` reads
`HOST_TARGETS` from `.env.dev` instead of naming hosts of its own
(`tests/test_deploy_scripts.py` fails the day any tunnel name or port returns). Current state: `gateway` runs its installed service, bound to the address
`gateway.reeselink.com` resolves to, and answers `/health` + `/snapshot` **on the host**
(re-verified 2026-10-18, and the token in `.env.dev` matches `/etc/dashboards-agent.token`),
but the laptop cannot reach its `:9120` — and that is now proven to be nobody's firewall here:
opening `9120/tcp` in that host's own firewalld (a runtime rule, added and reverted in the
phase-10 final pass; runtime and permanent port lists are back to their original set) changed
nothing, because the SYN never arrives — `ss -tan sport = :9120` on the host shows only the
`LISTEN` — while the same laptop reaches `:22` and `:80` on the same address in milliseconds.
The drop is the port-selective inter-VLAN filter on the lab gateway `10.2.0.1`, the same one
that eats 3051/3493. On `driveripper` and `deskwork` the ad-hoc `~/lab-agent/host_agent.py` agents
are gone (stopped with the old tunnel sessions, nothing replaced them, `:9120` is empty on
both) because their `deploy/agent-sudoers` grant is not installed yet, so the installer
cannot run there. Every `system`/`disks` source is therefore `DOWN` with its real reason
and renders as an `unreachable` stub row — the designed degradation, recorded rather than
hidden — which is also why 6 of the `live` tests fail on transport while both engines pass.
The per-host record, the exact root commands and the one firewall line each host still
needs are in `data/logs/NOTES-agent-install.md`, how far the NUT wire reaches from each
(`tests/test_deploy_scripts.py` fails the day any tunnel name or port returns).
**Fleet transport, re-verified from the laptop on 2026-10-19.** The port-selective inter-VLAN
drop phase 10 recorded on the lab gateway `10.2.0.1` is **gone**: `gateway` (`10.1.1.27:9120`)
and `deskwork` (`10.1.0.51:9120`) both take a plain TCP connect and answer `/snapshot`, and
`driveripper`'s upsd answers `LIST UPS` on `10.1.0.126:3051` from this same laptop. Two hosts
run the **installed** service — `systemctl is-active dashboards-agent` is `active` on both,
with `/usr/local/lib/dashboards-agent/host_agent.py` and a `--token-file` unit bound to the
address the dashboard dials. `driveripper` is the one host with no agent at all (unit
`inactive`, no installed file, nothing listening on `:9120` → `ConnectionRefused`), and it
renders as an `unreachable` stub row: the designed degradation, recorded rather than hidden.
`/api/health` today reads `host:deskwork ok`, `host:gateway ok`, `disks:deskwork ok`,
`disks:gateway ok`, `nut ok`, `llm:gufo ok`, `llm:vllm ok` with the two `driveripper` jobs
`DOWN`, and the `live` suite is 8 passed / 3 failed with every failure that one transport.
Phase 10's proof that the old drop was not this repo's firewall stands as history in
`data/logs/NOTES-verify.md` — do not re-add a tunnel on the strength of it, and re-run the
probe rather than trusting either note.
The per-host record and the exact root commands are in `data/logs/NOTES-agent-install.md`
(its firewall lines are the phase-10 state, not today's), how far the NUT wire reaches from each
network is in `data/logs/NOTES-upsd-acl.md`, and the end-to-end direct-dial verification
(health table, no-tunnel-port proof, live-suite tally, screenshots, and the probes that put
the block on the lab gateway's port allow-list rather than in this app) is in
`data/logs/NOTES-verify.md`.
`data/logs/NOTES-verify.md`, and the phase-11 login verification (the `curl` refusal codes, the
six screenshots, the two `meta` keys the database actually holds, and the operator's
deployment-repo actions) is in `data/logs/NOTES-auth.md`. Those notes are local evidence, not
tracked files: `data/` is gitignored (invariant 10's `data/logs/*.png` included), so a fresh
clone has the conclusions in the prose above and the captures live on the box that ran them —
re-run the probe rather than trusting either.
- Engines are reached directly: gufo `deskwork.reeselink.com:8731`, vLLM `driveripper.reeselink.com:8000`.
- NUT 2.8.5 on driveripper: **configs live in `/etc/ups/`, not `/etc/nut/`**, and **upsd is
on port 3051, not 3493**. Nothing listens on 3493 at all, so a default client reports
@@ -352,14 +549,15 @@ That scheme is the history API's key — treat it as a public interface.
2.8 does not refuse non-local clients the way older docs suggest: `LISTEN 0.0.0.0 3051` +
`LISTEN :: 3051` with `3051/tcp` in the `FedoraServer` zone answers any client routed to it
(reads are unauthenticated; `SET`/`INSTCMD` still need an `upsd.users` account). The host
range `10.1.0.0/21` does reach it today (verified from `deskwork`); the laptop's
`10.2.0.0/22` does **not** — the lab gateway `10.2.0.1` forwards 22/1234/8000-8004 and
silently drops 3051/3493 in both families, which no sudo on a fleet host can change, so the
laptop cannot yet dial `NUT_PORT=3051` (probe evidence and the operator's one rule:
`data/logs/NOTES-upsd-acl.md`).
Two UPSes are exported over USB and dialled directly as `NUT_HOST` + `NUT_PORT=3051` (from the
laptop that is still the blocked path above; from the host itself, and from the deployed
dashboard, it reads live):
range `10.1.0.0/21` reaches it, and **since 2026-10-19 so does the laptop**: a raw
`LIST UPS` to `10.1.0.126:3051` from the dev box answers `BEGIN LIST UPS`, which is why
`nut` reads `ok` in `./scripts/dev-server.sh status` with two real UPSes and no
`NUT_SIMULATE`. Phase 10 recorded `10.2.0.1` dropping 3051/3493 for this laptop; that is
history now, and the probes behind both statements are in `data/logs/NOTES-upsd-acl.md`.
Two UPSes are exported over USB and dialled directly as `NUT_HOST` + `NUT_PORT=3051`, and that
reads live from every vantage the dashboard has ever used — the host itself, the deployed
dashboard, and since 2026-10-19 the laptop (`/ups` shows both boxes at 100 % charge, `OL`,
with no `SIMULATED` badge anywhere):
`apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U (that is
`ups.description`, the operator's label — the driver's own `identity.model` answers
`GX1500U`, so the description is what identifies the box).
@@ -379,21 +577,22 @@ That scheme is the history API's key — treat it as a public interface.
`upsc`, `udevadm`, the `/etc/ups` reads, and the `tee /etc/ups/upsd.users|upsd.conf` +
`systemctl restart|reload nut-server` pair documented in `deploy/nut-sudoers`); `deskwork`
grants an unrelated set (`btrfs scrub`, `snapper`, `setfacl /srv/*`, `osbuild-auto.sh`).
Neither `driveripper` nor `deskwork` can run the installer or `visudo`, so those two installs
— and the `firewall-cmd` line each still needs — stay with the operator.
- **SMART is captured from real hardware; the live agents still cannot read it.** `smartctl`
7.5 is installed on deskwork and driveripper. The `ops` account (which runs the dev agents)
is in no `disk` group; driveripper grants it `(root) NOPASSWD: /usr/sbin/smartctl*` and
deskwork grants nothing, so the agent's own unprivileged `smartctl -i -A -H` — it shells the
binary and never calls sudo — answers *Permission denied* (exit 2) on every host and `smart`
stays `null` end to end: the designed degradation, not a bug. The agents deployed on
driveripper and deskwork also predate the SMART pass, so the `disks` source today yields one row
per physical disk (16 on driveripper, 1 on deskwork, `vda` on gateway) with `smart: null`,
`smart_errors: []` and `smartctl_available: null` — "nobody told us either way", which is why
that flag is three-way and not a boolean. `gateway`'s agent is the one exception as of phase 10:
it is the installed systemd service, which does know about the SMART pass and answers
`smartctl_available: false` there because smartmontools is not installed — the *install it*
half of the three-way flag, from a host the dashboard cannot yet reach (its firewall).
Neither host grants `ops` the installer or `visudo` (re-checked 2026-10-19), so every install
that has happened on those two went through root by the operator — including the one still
outstanding, `driveripper`'s.
- **SMART: real captures, and one live host finally reads them.** `smartctl` 7.5 is installed on
deskwork and driveripper. The `ops` account (which is who runs a hand-started dev agent) is in
no `disk` group, driveripper grants it `(root) NOPASSWD: /usr/sbin/smartctl*` and deskwork
grants nothing, and `host_agent.py` shells the binary **as itself** — it never calls sudo — so
an agent run as `ops` still answers *Permission denied* (exit 2) and reports `smart: null`.
What changed by 2026-10-19 is that `deskwork`'s **installed** service runs as
`dashboards-agent`, which is in the `disk` group (`getent group disk` → `disk:dashboards-agent`)
on an agent installed after the SMART pass: `/disks` shows real numbers there —
`smartctl_available: true`, `nvme0n1` healthy at 52 °C, 19 158 power-on hours, 11 % of rated
wear — so the wear bar and the alarm chips are drawn from live hardware, not only fixtures.
`gateway` answers `smartctl_available: false` (no smartmontools in its VM: the *install it*
half of the three-way flag), and `driveripper` answers nothing at all until its agent exists,
which is where `smartctl_available: null` — "nobody told us either way" — still comes from.
**Real data is in the repo now:** `tests/fixtures/smartctl_driveripper_<dev>.txt` is all 16
drives (8 NVMe, 8 spinning) read through that sudo rule, and `tests/test_agent.py` parses
every one — which is what found the parser's one real bug: this binary annotates the
@@ -403,15 +602,16 @@ That scheme is the history API's key — treat it as a public interface.
honest look at this fleet's drives: `nvme5n1` reports 13 media-and-data-integrity errors and
`nvme1n1` is at 58 % of its rated wear, so the alarm chip and the amber wear bar on `/disks`
are real features with real data behind them, not demo furniture.
**Lighting up the live page is two operator actions that travel together:** give the agent
user device read access — the `disk` group (`sudo usermod -aG disk <agent-user>`, which is
what `deploy/install-agent.sh` does on install), *not* a sudo rule, because the agent must
read the device itself — and redeploy `agent/host_agent.py` (never redeploy without the
grant: 16 permission-denied lines would age nothing but are pure noise, while with the grant
they are 16 drives' worth of temperature, hours and sector counters). Until then `/disks`
**Lighting up the rest of it is the same two operator actions that travel together:** install
the agent on `driveripper`, and give its agent user device read access — the `disk` group
(`sudo usermod -aG disk <agent-user>`, which is what `deploy/install-agent.sh` does on install),
*not* a sudo rule, because the agent must read the device itself — on an `agent/host_agent.py`
from phase 09 or later (never redeploy without the grant: 16 permission-denied lines would age
nothing but are pure noise, while with the grant they are 16 drives' worth of temperature,
hours and sector counters). `gateway` needs smartmontools first and will still read no SMART
after that, being a VM with no physical disk behind `vda`. A host that cannot read its drives
renders the designed degradation: one dimmed card per physical drive, every number `n/a`, and
the one-line why — which is also what gateway shows permanently, being a VM with no physical
disk to read and no smartmontools at all.
the one-line why.
`tests/fixtures/smartctl_reference_*.txt` stay as what they are: **reference shapes** (labels
verbatim from the deployed binary, values synthetic), kept beside the real captures so the
synthetic edge cases stay pinned. Re-record the real ones with
@@ -425,16 +625,19 @@ That scheme is the history API's key — treat it as a public interface.
```bash
uv sync --python 3.14 # deps + editable install
cp .env.example .env # production-shaped config
cp .env.example .env # production-shaped config (DASH_ADMIN_PASSWORD seeds
# the login on the first boot of a database — with no
# seed and no stored verifier the server exits non-zero)
cp .env.dev.example .env.dev # dev-only: the real hosts, tokens in URL userinfo
set -a; . ./.env.dev; set +a # dev dials the fleet the prod way, no tunnels
./scripts/dev-server.sh restart # keep http://127.0.0.1:8090 up
./scripts/dev-server.sh status # per-source ok/DOWN
./scripts/dev-server.sh restart # keep http://127.0.0.1:8090 up; sign in with the
# DASH_ADMIN_PASSWORD from .env.dev (`testpass`)
./scripts/dev-server.sh status # per-source ok/DOWN — unauthenticated: /api/health
./scripts/dev-server.sh logs
uv run pytest # hermetic: 793 passed + 11 live-skipped, no network
uv run pytest --cov # coverage gate (98.33%; floor 94%)
uv run pytest # hermetic: 1140 passed + 11 live-skipped, no network
uv run pytest --cov # coverage gate (98.6%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh # re-record ground truth (agents + UPS + gufo /slots)
@@ -512,6 +715,42 @@ branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only
test on hardware something is plugged into is not a test.
- The agent's real-`/proc` test (`tests/test_agent.py`) touches only the box running pytest,
so it stays in the default run and is *not* marked `live`.
- `tests/auth_contract.py` is why the HTTP suite can start a server at all: every app in a test
is seeded with `admin_password="testpass"` (the local-testing password — a server with no
stored verifier and no seed refuses to start, so an app under test needs a credential exactly
like a deployment does) and hashed at `pbkdf2_iterations=1000`, because the real 600 000 costs
~0.3 s per client and the suite starts a hundred-odd of them. The real figure stays pinned in
`test_auth.py` and `test_config.py`, so the cheap cost here cannot become the default there.
- **The perimeter is its own file, and it is paranoid in two specific ways.**
`tests/auth_client.py::sign_in(client)` is the one way a test gets a session: it posts
`testpass`, asserts the sign-in worked (so a suite cannot silently start from a 401 and fail
three lines later on something that looks like an unrelated bug), and puts the returned CSRF
token on the client's default headers — which is what let the ~200 pre-existing HTTP assertions
in `tests/test_api.py` stay untouched, asserting UPS behaviour rather than the new gate.
`tests/test_auth_api.py` then runs the gate through a real ASGI request with
`follow_redirects=False` on every anonymous call: `TestClient` follows redirects by default, so
an unsigned `GET /api/config` would fetch the login page and report `200` — the way a gated
suite ends up passing while testing nothing. Two of its tests are the ones to keep if you keep
none: `test_every_route_that_is_not_public_requires_a_session` walks the **served** route table
(`walk_routes` flattens pages, `include_router` wrappers and the `/static` mount, and raises on
anything it does not recognise, so the audit cannot silently shrink) and demands `401`/`303` for
every path outside `PUBLIC_PATHS`, GET and POST — the test that protects the next phase rather
than this one; and the UPS pair prove the CSRF refusal on **the wire** (a scripted
`nut.NutClient` records nothing for a tokenless write and does record it once the header is
there, and `driver.killpower` still needs `DANGER` behind the session), because 403 after
`instcmd` would be a power cut with an error message attached.
- **The setup surface has its own pins, because prose rots quietly.**
`tests/test_auth_docs.py` reads the artefacts an operator actually holds when a dashboard will
not come up: both tracked env templates (every `DASH_*` auth setting declared; every
secret-shaped *assignment* a placeholder — the same allow-list `.agents/validate.sh` 2b uses,
so `uv run pytest` catches a committed password on the working tree and not only in the gate;
the local-testing word allowed in prose and forbidden as a value), `scripts/dev-server.sh` (its
probe and its `status` ride `/api/health`, no `dash_session`/`/api/auth/`/`Authorization:`
anywhere, the password named but never assigned), and `README.md`/`AGENTS.md` (the *Signing in*
section before *Configuration*, the four public entries, the reset `DELETE FROM meta …` command
verbatim, the four config rows, the `DASH_COOKIE_SECURE=1` deployment note, the three auth
gaps). No server starts and no request is made: a doc that stops naming the reset command is a
locked-out admin, and that is worth a test even though it asserts nothing about Python.
- Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces
94% (`--cov-fail-under`), and ~98% is where we are; keep it there. New/modified Python
still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB,
@@ -878,16 +1117,65 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
field map is written.
3. **UPS history depth.** Events and the numeric series are stored; long-window battery
health (capacity fade over months) has the rollups but no dedicated view.
4. **The dev DB is big enough to refuse a write now and then.** The dev store (~610 MB, WAL
grown past 140 MB) logs `store-error: OperationalError: database is locked` a few times an
4. **The dev DB is big enough to refuse a write now and then.** The dev store (~640 MB on
2026-10-19, its WAL grown past 140 MB at its worst) logs `store-error: OperationalError:
database is locked` a few times an
hour across `system`/`llm`/`ups`; the poller handles it exactly as invariant 4 requires —
the source goes red with the real reason, ages, and writes `recording` when the next poll
lands — so nothing freezes, but those polls' samples are lost. The one cause this repo can
prove is two servers on one file, which is why `scripts/dev-server.sh stop` waits for the
old pid to die (its comment records the incident). A controlled 25 s external read
transaction during this pass produced none, so checkpoint starvation is not yet shown.
transaction during this pass produced none, and neither did the dev-server log window open on
2026-10-19 — the absence of a sample, not the absence of the pressure, since the file only
keeps growing.
Retention/`wal_checkpoint` tuning for a long-running store is its own phase, not a
phase-09 follow-up.
5. **The login throttle keys on the socket peer address** (phase 11), and the one
`LoginThrottle` instance now holds two keys per address: the bare one for
`POST /api/auth/login` and `password:<address>` for `POST /api/auth/password`. Behind nginx
every login shares one bucket, so anyone who can reach the login form can lock the operator out
of the *dashboard* — not the UPS, not the data, not a host — for at most
`LoginThrottle.max_seconds` (1 h by default) at a time, and a process restart clears the
counters. The two keys are separate on purpose, so neither form can be hammered at the other's
rate and a stolen cookie plus the change form is not an unthrottled oracle for the old
password; the cost of that choice is that one address can be blocked twice over. Accepted for a
single-admin box rather than becoming an `X-Forwarded-For` knob:
that alternative is spoofable by anyone the moment somebody forgets to configure which proxy
to trust, which would hand the attacker the throttle key instead of leaving it on the
network layer. PBKDF2-HMAC-SHA256 at `PBKDF2_ITERATIONS` (600 000) stays the real cost of
every guess; the throttle only makes hammering the form slower than useless.
6. **One admin, one factor — no second factor and no per-user accounts, by design** (phase 11).
The operator asked for a single admin, and that is what the model is: one verifier row, one
username in every signed session, one name in every `password-changed` event, no read-only
role and no per-viewer view. The consequence is worth stating out loud, because it is the
whole security budget of a public domain: the PBKDF2 cost plus the backoff is the only thing
between a stranger and the fleet's hostnames and the UPS command buttons, so there is no
defence in depth behind it — no TOTP, no WebAuthn, no per-IP allow-list, no "this login is
from somewhere new" notice. Adding a second human means more than a second row: the gate has
no notion of scope, so a colleague would be able to cut power unless somebody designs a
read-only session first. That is a phase, not an edit.
7. **The throttle is per-process, so a second uvicorn worker multiplies it** (phase 11).
`LoginThrottle` is a dict in one process's memory: `uvicorn --workers 4` hands anyone who
reaches the form four independent buckets and four times the attempts before a `429`, and a
restart clears all of them (the gap-5 note above). Nothing else in the auth design has that
problem — the session is an HMAC-signed cookie and the CSRF token derives from it, so both
are worker-safe by construction with no store to share and nothing to prune. The reason this
stays unfixed is that the app is single-process by nature anyway: every worker would run its
own poller loops against the same SQLite file (AGENTS.md invariant 4, and the `database is
locked` incident in gap 4 is what two processes on one store already looks like). If this is
ever revisited, the fix is a shared counter (the `meta` table would do, at login rates), not
a bigger number in one process.
8. **A logout ends this browser's cookie, not a copy of it** (phase 11). The session is an
HMAC-signed cookie with no server-side record — that is the design invariant 12 and gap 7 rest
on — so `POST /api/auth/logout` can only tell *this* browser to drop the cookie (verified
against a running server: a replayed token from a logged-out jar still authenticates until it
expires). The same is true of an `Authorization: Bearer` token, which was never in a cookie to
begin with. The only revocation this app has is rotating `auth.session_secret`, which
`POST /api/auth/password` does on purpose, and the reset `DELETE FROM meta …` does as the
lockout escape hatch. Accepted for a single-admin box on a domain nobody else signs in to —
the operator's own machine is the threat model here, not a shared kiosk — and the reason a
lost laptop is a password change and not a sign-out. A real revocation list would be a row per
session with a prune rule, which is the thing this design traded away.
## Locked decisions (and why)