services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 15_agent_mtls

Browse Source 
All green. Report:

**Verified / implemented (final pass of `15_agent_mtls`)**
- Wrote `data/logs/verify-15-tls-e2e.py`: a real `host_agent.py --cert-bundle` + a real dashboard process over a scratch DB (`HOST_TARGETS=loop=https://…`) — **27/27 PASS**, transcript `verify-15-tls-e2e.out`, notes `data/logs/NOTES-tls-rollout.md`.
- Read-only checks against the running dev store: 5 `certs.*` rows byte-identical across a restart; served bundle == the `meta` row (1430 B); 0 events containing `-----BEGIN`; unsigned `/certs`→`303 ?next=`, unsigned `/api/certs`→`401`, signed-in `200`; screenshots `data/logs/15-final-{certs,system}.png`.
- Docs (step 6, full review): README — added *Protecting an agent, host by host* (the rollout task-07 left out), a `tests/test_certs.py`/TLS-matrix/sandbox-installer paragraph, and fixed two stale “the one write endpoint in the app” claims (`NUT_ALLOW_ACTIONS`, Battery-test buttons). AGENTS.md — test count 1424→1427, bundle-fetch command in *Commands*, a Testing-policy bullet on structurally pinning a key-holding module, fleet transport bullet (token mode is now the compatibility path), gap 11 final state.

**Tests / lint (full suite 100% green)**
- `uv run pytest` → **1427 passed, 15 skipped**, exit 0 · `bash .agents/validate.sh` → **exit 0** (coverage **98.94%** vs 94% floor, `certs.py` 100%; ruff “All checks passed”; pyright 0 errors; `node --check` clean; hygiene clean)
- `uv run pytest -m live --live` → **11 passed / 0 failed** · `./scripts/dev-server.sh status` → all 9 sources `ok`
- `uv pip list --outdated` → empty after bumping `fastapi 0.152.0 → 0.153.0` (`uv.lock` only); full suite re-run green.

**Completion criteria**
- Fresh-DB root CA, idempotent: ✅ (tests + live store hashes) · agent TLS matrix incl. wrong-CA/certless 403/`/health` CA-only/revoked 403 + live denylist re-read: ✅
- `https` `host:`/`disks:` `ok` + revoked says “agent certificate revoked” with no dial: ✅ on loopback (real agent + real dashboard) and spy-pinned hermetically; **not** on the fleet (below)
- Signed-in `/certs` + rotate/revoke/root-rotate + byte-identical bundle download: ✅ · installer bundle/unit/`--status` over TLS: ✅ (0640, see deviation)
- `pytest`/`validate.sh`/`--outdated`: ✅ · token-mode byte-identical, engine/UPS untouched, public surface unchanged (route walk): ✅
- **Live fleet TLS (deskwork+gateway flip): NOT done** — no live changes. It needs root on hosts without the installer grant (`deskwork`), has no both-ways-working window (`http://` dashboard vs https-only agent), and would deploy the agent from this not-yet-committed tree. Exact procedure recorded in README + `data/logs/NOTES-tls-rollout.md`.

**Deviations**: bundle mode `0640 root:dashboards-agent` instead of the task file’s `0600` (the agent re-reads its own bundle for the denylist; 0600 breaks the unit — already recorded in AGENTS.md); the loopback dashboard substitutes for the fleet flip. Incidental: a stray `pkill` of mine killed the dev server mid-pass; restarted and verified healthy.

**Next pending phase**: `.agents/phases/todo/16_agent_smart_pass_budget` (then `17_installer_device_filter`).
This commit is contained in:
ducoterra
2026-10-20 10:02:16 -04:00
parent 569ca25fdb
commit cffbac5771
72 changed files with 7168 additions and 216 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+170 -20
View File
@@ -13,9 +13,11 @@ Four dashboards, one domain, one process, one SQLite file:
No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
runtime; every asset is served by this app.
Two more pages are not dashboards: `/login` (the one thing an anonymous visitor sees) and
Three more pages are not dashboards: `/login` (the one thing an anonymous visitor sees),
`/settings` — `⚙ Account` in the masthead of every dashboard — which shows this browser's
session, signs it out, and changes the admin password.
session, signs it out, and changes the admin password, and `/certs` — `⚷ Certs`, beside it —
which is the certificate authority the agents are dialled with: the root CA, this dashboard's
own client certificate, and one row per host with its rotate, revoke and bundle download.
Working on this repo? **[`AGENTS.md`](AGENTS.md)** holds the invariants (read-only by
construction, never invent a number, unconfigured ≠ broken), the locked decisions, the real
@@ -70,14 +72,16 @@ and running the installer without root prints that exact line. upsd is dialled a
`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` (everything, one
JSON document; `?top=N` trims the process lists), `GET /info` (slow-changing identity: OS,
kernel, CPU model, GPU list), `GET /schema` (the schema version plus this document) and
`GET /health` (the one route that never needs the token) as JSON, with a plain HTML status
`GET /health` (the one route that needs no credential) as JSON, with a plain HTML status
page at `GET /` so a browser pointed at an agent learns what it is looking at.
```bash
python3 host_agent.py --once # print one snapshot, exit
python3 host_agent.py --host 0.0.0.0 --port 9120 --token-file /etc/dashboards-agent.token
python3 host_agent.py --host 0.0.0.0 --port 9120 --cert-bundle /etc/dashboards-agent.certs
sudo deploy/install-agent.sh # systemd unit + user + token
deploy/install-agent.sh --status # unit state and /health
sudo deploy/install-agent.sh --cert-bundle agent.bundle # ...or the TLS install (bundle from /certs)
deploy/install-agent.sh --status # unit state and /health (https when TLS)
sudo deploy/install-agent.sh --uninstall
```
@@ -106,8 +110,41 @@ one installed before phase 09 has no SMART pass to run. Opt out silently with `-
The drive truth this is all developed against lives in `tests/fixtures/smartctl_driveripper_*.txt`:
16 real drives, recorded with serials scrubbed, each one parsed by a test.
Binding off-loopback without a token is refused (exit 2) rather than warned about. Prefer
`--token-file` over `--token`: argv is world-readable through `ps`, and this token guards
**TLS mode is `--cert-bundle` (env `AGENT_CERT_BUNDLE`)**: one installed PEM the dashboard
signed — this host's server certificate and key, the dashboard's root CA, a `# BUNDLE-REV <n>`
line and the `# REVOKED <serial>` denylist. The agent then serves `https://` and answers every
path but `/health` only to a **client certificate signed by that root and not on the list**: 403
`client certificate required` for a peer that proved nothing, 403 `client certificate revoked`
for one the operator withdrew (a revoked identity is refused on `/health` too — that path is open
to the installer's probe, which holds the CA and no certificate, not to a certificate that was
taken away). The denylist is re-read when the bundle file's stamp changes, so a re-deployed
bundle takes effect on the next request rather than the next restart; the certificate itself needs
a restart, because `ssl` cannot swap a chain under a live socket. The unpacked PEMs exist only
long enough to load into the context and are then deleted, and the private key is written `0600`
in the unit's `RuntimeDirectory` when it has one. In this mode `/snapshot` gains two additive
fields — `tls: true` and the `bundle_rev` the bundle carried, which is how the dashboard tells a
stale install from a current one (they reach `/api/system/snapshot` and `/api/disks/snapshot` as
`agent.tls` / `agent.bundle_rev`) — and **the bearer token is not read**: the handshake is the
auth, and a token that kept working after a revocation would defeat the list. The dashboard's half
is `collectors.AgentTls`: point that host's `HOST_TARGETS` entry at `https://` and the fetch
presents the dashboard's client certificate and verifies the agent against the same root, and a
host revoked in `certs.agents` is refused before a socket is opened. A bundle that will
not load is exit 2 naming the file; there is no "fall back to plain HTTP" path.
`deploy/install-agent.sh --cert-bundle <file>` is how that file gets there. It validates the
bundle first by the same text shape (two certificates and exactly one key, plus the
`# BUNDLE-REV` line) — before touching anything, before the root check, and without ever
dialling the dashboard — then installs it to `/etc/dashboards-agent.certs` as
`0640 root:dashboards-agent`, the token-file idiom: the agent opens the bundle *itself* at
startup and re-reads it when its stamp changes, so the service user must be able to read it
and no other account may. The unit gains `--cert-bundle` and a private `RuntimeDirectory` for
the PEM blocks, `--status` probes `https://…/health` with the bundle's CA and no client
certificate (python3 `ssl`, never an external TLS binary), `--uninstall` removes the bundle
with the token, and a re-run without the flag leaves an installed bundle in place — the
token-reuse idiom.
Binding off-loopback without a token *or* a bundle is refused (exit 2) rather than warned about.
Prefer `--token-file` over `--token`: argv is world-readable through `ps`, and this token guards
hostname, mounts, disk and GPU models, and process names. `deploy/install-agent.sh` creates
a system user, generates a token once (re-runs reuse it; `--rotate-token` for a new one),
stores it `0640 root:dashboards-agent` — the unit hands the agent the *path*, and the agent
@@ -116,13 +153,77 @@ and runs the agent under `ProtectSystem=strict`, `DevicePolicy=strict` with only
nodes and the `/dev/null` family reopened (SMART needs the former; without `/dev/null` in
the allow list systemd cannot start the service at all), with no capabilities.
### Managing the certificates
The dashboard is the authority, and `/certs` (`⚷ Certs` in the masthead) is its page: the **Root
CA** block (CN, serial, expiry as a local timestamp *and* as time left, and **Rotate root CA**),
the **Dashboard client** line (the one certificate every agent verifies the dial against —
display-only, because it is minted with the root and has no route of its own), and one row per
host with its status chip, serial, expiry, issued-vs-deployed bundle revision, the addresses its
certificate is valid for, and **Rotate** / **Revoke** / **bundle**.
Nothing there is coloured: `severity.py` rates no certificate, so an expiry is a date and a
bundle revision is a count. The one verdict on the page is the API's `status` word (`active` green,
`revoked` red, `unissued` grey — a certificate that does not exist never borrows a green pass).
Every write asks a `confirm()` first and prints the server's own sentence afterwards; a root
rotation answers `restart_required`, because the CA and the client pair are startup material — the
table is new at once, the running process keeps dialling with what it loaded, and every host then
needs its new bundle installed. A row whose host has left `HOST_TARGETS` is listed with no buttons
at all, because every per-host route refuses it.
Under the page is the same API a script would call. Every route below is signed-in (and every write
is CSRF-gated like any other), and **none of them dials a host** — the effect of a revocation is
that the *next poll* refuses to dial (see `collectors.AgentTls` above):
| Route | What it does |
| ----- | ------------ |
| `GET /api/certs` | `root` and `client` with serial and expiry, plus one row per host the operator could act on: every `HOST_TARGETS` name and every certificate still in the table. A configured host nobody has installed yet reads `status: "unissued"` with `null` serial rather than going missing; `bundle_rev` is what this dashboard issued and `deployed_bundle_rev` is what that host's agent reported back (so "the box is one bundle behind" is visible, `n/a` when nothing answered). Reads are pure — no page refresh ever mints a certificate |
| `GET /api/certs/<name>/bundle` | the exact bytes of the stored bundle as `application/x-pem-file` with `Content-Disposition: attachment; filename="agent-<name>.bundle"`. That byte-equality is the point: `install-agent.sh` is handed the file and compares nothing else, so a route that re-signed on each read would make "is this the bundle I installed?" unanswerable. It carries that host's **private key**, which is why it is a download behind the session and `no-store` |
| `POST /api/certs/<name>/rotate` | fresh key, fresh serial, `bundle_rev` +1, `status: active` — and it is also how a host gets its **first** bundle (`bundle_rev` 1). The SAN comes from the current `HOST_TARGETS` URLs plus the name, not from the old certificate, because the address a host is dialled by is a deployment fact |
| `POST /api/certs/<name>/revoke` | marks the identity revoked, and **re-seals every other bundle** because the `# REVOKED` denylist lives inside each of them — so the fleet has new bundles to collect, while their `bundle_rev` does not move (their identity did not change). There is no unrevoke: the fix is a rotate |
| `POST /api/certs/root/rotate` | a new root, a new client certificate, every agent re-issued, and `{"ok": true, "restart_required": true}` — the CA and the client pair are startup material, so the table is new at once while the running process keeps dialling with what it loaded. Saying so beats pretending a hot-swap that does not exist |
### Protecting an agent, host by host
Three steps, and they are order-dependent because the dashboard picks its dial from the URL:
```bash
# 1. one bundle per host, from the dashboard that owns the authority (a signed-in GET; it is
# byte-identical to the stored row and it carries that host's private key, so `curl -o` it
# somewhere private rather than `cat`ting it into a shell history)
curl -s -H "Authorization: Bearer $SESSION" -o agent-deskwork.bundle \
http://127.0.0.1:8090/api/certs/deskwork/bundle
# 2. install it beside the token — as root on the host, 0640 root:dashboards-agent. The unit
# gains `--cert-bundle` and its RuntimeDirectory, and restarts into TLS mode.
scp agent-deskwork.bundle <host>-ops:/tmp/
ssh <host>-ops sudo /tmp/install-agent.sh --cert-bundle /tmp/agent-deskwork.bundle
ssh <host>-ops sudo /tmp/install-agent.sh --status # https, with the CA and no client cert
# 3. point that host's HOST_TARGETS entry at https with NO userinfo token — the certificate is
# the auth now — and restart the dashboard. `/certs` should then read issued == deployed rev.
export HOST_TARGETS="deskwork=https://<addr>:9120 …"
```
`install-agent.sh --bind … --port 9120` prints the `https://<addr>:9120` line for a bundle install,
so step 3 is a copy rather than a guess. A host left on `http://` keeps working over its bearer
token — the rollout is per host, and one revoked or expired certificate never blinks the others.
A name that is not in `HOST_TARGETS` answers `404 {"configured": false}` **before** any signing, any
dial, and any `meta` write: a typo that minted a key would leave a certificate nobody can install
in the authority forever. Each accepted write leaves one fixed-text event —
`cert-rotated`, `cert-revoked`, `cert-root-rotated` under `source: "certs"`, readable at
`/api/events?source=certs` — and a certificate serial never leaves a `meta` row. Serials are served
as decimal **strings** for that reason: a 64-bit serial through a JS number is a serial the page
cannot match against a denylist.
## Signing in
One admin, one password, and the server **refuses to start** when it has neither a stored
verifier nor `DASH_ADMIN_PASSWORD` — the domain is public and the fleet's hostnames and UPS
controls sit behind this login, so there is no way to run it open by accident. There is no
second account and no per-user view: the five pages (`/`, `/system`, `/disks`, `/llm`, `/ups`),
the `/settings` page and every data route belong to that one operator.
the two admin pages (`/settings`, `/certs`) and every data route belong to that one operator.
A session is an **HMAC-signed cookie** (`dash_session`), not a row in a table: it carries the
signed-in name and its own expiry, is valid for `DASH_SESSION_HOURS` (default `720` = 30 days),
@@ -301,10 +402,10 @@ curl -s -H "Authorization: Bearer $SESSION" -X POST -H 'content-type: applicatio
-d '{"ups":"apc","command":"test.panel.start"}' http://127.0.0.1:8090/api/ups/command
```
That is the app's one write endpoint **that dials anything** (the `/api/auth/*` writes change only
this process and its SQLite `meta` table — invariant 1 is about the fleet), and the bearer header
changes nothing about its gates: a command that can cut output still needs `"confirm":"DANGER"`,
and is still refused while the UPS is on battery.
That is the app's one write endpoint **that dials anything** (the `/api/auth/*` and `/api/certs/*`
writes change only this process and its SQLite `meta` table — invariant 1 is about the fleet), and
the bearer header changes nothing about its gates: a command that can cut output still needs
`"confirm":"DANGER"`, and is still refused while the UPS is on battery.
### Lockout
@@ -328,7 +429,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
| Variable | Meaning |
| ------------------------------------- | -------------------------------------------------------------- |
| `HOST_TARGETS` | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`) |
| `HOST_TARGETS` | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`). An `https://` URL is dialled as a *protected* agent instead: verified against the dashboard's own root CA and presented with its client certificate, with a revoked host refused before the dial (`certs.py`, `collectors.AgentTls`) |
| `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
| `RATE_WINDOW_SECONDS` | the Δ window behind every measured rate — tokens/sec, bytes/sec, jiffies→CPU % (default 10); each value carries the window it actually got, and `/system`'s footnote prints that, not this default. A window the poll loop cannot honour — the ~0.2 s catch-up after a fetch slower than the interval, or a base older than 2× the window — answers one `n/a` instead of a spike, and a rate 8× above the last one is held until a window measured from a **different** sample confirms it (phase 13) |
| `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) |
@@ -339,7 +440,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
| `GUFO_API_KEY` / `VLLM_API_KEY` | only if an engine is behind a bearer token; they never leave the server |
| `NUT_HOST` / `NUT_PORT` | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT |
| `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) |
| `NUT_ALLOW_ACTIONS` | `1` enables the one write endpoint in the app |
| `NUT_ALLOW_ACTIONS` | `1` enables the one write endpoint that dials **anything** — the `/api/auth/*` and `/api/certs/*` writes touch only this process and its `meta` table |
| `NUT_SIMULATE` | `1` replaces upsd with a **labelled** simulator (development only) |
| `DASH_READ_ONLY` | `1` disables every mutating endpoint regardless of other flags |
| `DASH_DB` | SQLite path (default `data/dashboards.sqlite3`) |
@@ -372,6 +473,7 @@ sources/host.py · disks.py · llm.py · nut.py poller.py (one tas
store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event + meta ── api.py → web/
▲
auth.py ── verifier + session key, loaded once at startup (no row and no seed → no server)
certs.py ─ root CA + client cert, minted once at startup (unreadable row → no server)
```
* `poller.py` — every source gets its own asyncio task, its own health record, and its own
@@ -404,13 +506,40 @@ auth.py ── verifier + session key, loaded once at startup (no row and no see
once-a-minute write transaction from 5.4 s to a few milliseconds, which is what stopped a poller's
insert losing its 5 s `busy_timeout` (known gap 4).
Its `meta` table is the one place state that must outlive a restart goes (`get_meta`/
`set_meta`/`delete_meta`), and its first user is the admin credential below.
`set_meta`/`delete_meta`). Its users are the admin credential below and, since phase 15, the
certificate authority in `certs.py` — five `certs.*` rows and nothing else.
* `auth.py` — the admin login's decisions, in stdlib and without HTTP: a self-describing
PBKDF2-HMAC-SHA256 verifier, an HMAC-signed stateless session cookie (no session table, and
rotating the stored key is what evicts every other browser), a session-bound CSRF token, and
an in-process login backoff. `load_auth_state()` is the startup half: the environment seeds a
password **once**, the stored verifier wins from then on, and a server with neither refuses to
start (see *Signing in*).
* `certs.py` — the fleet's certificate authority, in the same shape as `auth.py`: a pure lifecycle
over the `meta` table, no clock it cannot be handed, no logging, and one fixed-text raise. The
first boot of a database mints a **root CA** (`certs.root.key`/`.cert`, self-signed
`CN=dashboards-root`, 10 years) and the dashboard's own **client** certificate
(`certs.client.key`/`.cert`, `CN=dashboards`, 1 year); every later boot reads those rows and
regenerates nothing, because a root that changes on restart untrusts every certificate already
installed on a host. `issue_agent_cert`/`reissue_agent_cert` sign one server certificate per
`HOST_TARGETS` name (`CN` = the host, SAN = the addresses it is dialled by) into
`certs.agents`, and each entry's **bundle** — the agent's cert, the agent's key, the root *cert*
(never the root key), a `# BUNDLE-REV <n>` line and the current `# REVOKED <serial>` lines — is
stored verbatim so the bytes an installer places are the bytes the row holds. `revoke` marks the
host and re-seals every bundle, because the denylist lives inside each of them; `rotate_root`
replaces the root, the client, and every agent at once. A stored PEM that will not parse (or a
client leaf the stored root did not sign) is a **fail-closed startup**, the admin-password rule:
the message is `sqlite3 "<db>" "DELETE FROM meta WHERE key LIKE 'certs.%'"`, after which every
agent needs a fresh bundle. Every certificate carries a `SubjectKeyIdentifier` and every leaf an
`AuthorityKeyIdentifier` naming that root, because OpenSSL's strict mode — the one Python's
`ssl.create_default_context()` turns on, which is what the dial below builds — refuses a chain
without them. The **dial** is `collectors.AgentTls` (task 03): with an `https://` `HOST_TARGETS`
entry, the fetch verifies the agent against this root and presents this dashboard's own client
certificate, so only this dashboard can read the host; a host marked `revoked` fails its `host:`
and `disks:` jobs with `agent certificate revoked` *before any socket opens*, and because the
status is a fresh `certs.agents` read on every poll, revoking takes effect on the next poll with
no restart. An `http://` entry is untouched — same shared client, same bearer token — so the
rollout is per host. The private keys are credentials like any password: never served, never
logged, and the root key never leaves the row that holds it.
* `severity.py` — the single green/amber/red table (`THRESHOLDS`, plus `INVERTED` for the
metrics where lower is worse). Rated values in the API payloads travel with a sibling
`severity` (`cpu.severity.usage_pct`, `memory.severity`, `disks.severity`, a UPS card's
@@ -436,9 +565,16 @@ auth.py ── verifier + session key, loaded once at startup (no row and no see
one per stale metric; the list dedupes per (source, metric) and is capped at 12 with a
marker that says how many it hid.
* `api.py` — the JSON every page reads: `/api/config`, `/api/health`, `/api/events`,
`/api/history`, `/api/metrics`, `/api/summary`, and one `snapshot` + `history` pair per
dashboard (`/api/system/*`, `/api/disks/*`, `/api/llm/*`, `/api/ups/*`). GETs everywhere
except the single `POST /api/ups/command`. Two rules make the pages legible: a source with
`/api/history`, `/api/metrics`, `/api/summary`, one `snapshot` + `history` pair per
dashboard (`/api/system/*`, `/api/disks/*`, `/api/llm/*`, `/api/ups/*`), and the certificate
authority's own surface (`/api/certs`, `/api/certs/<name>/bundle`, `POST …/rotate`, `POST
…/revoke`, `POST /api/certs/root/rotate`). GETs everywhere except `POST /api/ups/command`,
which stays the **only** route that dials anything: the `/api/certs/*` writes are `meta` row
writes through `certs.py`, the same class of write `POST /api/auth/password` is, and a
revocation's whole effect is that the next poll refuses to dial (see `certs.py` below).
Reads there are pure — `certs.read_state` is the request-time read and `certs.ensure_certs` the
startup bootstrap — so refreshing the list cannot mint a certificate, and it reports the
**stored** authority even after a root rotation the running process has not picked up yet. Two rules make the pages legible: a source with
no env config answers `404 {"configured": false}` **without dialling** (so a page says
"not configured" instead of going red), and a history query for a series that does not
exist answers an empty series instead of an error (so a card asks for every plot it could
@@ -458,7 +594,8 @@ auth.py ── verifier + session key, loaded once at startup (no row and no see
per host — counts, `smartctl_available`, and `failed_devices`, so a notifier learns *which*
drive the hardware called `FAILED` without a second call); it is additive, exactly as `alerts`
was, and the key paths of both are pinned with and without targets configured.
* `web/` — five static pages (`/`, `/system`, `/disks`, `/llm`, `/ups`), one stylesheet, hand-rolled
* `web/` — five static dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`) plus the three
admin pages (`/login`, `/settings`, `/certs`), one stylesheet, hand-rolled
SVG charts (`charts.js`), and a masthead status dot per dashboard job (`disks` has its own:
the same agent can answer the 10-second utilisation poll and have a wedged SMART pass). No
framework, no CDN, no bundler, and no narration: a page never explains its own pipeline, so
@@ -640,6 +777,18 @@ local-testing word stays legal in prose, which is the only place it can be usefu
the four public paths, the reset command, the four variables and the `DASH_COOKIE_SECURE=1`
deployment note. Nothing starts a server there; a sentence that quietly disappears is the bug.
`tests/test_certs.py` is the same kind of file for the certificate authority, and two of its tests
are structural rather than behavioural on purpose: they read `certs.py` itself and fail if it gains
a `logging` import or a raise that interpolates anything, because a stored PEM is a private key and
this app turns log lines and error strings into rows a browser reads. The rest runs the whole
lifecycle over a dict store — first boot persists exactly five rows, a second boot changes no byte,
rotate/revoke/root-rotate and their denylists, the bundle never carrying the root key — and one test
completes a real client-certificate handshake with the material a startup would write.
`tests/test_agent.py` runs the matching TLS matrix against the real agent handler, and
`tests/test_deploy_scripts.py` runs the installer in a sandbox where `install`/`chown` are faked, so
the `0640` mode, the byte-identical bundle and the `--status` probe (against a live TLS agent) are
proven without a root shell.
`tests/store_perf.py` is the second opt-in marker's machinery (`live` is the other). It seeds a
**production-shaped
SQLite file** — `SampleStore`'s own retention (`raw 48 h / 1m 30 d / 1h 400 d`), the fleet's real
@@ -699,8 +848,9 @@ the page never reads it.
## Battery-test buttons
Instant commands are the only mutating endpoint in the application (`POST /api/ups/command`)
and they are gated four times: `NUT_ALLOW_ACTIONS=1`, not `DASH_READ_ONLY`, an `upsd.users`
Instant commands are the only endpoint in the application that changes anything **outside this
process** (`POST /api/ups/command`; the session and certificate-authority writes change only this
app's own rows), and they are gated four times: `NUT_ALLOW_ACTIONS=1`, not `DASH_READ_ONLY`, an `upsd.users`
account that actually has the command, and a confirmation in the UI — destructive ones
(deep test, calibration, `load.off`) require typing `DANGER`. The server re-checks every
gate; hiding a button is not a safety control. Commands that would cut power are refused