services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 24_dev_mtls_rollout

Browse Source 
**Phase 24 final verification — all green**

- **Verified (read-only re-run, 62/62 PASS)** via `data/logs/verify-24-mtls.py`: nine sources `ok`; `/api/certs` all three `active` with `deployed_bundle_rev == bundle_rev == 2`; `agent.tls: true` + `agent.version 1.1.0` on every host; driveripper 16/16 drives live; gateway absent from `/disks` but `disks:gateway ok`; all served target URLs `https://`, no userinfo; pre-flip backup present; no transport/cert alert.
- **Criterion 1 (export):** `data/certs/{driveripper,deskwork,gateway}.bundle` re-checked byte-identical (sha256) to `GET /api/certs/<name>/bundle`, mode `0600`, under gitignored `data/`; live arms re-probed (issued→file, unknown/`issued:false`→one-line note + no file, wrong password→exit 2, nothing written).
- **Criterion 2 (`--fetch-bundle`):** 54 sandbox pins in `tests/test_deploy_scripts.py` pass — byte-identical `0640` install + TLS unit lines, refusals (unissued / 401 / non-bundle body / dead dashboard) before `need_root` with no write, token absent from token file, unit, stdout and sudoers hint.
- **Criterion 3–4:** `./scripts/dev-server.sh status` nine `ok` + bundles block; screenshots `data/logs/24-task04-{certs,disks,system}.png` exist.
- **Criterion 7:** no behavior change — `agent/` untouched, `src/` diff is a `certs.ensure_certs` docstring only; token-mode, revocation-before-dial and bundle-route tests green.
- **Tests/lint:** `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1542 passed, 15 skipped, coverage 98.94%**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; `uv run pytest -m live --live` → **11 passed**.
- **Docs:** README.md (all 1144 lines) + AGENTS.md reviewed against the live project — every config.py env var is in README and both templates, route set / severity thresholds / test counts / coverage figure match, all repo paths, cited test names and `Dash.*`/`DashCharts.*` helpers resolve, phase-24 doc pins green. **Verified consistent — no changes needed.**
- **Notable:** left `fastapi 0.153.0 → 0.158.0` (validate reports it as advisory; no dependency was touched, rule 8 conditions the check on deps moving). Killed two abandoned `/tmp/fbtest` loopback stub processes left by task 02's manual probe; dev server unaffected.
- **Next pending phase:** `.agents/phases/todo/25_masthead_control_placement/`.
This commit is contained in:
ducoterra
2026-10-21 13:47:45 -04:00
parent 5f3ed9bcd4
commit 89d9fb9f89
36 changed files with 2378 additions and 146 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+232 -55
View File
@@ -103,6 +103,19 @@ page still renders.
no line of the stored key), the root key is never written to the temp dir the client material
lives in, and a bundle carries the root *certificate* — never the key — because an agent only
has to verify its clients and a stolen agent install must not be a stolen CA.
**A session token used to fetch a bundle is the same case** (phase 24): the installer's
`--fetch-bundle <url> --cert-name <name> --session <token>` sends that token on exactly one
request, hands it to its `python3` child through the **environment** rather than argv, prints
only the URL in its log line, `unset`s it before anything is written, and never repeats it in
the sudoers hint's re-quoted command (which names `--bind`/`--port` and nothing else).
`tests/test_deploy_scripts.py` signs in for a real token and then greps every file the run
could have touched plus both output streams.
**A bundle exported for the dev loop is the same case** (phase 24):
`scripts/dev-certs-export.py` writes only into the gitignored `data/certs/` tree, creates each
file with `os.open` so `0600` is its *birth* mode (and re-chmods a file that already existed
looser), dials `GET /api/certs/<name>/bundle` over HTTP instead of importing `ensure_certs` — so
it cannot mint even by accident — and prints one line per host with no PEM, no token and no
password in it (`tests/test_deploy_scripts.py` greps the output for all three).
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
`sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
@@ -463,7 +476,14 @@ Responsibilities, one line each:
a write, so a page refresh cannot bootstrap an authority the operator deleted. The material is
read with (task 03): `collectors.agent_tls_map` turns `AppContext.certs` into the per-host dial
pair, and because the revocation answer is `agent_status()` called per poll, a revoke in this
module is what stops the dial — no restart, no cache, no second list to keep in sync.
module is what stops the dial — no restart, no cache, no second list to keep in sync. There is a
**third door and it is deliberately outside this module** (phase 24):
`scripts/dev-certs-export.py` is the dev loop's exporter — stdlib `urllib`, signs in through
`POST /api/auth/login`, reads the token out of the `Set-Cookie`, and downloads each
`HOST_TARGETS` name's bundle into gitignored `data/certs/` at `0600`. It cannot mint because it
imports neither read; it dials the route the `/certs` page downloads from, which is what makes the
exported file byte-identical to the row. `scripts/dev-server.sh` calls it after the readiness
probe and treats a failure as a warning line — the server being up is the thing the probe proved.
- `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric,
`classify()` → `ok`/`warn`/`fail`/`None`, plus `attach()`/`with_severity()` to put the verdict
beside a value. No measurement, no verdict: `None` in → `None` out, never a green pass for
@@ -917,8 +937,10 @@ That scheme is the history API's key — treat it as a public interface.
- **Transport is direct dial: dev is the prod topology, repo-wide (phase 10 task 05).** The
designed shape is one `dashboards-agent` **systemd service per host**, installed by `ops`
through the `deploy/agent-sudoers` grant with `deploy/install-agent.sh`, bound to
the address the dashboard dials, its bearer token carried in the `HOST_TARGETS` URL
userinfo; driveripper's upsd serves the LAN on 3051 with the host and laptop ranges allowed
the address the dashboard dials — since phase 24 registered **with a certificate bundle**
(`--fetch-bundle`, so its `HOST_TARGETS` URL is `https://` and carries no userinfo); what was
once the token in the URL is the compatibility path for a host nobody has bundled. driveripper's
upsd serves the LAN on 3051 with the host and laptop ranges allowed
by the **firewall**, never by a `upsd.conf` line. The old ssh-tunnel script is deleted and
neither `.env.dev` nor `.env.dev.example` declares a forwarded port — dev dials the same
addresses the deployed dashboard does, and `scripts/capture-fixtures.sh` reads
@@ -953,11 +975,28 @@ That scheme is the history API's key — treat it as a public interface.
`/api/health` today reads **all nine sources `ok`**: `host:driveripper`, `host:deskwork`,
`host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `nut`, `llm:gufo`,
`llm:vllm`.
**Since phase 15 that bearer-token transport is the compatibility path, not the designed one:**
the agent can serve `https://` from an installed bundle and the dashboard dials it with its own
client certificate, but no bundle is installed on any host here, so `HOST_TARGETS` stays
`http://` + userinfo until somebody runs the rollout in README's *Protecting an agent, host by
host* (gap 11 says what that needs and why flipping the URL first turns a host red).
**The fleet is mTLS since 2026-10-21 (phase 24 task 03), so the bearer-token transport is now
history rather than the designed shape.** All three hosts run `agent/host_agent.py` **1.1.0**
from an installed bundle, every unit carries `--cert-bundle /etc/dashboards-agent.certs` (plus
`RuntimeDirectory=dashboards-agent`, and still its `--token-file` line), and `HOST_TARGETS` is
`"driveripper=https://10.1.0.126:9120 deskwork=https://10.1.0.51:9120
gateway=https://10.1.1.27:9120"` — **no `http://` entry and no userinfo anywhere**: the
certificate is the credential (the pre-flip `.env.dev` is backed up in `data/env-backup/`). `agent.tls: true` + `agent.bundle_rev: 2` in
every payload, `deployed_bundle_rev == bundle_rev` for all three on `/certs`, and
`./scripts/dev-server.sh status` reads the nine sources `ok`. The per-host order was the safety:
install agent 1.1.0 + bundle → `systemctl restart` → the CA-only `--status` probe answers →
*then* that host's URL flips, so no source was red longer than its own host's restart.
`data/logs/NOTES-24-rollout.md` is the transcript (sha256 of what was staged where, the unit
diffs the re-run guard printed, the before/after SMART tables, the screenshots). Two facts in it
are load-bearing for the next host: the fetch needed an **SSH reverse tunnel** (no monitored host
can route back to a dev box, so `ssh -R 18099:127.0.0.1:8090 <host>` +
`--fetch-bundle http://127.0.0.1:18099` is how registration reaches the dashboard), and
`--status` takes its address from `--bind`, so run bare it dials `127.0.0.1` and reports "no
answer" about a healthy host. **Rolling a host back is not just the URL**: a TLS-mode agent
serves https only, so the `--cert-bundle` line and `/etc/dashboards-agent.certs` both have to go
with it. And the fleet now trusts **this** dashboard's CA — a second deployment with its own
`certs.*` rows cannot dial these agents until it carries the same authority or re-issues and
re-installs every bundle.
Phase 10's proof that the old drop was not this repo's firewall stands as history in
`data/logs/NOTES-verify.md` — do not re-add a tunnel on the strength of it, and re-run the
probe rather than trusting either note.
@@ -1015,16 +1054,25 @@ That scheme is the history API's key — treat it as a public interface.
`upsc`, `udevadm`, the `/etc/ups` reads, and the `tee /etc/ups/upsd.users|upsd.conf` +
`systemctl restart|reload nut-server` pair documented in `deploy/nut-sudoers`); `deskwork`
grants an unrelated set (`btrfs scrub`, `snapper`, `setfacl /srv/*`, `osbuild-auto.sh`).
`deskwork` grants `ops` neither the installer nor `visudo`, so its install went through root by
the operator. **`driveripper` does grant the installer** — `(root) NOPASSWD:
/home/ops/agent-install/deploy/install-agent.sh`, an Oct-18 snapshot of the script staged under
the operator's home (`sudo -n -l`, re-checked 2026-10-20) — and that grant is still the trap,
not the shortcut: it points at that Oct-18 **snapshot**, which carries neither this phase's
flags, nor the filter removal, nor the re-run guard, so running the granted path replaces the
`User=root` + `CapabilityBoundingSet` unit that gap 10 resolved with the snapshot's own
`User=dashboards-agent`, capability-less unit and re-breaks all 16 SMART reads — **silently**,
because the guard that refuses a differing re-run (task 03) lives only in this repo's script,
which therefore has to be re-staged before any of it means anything there. The template now writes the live unit's shape: no device filter at all
Neither `deskwork` nor `driveripper` grants `visudo`, so a sudoers change is the operator's
action. **`deskwork` and `driveripper` both grant the installer** —
`(root) NOPASSWD: /home/ops/agent-install/deploy/install-agent.sh` — and both also grant
`(root) NOPASSWD: /usr/bin/systemctl restart *`. The earlier record
here said deskwork's `ops` had no installer grant and that its install went through the
operator's root; that is out of date — `sudo -n -l` on 2026-10-21 lists the grant, and phase 24
task 03 installed deskwork's bundle as `ops`. The grant names a **path**, and until phase 24
that path held an Oct-18 snapshot of the script which carried neither the later flags, nor the
filter removal, nor the re-run guard — running it would have replaced driveripper's `User=root`
+ `CapabilityBoundingSet` unit with the snapshot's capability-less one, **silently**. That trap
is now closed on all three hosts: `/home/ops/agent-install/` is `ops`-owned, so the repo's
`deploy/install-agent.sh` (sha256 `ffbcd91f…`) and `agent/host_agent.py` (**1.1.0**) were staged
over it without root, and each snapshot was left beside the grant as
`install-agent.sh.oct18-backup`, which sudo will not run. **Re-staging is therefore the first
step of any install, not an optional one** — the grant is only as current as the file it points
at. The `systemctl restart *` grant on those two hosts is the other half of an unattended
install: a bundle changes the unit, and the unit only takes effect on a restart that
`systemctl enable --now` will not perform.
The template now writes the live unit's shape: no device filter at all
(phase 17 task 01 — systemd's default `auto`, with `PrivateDevices=false`) and `--as-root`
plus `--caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` for the `User=root` and
`CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` lines (task 02). The old allow-list's
@@ -1121,15 +1169,24 @@ uv sync --python 3.14 # deps + editable install
cp .env.example .env # production-shaped config (DASH_ADMIN_PASSWORD seeds
# the login on the first boot of a database — with no
# seed and no stored verifier the server exits non-zero)
cp .env.dev.example .env.dev # dev-only: the real hosts, tokens in URL userinfo
cp .env.dev.example .env.dev # dev-only: the real hosts (this fleet's URLs are
# `https://` with no userinfo — the bundle is the cred)
set -a; . ./.env.dev; set +a # dev dials the fleet the prod way, no tunnels
./scripts/dev-server.sh restart # keep http://127.0.0.1:8090 up; sign in with the
# DASH_ADMIN_PASSWORD from .env.dev (`testpass`)
# DASH_ADMIN_PASSWORD from .env.dev (`testpass`), then
# export every issued HOST_TARGETS bundle to data/certs/
./scripts/dev-server.sh status # per-source ok/DOWN — unauthenticated: /api/health
./scripts/dev-server.sh bundles # what that export left: names, sizes, modes
./scripts/dev-server.sh logs
uv run pytest # hermetic: 1494 passed + 15 skipped (11 live,
uv run python scripts/dev-certs-export.py --url http://127.0.0.1:8090 \
--password "$DASH_ADMIN_PASSWORD" --names <the HOST_TARGETS names>
# the export on its own (phase 24): reads
# GET /api/certs/<name>/bundle and writes
# data/certs/<name>.bundle at 0600 — mints nothing
uv run pytest # hermetic: 1542 passed + 15 skipped (11 live,
# 4 store_perf), no network
uv run pytest --cov # coverage gate (98.94%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
@@ -1177,12 +1234,13 @@ bash .agents/validate.sh # the phase gate: the four checks abov
Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on `0.0.0.0:8090`,
SQLite under `/var/lib/dashboards` — put a volume there) and `deploy/install-agent.sh`
per monitored host (systemd unit, own user, `--bind`/`--port`/`--token`/`--rotate-token`/
`--cert-bundle`/`--as-root`/`--caps`/`--force`/`--status`/`--uninstall`; it stores the token
`--cert-bundle`/`--fetch-bundle`/`--cert-name`/`--session`/`--as-root`/`--caps`/`--force`/
`--status`/`--uninstall`; it stores the token
`0640 root:dashboards-agent` — the
unit hands the agent the *path* and the agent opens it as its own user, so that user must be able
to read it and no other account may, and a `--cert-bundle` file is installed to
`/etc/dashboards-agent.certs` at that same mode after a shape check that runs before anything is
touched and never dials the dashboard (the agent re-reads the bundle live for the denylist, so
touched (with a local file this script dials nothing; the fetched path is the one dial, below), so
0600 would fail the unit exactly like it fails the token file; the task file's `0600` is the
intent — *root and the agent's own user only* — and 0640 is the mode that achieves it) — never
puts a token on argv, and runs under `ProtectSystem=strict` with **no device
@@ -1202,7 +1260,19 @@ PEM blocks there and the agent deletes the key once the SSL context holds it), k
bundle's CA with **no** client certificate via a `python3` `ssl` one-liner — task 02's
`CERT_OPTIONAL` + `/health` exemption is what makes that probeable, and there is no external TLS
binary. A re-run without the flag leaves an installed bundle in place, and `--uninstall` removes
it with the token — and neither is guarded, because both are deliberate. What is guarded is the
it with the token — and neither is guarded, because both are deliberate. **Registration can fetch
that bundle** (phase 24 task 02): `--fetch-bundle <dashboard url> --cert-name <HOST_TARGETS name>
--session <signed-in token>` dials `GET /api/certs/<name>/bundle` with `python3` `urllib` (no
`curl` assumption on a monitored host) in the install run and hands the bytes to that same
`--cert-bundle` path, so the shape check, the `0640` install, the unit's TLS lines and the
`--status` probe behave identically for a fetched and a downloaded bundle. The three flags are
all-or-nothing, are refused together with a local `--cert-bundle` and with `--status`/`--uninstall`,
and the fetch plus the shape check run **before** `need_root`: a 401, an unissued name, a proxy's
200-with-HTML or a dashboard that never answers is one line naming the status and no partial
install. The session is a **fetch credential only** — it travels to python in the environment (argv
is world-readable through `ps`, which is why the agent's token is a file), is `unset` as soon as
the bytes are in hand, appears in no written artefact and not even in the sudoers hint's re-quoted
command, and the fetched temp file is removed by an EXIT trap on every path. What is guarded is the
unit itself (phase 17 task 03): it is rendered to a temp file first, and if an installed `$UNIT`
differs the run prints `refusing to overwrite …` plus the `diff -u` on stdout and exits 1 before
the unit write, the `daemon-reload` and the `enable`, so a redeploy cannot undo a hand-fix and an
@@ -1212,7 +1282,9 @@ which is what the TLS rollout's `--cert-bundle` step needs on a token-mode host,
so a run that rewrote `ExecStart` leaves the already-running agent on its old settings until
`systemctl restart dashboards-agent` — and `--status` chooses http from https by reading the unit
file, so probing before that restart reports "no answer" for a change that landed correctly (the
rollout in README restarts for exactly this reason). No root shell? The one sudoers
rollout in README restarts for exactly this reason). It takes its **address** from the flags rather
than the unit, so a bare `--status` probes `127.0.0.1:9120` and answers "no answer" about an agent
bound to a LAN address that is healthy — always pass the same `--bind`/`--port` the install did. No root shell? The one sudoers
line that lets `ops` run the installer is `deploy/agent-sudoers`; run the script without root and
it prints that line with its own resolved path.)
The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other
@@ -1293,6 +1365,16 @@ branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only
fleet, engines, and upsd. Skipped unless `--live`; each test skips when its source is
unset, so partial fleet availability is normal. **Live tests must not write** — a battery
test on hardware something is plugged into is not a test.
**An `https://` agent is dialled with the stored authority, through `fetch_snapshot` and never a
hand-rolled `client.get`** (phase 24 task 03). The module-scoped `agent_tls` fixture reads the
`certs.*` rows out of the same SQLite file the app uses — through a read-only `sqlite3` URI whose
`set_meta`/`delete_meta` **raise**, because "live tests do not write" includes not minting or
repairing an authority from a test — and hands `collectors.agent_tls_map` the material
`certs.write_material` laid down, exactly as `app.build_hub` does. The failure this removes is the
misleading kind: a raw `get` against a healthy mTLS agent dies with `CERTIFICATE_VERIFY_FAILED:
self-signed certificate in certificate chain`, which reads like a dead host and is really a test
that forgot to present a certificate. No store file, or no authority in it, skips — the same
partial-environment rule as an unreachable host.
- The agent's real-`/proc` test (`tests/test_agent.py`) touches only the box running pytest,
so it stays in the default run and is *not* marked `live`.
- `tests/auth_contract.py` is why the HTTP suite can start a server at all: every app in a test
@@ -1346,6 +1428,29 @@ branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only
faked, which is how a `0640` mode, a byte-identical bundle and the https `--status` probe are proven
without a root shell or a host. **A `0600` in a task file is not a fact about this repo**: the agent
opens its own bundle and re-reads it for the denylist, so the group bit is load-bearing.
- **A bundle source is validated once, whichever source it is** (phase 24 task 02). The
installer's `--fetch-bundle` is tested through the same `served_dashboard` socket as the export,
because the claim is the same claim: the file a host installs is the route's bytes. The pins that
go with it are the ones a second install path would break — byte-identical `0640` install with
the unit's TLS lines; `issued: false`, a 401 and a 200-with-HTML each refused with their own
sentence, no stub call logged and no file written (the stub log is the proof the refusal came
*before* the root boundary); a temp `TMPDIR` left empty on the success path and on every refusal
path, including the one where the shape check is what stopped the run; the three flags
all-or-nothing and refused beside a local `--cert-bundle`, beside `--status`/`--uninstall`; and
one label-aware validator, pinned by demanding that the local path still reads `--cert-bundle …`
and never names the flag that was not used.
- **A script that is an HTTP client is tested over a real socket** (phase 24). The dev bundle
exporter is run as a subprocess against a real `create_app(...)` served by uvicorn on
`127.0.0.1:0` in a thread, not through `TestClient`: what has to be proven is that it signs in
through the public route, reads the session token back out of a `Set-Cookie` header, and sends it
as `Authorization: Bearer` — none of which a transport-level test client has to serialise. The
pins that go with it are the ones that keep the artifact honest: the file byte-identical to the
route's bytes, `0600` (including the re-`chmod` of a file somebody loosened), no file and one
note for `issued: false` and for `configured: false`, exit 2 with nothing written for a refused
sign-in, **no mint** (the serials, revisions and root are the same after the run), and no PEM,
token or password in the output. The dev-server composition is pinned on the served text too —
export after the readiness probe, failure as `|| echo` and no `return 1` — because "it runs
later, and never fails `start`" is the part a refactor loses.
- **A live measurement is re-derivable, so a verdict can be re-checked without re-watching**
(phase 23). `scripts/watch-llm-window.py` writes the series it judged (value, the `Δ<window>`
the payload carried, the engine's own busy count, the raw token counter) beside its verdict, and
@@ -2086,19 +2191,73 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
`/snapshot`. So the operator's `--force` run lands the new rationale prose and changes nothing
systemd acts on (the three captures and the exact probe list are in
`data/logs/NOTES-17-installer-repro.md`; no live change was made while pinning this phase). Two facts that came out of reading the live host, both load-bearing for the
next install: the sudo-granted `/home/ops/agent-install` snapshot is the Oct-18 file (no
next install: the sudo-granted `/home/ops/agent-install` path held the Oct-18 file (no
`--as-root`, no `--caps`, no `--force`, no `--cert-bundle`, still nine `DeviceAllow` lines) and
it **refuses** the recipe with `unknown argument: --as-root` and exit 2 before `need_root`, so
that path cannot apply this recipe at all until the repo's script is re-staged — while a
re-run of it *without* the flags is still the silent clobber the guard exists for. (The other
it **refused** the recipe with `unknown argument: --as-root` and exit 2 before `need_root`, so
that path could not apply this recipe at all — while a re-run of it *without* the flags was
still the silent clobber the guard exists for. **Resolved on 2026-10-21:** phase 24 task 03
staged the repo's script and agent 1.1.0 into `/home/ops/agent-install/` on all three hosts
(the directory is `ops`-owned, so no root needed) before running the granted path, and left the
snapshot beside it as a file sudo will not execute. The rule that came out of it is in the
fleet record above: re-stage first, because a path grant is only as current as the file. (The other
fact from that read — an applied change reaches the file but not the running process until
something restarts it — is in the Deployment paragraph, and README's rollout now restarts.)
`tests/fixtures/smartctl_driveripper_*.txt` (captured via `sudo`) remain the ground truth
for the drives' shape.
11. **The CA is durable, the collector dials with it, and no bundle is installed with one**
(phase 15 tasks 01-05, landed 2026-10-20). `certs.py` mints the CA and the dashboard's client
certificate into `meta` on the first boot of a database — verified against the running dev
store: the five `certs.*` rows exist
11. **The fleet runs on installed bundles; what is left is rotation hygiene**
(phase 15 tasks 01-05 landed 2026-10-20; phase 24 task 03 rolled the fleet out 2026-10-21).
The rollout is done and verified: gateway, deskwork and driveripper each run `agent/host_agent.py`
**1.1.0** with `/etc/dashboards-agent.certs` installed `0640 root:dashboards-agent`, their units
carry `--cert-bundle` + `RuntimeDirectory=dashboards-agent` (and keep `--token-file`),
`HOST_TARGETS` is `https://` for all three with no userinfo, `agent.bundle_rev` reads `2` on
every host and equals what `/api/certs` issued, and the nine sources are `ok`. The transcript is
`data/logs/NOTES-24-rollout.md`, and task 04 re-read the whole end state **read-only** afterwards
(`data/logs/verify-24-mtls.py` → `24-task04-endstate.txt`, plus the rendered proof
`data/logs/24-task04-verify.cjs` → `24-task04-{certs,disks,system}.png`): nine sources `ok`,
`status: active` with `deployed_bundle_rev == bundle_rev` on all three rows, `agent.tls` +
`agent.bundle_rev` + `agent.version 1.1.0` in every host payload, driveripper's 16 drives and
deskwork's `nvme0n1` reporting temperature, hours and wear, deskwork's GPU card drawn on
`/system`, gateway still absent from `/disks` with `disks:gateway` still `ok`, no `http://` and
no userinfo in any URL the dashboard serves, and **no transport or certificate row in
`/api/summary`'s alerts**. It revoked nothing: the standing negative control for "a revoked
serial is refused" remains phase 15's `data/logs/verify-15-tls-e2e.py` (transcript
`verify-15-tls-e2e.out`) plus the two suite pins — `tests/test_collectors.py`'s
`test_a_revoked_agent_is_refused_before_anything_is_dialled` and `tests/test_api.py`'s
`test_a_revoked_agent_fails_both_of_its_sources_without_a_dial`.
The **docs** are part of this end state, because a fleet comes back to `http://` through a
setup file somebody copies: `tests/test_deploy_scripts.py` now pins that `README.md`,
`AGENTS.md` and both tracked env templates name the registration trio (`--fetch-bundle`,
`--cert-name`, `--session`), and that every `HOST_TARGETS` example line in the templates is
`https://` with no userinfo (an empty `HOST_TARGETS=` stays allowed — unconfigured is not
misconfigured). The token-mode dial itself is unchanged and still documented as the
compatibility path; what may not come back is it being the shape a template seeds.
Two things that gap learned belong with this record, because both bite before anybody reads a
dashboard:
* **A `certs.root.cert` row older than the SKI/AKI fix signs chains nobody can verify.** "A
stored row always wins" is the invariant that keeps a CA from rotating with the process, and
its cost is that a root minted before the extension keeps being used forever: a leaf issued
afterwards carries a correct AKI pointing at an identifier its issuer does not have, and
`ssl.create_default_context()` — strict mode, the context `AgentTls.ssl_context()` builds —
answers `CERTIFICATE_VERIFY_FAILED: Missing Subject Key Identifier`. The agent is healthy,
`--status` answers over TLS, and the dashboard calls the host unreachable. The repair is
`POST /api/certs/root/rotate`, a dashboard restart (the CA and client pair are startup
material) and a fresh bundle per host — never `DELETE FROM meta`, which throws away the
authority instead of replacing it. Pinned by
`tests/test_certs.py::test_a_stored_root_without_an_identifier_still_boots_and_root_rotate_repairs_it`,
which also pins the other half: reading such a table must **not** fail closed, because
`/certs` is the page that fixes it.
* **These bundles came from the dev-box authority.** The fleet trusts the `certs.*` rows in the
database `./scripts/dev-server.sh` runs against, and nothing else dials these agents (a
`ss` sweep on each host saw only the dev box on 9120). A second deployment with its own
authority cannot read this fleet until it carries the same root or re-issues and reinstalls;
that is a deployment-repo decision, and the honest version is one authority per fleet.
Everything below is the phase-15/24 history that led here.
`certs.py` mints the CA and the dashboard's client certificate into `meta` on the first boot of
a database — verified against the running dev store: the five `certs.*` rows exist
and did not change across two restarts — signs per-host server certificates into
`certs.agents`, and does reissue, revocation, and root rotation. Task 02 gave
`agent/host_agent.py` its TLS mode (see the agent bullet above: one installed bundle, the
@@ -2109,9 +2268,22 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
socket opens** — spy-pinned in `tests/test_collectors.py` and again through the real
`build_hub` wiring in `tests/test_api.py`, where the revocation is written into the same `meta`
table the running app reads. So both halves are proven against each other with no network and
no hardware. What is still missing is an **installed** bundle: `HOST_TARGETS` is `http://` for
every host here and no agent runs `--cert-bundle`, so deleting `certs.*` costs nothing today
and everything once bundles are placed (which is what the reset message warns about). Proving
no hardware. What phase 15 left missing — an **installed** bundle, which is what makes
`DELETE FROM meta` the expensive command the reset message warns about — is what task 03
supplied: deleting `certs.*` now costs the fleet's identity, and every host has to be
re-bundled. Phase 24
task 01 has since closed the **dev-side** half of that sentence: `./scripts/dev-server.sh
restart` exports every issued bundle into gitignored `data/certs/<name>.bundle` at `0600`, so
putting one on a host is an `scp` plus `install-agent.sh --cert-bundle` rather than a hand-run
signed `curl` (`scripts/dev-certs-export.py`, invariant 6's export rules, pinned by
`tests/test_deploy_scripts.py` against a real uvicorn-served `create_app`). Task 02 has since
removed even that hand-run from the rollout: `install-agent.sh --fetch-bundle <url> --cert-name
<name> --session <token>` fetches `GET /api/certs/<name>/bundle` during the install and feeds
the bytes to the unchanged validator + `0640` install + TLS-unit path, refusing a 401, an
unissued name or a non-bundle body before it writes anything, with the session token absent from
every artefact the run leaves (pinned the same way, over the same socket) — which is the path
all three hosts were installed through, over an SSH reverse tunnel because no monitored host
can route back to a laptop. Proving
the dial also found one defect in task 01's output: the leaves carried no
`AuthorityKeyIdentifier`, and OpenSSL's strict mode — which is what Python's
`ssl.create_default_context()` turns on, and that is the context this dial builds — refuses
@@ -2155,23 +2327,25 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
followed by `agent certificate revoked` on both jobs with no dial, the dashboard's own client
identity untouched by a *host* revoke (the list is serials, not hosts), and a revoked **client**
serial refused 403 by the running agent on `/health` too the moment the re-sealed bundle is in
place (`data/logs/verify-15-tls-e2e.py`, transcript `verify-15-tls-e2e.out`). What is left of
the phase is the live rollout (07): the fleet's agents still run token mode, so `HOST_TARGETS`
stays `http://` here until somebody installs a bundle on a host — which needs root on that host
(`ops` has no installer grant on `deskwork`, and `driveripper`'s unit is reproducible only with
phase 17's `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` line, gap 10), and flipping
the URL before the bundle is installed turns that host red by construction.
Re-checked from the dev box the same day, and worth knowing before that rollout: all three
agents answer `/health` with `version` **1.0.0** (`agent_version` is the `/snapshot` spelling),
so none of them carries task 02's TLS code either — the rollout ships the repo's
`agent/host_agent.py` (1.1.0) *together with* the
bundle, never the bundle alone. `driveripper`'s installed copy differs from the repo's by
exactly that TLS block; its `_smart_pass` and the phase-16 budget scale-up are already there.
Copying 1.1.0 into place is not an `install-agent.sh` re-run on that host (gap 10) — even
though `ops` there is sudo-granted a staged copy of the installer (see the fleet record above),
which is the one route that must not be taken until that snapshot is re-staged from this repo:
the Oct-18 copy predates phase 17's template *and* its re-run guard, so it would rewrite that
unit silently — and no live change was made while pinning phase 16.
place (`data/logs/verify-15-tls-e2e.py`, transcript `verify-15-tls-e2e.out`).
What was left was the live rollout, and phase 24 task 03 ran it on 2026-10-21, host by host, in
the order that keeps every number honest: install **1.1.0 together with the bundle** (never the
bundle alone — the installed agents were 1.0.0 and do not know `--cert-bundle`), restart the
unit, let the CA-only `--status` probe answer, and only *then* flip that host's URL, because a
URL flipped before its bundle is installed turns the host red by construction. All three: the
trivial `gateway`, the drop-in-carrying `deskwork`, and `driveripper` with the gap-10 line
(`--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force`), whose dry-run diff moved comments and
the TLS lines only — `User=root` and `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` came
through byte-identical, and all 16 drives read live on both sides of the rewrite (same single
`nvme5n1` alarm chip). Every host ran the installer **twice**, once without `--force` to read
what task 17's guard would refuse before letting it apply. Staging the repo's script and agent
into the sudo-granted `/home/ops/agent-install/` path was step one on all three (gap 10), and
fetching was reached through an SSH reverse tunnel because no monitored host can route back to
the dev box. `data/logs/NOTES-24-rollout.md` is the transcript: commands, unit diffs, the
before/after SMART tables, the screenshots, and the two operational traps named at the head of
this gap. What the fleet now *never* does is answer an agent over plain `http` or accept a
bearer token on the certificate path — the TODO's "the client should never allow non-mtls
communication" is the shipped state, and a new host has to be registered the same way.
12. **The nav collapses on a phone; the page content below it does not** (phase 18's verification
pass, 2026-10-20). At 360px `/`'s fleet table measures 450-510 px inside a 320 px
`section.card`, and `.card { overflow: hidden }` clips the tail columns (`WORST DISK`,
@@ -2191,8 +2365,11 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
left-justified rows at every width, it is no longer sticky, the phone menu slides, and both
`⚙ Account` and `⚷ Certs` render on all seven signed-in pages including the two they name
(task 04, reversing phase 15's "no self-link" rule at the operator's request). The other,
`24_dev_mtls_rollout` (never accepting a non-mTLS agent — fleet gap 11's rollout, half of it),
is still in `todo/`.
`24_dev_mtls_rollout` (never accepting a non-mTLS agent) is closed: its fleet half landed on
2026-10-21 and task 04 re-read the end state read-only the same day — gap 11. No `http://`
agent URL and no token-mode agent is left on this fleet, and the setup templates no longer
seed one (`tests/test_deploy_scripts.py` keeps it that way). What remains of phase 24 is
ordinary rotation hygiene, not a missing step.
## Locked decisions (and why)