services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 24_dev_mtls_rollout

Browse Source 
**Phase 24 final verification — all green**

- **Verified (read-only re-run, 62/62 PASS)** via `data/logs/verify-24-mtls.py`: nine sources `ok`; `/api/certs` all three `active` with `deployed_bundle_rev == bundle_rev == 2`; `agent.tls: true` + `agent.version 1.1.0` on every host; driveripper 16/16 drives live; gateway absent from `/disks` but `disks:gateway ok`; all served target URLs `https://`, no userinfo; pre-flip backup present; no transport/cert alert.
- **Criterion 1 (export):** `data/certs/{driveripper,deskwork,gateway}.bundle` re-checked byte-identical (sha256) to `GET /api/certs/<name>/bundle`, mode `0600`, under gitignored `data/`; live arms re-probed (issued→file, unknown/`issued:false`→one-line note + no file, wrong password→exit 2, nothing written).
- **Criterion 2 (`--fetch-bundle`):** 54 sandbox pins in `tests/test_deploy_scripts.py` pass — byte-identical `0640` install + TLS unit lines, refusals (unissued / 401 / non-bundle body / dead dashboard) before `need_root` with no write, token absent from token file, unit, stdout and sudoers hint.
- **Criterion 3–4:** `./scripts/dev-server.sh status` nine `ok` + bundles block; screenshots `data/logs/24-task04-{certs,disks,system}.png` exist.
- **Criterion 7:** no behavior change — `agent/` untouched, `src/` diff is a `certs.ensure_certs` docstring only; token-mode, revocation-before-dial and bundle-route tests green.
- **Tests/lint:** `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1542 passed, 15 skipped, coverage 98.94%**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; `uv run pytest -m live --live` → **11 passed**.
- **Docs:** README.md (all 1144 lines) + AGENTS.md reviewed against the live project — every config.py env var is in README and both templates, route set / severity thresholds / test counts / coverage figure match, all repo paths, cited test names and `Dash.*`/`DashCharts.*` helpers resolve, phase-24 doc pins green. **Verified consistent — no changes needed.**
- **Notable:** left `fastapi 0.153.0 → 0.158.0` (validate reports it as advisory; no dependency was touched, rule 8 conditions the check on deps moving). Killed two abandoned `/tmp/fbtest` loopback stub processes left by task 02's manual probe; dev server unaffected.
- **Next pending phase:** `.agents/phases/todo/25_masthead_control_placement/`.
This commit is contained in:
ducoterra
2026-10-21 13:47:45 -04:00
parent 5f3ed9bcd4
commit 89d9fb9f89
36 changed files with 2378 additions and 146 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+139 -35
View File
@@ -44,25 +44,46 @@ Developing against the real fleet from a laptop — dev dials exactly the addres
dashboard dials, so there is nothing to tunnel and no port to forward:
```bash
cp .env.dev.example .env.dev # your host addresses + agent tokens (chmod 0600)
sudo deploy/install-agent.sh --bind <addr> --port 9120
cp .env.dev.example .env.dev # your host addresses (chmod 0600); this fleet's
# entries are `https://` and carry no userinfo
sudo deploy/install-agent.sh --bind <addr> --port 9120 --fetch-bundle <dashboard-url> \
--cert-name <name> --session <signed-in token>
# once per monitored host: the agent as a systemd
# service, printing its bearer token exactly once
# service, registered with the bundle this dashboard
# signs for it (the token-mode install — no
# `--cert-bundle` — still works and prints its bearer
# token exactly once; see *Protecting an agent*)
set -a; . ./.env.dev; set +a
uv run dashboards # foreground
./scripts/dev-server.sh restart # or backgrounded, with a pidfile + logs
./scripts/dev-server.sh bundles # the bundles it exported, with sizes and modes
```
Local dev signs in with the `DASH_ADMIN_PASSWORD` in that file — the word the template's comment
names (`testpass`, what the hermetic suite uses) — and without it a fresh database exits non-zero
for the same reason production does. `.env.dev` is gitignored and `0600`: it holds the agent
tokens and the upsd password too.
for the same reason production does. `.env.dev` is gitignored and `0600`: it holds the upsd
password, and it held the agent bearer tokens until every host of this fleet moved onto an
installed bundle — a TLS `HOST_TARGETS` entry carries no userinfo, so the credential that used to
sit in the URL is now a certificate on the host and a `meta` row here.
Each `HOST_TARGETS` entry carries that token as URL userinfo
`dev-server.sh restart` also **exports the agent bundles** while it is up: one line per
`HOST_TARGETS` name, and each host's stored bundle written to `data/certs/<name>.bundle` at `0600`
(`data/` is gitignored, so no key this writes can be committed). It is a read of
`GET /api/certs/<name>/bundle` and nothing else — the authority stays the one in the database — so
the file is byte-identical to the row and ready to `scp` into `install-agent.sh --cert-bundle` —
though the normal path now needs neither: `install-agent.sh --fetch-bundle <url> --cert-name <name>
--session <token>` downloads that same route on the host during its own install (see
*Protecting an agent, host by host*). A host with nothing issued writes no file and prints the route
that issues one. See *Managing the certificates*.
Each **token-mode** `HOST_TARGETS` entry carries its token as URL userinfo
(`driveripper=http://<token>@<addr>:9120`); `parse_targets` moves it into an `Authorization`
header, so `Target.url`, `/api/config` and every error message stay credential-free. No root
shell on a host? `deploy/agent-sudoers` is the one-line grant that lets `ops` run the installer,
and running the installer without root prints that exact line. upsd is dialled at
header, so `Target.url`, `/api/config` and every error message stay credential-free. A host that
runs from an installed bundle is `https://<addr>:9120` with **no userinfo at all** — the client
certificate is the credential there, and this fleet is all `https://` today (see
*Protecting an agent, host by host*). No root shell on a host? `deploy/agent-sudoers` is the
one-line grant that lets `ops` run the installer, and running the installer without root prints
that exact line. upsd is dialled at
`NUT_HOST`/`NUT_PORT` on the port it really answers on (3051 on driveripper, not the stock
3493) — see the NUT section below for who may connect.
@@ -81,6 +102,9 @@ python3 host_agent.py --host 0.0.0.0 --port 9120 --token-file /etc/dashboards-ag
python3 host_agent.py --host 0.0.0.0 --port 9120 --cert-bundle /etc/dashboards-agent.certs
sudo deploy/install-agent.sh # systemd unit + user + token
sudo deploy/install-agent.sh --cert-bundle agent.bundle # ...or the TLS install (bundle from /certs)
sudo deploy/install-agent.sh --fetch-bundle https://dashboards.example.com \
--cert-name deskwork --session <signed-in token> --bind <addr> --port 9120
# ...the same TLS install, bundle fetched in the run
sudo deploy/install-agent.sh --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN
# ...or a kernel-gated host (SMART passthrough, below)
sudo deploy/install-agent.sh --force # ...only when it must overwrite a unit it would change
@@ -123,6 +147,19 @@ one installed before phase 09 has no SMART pass to run. Opt out silently with `-
The drive truth this is all developed against lives in `tests/fixtures/smartctl_driveripper_*.txt`:
16 real drives, recorded with serials scrubbed, each one parsed by a test.
`deploy/install-agent.sh --fetch-bundle <url> --cert-name <name> --session <token>` is the same
TLS install with the download done inside the registration run: it asks
`GET <url>/api/certs/<name>/bundle`, writes the answer to a temp file, and hands it to the
identical validator + `0640` install + unit path. The three flags are all-or-nothing, they are
refused together with a local `--cert-bundle` (one install, one bundle source) and with
`--status`/`--uninstall`, and the fetch and the shape check both run **before** the root
boundary — a dashboard that answers 401, or that has issued nothing for that name, stops the run
with its own one-line sentence and no partial install. The session token is a *fetch credential
only*: it is sent on that one request, never printed (the sudoers hint re-quotes the command
without it), never written to the token file, the unit, the bundle or the log, and unset the
moment the bytes are in hand. The temp copy is removed on every exit path, so the installed
`/etc/dashboards-agent.certs` is the only place the fetched key survives.
**TLS mode is `--cert-bundle` (env `AGENT_CERT_BUNDLE`)**: one installed PEM the dashboard
signed — this host's server certificate and key, the dashboard's root CA, a `# BUNDLE-REV <n>`
line and the `# REVOKED <serial>` denylist. The agent then serves `https://` and answers every
@@ -219,42 +256,101 @@ that the *next poll* refuses to dial (see `collectors.AgentTls` above):
| `POST /api/certs/<name>/revoke` | marks the identity revoked, and **re-seals every other bundle** because the `# REVOKED` denylist lives inside each of them — so the fleet has new bundles to collect, while their `bundle_rev` does not move (their identity did not change). There is no unrevoke: the fix is a rotate |
| `POST /api/certs/root/rotate` | a new root, a new client certificate, every agent re-issued, and `{"ok": true, "restart_required": true}` — the CA and the client pair are startup material, so the table is new at once while the running process keeps dialling with what it loaded. Saying so beats pretending a hot-swap that does not exist |
**During development there is no second authority.** The CA the first boot minted into the
`meta` table is the one everything reads, and `scripts/dev-certs-export.py` is the dev loop's half
of that: `./scripts/dev-server.sh restart` runs it after the readiness probe, it signs in to the
local server, downloads every `HOST_TARGETS` name's bundle through `GET /api/certs/<name>/bundle`,
and writes each one to `data/certs/<name>.bundle` at `0600` (`os.open`, so the mode is the file's
birth mode) in gitignored `data/`. Per host it prints exactly one line — the file it wrote with its
size, or `no bundle issued yet — issue one on /certs` — and nothing at all is written for a host
with nothing issued. It is a **read** of the same route the `/certs` download uses, never the
startup path, so a restart cannot rotate an identity: the bytes on disk are the bytes in the row.
A sign-in that fails is exit 2 and a `!` line (a `429` says so with its wait), and
`dev-server.sh` prints that as a warning rather than failing `start`, because the server is
provably up and the authority is untouched. `status` — or `dev-server.sh bundles` — lists what is
there with sizes and modes.
### Protecting an agent, host by host
Three steps, and they are order-dependent because the dashboard picks its dial from the URL:
Four steps, and they are order-dependent because the dashboard picks its dial from the URL. **A URL
flipped before its bundle is installed turns that host red by construction**, so the URL flips
last, after the probe answers.
```bash
# 1. one bundle per host, from the dashboard that owns the authority (a signed-in GET; it is
# byte-identical to the stored row and it carries that host's private key, so `curl -o` it
# somewhere private rather than `cat`ting it into a shell history)
curl -s -H "Authorization: Bearer $SESSION" -o agent-deskwork.bundle \
http://127.0.0.1:8090/api/certs/deskwork/bundle
# 0. issue the host a bundle if it has none: POST /api/certs/<name>/rotate (the Rotate button on
# /certs) is also how a host gets its *first* one, and `GET …/bundle` then answers the bytes.
# An unissued name is what makes step 1 say "no bundle issued for <name>".
# 2. install it beside the token — as root on the host, 0640 root:dashboards-agent. The same run
# copies the repo's `agent/host_agent.py`, which is how a pre-1.1.0 agent gains TLS mode at all
# (an older file does not know the `--cert-bundle` flag). The unit gains `--cert-bundle` and its
# RuntimeDirectory — and then restart the service as root, because the installer's
# `systemctl enable --now` only *starts* a unit, so an already-running agent keeps serving its
# old mode until something restarts it.
scp agent-deskwork.bundle <host>-ops:/tmp/
ssh <host>-ops sudo /tmp/install-agent.sh --cert-bundle /tmp/agent-deskwork.bundle --force
# 1. register the host: one run installs the agent *and* fetches its bundle from the dashboard
# that owns the authority. The name is the HOST_TARGETS name; the token is a signed-in
# session and it buys this one GET — it is not stored on the host. As root, 0640
# root:dashboards-agent, after a shape check on the fetched bytes that runs before anything
# is written, so a 401 or an unissued name leaves this host exactly as it was.
# Stage the repo's script and agent first (`../agent/host_agent.py` is where the installer
# reads the agent from, and the fleet's sudo grant names a path, not a checkout):
scp deploy/install-agent.sh <host>-ops:/home/ops/agent-install/deploy/
scp agent/host_agent.py <host>-ops:/home/ops/agent-install/agent/
ssh <host>-ops sudo /home/ops/agent-install/deploy/install-agent.sh --bind <addr> --port 9120 \
--fetch-bundle https://dashboards.example.com --cert-name deskwork --session "$SESSION" \
--force
# the fallback, for a host that cannot reach the dashboard: the signed-in GET by hand (its
# bytes are identical to the stored row and it carries that host's private key, so `curl -o`
# it somewhere private rather than `cat`ting it into a shell history — and against a local dev
# server this file already exists, written by `./scripts/dev-server.sh restart`), then
# --cert-bundle <file> instead of --fetch-bundle.
# curl -s -H "Authorization: Bearer $SESSION" -o agent-deskwork.bundle \
# http://127.0.0.1:8090/api/certs/deskwork/bundle
# scp agent-deskwork.bundle <host>-ops:/tmp/
# A *dev* dashboard on a laptop is usually unreachable from a monitored host in the other
# direction too (the tunnel route is one-way). An SSH reverse tunnel keeps the registration
# path rather than the scp: `ssh -R 18099:127.0.0.1:8090 <host>-ops` and then
# `--fetch-bundle http://127.0.0.1:18099`.
# 2. restart it as root: the same run copies the repo's `agent/host_agent.py`, which is how a
# pre-1.1.0 agent gains TLS mode at all (an older file does not know the `--cert-bundle`
# flag), and the unit gains `--cert-bundle` and its RuntimeDirectory — but `systemctl enable
# --now` only *starts* a unit, so an already-running agent keeps serving its old mode until
# something restarts it.
ssh <host>-ops sudo systemctl restart dashboards-agent
ssh <host>-ops sudo /tmp/install-agent.sh --status # https, with the CA and no client cert
# the CA-only https probe — pass the SAME --bind the unit uses, because the probe takes its
# address from the flags and would otherwise dial 127.0.0.1 and report "no answer" for an
# agent bound to a LAN address that is in fact healthy.
ssh <host>-ops sudo /home/ops/agent-install/deploy/install-agent.sh --bind <addr> --port 9120 --status
# 3. point that host's HOST_TARGETS entry at https with NO userinfo token — the certificate is
# the auth now — and restart the dashboard. `/certs` should then read issued == deployed rev.
export HOST_TARGETS="deskwork=https://<addr>:9120 …"
```
`--force` in step 2 is not a shortcut: installing a bundle *is* a change to that host's unit (the
`--force` in step 1 is not a shortcut: installing a bundle *is* a change to that host's unit (the
`--cert-bundle` flag and its `RuntimeDirectory`), so without it the run prints the diff and stops
by design — read it, then re-run with the flag. The same rule covers every change the installer
applies, which is why the `--status` line above comes *after* the restart: the probe reads the
unit file to decide between http and https, so pointing it at a process that has not been
restarted yet reports "no answer" for a bundle that installed perfectly.
`install-agent.sh --bind … --port 9120` prints the `https://<addr>:9120` line for a bundle install,
so step 3 is a copy rather than a guess. A host left on `http://` keeps working over its bearer
token — the rollout is per host, and one revoked or expired certificate never blinks the others.
by design — read it, then re-run with the flag. Reading it first is cheap and worth doing on a host
with a hand-fixed unit (run the same command once without the flag, study the diff, run it again
with it). The same rule covers every change the installer applies, which is why the `--status` line
above comes *after* the restart: the probe reads the unit file to decide between http and https, so
pointing it at a process that has not been restarted yet reports "no answer" for a bundle that
installed perfectly. `install-agent.sh --bind … --port 9120` prints the `https://<addr>:9120` line
for a bundle install, so step 3 is a copy rather than a guess. A host left on `http://` keeps
working over its bearer token — the rollout is per host, and one revoked or expired certificate
never blinks the others.
Two things the first full rollout learned, both cheap to hit and expensive to guess at:
* **A CA row older than the identifier fix must be rotated before any bundle is installed.** A
root minted without a `SubjectKeyIdentifier` is still readable, still starts, and still signs —
but `ssl.create_default_context()` (the strict mode every dial here builds) rejects every chain
it produces, with `CERTIFICATE_VERIFY_FAILED: Missing Subject Key Identifier`, so the host comes
up healthy on `https` and reads `unreachable (ConnectError)` from the dashboard. `POST
/api/certs/root/rotate` is the repair — a new root with its identifiers, a new client identity,
every agent re-issued, then a dashboard restart and a fresh bundle per host. Never a
`DELETE FROM meta` for this: the rotate button exists precisely so the operator keeps the
authority in one place.
* **Rolling back is not just the URL.** A TLS-mode agent serves `https` only, and the unit keeps
its `--token-file` line (so the token survives on the box, ready for this). Restoring an old
`http://<token>@…` entry does not bring the host back on its own: the `--cert-bundle` line has to
come out of the unit *and* `/etc/dashboards-agent.certs` has to go, because the installer reuses
an installed bundle on a run that was not given the flag.
A name that is not in `HOST_TARGETS` answers `404 {"configured": false}` **before** any signing, any
dial, and any `meta` write: a typo that minted a key would leave a certificate nobody can install
@@ -476,7 +572,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
| Variable | Meaning |
| ------------------------------------- | -------------------------------------------------------------- |
| `HOST_TARGETS` | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`). An `https://` URL is dialled as a *protected* agent instead: verified against the dashboard's own root CA and presented with its client certificate, with a revoked host refused before the dial (`certs.py`, `collectors.AgentTls`) |
| `HOST_TARGETS` | `name=url` per host running the agent. The shape a new host gets is `https://host:9120` with **no userinfo** — a certificate-authenticated agent, verified against this app's own root CA and presented with its client certificate, with a revoked host refused before the dial (`certs.py`, `collectors.AgentTls`). The compatibility shape is a token-mode agent's URL userinfo (`http://TOKEN@host:9120`), moved into an `Authorization` header by `parse_targets` so it never reaches `/api/config` or an error message |
| `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
| `RATE_WINDOW_SECONDS` | the **host's** Δ window (default 10) — bytes/sec and jiffies→CPU %, measured over what a 10 s poll can fill; each value carries the window it actually got, and `/system`'s footnote prints that, not this default. A window the poll loop cannot honour — the ~0.2 s catch-up after a fetch slower than the interval, or a base older than 2× the window — answers one `n/a` instead of a spike, and a rate 8× above the last one is held until a window measured from a **different** sample confirms it (phase 13). The LLM rates do **not** use it: they are measured over `LLM_INTERVAL_SECONDS` (phase 23) |
| `LLM_INTERVAL_SECONDS` | the `/llm` poll cadence (default 2 s) **and** the Δ window behind every token rate — one estimator per measuring cadence, so a slot that starts reads steady within two polls instead of ramping over five. Slowing the poll slows the window with it; there is no separate LLM window knob, and a window narrower than the interval would answer `n/a` every poll |
@@ -806,7 +902,10 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
uv run pytest # unit + API tests, fully mocked, no network
uv run pytest --cov # coverage gate
set -a; . ./.env.dev; set +a # the live suite reads the same env as the app
uv run pytest -m live --live # optional: real fleet / engines / upsd
uv run pytest -m live --live # optional: real fleet / engines / upsd — an
# `https://` HOST_TARGETS entry is dialled with the
# authority in DASH_DB (read-only), so a box with no
# store skips the host tests instead of failing them
uv run pytest -m store_perf --store-perf -s # optional: the store's hot-path timings (phase 14)
uv run python scripts/store_perf_probe.py # the same measurements as a report in data/logs/
DASH_WATCH_SECONDS=1500 uv run python scripts/watch-store-perf.py
@@ -885,7 +984,12 @@ completes a real client-certificate handshake with the material a startup would
`tests/test_agent.py` runs the matching TLS matrix against the real agent handler, and
`tests/test_deploy_scripts.py` runs the installer in a sandbox where `install`/`chown` are faked, so
the `0640` mode, the byte-identical bundle and the `--status` probe (against a live TLS agent) are
proven without a root shell.
proven without a root shell. The same file runs `scripts/dev-certs-export.py` against a real
`create_app(...)` served by uvicorn on a loopback port (phase 24) — a thread and a socket rather
than `TestClient`, because the unit under test is an HTTP client that reads its session out of a
`Set-Cookie` header — and pins the export against the route: `0600`, byte-identical, one line per
host, no file for an unissued one, no mint and no PEM in the output, and a wrong password exiting
non-zero having written nothing.
`tests/store_perf.py` is the second opt-in marker's machinery (`live` is the other). It seeds a
**production-shaped