services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 24_dev_mtls_rollout

Browse Source 
**Phase 24 final verification — all green**

- **Verified (read-only re-run, 62/62 PASS)** via `data/logs/verify-24-mtls.py`: nine sources `ok`; `/api/certs` all three `active` with `deployed_bundle_rev == bundle_rev == 2`; `agent.tls: true` + `agent.version 1.1.0` on every host; driveripper 16/16 drives live; gateway absent from `/disks` but `disks:gateway ok`; all served target URLs `https://`, no userinfo; pre-flip backup present; no transport/cert alert.
- **Criterion 1 (export):** `data/certs/{driveripper,deskwork,gateway}.bundle` re-checked byte-identical (sha256) to `GET /api/certs/<name>/bundle`, mode `0600`, under gitignored `data/`; live arms re-probed (issued→file, unknown/`issued:false`→one-line note + no file, wrong password→exit 2, nothing written).
- **Criterion 2 (`--fetch-bundle`):** 54 sandbox pins in `tests/test_deploy_scripts.py` pass — byte-identical `0640` install + TLS unit lines, refusals (unissued / 401 / non-bundle body / dead dashboard) before `need_root` with no write, token absent from token file, unit, stdout and sudoers hint.
- **Criterion 3–4:** `./scripts/dev-server.sh status` nine `ok` + bundles block; screenshots `data/logs/24-task04-{certs,disks,system}.png` exist.
- **Criterion 7:** no behavior change — `agent/` untouched, `src/` diff is a `certs.ensure_certs` docstring only; token-mode, revocation-before-dial and bundle-route tests green.
- **Tests/lint:** `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1542 passed, 15 skipped, coverage 98.94%**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; `uv run pytest -m live --live` → **11 passed**.
- **Docs:** README.md (all 1144 lines) + AGENTS.md reviewed against the live project — every config.py env var is in README and both templates, route set / severity thresholds / test counts / coverage figure match, all repo paths, cited test names and `Dash.*`/`DashCharts.*` helpers resolve, phase-24 doc pins green. **Verified consistent — no changes needed.**
- **Notable:** left `fastapi 0.153.0 → 0.158.0` (validate reports it as advisory; no dependency was touched, rule 8 conditions the check on deps moving). Killed two abandoned `/tmp/fbtest` loopback stub processes left by task 02's manual probe; dev server unaffected.
- **Next pending phase:** `.agents/phases/todo/25_masthead_control_placement/`.
This commit is contained in:
ducoterra
2026-10-21 13:47:45 -04:00
parent 5f3ed9bcd4
commit 89d9fb9f89
36 changed files with 2378 additions and 146 deletions
Download Patch File Download Diff File Expand all files Collapse all files
deploy/install-agent.sh
+201 -12
View File
@@ -4,6 +4,9 @@
# sudo ./deploy/install-agent.sh # loopback only (read it locally)
# sudo ./deploy/install-agent.sh --bind 10.0.0.5 # serve the LAN, token required
# sudo ./deploy/install-agent.sh --cert-bundle agent.bundle # serve TLS (phase 15)
# sudo ./deploy/install-agent.sh --fetch-bundle https://dashboards.example.com \
# --cert-name deskwork --session <signed-in token> --bind 10.1.0.51 --port 9120
# # registration fetches its own bundle
# sudo ./deploy/install-agent.sh --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN
# # a kernel-gated host (see below)
# ./deploy/install-agent.sh --status # unit state + /health, no root
@@ -23,7 +26,7 @@
# The certificate bundle (--cert-bundle <file>, fetched by the operator from the dashboard's
# GET /api/certs/<name>/bundle or the /certs page) is the TLS install: one PEM holding this
# host's server certificate and key, the dashboard's root CA, a `# BUNDLE-REV <n>` line and
# the revocation denylist. This script never dials the dashboard for it. The bundle is
# the revocation denylist. With a local file this script dials nothing at all. The bundle is
# validated before anything is touched — a bundle the agent could not load must stop the
# install with its reason, not leave a unit that exits at startup — and then installed to
# /etc/dashboards-agent.certs, the token-file idiom: the agent opens the bundle *itself* at
@@ -35,6 +38,31 @@
# the agent exempts for probes. Re-running without the flag leaves an installed bundle in
# place, the token-reuse idiom.
#
# --fetch-bundle is that same install with the download done for you, because registration is
# the moment a client should get its identity (TODO.md:3 — "distribute them to clients during
# registration so we never need non-mtls servers"). It pulls
# GET <url>/api/certs/<cert-name>/bundle and hands the bytes to the path above, so there is no
# curl-a-key-across-the-network step for an operator to forget; the shape check, the 0640
# install, the unit's TLS lines and the --status probe all then run on the fetched bytes
# unchanged.
#
# --fetch-bundle URL the dashboard's own base URL — this is the one dial this script makes
# --cert-name NAME the HOST_TARGETS name the certificate is issued for
# --session TOKEN the signed-in session token, a **fetch credential only**
#
# The three are all-or-nothing, and they are refused together with a local --cert-bundle: one
# install, one bundle source. The token is used for that one request and nothing else — never
# printed, never written to the token file, the unit, the bundle or a log, and unset as soon as
# the bundle is in hand (it is a bearer credential for a signed-in surface, so `ps` exposure
# for the seconds of the fetch is the accepted cost on a single-admin box; the file it buys is
# the artifact, not the credential). The fetch and the shape check both run before the root
# boundary, so a dashboard that answers 401 — or says nothing is issued for that name — stops
# the run with its own sentence and no partial install:
#
# sudo ./deploy/install-agent.sh --bind <addr> --port 9120 \
# --fetch-bundle https://dashboards.example.com --cert-name <HOST_TARGETS name> \
# --session <signed-in session token>
#
# The agent is read-only by construction (it answers GET and writes nothing), runs as its own
# system user (unless --as-root says otherwise, below), and gets the usual systemd lockdown. It
# still needs to read /proc, /sys and statvfs(), which the hardening below leaves alone.
@@ -111,6 +139,17 @@ PORT="9120"
TOKEN="${AGENT_TOKEN:-}"
ROTATE=0
BUNDLE_SRC=""
# Which flag the bytes in $BUNDLE_SRC came from, so a shape refusal names the flag the operator
# typed. Empty means `--cert-bundle` (the default inside `validate_bundle`).
BUNDLE_LABEL=""
# --fetch-bundle: the three parts of "fetch my bundle in this install run". All three or none
# (checked below, before any write), and never together with a local --cert-bundle.
FETCH_URL=""
CERT_NAME=""
SESSION_TOKEN=""
# The temp file the fetched bytes land in, and the trap that guarantees it is gone whatever the
# rest of the run does. Its name is never printed.
FETCH_TMP=""
# --as-root / --caps: the kernel-gate variant of the unit (phase 17 task 02). Both off means
# the default install — dedicated user, empty CapabilityBoundingSet — byte-for-byte what this
# script wrote before the flags existed.
@@ -134,6 +173,9 @@ while [ $# -gt 0 ]; do
--token) TOKEN="$2"; shift 2 ;;
--rotate-token) ROTATE=1; shift ;;
--cert-bundle) BUNDLE_SRC="$2"; shift 2 ;;
--fetch-bundle) FETCH_URL="$2"; shift 2 ;;
--cert-name) CERT_NAME="$2"; shift 2 ;;
--session) SESSION_TOKEN="$2"; shift 2 ;;
--as-root) AS_ROOT=1; shift ;;
--caps)
[ $# -ge 2 ] || {
@@ -167,6 +209,43 @@ if [ -n "$CAPS" ] && [ "$AS_ROOT" -ne 1 ]; then
exit 1
fi
# The same reasoning for the fetch: a flag combination is a mistake about what to install, and
# it gets its own sentence before anything else happens. Three parts because a URL without a
# name is a route nobody can guess and a name without a token is a 401 the operator would have
# to diagnose — so all three or none. And `--fetch-bundle` beside `--cert-bundle` is two
# answers to "where is the bundle?", which is a contradiction rather than a preference.
FETCH_GIVEN=0
for _flag in "$FETCH_URL" "$CERT_NAME" "$SESSION_TOKEN"; do
# an `if`, not `[ -n … ] && FETCH_GIVEN=…`: under `set -e` a failing test as the last command
# of the loop body would end the run on the first unset flag instead of counting them.
if [ -n "$_flag" ]; then
FETCH_GIVEN=$((FETCH_GIVEN + 1))
fi
done
if [ "$FETCH_GIVEN" -gt 0 ] && [ "$FETCH_GIVEN" -lt 3 ]; then
echo "--fetch-bundle, --cert-name and --session go together (see --help)" >&2
exit 1
fi
if [ "$FETCH_GIVEN" -eq 3 ] && [ -n "$BUNDLE_SRC" ]; then
echo "--fetch-bundle and --cert-bundle are two bundle sources; pick one" >&2
exit 1
fi
# The fetch belongs to an install. `--status` probes a port and `--uninstall` deletes files; a
# run of either would fetch a private key for nothing and leave it in a temp file.
if [ "$FETCH_GIVEN" -eq 3 ] && [ "$MODE" != "install" ]; then
echo "--fetch-bundle needs an install (it means nothing to --status or --uninstall)" >&2
exit 1
fi
cleanup_fetch() {
# The fetched bytes are somebody's private key. Registered as an EXIT trap when the fetch
# starts, so no path — a refused shape, a refused unit re-run, a failing systemctl — leaves
# them behind in /tmp. Runs before the trap is armed too, and does nothing then.
if [ -n "$FETCH_TMP" ] && [ -e "$FETCH_TMP" ]; then
rm -f "$FETCH_TMP"
fi
}
need_root() {
if [ "$(id -u)" -ne 0 ]; then
# The dev workflow deploys over ssh as the unprivileged `ops` account, so "run as root"
@@ -269,23 +348,28 @@ validate_bundle() {
# stdlib; a monitored host has no cryptography) and the dashboard is the only signer, so
# a second copy of this shape check would only ever disagree about nothing. The three PEM
# blocks are positional: agent certificate, agent key, root CA certificate.
#
# $2 names the flag the bytes came from, so a refusal says what the operator typed rather than
# a temp file they did not. One validator, two bundle sources: a local --cert-bundle file and
# the fetched bytes of --fetch-bundle have to satisfy exactly the same shape, because the agent
# cannot tell them apart either.
if ! command -v python3 >/dev/null 2>&1; then
echo "--cert-bundle needs python3 (3.10 or newer) to be validated" >&2
echo "${2:---cert-bundle} needs python3 (3.10 or newer) to validate the bundle" >&2
exit 2
fi
if python3 - "$1" <<'PY' 1>&2
if python3 - "$1" "${2:---cert-bundle}" <<'PY' 1>&2
import re
import sys
path = sys.argv[1]
path, flag = sys.argv[1], sys.argv[2]
try:
with open(path, encoding="utf-8", errors="replace") as handle:
text = handle.read()
except OSError as exc:
print(f"--cert-bundle {path} cannot be read: {exc.strerror or exc}")
print(f"{flag} {path} cannot be read: {exc.strerror or exc}")
raise SystemExit(1)
if not text.strip():
print(f"--cert-bundle {path} is empty")
print(f"{flag} {path} is empty")
raise SystemExit(1)
revision = None
for line in text.splitlines():
@@ -299,7 +383,7 @@ for line in text.splitlines():
revision = None
break
if revision is None:
print(f"--cert-bundle {path} has no `# BUNDLE-REV <n>` line — that line is how the"
print(f"{flag} {path} has no `# BUNDLE-REV <n>` line — that line is how the"
" dashboard tells the bundle it issued from a stale one, so a bundle without it"
" is not one this app signed")
raise SystemExit(1)
@@ -307,7 +391,7 @@ blocks = re.findall(r"-----BEGIN [A-Z0-9 ]+-----.*?-----END [A-Z0-9 ]+-----", te
certificates = [b for b in blocks if b.startswith("-----BEGIN CERTIFICATE-----")]
keys = [b for b in blocks if not b.startswith("-----BEGIN CERTIFICATE-----")]
if len(certificates) != 2 or len(keys) != 1:
print(f"--cert-bundle {path} must hold exactly the agent certificate, the agent key and"
print(f"{flag} {path} must hold exactly the agent certificate, the agent key and"
f" the root CA certificate — found {len(certificates)} certificates and"
f" {len(keys)} keys. Re-download it from GET /api/certs/<name>/bundle")
raise SystemExit(1)
@@ -318,15 +402,118 @@ PY
exit 2
}
fetch_bundle() {
# Registration is where a client gets its identity, so the install run pulls this host's
# bundle from the dashboard instead of an operator carrying a file across the network by hand
# (phase 24: "distribute them to clients during registration so we never need non-mtls
# servers"). Two properties are load-bearing, and both are the validator's reasoning.
#
# * **Before `need_root`.** A dashboard that answers 401, or says nothing is issued for that
# name, or does not answer at all, stops the run here with its own sentence — no user, no
# files, no unit. A refusal is not a sudoers problem, and a half-install is worse than none.
# * **Then the existing shape check.** The bytes land in a temp file and become `$BUNDLE_SRC`,
# so the PEM check, the `0640` install, the unit's TLS lines and the `--status` probe see a
# fetched bundle exactly as they see a downloaded one. One install path, two sources.
#
# python3 is already required (the shape check and the TLS probe both use it), so this adds no
# dependency and no assumption that a monitored host has `curl`. The token reaches python in
# the **environment**, never on its argv: argv is world-readable through `ps`, which is the
# same reason the agent's own token is a file rather than a flag — a second copy of a bearer
# credential on a command line would only widen what `ps` shows.
if ! command -v python3 >/dev/null 2>&1; then
echo "--fetch-bundle needs python3 (3.10 or newer) to fetch the bundle" >&2
exit 2
fi
FETCH_TMP="$(mktemp "${TMPDIR:-/tmp}/dashboards-agent-bundle.XXXXXX")"
trap cleanup_fetch EXIT
if ! FETCH_SESSION="$SESSION_TOKEN" python3 - "$FETCH_URL" "$CERT_NAME" "$FETCH_TMP" <<'PY' 1>&2
import json
import os
import sys
import urllib.error
import urllib.parse
import urllib.request
base, name, destination = sys.argv[1].rstrip("/"), sys.argv[2], sys.argv[3]
token = os.environ.get("FETCH_SESSION", "")
route = f"/api/certs/{urllib.parse.quote(name, safe='')}/bundle"
request = urllib.request.Request(f"{base}{route}",
headers={"authorization": f"Bearer {token}"})
try:
with urllib.request.urlopen(request, timeout=20) as response:
status, body = response.status, response.read()
except urllib.error.HTTPError as refused:
# A refusal is the answer, and it carries the reason: this app's 404 body says whether the
# name is configured and whether anything was issued, which are three different operator
# actions (fix the URL, issue a certificate, re-sign in).
status, body = refused.code, refused.read()
except (urllib.error.URLError, TimeoutError, OSError) as unreachable:
reason = getattr(unreachable, "reason", None) or unreachable
print(f"--fetch-bundle: no dashboard answered at {base}: {reason}")
raise SystemExit(1)
try:
parsed = json.loads(body.decode("utf-8", "replace"))
except ValueError:
parsed = None
payload = parsed if isinstance(parsed, dict) else None
def detail() -> str:
"""The dashboard's own sentence, or the first bytes of whatever it sent instead.
Every refusal this prints is text the app writes for an operator, and no response body of
a *successful* fetch ever reaches the output — only the bytes land in the file.
"""
if isinstance(payload, dict) and isinstance(payload.get("detail"), str) and payload["detail"]:
return str(payload["detail"])
text = body.decode("utf-8", "replace").strip().replace("\n", " ")
return text[:160] or "no body"
if status == 200:
if not body.strip():
print(f"--fetch-bundle: GET {route} answered 200 with an empty body — nothing to install")
raise SystemExit(1)
with open(destination, "wb") as handle:
handle.write(body)
raise SystemExit(0)
if (status == 404 and isinstance(payload, dict)
and payload.get("configured") is True and payload.get("issued") is False):
# The invariant-3 shape: a configured host with no certificate yet. Its own sentence, because
# the fix is a click on /certs and not a re-typed URL or a new token.
print(f"--fetch-bundle: no bundle issued for {name} — issue it from /certs first"
f" (POST /api/certs/{name}/rotate mints one; GET {route} answered 404)")
raise SystemExit(1)
print(f"--fetch-bundle: GET {route} answered {status}: {detail()}")
raise SystemExit(1)
PY
then
exit 2
fi
# A fetch credential only: from here the install needs the bytes, not the session. The env
# assignment above was per-command, so the only copy left is the shell variable.
unset SESSION_TOKEN
BUNDLE_SRC="$FETCH_TMP"
BUNDLE_LABEL="--fetch-bundle"
echo " fetched the certificate bundle for $CERT_NAME from $FETCH_URL"
}
# NOT named `install`: bash resolves a function before an external command of the same name,
# so an `install()` here turns the `install -d` / `install -m` calls below into infinite
# recursion instead of creating $AGENT_DIR. Phase 10 task 02 found exactly that on a live
# host: the run looped on the SMART block forever and installed nothing.
do_install() {
# Registration fetches its own bundle, before the root boundary for the same reason the shape
# check is there: a dashboard that refuses is answered with its own sentence, and a run that
# cannot get a bundle writes nothing at all. On success this is just another $BUNDLE_SRC.
if [ -n "$FETCH_URL" ]; then
fetch_bundle
fi
# Validate before need_root: a bad bundle is refused with its own reason whether or not
# this shell is root, so a wrong file says "wrong file" and not "add a sudoers line".
if [ -n "$BUNDLE_SRC" ]; then
validate_bundle "$BUNDLE_SRC"
validate_bundle "$BUNDLE_SRC" "$BUNDLE_LABEL"
fi
need_root --bind "$BIND" --port "$PORT"
[ -f "$AGENT_SOURCE" ] || { echo "no agent next to this script: $AGENT_SOURCE" >&2; exit 1; }
@@ -534,9 +721,11 @@ none:
HOST_TARGETS="hostname=https://$BIND:$PORT"
After a rotate or a root rotation on the dashboard, re-download the bundle and re-run this
script with --cert-bundle; the agent picks a changed denylist up without a restart, but a
new certificate needs one.
After a rotate or a root rotation on the dashboard, fetch the new bundle and re-run this
script: --fetch-bundle <url> --cert-name <name> --session <token> does the download and the
install in one go, and --cert-bundle <file> still works from a file already on this box. The
agent picks a changed denylist up without a restart, but a new certificate needs one, and the
run above only *starts* the unit — restart the service to put the new bundle into effect.
NEXT
else
cat <<NEXT