phase: 24_dev_mtls_rollout
**Phase 24 final verification — all green**
- **Verified (read-only re-run, 62/62 PASS)** via `data/logs/verify-24-mtls.py`: nine sources `ok`; `/api/certs` all three `active` with `deployed_bundle_rev == bundle_rev == 2`; `agent.tls: true` + `agent.version 1.1.0` on every host; driveripper 16/16 drives live; gateway absent from `/disks` but `disks:gateway ok`; all served target URLs `https://`, no userinfo; pre-flip backup present; no transport/cert alert.
- **Criterion 1 (export):** `data/certs/{driveripper,deskwork,gateway}.bundle` re-checked byte-identical (sha256) to `GET /api/certs/<name>/bundle`, mode `0600`, under gitignored `data/`; live arms re-probed (issued→file, unknown/`issued:false`→one-line note + no file, wrong password→exit 2, nothing written).
- **Criterion 2 (`--fetch-bundle`):** 54 sandbox pins in `tests/test_deploy_scripts.py` pass — byte-identical `0640` install + TLS unit lines, refusals (unissued / 401 / non-bundle body / dead dashboard) before `need_root` with no write, token absent from token file, unit, stdout and sudoers hint.
- **Criterion 3–4:** `./scripts/dev-server.sh status` nine `ok` + bundles block; screenshots `data/logs/24-task04-{certs,disks,system}.png` exist.
- **Criterion 7:** no behavior change — `agent/` untouched, `src/` diff is a `certs.ensure_certs` docstring only; token-mode, revocation-before-dial and bundle-route tests green.
- **Tests/lint:** `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1542 passed, 15 skipped, coverage 98.94%**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; `uv run pytest -m live --live` → **11 passed**.
- **Docs:** README.md (all 1144 lines) + AGENTS.md reviewed against the live project — every config.py env var is in README and both templates, route set / severity thresholds / test counts / coverage figure match, all repo paths, cited test names and `Dash.*`/`DashCharts.*` helpers resolve, phase-24 doc pins green. **Verified consistent — no changes needed.**
- **Notable:** left `fastapi 0.153.0 → 0.158.0` (validate reports it as advisory; no dependency was touched, rule 8 conditions the check on deps moving). Killed two abandoned `/tmp/fbtest` loopback stub processes left by task 02's manual probe; dev server unaffected.
- **Next pending phase:** `.agents/phases/todo/25_masthead_control_placement/`.
This commit is contained in:
+201
-12
@@ -4,6 +4,9 @@
|
||||
# sudo ./deploy/install-agent.sh # loopback only (read it locally)
|
||||
# sudo ./deploy/install-agent.sh --bind 10.0.0.5 # serve the LAN, token required
|
||||
# sudo ./deploy/install-agent.sh --cert-bundle agent.bundle # serve TLS (phase 15)
|
||||
# sudo ./deploy/install-agent.sh --fetch-bundle https://dashboards.example.com \
|
||||
# --cert-name deskwork --session <signed-in token> --bind 10.1.0.51 --port 9120
|
||||
# # registration fetches its own bundle
|
||||
# sudo ./deploy/install-agent.sh --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN
|
||||
# # a kernel-gated host (see below)
|
||||
# ./deploy/install-agent.sh --status # unit state + /health, no root
|
||||
@@ -23,7 +26,7 @@
|
||||
# The certificate bundle (--cert-bundle <file>, fetched by the operator from the dashboard's
|
||||
# GET /api/certs/<name>/bundle or the /certs page) is the TLS install: one PEM holding this
|
||||
# host's server certificate and key, the dashboard's root CA, a `# BUNDLE-REV <n>` line and
|
||||
# the revocation denylist. This script never dials the dashboard for it. The bundle is
|
||||
# the revocation denylist. With a local file this script dials nothing at all. The bundle is
|
||||
# validated before anything is touched — a bundle the agent could not load must stop the
|
||||
# install with its reason, not leave a unit that exits at startup — and then installed to
|
||||
# /etc/dashboards-agent.certs, the token-file idiom: the agent opens the bundle *itself* at
|
||||
@@ -35,6 +38,31 @@
|
||||
# the agent exempts for probes. Re-running without the flag leaves an installed bundle in
|
||||
# place, the token-reuse idiom.
|
||||
#
|
||||
# --fetch-bundle is that same install with the download done for you, because registration is
|
||||
# the moment a client should get its identity (TODO.md:3 — "distribute them to clients during
|
||||
# registration so we never need non-mtls servers"). It pulls
|
||||
# GET <url>/api/certs/<cert-name>/bundle and hands the bytes to the path above, so there is no
|
||||
# curl-a-key-across-the-network step for an operator to forget; the shape check, the 0640
|
||||
# install, the unit's TLS lines and the --status probe all then run on the fetched bytes
|
||||
# unchanged.
|
||||
#
|
||||
# --fetch-bundle URL the dashboard's own base URL — this is the one dial this script makes
|
||||
# --cert-name NAME the HOST_TARGETS name the certificate is issued for
|
||||
# --session TOKEN the signed-in session token, a **fetch credential only**
|
||||
#
|
||||
# The three are all-or-nothing, and they are refused together with a local --cert-bundle: one
|
||||
# install, one bundle source. The token is used for that one request and nothing else — never
|
||||
# printed, never written to the token file, the unit, the bundle or a log, and unset as soon as
|
||||
# the bundle is in hand (it is a bearer credential for a signed-in surface, so `ps` exposure
|
||||
# for the seconds of the fetch is the accepted cost on a single-admin box; the file it buys is
|
||||
# the artifact, not the credential). The fetch and the shape check both run before the root
|
||||
# boundary, so a dashboard that answers 401 — or says nothing is issued for that name — stops
|
||||
# the run with its own sentence and no partial install:
|
||||
#
|
||||
# sudo ./deploy/install-agent.sh --bind <addr> --port 9120 \
|
||||
# --fetch-bundle https://dashboards.example.com --cert-name <HOST_TARGETS name> \
|
||||
# --session <signed-in session token>
|
||||
#
|
||||
# The agent is read-only by construction (it answers GET and writes nothing), runs as its own
|
||||
# system user (unless --as-root says otherwise, below), and gets the usual systemd lockdown. It
|
||||
# still needs to read /proc, /sys and statvfs(), which the hardening below leaves alone.
|
||||
@@ -111,6 +139,17 @@ PORT="9120"
|
||||
TOKEN="${AGENT_TOKEN:-}"
|
||||
ROTATE=0
|
||||
BUNDLE_SRC=""
|
||||
# Which flag the bytes in $BUNDLE_SRC came from, so a shape refusal names the flag the operator
|
||||
# typed. Empty means `--cert-bundle` (the default inside `validate_bundle`).
|
||||
BUNDLE_LABEL=""
|
||||
# --fetch-bundle: the three parts of "fetch my bundle in this install run". All three or none
|
||||
# (checked below, before any write), and never together with a local --cert-bundle.
|
||||
FETCH_URL=""
|
||||
CERT_NAME=""
|
||||
SESSION_TOKEN=""
|
||||
# The temp file the fetched bytes land in, and the trap that guarantees it is gone whatever the
|
||||
# rest of the run does. Its name is never printed.
|
||||
FETCH_TMP=""
|
||||
# --as-root / --caps: the kernel-gate variant of the unit (phase 17 task 02). Both off means
|
||||
# the default install — dedicated user, empty CapabilityBoundingSet — byte-for-byte what this
|
||||
# script wrote before the flags existed.
|
||||
@@ -134,6 +173,9 @@ while [ $# -gt 0 ]; do
|
||||
--token) TOKEN="$2"; shift 2 ;;
|
||||
--rotate-token) ROTATE=1; shift ;;
|
||||
--cert-bundle) BUNDLE_SRC="$2"; shift 2 ;;
|
||||
--fetch-bundle) FETCH_URL="$2"; shift 2 ;;
|
||||
--cert-name) CERT_NAME="$2"; shift 2 ;;
|
||||
--session) SESSION_TOKEN="$2"; shift 2 ;;
|
||||
--as-root) AS_ROOT=1; shift ;;
|
||||
--caps)
|
||||
[ $# -ge 2 ] || {
|
||||
@@ -167,6 +209,43 @@ if [ -n "$CAPS" ] && [ "$AS_ROOT" -ne 1 ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# The same reasoning for the fetch: a flag combination is a mistake about what to install, and
|
||||
# it gets its own sentence before anything else happens. Three parts because a URL without a
|
||||
# name is a route nobody can guess and a name without a token is a 401 the operator would have
|
||||
# to diagnose — so all three or none. And `--fetch-bundle` beside `--cert-bundle` is two
|
||||
# answers to "where is the bundle?", which is a contradiction rather than a preference.
|
||||
FETCH_GIVEN=0
|
||||
for _flag in "$FETCH_URL" "$CERT_NAME" "$SESSION_TOKEN"; do
|
||||
# an `if`, not `[ -n … ] && FETCH_GIVEN=…`: under `set -e` a failing test as the last command
|
||||
# of the loop body would end the run on the first unset flag instead of counting them.
|
||||
if [ -n "$_flag" ]; then
|
||||
FETCH_GIVEN=$((FETCH_GIVEN + 1))
|
||||
fi
|
||||
done
|
||||
if [ "$FETCH_GIVEN" -gt 0 ] && [ "$FETCH_GIVEN" -lt 3 ]; then
|
||||
echo "--fetch-bundle, --cert-name and --session go together (see --help)" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ "$FETCH_GIVEN" -eq 3 ] && [ -n "$BUNDLE_SRC" ]; then
|
||||
echo "--fetch-bundle and --cert-bundle are two bundle sources; pick one" >&2
|
||||
exit 1
|
||||
fi
|
||||
# The fetch belongs to an install. `--status` probes a port and `--uninstall` deletes files; a
|
||||
# run of either would fetch a private key for nothing and leave it in a temp file.
|
||||
if [ "$FETCH_GIVEN" -eq 3 ] && [ "$MODE" != "install" ]; then
|
||||
echo "--fetch-bundle needs an install (it means nothing to --status or --uninstall)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cleanup_fetch() {
|
||||
# The fetched bytes are somebody's private key. Registered as an EXIT trap when the fetch
|
||||
# starts, so no path — a refused shape, a refused unit re-run, a failing systemctl — leaves
|
||||
# them behind in /tmp. Runs before the trap is armed too, and does nothing then.
|
||||
if [ -n "$FETCH_TMP" ] && [ -e "$FETCH_TMP" ]; then
|
||||
rm -f "$FETCH_TMP"
|
||||
fi
|
||||
}
|
||||
|
||||
need_root() {
|
||||
if [ "$(id -u)" -ne 0 ]; then
|
||||
# The dev workflow deploys over ssh as the unprivileged `ops` account, so "run as root"
|
||||
@@ -269,23 +348,28 @@ validate_bundle() {
|
||||
# stdlib; a monitored host has no cryptography) and the dashboard is the only signer, so
|
||||
# a second copy of this shape check would only ever disagree about nothing. The three PEM
|
||||
# blocks are positional: agent certificate, agent key, root CA certificate.
|
||||
#
|
||||
# $2 names the flag the bytes came from, so a refusal says what the operator typed rather than
|
||||
# a temp file they did not. One validator, two bundle sources: a local --cert-bundle file and
|
||||
# the fetched bytes of --fetch-bundle have to satisfy exactly the same shape, because the agent
|
||||
# cannot tell them apart either.
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
echo "--cert-bundle needs python3 (3.10 or newer) to be validated" >&2
|
||||
echo "${2:---cert-bundle} needs python3 (3.10 or newer) to validate the bundle" >&2
|
||||
exit 2
|
||||
fi
|
||||
if python3 - "$1" <<'PY' 1>&2
|
||||
if python3 - "$1" "${2:---cert-bundle}" <<'PY' 1>&2
|
||||
import re
|
||||
import sys
|
||||
|
||||
path = sys.argv[1]
|
||||
path, flag = sys.argv[1], sys.argv[2]
|
||||
try:
|
||||
with open(path, encoding="utf-8", errors="replace") as handle:
|
||||
text = handle.read()
|
||||
except OSError as exc:
|
||||
print(f"--cert-bundle {path} cannot be read: {exc.strerror or exc}")
|
||||
print(f"{flag} {path} cannot be read: {exc.strerror or exc}")
|
||||
raise SystemExit(1)
|
||||
if not text.strip():
|
||||
print(f"--cert-bundle {path} is empty")
|
||||
print(f"{flag} {path} is empty")
|
||||
raise SystemExit(1)
|
||||
revision = None
|
||||
for line in text.splitlines():
|
||||
@@ -299,7 +383,7 @@ for line in text.splitlines():
|
||||
revision = None
|
||||
break
|
||||
if revision is None:
|
||||
print(f"--cert-bundle {path} has no `# BUNDLE-REV <n>` line — that line is how the"
|
||||
print(f"{flag} {path} has no `# BUNDLE-REV <n>` line — that line is how the"
|
||||
" dashboard tells the bundle it issued from a stale one, so a bundle without it"
|
||||
" is not one this app signed")
|
||||
raise SystemExit(1)
|
||||
@@ -307,7 +391,7 @@ blocks = re.findall(r"-----BEGIN [A-Z0-9 ]+-----.*?-----END [A-Z0-9 ]+-----", te
|
||||
certificates = [b for b in blocks if b.startswith("-----BEGIN CERTIFICATE-----")]
|
||||
keys = [b for b in blocks if not b.startswith("-----BEGIN CERTIFICATE-----")]
|
||||
if len(certificates) != 2 or len(keys) != 1:
|
||||
print(f"--cert-bundle {path} must hold exactly the agent certificate, the agent key and"
|
||||
print(f"{flag} {path} must hold exactly the agent certificate, the agent key and"
|
||||
f" the root CA certificate — found {len(certificates)} certificates and"
|
||||
f" {len(keys)} keys. Re-download it from GET /api/certs/<name>/bundle")
|
||||
raise SystemExit(1)
|
||||
@@ -318,15 +402,118 @@ PY
|
||||
exit 2
|
||||
}
|
||||
|
||||
fetch_bundle() {
|
||||
# Registration is where a client gets its identity, so the install run pulls this host's
|
||||
# bundle from the dashboard instead of an operator carrying a file across the network by hand
|
||||
# (phase 24: "distribute them to clients during registration so we never need non-mtls
|
||||
# servers"). Two properties are load-bearing, and both are the validator's reasoning.
|
||||
#
|
||||
# * **Before `need_root`.** A dashboard that answers 401, or says nothing is issued for that
|
||||
# name, or does not answer at all, stops the run here with its own sentence — no user, no
|
||||
# files, no unit. A refusal is not a sudoers problem, and a half-install is worse than none.
|
||||
# * **Then the existing shape check.** The bytes land in a temp file and become `$BUNDLE_SRC`,
|
||||
# so the PEM check, the `0640` install, the unit's TLS lines and the `--status` probe see a
|
||||
# fetched bundle exactly as they see a downloaded one. One install path, two sources.
|
||||
#
|
||||
# python3 is already required (the shape check and the TLS probe both use it), so this adds no
|
||||
# dependency and no assumption that a monitored host has `curl`. The token reaches python in
|
||||
# the **environment**, never on its argv: argv is world-readable through `ps`, which is the
|
||||
# same reason the agent's own token is a file rather than a flag — a second copy of a bearer
|
||||
# credential on a command line would only widen what `ps` shows.
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
echo "--fetch-bundle needs python3 (3.10 or newer) to fetch the bundle" >&2
|
||||
exit 2
|
||||
fi
|
||||
FETCH_TMP="$(mktemp "${TMPDIR:-/tmp}/dashboards-agent-bundle.XXXXXX")"
|
||||
trap cleanup_fetch EXIT
|
||||
if ! FETCH_SESSION="$SESSION_TOKEN" python3 - "$FETCH_URL" "$CERT_NAME" "$FETCH_TMP" <<'PY' 1>&2
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
base, name, destination = sys.argv[1].rstrip("/"), sys.argv[2], sys.argv[3]
|
||||
token = os.environ.get("FETCH_SESSION", "")
|
||||
route = f"/api/certs/{urllib.parse.quote(name, safe='')}/bundle"
|
||||
request = urllib.request.Request(f"{base}{route}",
|
||||
headers={"authorization": f"Bearer {token}"})
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=20) as response:
|
||||
status, body = response.status, response.read()
|
||||
except urllib.error.HTTPError as refused:
|
||||
# A refusal is the answer, and it carries the reason: this app's 404 body says whether the
|
||||
# name is configured and whether anything was issued, which are three different operator
|
||||
# actions (fix the URL, issue a certificate, re-sign in).
|
||||
status, body = refused.code, refused.read()
|
||||
except (urllib.error.URLError, TimeoutError, OSError) as unreachable:
|
||||
reason = getattr(unreachable, "reason", None) or unreachable
|
||||
print(f"--fetch-bundle: no dashboard answered at {base}: {reason}")
|
||||
raise SystemExit(1)
|
||||
|
||||
try:
|
||||
parsed = json.loads(body.decode("utf-8", "replace"))
|
||||
except ValueError:
|
||||
parsed = None
|
||||
payload = parsed if isinstance(parsed, dict) else None
|
||||
|
||||
|
||||
def detail() -> str:
|
||||
"""The dashboard's own sentence, or the first bytes of whatever it sent instead.
|
||||
|
||||
Every refusal this prints is text the app writes for an operator, and no response body of
|
||||
a *successful* fetch ever reaches the output — only the bytes land in the file.
|
||||
"""
|
||||
if isinstance(payload, dict) and isinstance(payload.get("detail"), str) and payload["detail"]:
|
||||
return str(payload["detail"])
|
||||
text = body.decode("utf-8", "replace").strip().replace("\n", " ")
|
||||
return text[:160] or "no body"
|
||||
|
||||
|
||||
if status == 200:
|
||||
if not body.strip():
|
||||
print(f"--fetch-bundle: GET {route} answered 200 with an empty body — nothing to install")
|
||||
raise SystemExit(1)
|
||||
with open(destination, "wb") as handle:
|
||||
handle.write(body)
|
||||
raise SystemExit(0)
|
||||
if (status == 404 and isinstance(payload, dict)
|
||||
and payload.get("configured") is True and payload.get("issued") is False):
|
||||
# The invariant-3 shape: a configured host with no certificate yet. Its own sentence, because
|
||||
# the fix is a click on /certs and not a re-typed URL or a new token.
|
||||
print(f"--fetch-bundle: no bundle issued for {name} — issue it from /certs first"
|
||||
f" (POST /api/certs/{name}/rotate mints one; GET {route} answered 404)")
|
||||
raise SystemExit(1)
|
||||
print(f"--fetch-bundle: GET {route} answered {status}: {detail()}")
|
||||
raise SystemExit(1)
|
||||
PY
|
||||
then
|
||||
exit 2
|
||||
fi
|
||||
# A fetch credential only: from here the install needs the bytes, not the session. The env
|
||||
# assignment above was per-command, so the only copy left is the shell variable.
|
||||
unset SESSION_TOKEN
|
||||
BUNDLE_SRC="$FETCH_TMP"
|
||||
BUNDLE_LABEL="--fetch-bundle"
|
||||
echo " fetched the certificate bundle for $CERT_NAME from $FETCH_URL"
|
||||
}
|
||||
|
||||
# NOT named `install`: bash resolves a function before an external command of the same name,
|
||||
# so an `install()` here turns the `install -d` / `install -m` calls below into infinite
|
||||
# recursion instead of creating $AGENT_DIR. Phase 10 task 02 found exactly that on a live
|
||||
# host: the run looped on the SMART block forever and installed nothing.
|
||||
do_install() {
|
||||
# Registration fetches its own bundle, before the root boundary for the same reason the shape
|
||||
# check is there: a dashboard that refuses is answered with its own sentence, and a run that
|
||||
# cannot get a bundle writes nothing at all. On success this is just another $BUNDLE_SRC.
|
||||
if [ -n "$FETCH_URL" ]; then
|
||||
fetch_bundle
|
||||
fi
|
||||
# Validate before need_root: a bad bundle is refused with its own reason whether or not
|
||||
# this shell is root, so a wrong file says "wrong file" and not "add a sudoers line".
|
||||
if [ -n "$BUNDLE_SRC" ]; then
|
||||
validate_bundle "$BUNDLE_SRC"
|
||||
validate_bundle "$BUNDLE_SRC" "$BUNDLE_LABEL"
|
||||
fi
|
||||
need_root --bind "$BIND" --port "$PORT"
|
||||
[ -f "$AGENT_SOURCE" ] || { echo "no agent next to this script: $AGENT_SOURCE" >&2; exit 1; }
|
||||
@@ -534,9 +721,11 @@ none:
|
||||
|
||||
HOST_TARGETS="hostname=https://$BIND:$PORT"
|
||||
|
||||
After a rotate or a root rotation on the dashboard, re-download the bundle and re-run this
|
||||
script with --cert-bundle; the agent picks a changed denylist up without a restart, but a
|
||||
new certificate needs one.
|
||||
After a rotate or a root rotation on the dashboard, fetch the new bundle and re-run this
|
||||
script: --fetch-bundle <url> --cert-name <name> --session <token> does the download and the
|
||||
install in one go, and --cert-bundle <file> still works from a file already on this box. The
|
||||
agent picks a changed denylist up without a restart, but a new certificate needs one, and the
|
||||
run above only *starts* the unit — restart the service to put the new bundle into effect.
|
||||
NEXT
|
||||
else
|
||||
cat <<NEXT
|
||||
|
||||
Reference in New Issue
Block a user