phase: 15_agent_mtls
All green. Report:
**Verified / implemented (final pass of `15_agent_mtls`)**
- Wrote `data/logs/verify-15-tls-e2e.py`: a real `host_agent.py --cert-bundle` + a real dashboard process over a scratch DB (`HOST_TARGETS=loop=https://…`) — **27/27 PASS**, transcript `verify-15-tls-e2e.out`, notes `data/logs/NOTES-tls-rollout.md`.
- Read-only checks against the running dev store: 5 `certs.*` rows byte-identical across a restart; served bundle == the `meta` row (1430 B); 0 events containing `-----BEGIN`; unsigned `/certs`→`303 ?next=`, unsigned `/api/certs`→`401`, signed-in `200`; screenshots `data/logs/15-final-{certs,system}.png`.
- Docs (step 6, full review): README — added *Protecting an agent, host by host* (the rollout task-07 left out), a `tests/test_certs.py`/TLS-matrix/sandbox-installer paragraph, and fixed two stale “the one write endpoint in the app” claims (`NUT_ALLOW_ACTIONS`, Battery-test buttons). AGENTS.md — test count 1424→1427, bundle-fetch command in *Commands*, a Testing-policy bullet on structurally pinning a key-holding module, fleet transport bullet (token mode is now the compatibility path), gap 11 final state.
**Tests / lint (full suite 100% green)**
- `uv run pytest` → **1427 passed, 15 skipped**, exit 0 · `bash .agents/validate.sh` → **exit 0** (coverage **98.94%** vs 94% floor, `certs.py` 100%; ruff “All checks passed”; pyright 0 errors; `node --check` clean; hygiene clean)
- `uv run pytest -m live --live` → **11 passed / 0 failed** · `./scripts/dev-server.sh status` → all 9 sources `ok`
- `uv pip list --outdated` → empty after bumping `fastapi 0.152.0 → 0.153.0` (`uv.lock` only); full suite re-run green.
**Completion criteria**
- Fresh-DB root CA, idempotent: ✅ (tests + live store hashes) · agent TLS matrix incl. wrong-CA/certless 403/`/health` CA-only/revoked 403 + live denylist re-read: ✅
- `https` `host:`/`disks:` `ok` + revoked says “agent certificate revoked” with no dial: ✅ on loopback (real agent + real dashboard) and spy-pinned hermetically; **not** on the fleet (below)
- Signed-in `/certs` + rotate/revoke/root-rotate + byte-identical bundle download: ✅ · installer bundle/unit/`--status` over TLS: ✅ (0640, see deviation)
- `pytest`/`validate.sh`/`--outdated`: ✅ · token-mode byte-identical, engine/UPS untouched, public surface unchanged (route walk): ✅
- **Live fleet TLS (deskwork+gateway flip): NOT done** — no live changes. It needs root on hosts without the installer grant (`deskwork`), has no both-ways-working window (`http://` dashboard vs https-only agent), and would deploy the agent from this not-yet-committed tree. Exact procedure recorded in README + `data/logs/NOTES-tls-rollout.md`.
**Deviations**: bundle mode `0640 root:dashboards-agent` instead of the task file’s `0600` (the agent re-reads its own bundle for the denylist; 0600 breaks the unit — already recorded in AGENTS.md); the loopback dashboard substitutes for the fleet flip. Incidental: a stray `pkill` of mine killed the dev server mid-pass; restarted and verified healthy.
**Next pending phase**: `.agents/phases/todo/16_agent_smart_pass_budget` (then `17_installer_device_filter`).
This commit is contained in:
+171
-5
@@ -3,6 +3,7 @@
|
||||
#
|
||||
# sudo ./deploy/install-agent.sh # loopback only (read it locally)
|
||||
# sudo ./deploy/install-agent.sh --bind 10.0.0.5 # serve the LAN, token required
|
||||
# sudo ./deploy/install-agent.sh --cert-bundle agent.bundle # serve TLS (phase 15)
|
||||
# ./deploy/install-agent.sh --status # unit state + /health, no root
|
||||
# sudo ./deploy/install-agent.sh --uninstall
|
||||
#
|
||||
@@ -16,6 +17,21 @@
|
||||
# names. Re-running the installer reuses the existing token — use --rotate-token when the old
|
||||
# one leaked.
|
||||
#
|
||||
# The certificate bundle (--cert-bundle <file>, fetched by the operator from the dashboard's
|
||||
# GET /api/certs/<name>/bundle or the /certs page) is the TLS install: one PEM holding this
|
||||
# host's server certificate and key, the dashboard's root CA, a `# BUNDLE-REV <n>` line and
|
||||
# the revocation denylist. This script never dials the dashboard for it. The bundle is
|
||||
# validated before anything is touched — a bundle the agent could not load must stop the
|
||||
# install with its reason, not leave a unit that exits at startup — and then installed to
|
||||
# /etc/dashboards-agent.certs, the token-file idiom: the agent opens the bundle *itself* at
|
||||
# startup and re-reads it when its stamp changes (that is how a re-deployed revocation list
|
||||
# lands without a restart), so the mode must let the agent's own service user read it and no
|
||||
# other account. The unit then runs https with --cert-bundle and a RuntimeDirectory for the
|
||||
# PEM blocks the agent unpacks (it deletes the key once the SSL context holds it), and
|
||||
# --status probes /health over TLS with the bundle's CA and no client certificate — the path
|
||||
# the agent exempts for probes. Re-running without the flag leaves an installed bundle in
|
||||
# place, the token-reuse idiom.
|
||||
#
|
||||
# The agent is read-only by construction (it answers GET and writes nothing), runs as its
|
||||
# own system user, and gets the usual systemd lockdown. It still needs to read /proc, /sys
|
||||
# and statvfs(), which the hardening below leaves alone.
|
||||
@@ -34,6 +50,7 @@ AGENT_USER="dashboards-agent"
|
||||
AGENT_DIR="/usr/local/lib/dashboards-agent"
|
||||
AGENT_TARGET="$AGENT_DIR/host_agent.py"
|
||||
TOKEN_FILE="/etc/dashboards-agent.token"
|
||||
CERT_FILE="/etc/dashboards-agent.certs"
|
||||
UNIT="/etc/systemd/system/$SERVICE.service"
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
AGENT_SOURCE="$SCRIPT_DIR/../agent/host_agent.py"
|
||||
@@ -51,6 +68,7 @@ BIND="127.0.0.1"
|
||||
PORT="9120"
|
||||
TOKEN="${AGENT_TOKEN:-}"
|
||||
ROTATE=0
|
||||
BUNDLE_SRC=""
|
||||
MODE="install"
|
||||
|
||||
usage() {
|
||||
@@ -65,6 +83,7 @@ while [ $# -gt 0 ]; do
|
||||
--port) PORT="$2"; shift 2 ;;
|
||||
--token) TOKEN="$2"; shift 2 ;;
|
||||
--rotate-token) ROTATE=1; shift ;;
|
||||
--cert-bundle) BUNDLE_SRC="$2"; shift 2 ;;
|
||||
--status) MODE="status"; shift ;;
|
||||
--uninstall) MODE="uninstall"; shift ;;
|
||||
-h|--help) usage 0 ;;
|
||||
@@ -81,7 +100,7 @@ need_root() {
|
||||
cat >&2 <<HINT
|
||||
$SCRIPT_PATH has to run as root: it writes $UNIT,
|
||||
creates the $AGENT_USER user and its disk-group membership, and stores
|
||||
the token in $TOKEN_FILE.
|
||||
the token in $TOKEN_FILE (and, with --cert-bundle, the certificate bundle in $CERT_FILE).
|
||||
|
||||
Add this single sudoers line on this host (it is the whole grant; see $SUDOERS_SOURCE):
|
||||
|
||||
@@ -105,7 +124,47 @@ HINT
|
||||
status() {
|
||||
systemctl status "$SERVICE" --no-pager 2>&1 || true
|
||||
echo
|
||||
if command -v curl >/dev/null 2>&1; then
|
||||
if [ -r "$UNIT" ] && grep -q -- '--cert-bundle' "$UNIT"; then
|
||||
# TLS install (phase 15): probe over https with the bundle's root CA and *no* client
|
||||
# certificate. The agent's CERT_OPTIONAL handshake plus the /health exemption is what
|
||||
# makes this possible — the installer holds no client identity, and a service that
|
||||
# cannot health-check is a dead install. The CA is the bundle's last certificate block
|
||||
# (the format is positional: agent cert, agent key, root CA). Chain-to-our-root is the
|
||||
# claim; the leaf names the dial addresses rather than loopback, so the name check is
|
||||
# off — forging the chain would still need the root key. python3 is the unit's own
|
||||
# interpreter: there is no external TLS binary to install on a monitored host.
|
||||
printf 'health: '
|
||||
python3 - "$BIND" "$PORT" "$CERT_FILE" <<'PY' || echo "(no answer)"
|
||||
import re
|
||||
import ssl
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
host, port, bundle = sys.argv[1], sys.argv[2], sys.argv[3]
|
||||
try:
|
||||
with open(bundle, encoding="utf-8", errors="replace") as handle:
|
||||
text = handle.read()
|
||||
except OSError as exc:
|
||||
print(f"(cannot read {bundle}: {exc.strerror or exc} — run --status as root)")
|
||||
raise SystemExit(0)
|
||||
blocks = re.findall(r"-----BEGIN CERTIFICATE-----.*?-----END CERTIFICATE-----", text, re.DOTALL)
|
||||
if len(blocks) < 2:
|
||||
print(f"(no root CA block in {bundle})")
|
||||
raise SystemExit(0)
|
||||
context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
|
||||
context.check_hostname = False
|
||||
context.verify_mode = ssl.CERT_REQUIRED
|
||||
context.load_verify_locations(cadata=blocks[-1])
|
||||
try:
|
||||
with urllib.request.urlopen(
|
||||
f"https://{host}:{port}/health", timeout=4, context=context
|
||||
) as response:
|
||||
print(response.read().decode("utf-8", "replace").strip())
|
||||
except Exception as exc: # the probe reports, it never propagates
|
||||
print(f"(no answer: {exc})")
|
||||
PY
|
||||
elif command -v curl >/dev/null 2>&1; then
|
||||
# token mode: the plain probe, unchanged
|
||||
printf 'health: '
|
||||
curl -s --max-time 4 "http://$BIND:$PORT/health" || echo "(no answer)"
|
||||
echo
|
||||
@@ -115,7 +174,7 @@ status() {
|
||||
uninstall() {
|
||||
need_root --uninstall
|
||||
systemctl disable --now "$SERVICE" 2>/dev/null || true
|
||||
rm -f "$UNIT" "$TOKEN_FILE" "$AGENT_TARGET"
|
||||
rm -f "$UNIT" "$TOKEN_FILE" "$CERT_FILE" "$AGENT_TARGET"
|
||||
rmdir "$AGENT_DIR" 2>/dev/null || true
|
||||
systemctl daemon-reload
|
||||
echo "removed $SERVICE (the $AGENT_USER user was left in place; remove it with userdel)"
|
||||
@@ -127,11 +186,72 @@ new_token() {
|
||||
python3 -c 'import secrets; print(secrets.token_urlsafe(32).rstrip("="))'
|
||||
}
|
||||
|
||||
validate_bundle() {
|
||||
# The `resolve_bundle` idiom from agent/host_agent.py, run before anything is touched:
|
||||
# a bundle the agent could not load must stop the install with its reason, not leave a
|
||||
# unit that exits at startup. Shape only, by text — the agent parses the same way (it is
|
||||
# stdlib; a monitored host has no cryptography) and the dashboard is the only signer, so
|
||||
# a second copy of this shape check would only ever disagree about nothing. The three PEM
|
||||
# blocks are positional: agent certificate, agent key, root CA certificate.
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
echo "--cert-bundle needs python3 (3.10 or newer) to be validated" >&2
|
||||
exit 2
|
||||
fi
|
||||
if python3 - "$1" <<'PY' 1>&2
|
||||
import re
|
||||
import sys
|
||||
|
||||
path = sys.argv[1]
|
||||
try:
|
||||
with open(path, encoding="utf-8", errors="replace") as handle:
|
||||
text = handle.read()
|
||||
except OSError as exc:
|
||||
print(f"--cert-bundle {path} cannot be read: {exc.strerror or exc}")
|
||||
raise SystemExit(1)
|
||||
if not text.strip():
|
||||
print(f"--cert-bundle {path} is empty")
|
||||
raise SystemExit(1)
|
||||
revision = None
|
||||
for line in text.splitlines():
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("# BUNDLE-REV "):
|
||||
fields = stripped[len("# BUNDLE-REV "):].split()
|
||||
if fields:
|
||||
try:
|
||||
revision = int(fields[0])
|
||||
except ValueError:
|
||||
revision = None
|
||||
break
|
||||
if revision is None:
|
||||
print(f"--cert-bundle {path} has no `# BUNDLE-REV <n>` line — that line is how the"
|
||||
" dashboard tells the bundle it issued from a stale one, so a bundle without it"
|
||||
" is not one this app signed")
|
||||
raise SystemExit(1)
|
||||
blocks = re.findall(r"-----BEGIN [A-Z0-9 ]+-----.*?-----END [A-Z0-9 ]+-----", text, re.DOTALL)
|
||||
certificates = [b for b in blocks if b.startswith("-----BEGIN CERTIFICATE-----")]
|
||||
keys = [b for b in blocks if not b.startswith("-----BEGIN CERTIFICATE-----")]
|
||||
if len(certificates) != 2 or len(keys) != 1:
|
||||
print(f"--cert-bundle {path} must hold exactly the agent certificate, the agent key and"
|
||||
f" the root CA certificate — found {len(certificates)} certificates and"
|
||||
f" {len(keys)} keys. Re-download it from GET /api/certs/<name>/bundle")
|
||||
raise SystemExit(1)
|
||||
PY
|
||||
then
|
||||
return 0
|
||||
fi
|
||||
exit 2
|
||||
}
|
||||
|
||||
# NOT named `install`: bash resolves a function before an external command of the same name,
|
||||
# so an `install()` here turns the `install -d` / `install -m` calls below into infinite
|
||||
# recursion instead of creating $AGENT_DIR. Phase 10 task 02 found exactly that on a live
|
||||
# host: the run looped on the SMART block forever and installed nothing.
|
||||
do_install() {
|
||||
# Validate before need_root: a bad bundle is refused with its own reason whether or not
|
||||
# this shell is root, so a wrong file says "wrong file" and not "add a sudoers line".
|
||||
if [ -n "$BUNDLE_SRC" ]; then
|
||||
validate_bundle "$BUNDLE_SRC"
|
||||
fi
|
||||
need_root --bind "$BIND" --port "$PORT"
|
||||
[ -f "$AGENT_SOURCE" ] || { echo "no agent next to this script: $AGENT_SOURCE" >&2; exit 1; }
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
@@ -177,6 +297,36 @@ do_install() {
|
||||
chmod 0640 "$TOKEN_FILE"
|
||||
chown root:"$AGENT_USER" "$TOKEN_FILE"
|
||||
|
||||
# The certificate bundle is the token's twin and the mode is the whole design. It holds
|
||||
# the agent's *private key*, so every other account — ops included — must not read it;
|
||||
# but the agent process (running as $AGENT_USER) opens the bundle itself at startup and
|
||||
# re-reads it when its stamp changes (that live re-read is how a re-deployed denylist
|
||||
# lands without a restart), so a 0600 root-owned file would fail the unit exactly the way
|
||||
# it would fail the token file. 0640 root:$AGENT_USER is the smallest grant that works.
|
||||
# An existing bundle survives a re-run that was not given the flag — the token-reuse
|
||||
# idiom — and this script never dials the dashboard to fetch one.
|
||||
if [ -n "$BUNDLE_SRC" ]; then
|
||||
install -m 0640 "$BUNDLE_SRC" "$CERT_FILE"
|
||||
chown root:"$AGENT_USER" "$CERT_FILE"
|
||||
echo " installed the certificate bundle in $CERT_FILE"
|
||||
elif [ -f "$CERT_FILE" ]; then
|
||||
echo "reusing the installed certificate bundle in $CERT_FILE (--cert-bundle <file> for a new one)"
|
||||
fi
|
||||
|
||||
# TLS mode: the unit runs https with the installed bundle. The --token-file line stays —
|
||||
# the agent skips the token check on the TLS path — so a rollout happens host by host
|
||||
# instead of on a flag day.
|
||||
TLS_EXEC_ARGS=""
|
||||
TLS_UNIT_LINES=""
|
||||
if [ -f "$CERT_FILE" ]; then
|
||||
TLS_EXEC_ARGS=" --cert-bundle $CERT_FILE"
|
||||
TLS_UNIT_LINES="# The TLS bundle unpacks its PEM blocks here: tmpfs, private to the unit,
|
||||
# gone when it stops. ProtectSystem=strict leaves the agent nothing else writable, and the
|
||||
# agent deletes the key the moment the SSL context holds it.
|
||||
RuntimeDirectory=dashboards-agent
|
||||
RuntimeDirectoryMode=0700"
|
||||
fi
|
||||
|
||||
cat > "$UNIT" <<UNIT_FILE
|
||||
[Unit]
|
||||
Description=Homelab host agent (read-only system metrics)
|
||||
@@ -186,11 +336,12 @@ Wants=network-online.target
|
||||
[Service]
|
||||
Type=simple
|
||||
User=$AGENT_USER
|
||||
ExecStart=/usr/bin/env python3 $AGENT_TARGET --host $BIND --port $PORT --token-file $TOKEN_FILE
|
||||
ExecStart=/usr/bin/env python3 $AGENT_TARGET --host $BIND --port $PORT --token-file ${TOKEN_FILE}${TLS_EXEC_ARGS}
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
Environment=PYTHONUNBUFFERED=1
|
||||
SyslogIdentifier=$SERVICE
|
||||
$TLS_UNIT_LINES
|
||||
# The agent reads /proc, /sys and statvfs() and writes nothing, so everything writable can
|
||||
# be taken away. smartctl is the one thing that needs a real /dev node, so instead of
|
||||
# PrivateDevices the unit closes all devices and reopens only the disk nodes, read/write
|
||||
@@ -242,7 +393,21 @@ UNIT_FILE
|
||||
journalctl -u "$SERVICE" -n 20 --no-pager >&2
|
||||
exit 1
|
||||
fi
|
||||
cat <<NEXT
|
||||
if [ -n "$TLS_EXEC_ARGS" ]; then
|
||||
cat <<NEXT
|
||||
|
||||
The unit now serves https with the certificate bundle in $CERT_FILE. Point the dashboard at
|
||||
it over TLS — the bearer token plays no part on the certificate path, so the URL carries
|
||||
none:
|
||||
|
||||
HOST_TARGETS="hostname=https://$BIND:$PORT"
|
||||
|
||||
After a rotate or a root rotation on the dashboard, re-download the bundle and re-run this
|
||||
script with --cert-bundle; the agent picks a changed denylist up without a restart, but a
|
||||
new certificate needs one.
|
||||
NEXT
|
||||
else
|
||||
cat <<NEXT
|
||||
|
||||
Point the dashboard at it (the token is the URL userinfo, so it never needs a variable of
|
||||
its own, and the API strips it before anything reaches a browser):
|
||||
@@ -254,6 +419,7 @@ Paste the token now if this is the first install — it is not printed again wit
|
||||
|
||||
$TOKEN
|
||||
NEXT
|
||||
fi
|
||||
}
|
||||
|
||||
case "$MODE" in
|
||||
|
||||
Reference in New Issue
Block a user