phase: 15_agent_mtls

All green. Report:

**Verified / implemented (final pass of `15_agent_mtls`)**
- Wrote `data/logs/verify-15-tls-e2e.py`: a real `host_agent.py --cert-bundle` + a real dashboard process over a scratch DB (`HOST_TARGETS=loop=https://…`) — **27/27 PASS**, transcript `verify-15-tls-e2e.out`, notes `data/logs/NOTES-tls-rollout.md`.
- Read-only checks against the running dev store: 5 `certs.*` rows byte-identical across a restart; served bundle == the `meta` row (1430 B); 0 events containing `-----BEGIN`; unsigned `/certs`→`303 ?next=`, unsigned `/api/certs`→`401`, signed-in `200`; screenshots `data/logs/15-final-{certs,system}.png`.
- Docs (step 6, full review): README — added *Protecting an agent, host by host* (the rollout task-07 left out), a `tests/test_certs.py`/TLS-matrix/sandbox-installer paragraph, and fixed two stale “the one write endpoint in the app” claims (`NUT_ALLOW_ACTIONS`, Battery-test buttons). AGENTS.md — test count 1424→1427, bundle-fetch command in *Commands*, a Testing-policy bullet on structurally pinning a key-holding module, fleet transport bullet (token mode is now the compatibility path), gap 11 final state.

**Tests / lint (full suite 100% green)**
- `uv run pytest` → **1427 passed, 15 skipped**, exit 0 · `bash .agents/validate.sh` → **exit 0** (coverage **98.94%** vs 94% floor, `certs.py` 100%; ruff “All checks passed”; pyright 0 errors; `node --check` clean; hygiene clean)
- `uv run pytest -m live --live` → **11 passed / 0 failed** · `./scripts/dev-server.sh status` → all 9 sources `ok`
- `uv pip list --outdated` → empty after bumping `fastapi 0.152.0 → 0.153.0` (`uv.lock` only); full suite re-run green.

**Completion criteria**
- Fresh-DB root CA, idempotent: ✅ (tests + live store hashes) · agent TLS matrix incl. wrong-CA/certless 403/`/health` CA-only/revoked 403 + live denylist re-read: ✅
- `https` `host:`/`disks:` `ok` + revoked says “agent certificate revoked” with no dial: ✅ on loopback (real agent + real dashboard) and spy-pinned hermetically; **not** on the fleet (below)
- Signed-in `/certs` + rotate/revoke/root-rotate + byte-identical bundle download: ✅ · installer bundle/unit/`--status` over TLS: ✅ (0640, see deviation)
- `pytest`/`validate.sh`/`--outdated`: ✅ · token-mode byte-identical, engine/UPS untouched, public surface unchanged (route walk): ✅
- **Live fleet TLS (deskwork+gateway flip): NOT done** — no live changes. It needs root on hosts without the installer grant (`deskwork`), has no both-ways-working window (`http://` dashboard vs https-only agent), and would deploy the agent from this not-yet-committed tree. Exact procedure recorded in README + `data/logs/NOTES-tls-rollout.md`.

**Deviations**: bundle mode `0640 root:dashboards-agent` instead of the task file’s `0600` (the agent re-reads its own bundle for the denylist; 0600 breaks the unit — already recorded in AGENTS.md); the loopback dashboard substitutes for the fleet flip. Incidental: a stray `pkill` of mine killed the dev server mid-pass; restarted and verified healthy.

**Next pending phase**: `.agents/phases/todo/16_agent_smart_pass_budget` (then `17_installer_device_filter`).
This commit is contained in:
2026-10-20 10:02:16 -04:00
parent 569ca25fdb
commit cffbac5771
72 changed files with 7168 additions and 216 deletions
+350 -11
View File
@@ -42,6 +42,8 @@ SCHEMA (GET /snapshot)
ts float unix seconds (agent clock)
duration_ms float collection cost
agent_version str
tls true only when serving with --cert-bundle
bundle_rev int the `# BUNDLE-REV` of the bundle it loaded
host {hostname, os_pretty, kernel, arch, python, boot_ts, uptime_seconds,
cpu_model, cpu_cores_logical, cpu_cores_physical, container}
cpu {modes: {user,nice,system,idle,iowait,irq,softirq,steal,guest,
@@ -81,17 +83,26 @@ SCHEMA (GET /snapshot)
errors [str] reads that failed; non-empty means the snapshot is partial
Deployment: see deploy/install-agent.sh (systemd unit, binds 127.0.0.1 unless you pass
--host, requires a bearer token via --token or AGENT_TOKEN when bound off-loopback).
--host, requires a bearer token via --token or AGENT_TOKEN when bound off-loopback). The
other protection is `--cert-bundle`/AGENT_CERT_BUNDLE: one installed PEM (server cert, server
key, the dashboard's root CA, `# BUNDLE-REV <n>` and `# REVOKED <serial>` lines) turns the
server into HTTPS that answers every path but `/health` only to a client certificate signed
by that root, and refuses one on the denylist on every path — the certificate is the auth, so
no token is read on that path.
"""
from __future__ import annotations
import argparse
import contextlib
import json
import os
import re
import shutil
import socket
import ssl
import subprocess
import tempfile
import threading
import time
from collections.abc import Callable
@@ -99,7 +110,7 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from typing import Any
AGENT_VERSION = "1.0.0"
AGENT_VERSION = "1.1.0"
SCHEMA_VERSION = 1
PAGE_SIZE = 0 # resolved from sysconf at runtime
@@ -1136,6 +1147,267 @@ def _split_stat(stat: str) -> list[str] | None:
return [match.group("comm"), *match.group("rest").split()]
# --------------------------------------------------------------------------- #
# TLS bundle: what the dashboard signs, what the installer places
# --------------------------------------------------------------------------- #
#: The two metadata lines inside a bundle, spelled exactly as `src/dashboards/certs.py` writes
#: them. The agent stays stdlib (no `cryptography` on a monitored host), so the format is agreed
#: here by text rather than imported, and `tests/test_agent.py` parses a bundle that `certs.py`
#: actually produced — the only way the two halves can stay honest.
BUNDLE_REV_LINE = "# BUNDLE-REV "
REVOKED_LINE = "# REVOKED "
#: Where the bundle is unpacked when systemd gave the unit a `RuntimeDirectory` (task 06): tmpfs,
#: private to the service user, gone when the unit stops. Otherwise a `mkdtemp` directory that is
#: removed as soon as the SSL context holds the material.
BUNDLE_RUNTIME_DIR = Path("/run/dashboards-agent")
_PEM_BLOCK_RE = re.compile(r"-----BEGIN [A-Z0-9 ]+-----.*?-----END [A-Z0-9 ]+-----\n", re.DOTALL)
_CERT_HEADER = "-----BEGIN CERTIFICATE-----"
#: The unpacked file names, in the order `_bundle_blocks` hands the blocks back.
_BUNDLE_FILES = ("agent-cert.pem", "agent-key.pem", "root-ca.pem")
def _line_value(line: str, prefix: str) -> str:
"""The first token after a bundle line's prefix, or "" when the line carries nothing."""
rest = line[len(prefix):].strip()
fields = rest.split()
return fields[0] if fields else ""
def bundle_revision(text: str) -> int | None:
"""The `# BUNDLE-REV <n>` number, or None when the line is absent or is not a number.
Absent is an error rather than a default of 0: the revision is how the dashboard tells a stale
bundle from the one it just issued, so an agent serving "rev 0" would be reporting a number it
never had.
"""
for line in text.splitlines():
stripped = line.strip()
if not stripped.startswith(BUNDLE_REV_LINE):
continue
token = _line_value(stripped, BUNDLE_REV_LINE)
try:
return int(token)
except ValueError:
return None
return None
def bundle_revoked(text: str) -> frozenset[str]:
"""Every `# REVOKED <serial>` value in a bundle, kept exactly as written."""
serials: set[str] = set()
for line in text.splitlines():
stripped = line.strip()
if stripped.startswith(REVOKED_LINE):
serial = _line_value(stripped, REVOKED_LINE)
if serial:
serials.add(serial)
return frozenset(serials)
def _bundle_blocks(text: str) -> tuple[str, str, str]:
"""(agent certificate, agent key, root certificate) — positionally, because that is the format.
Two certificates and exactly one private key is the shape of a bundle. A bundle with the root
key in it is the failure this catches: an exporter needs to *verify* its clients, which is a
public-key operation, so a stolen agent install must never be a stolen CA.
"""
blocks = _PEM_BLOCK_RE.findall(text)
certs = [block for block in blocks if block.startswith(_CERT_HEADER)]
keys = [block for block in blocks if not block.startswith(_CERT_HEADER)]
if len(certs) != 2 or len(keys) != 1:
raise ValueError("expected the agent certificate, the agent key and the root certificate")
return certs[0], keys[0], certs[1]
def _file_stamp(path: str) -> tuple[int, int] | None:
"""(mtime_ns, size) for the denylist re-read, or None when the file cannot be stated.
Nanoseconds rather than the float `st_mtime`: two writes in the same millisecond are a real
redeploy, and a change this agent misses is a revoked certificate that keeps working.
"""
try:
info = os.stat(path)
except OSError:
return None
return (info.st_mtime_ns, info.st_size)
def _material_dir() -> Path:
"""The private directory to unpack into: the unit's RuntimeDirectory when usable, else tmp."""
if BUNDLE_RUNTIME_DIR.is_dir() and os.access(BUNDLE_RUNTIME_DIR, os.W_OK | os.X_OK):
return BUNDLE_RUNTIME_DIR
directory = Path(tempfile.mkdtemp(prefix="dashboards-agent-"))
os.chmod(directory, 0o700) # mkdtemp is 0700 already; the claim is this agent's, not the umask's
return directory
def _unlink(path: Path) -> None:
with contextlib.suppress(FileNotFoundError):
path.unlink()
def _write_private(path: Path, text: str) -> None:
"""Create `path` at 0600 in one `open()`, so a private key never exists at the umask default."""
_unlink(path) # O_EXCL would refuse a leftover from a killed run, and a symlink would be worse
handle = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
with os.fdopen(handle, "w", encoding="ascii") as stream:
stream.write(text)
def resolve_bundle(path: str) -> tuple[ssl.SSLContext | None, int, frozenset[str], str | None]:
"""(context, bundle_rev, revoked serials, error) for one installed bundle.
The `resolve_token` idiom: no globals, no server state, every failure an error string the
caller prints and exits on. A bundle this agent cannot load must stop the startup rather than
answer plain HTTP behind a token nobody configured — "degraded to no TLS" is the one answer
that is worse than a crash.
The three PEM blocks are grepped out by text and written into a private directory, because
`ssl` loads a chain from paths; they are unlinked as soon as the context has them, so the key
then lives in this process's memory and nowhere else (and there is no shutdown path that could
forget to remove it). The root CA goes in as the *verify* list, which is the whole of what an
exporter needs: it authenticates its clients, it is never authenticated by them.
"""
try:
text = Path(path).read_text(encoding="utf-8", errors="replace")
except OSError as exc:
return None, 0, frozenset(), f"cannot read --cert-bundle {path}: {exc}"
if not text.strip():
return None, 0, frozenset(), f"--cert-bundle {path} is empty"
revision = bundle_revision(text)
if revision is None:
return None, 0, frozenset(), (
f"--cert-bundle {path} has no readable `# BUNDLE-REV <n>` line — the revision is how "
"the dashboard tells this bundle from a stale one, so a bundle without it is not one "
"this app signed")
try:
server_cert, server_key, root_cert = _bundle_blocks(text)
except ValueError as exc:
return None, 0, frozenset(), f"--cert-bundle {path} is not a bundle: {exc}"
directory: Path | None = None
ephemeral = False
try:
directory = _material_dir()
ephemeral = directory != BUNDLE_RUNTIME_DIR
written = [directory / name for name in _BUNDLE_FILES]
for target, body in zip(written, (server_cert, server_key, root_cert), strict=True):
_write_private(target, body)
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
context.minimum_version = ssl.TLSVersion.TLSv1_2
context.load_cert_chain(str(written[0]), str(written[1]))
context.load_verify_locations(str(written[2]))
# Optional at the handshake, enforced by the handler: `CERT_REQUIRED` would make the
# installer's `--status` and systemd's probe fail, because neither holds a client
# certificate the installer was never given, and a service that cannot health-check is a
# dead install. See `Handler._certificate_allows` for who may read what.
context.verify_mode = ssl.CERT_OPTIONAL
except (OSError, ValueError, ssl.SSLError) as exc:
return None, 0, frozenset(), f"cannot load --cert-bundle {path}: {exc}"
finally:
if directory is not None:
for name in _BUNDLE_FILES:
_unlink(directory / name)
if ephemeral:
shutil.rmtree(directory, ignore_errors=True)
return context, revision, bundle_revoked(text), None
class TlsBundle:
"""One loaded bundle: the SSL context, the revision it carries, and the live denylist.
The context and the revision are fixed for the process's life — `ssl` cannot swap a chain
under a live socket, so a new certificate needs a restart (and says so on `/certs`). The
`# REVOKED` list is not fixed: re-reading one small text file when its stamp changes costs
nothing per poll and is the difference between "revoked" and "revoked on the next deploy",
which is the only reason the lines live in the bundle instead of in the dashboard alone.
"""
def __init__(self, path: str, context: ssl.SSLContext, bundle_rev: int,
revoked: frozenset[str]) -> None:
self.path = path
self.context = context
self.bundle_rev = bundle_rev
self._lock = threading.Lock() # ThreadingHTTPServer may re-read on several requests
self._revoked = revoked
self._stamp = _file_stamp(path)
@classmethod
def load(cls, path: str) -> tuple[TlsBundle | None, str | None]:
"""`resolve_bundle` plus the file stamp the denylist re-read compares against."""
context, revision, revoked, error = resolve_bundle(path)
if error is not None or context is None:
return None, error
return cls(path, context, revision, revoked), None
def revoked_serials(self) -> frozenset[str]:
"""The denylist as the file holds it right now, re-read only when the file changed."""
stamp = _file_stamp(self.path)
with self._lock:
if stamp is None or stamp == self._stamp:
return self._revoked
try:
text = Path(self.path).read_text(encoding="utf-8", errors="replace")
except OSError:
# Unreadable mid-redeploy: keep the list this agent already had. Forgetting a
# revocation because a file vanished for one second is the wrong direction.
return self._revoked
self._revoked = bundle_revoked(text)
self._stamp = stamp
return self._revoked
def _peer_certificate(connection: Any) -> dict[str, Any]:
"""The verified client certificate of this connection, or {} when there is none.
`getpeercert()` is only there on a TLS socket and only answers after a handshake, so a
plain-HTTP connection to a bundle-configured agent (a scanner, a port probe) reads as "no
certificate" and gets the same 403 as a certless TLS client rather than a traceback.
"""
getter = getattr(connection, "getpeercert", None)
if getter is None:
return {}
try:
peer = getter()
except (OSError, ValueError): # ValueError is `ssl` for "the handshake has not run"
return {}
return peer or {}
def _serial_forms(raw: str) -> frozenset[str]:
"""Every spelling one certificate serial can be written in.
`getpeercert()` renders a serial as **hexadecimal** text while `certs.py` writes the **decimal**
number into the bundle's `# REVOKED` line, and an agent that compared one spelling to the other
would accept a revoked client forever — a revocation that silently does nothing is worse than
no revocation list, because the operator believes it happened. Accepting both spellings on both
sides can only ever deny a certificate that is not revoked (two different serials whose hex and
decimal renderings coincide), never allow one that is.
"""
token = (raw or "").strip()
if not token:
return frozenset()
forms = {token}
for base in (10, 16):
try:
value = int(token, base)
except ValueError:
continue
forms.add(str(value))
forms.add(format(value, "X"))
return frozenset(forms)
def serial_is_revoked(cert: dict[str, Any], revoked: frozenset[str]) -> bool:
"""Is this peer certificate on the bundle's denylist?"""
peer = _serial_forms(str(cert.get("serialNumber") or ""))
if not peer:
return False
return any(_serial_forms(serial) & peer for serial in revoked)
# --------------------------------------------------------------------------- #
# HTTP server
# --------------------------------------------------------------------------- #
@@ -1165,10 +1437,18 @@ class Handler(BaseHTTPRequestHandler):
server_version = f"host_agent/{AGENT_VERSION}"
collector: Collector
token: str | None = None
bundle: TlsBundle | None = None
def do_GET(self) -> None: # the BaseHTTPRequestHandler API name — keep it
path = self.path.split("?", 1)[0].rstrip("/") or "/"
if self.token and path != "/health":
bundle = self.bundle
if bundle is not None:
# The handshake is the auth on this path, so the bearer check below never runs: a
# bundle *and* a token would let a stolen token read a host whose certificate the
# operator just revoked, which is the one thing the denylist exists to prevent.
if not self._certificate_allows(bundle, path):
return
elif self.token and path != "/health":
given = self.headers.get("Authorization", "")
if given != f"Bearer {self.token}":
self._send(401, {"ok": False, "detail": "missing or bad bearer token"})
@@ -1192,10 +1472,42 @@ class Handler(BaseHTTPRequestHandler):
top = max(0, int(part[4:]))
except ValueError:
top = None
self._send(200, {"ok": True, **self.collector.snapshot(top=top)})
body: dict[str, Any] = {"ok": True, **self.collector.snapshot(top=top)}
if bundle is not None:
# Additive, and only in TLS mode: the dashboard (task 03) reads the revision to
# tell a stale installed bundle from the one `/certs` just issued, and an absent
# field means "this agent predates certificates" rather than "revision 0".
body["tls"] = True
body["bundle_rev"] = bundle.bundle_rev
self._send(200, body)
else:
self._send(404, {"ok": False, "detail": f"no such path: {path}"})
def _certificate_allows(self, bundle: TlsBundle, path: str) -> bool:
"""Does this connection's client certificate authorise `path`? Sends the refusal if not.
Two answers, because they are two different operator actions: 403 "required" is a client
that never proved who it is (nothing installed, or the wrong CA), 403 "revoked" is a client
that proved it and was withdrawn.
The order is the design. A revoked identity is refused on **every** path, `/health` first:
the exemption is for the probe that holds the root CA and no client certificate at all, not
for a certificate the operator has just taken away. Then `/health` passes a certless probe,
because `install-agent.sh --status` and systemd's probe were never handed a client
certificate and a service that cannot health-check is a dead install.
"""
peer = _peer_certificate(self.connection)
present = bool(peer and peer.get("subject"))
if present and serial_is_revoked(peer, bundle.revoked_serials()):
self._send(403, {"ok": False, "detail": "client certificate revoked"})
return False
if path == "/health":
return True
if not present:
self._send(403, {"ok": False, "detail": "client certificate required"})
return False
return True
def _status_page(self) -> None:
info = self.collector.host_info()
uptime = info.get("uptime_seconds")
@@ -1232,12 +1544,26 @@ class Handler(BaseHTTPRequestHandler):
print(f"[agent] {self.address_string()} {fmt}", flush=True)
def serve(host: str, port: int, collector: Collector, token: str | None) -> None:
def serve(host: str, port: int, collector: Collector, token: str | None,
bundle: TlsBundle | None = None) -> None:
"""Serve forever on host:port — plain HTTP with an optional token, or HTTPS with a bundle.
All three class attributes are assigned on every call: `Handler` is process-wide state, so a
server started without a bundle must not inherit one from a server that had it (that would
answer a plain-HTTP socket with 403 on every path, or worse, accept a client certificate the
operator revoked).
"""
Handler.collector = collector
Handler.token = token
Handler.bundle = bundle
httpd = ThreadingHTTPServer((host, port), Handler)
print(f"[agent] listening on http://{host}:{port} "
f"({'token required' if token else 'no token'}); pid {os.getpid()}", flush=True)
if bundle is not None:
httpd.socket = bundle.context.wrap_socket(httpd.socket, server_side=True)
print(f"[agent] listening on https://{host}:{port} with tls "
f"(client cert required, rev {bundle.bundle_rev}); pid {os.getpid()}", flush=True)
else:
print(f"[agent] listening on http://{host}:{port} "
f"({'token required' if token else 'no token'}); pid {os.getpid()}", flush=True)
try:
httpd.serve_forever()
except KeyboardInterrupt:
@@ -1271,6 +1597,12 @@ def main(argv: list[str] | None = None) -> int:
parser.add_argument("--token-file", default=os.environ.get("AGENT_TOKEN_FILE", ""),
help="read the bearer token from a file (keeps it out of ps); wins "
"over --token")
parser.add_argument("--cert-bundle", default=os.environ.get("AGENT_CERT_BUNDLE", ""),
help="one installed PEM from the dashboard: this agent's server "
"certificate and key, the dashboard's root CA, a `# BUNDLE-REV` "
"line and the `# REVOKED` denylist. Serves https and answers "
"every path but /health only to a client certificate signed by "
"that root and not on the list; on that path the token is not read")
parser.add_argument("--root", default="/", help="filesystem root to read (for tests)")
parser.add_argument("--top", type=int, default=5, help="processes per top-N list")
parser.add_argument("--net-all", action="store_true", help="include veth/bridge interfaces")
@@ -1292,11 +1624,18 @@ def main(argv: list[str] | None = None) -> int:
if error:
print(f"[agent] {error}", flush=True)
return 2
if args.host not in ("127.0.0.1", "localhost", "::1") and not token:
print("[agent] refusing to bind off-loopback without --token (host metrics are "
"sensitive); pass --token, --token-file or set AGENT_TOKEN", flush=True)
bundle: TlsBundle | None = None
if args.cert_bundle:
bundle, bundle_error = TlsBundle.load(args.cert_bundle)
if bundle_error is not None:
print(f"[agent] {bundle_error}", flush=True)
return 2
if args.host not in ("127.0.0.1", "localhost", "::1") and not token and bundle is None:
print("[agent] refusing to bind off-loopback without --token or --cert-bundle (host "
"metrics are sensitive); pass --token, --token-file, --cert-bundle or set "
"AGENT_TOKEN / AGENT_CERT_BUNDLE", flush=True)
return 2
serve(args.host, args.port, collector, token)
serve(args.host, args.port, collector, token, bundle)
return 0