phase: 15_agent_mtls
All green. Report:
**Verified / implemented (final pass of `15_agent_mtls`)**
- Wrote `data/logs/verify-15-tls-e2e.py`: a real `host_agent.py --cert-bundle` + a real dashboard process over a scratch DB (`HOST_TARGETS=loop=https://…`) — **27/27 PASS**, transcript `verify-15-tls-e2e.out`, notes `data/logs/NOTES-tls-rollout.md`.
- Read-only checks against the running dev store: 5 `certs.*` rows byte-identical across a restart; served bundle == the `meta` row (1430 B); 0 events containing `-----BEGIN`; unsigned `/certs`→`303 ?next=`, unsigned `/api/certs`→`401`, signed-in `200`; screenshots `data/logs/15-final-{certs,system}.png`.
- Docs (step 6, full review): README — added *Protecting an agent, host by host* (the rollout task-07 left out), a `tests/test_certs.py`/TLS-matrix/sandbox-installer paragraph, and fixed two stale “the one write endpoint in the app” claims (`NUT_ALLOW_ACTIONS`, Battery-test buttons). AGENTS.md — test count 1424→1427, bundle-fetch command in *Commands*, a Testing-policy bullet on structurally pinning a key-holding module, fleet transport bullet (token mode is now the compatibility path), gap 11 final state.
**Tests / lint (full suite 100% green)**
- `uv run pytest` → **1427 passed, 15 skipped**, exit 0 · `bash .agents/validate.sh` → **exit 0** (coverage **98.94%** vs 94% floor, `certs.py` 100%; ruff “All checks passed”; pyright 0 errors; `node --check` clean; hygiene clean)
- `uv run pytest -m live --live` → **11 passed / 0 failed** · `./scripts/dev-server.sh status` → all 9 sources `ok`
- `uv pip list --outdated` → empty after bumping `fastapi 0.152.0 → 0.153.0` (`uv.lock` only); full suite re-run green.
**Completion criteria**
- Fresh-DB root CA, idempotent: ✅ (tests + live store hashes) · agent TLS matrix incl. wrong-CA/certless 403/`/health` CA-only/revoked 403 + live denylist re-read: ✅
- `https` `host:`/`disks:` `ok` + revoked says “agent certificate revoked” with no dial: ✅ on loopback (real agent + real dashboard) and spy-pinned hermetically; **not** on the fleet (below)
- Signed-in `/certs` + rotate/revoke/root-rotate + byte-identical bundle download: ✅ · installer bundle/unit/`--status` over TLS: ✅ (0640, see deviation)
- `pytest`/`validate.sh`/`--outdated`: ✅ · token-mode byte-identical, engine/UPS untouched, public surface unchanged (route walk): ✅
- **Live fleet TLS (deskwork+gateway flip): NOT done** — no live changes. It needs root on hosts without the installer grant (`deskwork`), has no both-ways-working window (`http://` dashboard vs https-only agent), and would deploy the agent from this not-yet-committed tree. Exact procedure recorded in README + `data/logs/NOTES-tls-rollout.md`.
**Deviations**: bundle mode `0640 root:dashboards-agent` instead of the task file’s `0600` (the agent re-reads its own bundle for the denylist; 0600 breaks the unit — already recorded in AGENTS.md); the loopback dashboard substitutes for the fleet flip. Incidental: a stray `pkill` of mine killed the dev server mid-pass; restarted and verified healthy.
**Next pending phase**: `.agents/phases/todo/16_agent_smart_pass_budget` (then `17_installer_device_filter`).
This commit is contained in:
+642
-7
@@ -1,26 +1,49 @@
|
||||
"""The host agent: stdlib-only, read-only by construction, and wrong on no box.
|
||||
|
||||
Three things are asserted here:
|
||||
Four things are asserted here:
|
||||
* parsing of /proc and /sys files exactly as Linux writes them (fake roots, so the
|
||||
weird cases are testable on any machine);
|
||||
* the collector produces a complete, non-crashing snapshot against the *real* /proc of
|
||||
whatever runs the tests, because that is the code that ships;
|
||||
* the HTTP surface answers GETs, needs a token when it is exposed, and has no write path.
|
||||
* the HTTP surface answers GETs, needs a token when it is exposed, and has no write path;
|
||||
* the TLS mode (phase 15) answers a real handshake only to a client certificate the bundle's
|
||||
root signed and the bundle's denylist does not name, with certificates minted by the
|
||||
dashboard's own `certs.py` — there is no second generator to disagree with.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import http.client
|
||||
import importlib.util
|
||||
import json
|
||||
import re
|
||||
import socket
|
||||
import ssl
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from http.server import ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from typing import Any, cast
|
||||
|
||||
import pytest
|
||||
from cryptography import x509
|
||||
from cryptography.hazmat.primitives.asymmetric import ec
|
||||
from cryptography.hazmat.primitives.asymmetric.ec import EllipticCurvePrivateKey
|
||||
from cryptography.hazmat.primitives.hashes import SHA256
|
||||
from cryptography.hazmat.primitives.serialization import (
|
||||
Encoding,
|
||||
NoEncryption,
|
||||
PrivateFormat,
|
||||
load_pem_private_key,
|
||||
)
|
||||
from cryptography.x509.oid import NameOID
|
||||
|
||||
from dashboards import certs
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
AGENT_PATH = ROOT / "agent" / "host_agent.py"
|
||||
@@ -414,6 +437,141 @@ def test_the_pass_budget_skips_the_rest_with_one_error(tmp_path, monkeypatch) ->
|
||||
assert any("budget" in e and "3 disk(s) skipped" in e for e in snap["errors"])
|
||||
|
||||
|
||||
class _TimeProxy:
|
||||
"""The agent's `time` module with one instrumented function: `monotonic`.
|
||||
|
||||
Everything else (`time()`, `sleep()`) stays the stdlib one, so patching the clock cannot
|
||||
leak into the rest of the snapshot or into pytest.
|
||||
"""
|
||||
|
||||
def __init__(self, clock: FakeSmartPassClock) -> None:
|
||||
self._clock = clock
|
||||
|
||||
def monotonic(self) -> float:
|
||||
return self._clock.now
|
||||
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
"""Everything the agent's `time` module offers besides `monotonic`, unchanged."""
|
||||
return getattr(time, name)
|
||||
|
||||
|
||||
class FakeSmartPassClock:
|
||||
"""A clock plus a `smartctl` spawn the test drives, so a **deadline** is assertable.
|
||||
|
||||
The budget arithmetic in `_smart_pass` is invisible to a fake binary: `subprocess.run`'s
|
||||
`timeout` never reaches the child, and burning 80 real seconds to watch a 16-disk pass
|
||||
truncate is not a measurement anybody keeps. So this drives the two things the pass
|
||||
actually reads — `time.monotonic()` and `subprocess.run()` — and records the timeout each
|
||||
device was given, which is `min(deadline - now, 5 s)`: the pass's own view of how much
|
||||
budget was left when it asked. Both patches go on `host_agent`'s own module globals
|
||||
through proxies, never onto the stdlib modules themselves.
|
||||
"""
|
||||
|
||||
def __init__(self, *, seconds_per_device: float, start: float = 10_000.0) -> None:
|
||||
self.seconds_per_device = seconds_per_device
|
||||
self.now = start
|
||||
self.calls: list[tuple[str, float]] = []
|
||||
self.output = read_fixture(NVME_CAPTURE)
|
||||
|
||||
def run(self, argv: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]:
|
||||
device = str(argv[-1]).rsplit("/", 1)[-1]
|
||||
timeout = float(kwargs["timeout"])
|
||||
self.calls.append((device, timeout))
|
||||
self.now += self.seconds_per_device
|
||||
return subprocess.CompletedProcess(argv, 0, self.output, "")
|
||||
|
||||
def install(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setattr(host_agent, "time", _TimeProxy(self))
|
||||
monkeypatch.setattr(host_agent, "subprocess", SimpleNamespace(
|
||||
run=self.run,
|
||||
TimeoutExpired=subprocess.TimeoutExpired,
|
||||
CompletedProcess=subprocess.CompletedProcess))
|
||||
|
||||
|
||||
# driveripper's real inventory: 8 NVMe + 8 spinning, which is the shape the budget scales for
|
||||
SHELF_DISKS = [f"nvme{i}n1" for i in range(8)] + [f"sd{c}" for c in "abcdefgh"]
|
||||
|
||||
|
||||
def write_disk_shelf(root: Path, names: list[str]) -> None:
|
||||
for name in names:
|
||||
write_fake_disk(root, name, model=f"SYSFS {name}")
|
||||
|
||||
|
||||
def test_the_pass_budget_scales_with_the_disk_count(tmp_path, monkeypatch) -> None:
|
||||
"""A 16-disk pass gets 5 s x 16, not the 20 s floor — because a truncated pass nulls the
|
||||
tail drives on every single poll (the bug that put this on the operator's screen on
|
||||
2026-10-19). The clock here burns 1.5 s per device, so the pass costs 24 s of budget:
|
||||
over the floor, far under the scale. With the flat budget this test truncates.
|
||||
"""
|
||||
root = tmp_path / "root"
|
||||
write_disk_shelf(root, SHELF_DISKS)
|
||||
clock = FakeSmartPassClock(seconds_per_device=1.5)
|
||||
clock.install(monkeypatch)
|
||||
collector = host_agent.Collector(str(root))
|
||||
snap = collector.snapshot()
|
||||
|
||||
asked = [name for name, _ in clock.calls]
|
||||
assert asked == snap["disk_io"]["device_names"], "every disk got its smartctl run"
|
||||
assert len(asked) == 16
|
||||
assert clock.seconds_per_device * len(asked) > host_agent.SMARTCTL_BUDGET_SECONDS, (
|
||||
"this pass costs more than the 20 s floor — that is what makes it a scaling test")
|
||||
assert [e for e in snap["errors"] if "budget" in e] == [], snap["errors"]
|
||||
assert all(device["smart"] is not None for device in snap["disk_io"]["devices"])
|
||||
# the *late* devices still hold the whole per-device bound, which is the point of scaling
|
||||
assert clock.calls[-1][1] == host_agent.SMARTCTL_TIMEOUT_SECONDS
|
||||
assert clock.calls[0][1] == host_agent.SMARTCTL_TIMEOUT_SECONDS
|
||||
|
||||
|
||||
def test_the_pass_budget_floor_is_twenty_seconds(tmp_path, monkeypatch) -> None:
|
||||
"""Scaling is `max(20 s, 5 s x disks)`, so one or two disks still get the 20 s floor and
|
||||
not a 5 s or 10 s window. The second device's timeout is the visible half: with a 20 s
|
||||
floor it is `min(20 - 18, 5)` = 2 s, where a budget of 5 s x 2 would have skipped it and a
|
||||
30 s floor would still have handed out the full 5 s.
|
||||
"""
|
||||
root = tmp_path / "root"
|
||||
write_disk_shelf(root, ["nvme0n1", "sda"])
|
||||
clock = FakeSmartPassClock(seconds_per_device=18.0)
|
||||
clock.install(monkeypatch)
|
||||
collector = host_agent.Collector(str(root))
|
||||
snap = collector.snapshot()
|
||||
|
||||
assert [name for name, _ in clock.calls] == ["nvme0n1", "sda"]
|
||||
assert clock.calls[0][1] == host_agent.SMARTCTL_TIMEOUT_SECONDS
|
||||
assert clock.calls[1][1] == pytest.approx(
|
||||
host_agent.SMARTCTL_BUDGET_SECONDS - clock.seconds_per_device)
|
||||
assert [e for e in snap["errors"] if "budget" in e] == []
|
||||
assert all(device["smart"] is not None for device in snap["disk_io"]["devices"])
|
||||
|
||||
|
||||
def test_the_pass_budget_error_line_names_the_scaled_figure(tmp_path, monkeypatch) -> None:
|
||||
"""When a scaled pass does run out, the operator's one line says *which* budget ran out:
|
||||
80 s on a 16-disk shelf, never the 20 s floor. A test that asserted only "a budget error
|
||||
appears" would pass on the pre-scaling code; this one cannot.
|
||||
"""
|
||||
root = tmp_path / "root"
|
||||
write_disk_shelf(root, SHELF_DISKS)
|
||||
clock = FakeSmartPassClock(seconds_per_device=6.0)
|
||||
clock.install(monkeypatch)
|
||||
collector = host_agent.Collector(str(root))
|
||||
snap = collector.snapshot()
|
||||
|
||||
scaled = max(host_agent.SMARTCTL_BUDGET_SECONDS,
|
||||
host_agent.SMARTCTL_TIMEOUT_SECONDS * len(SHELF_DISKS))
|
||||
assert scaled == 80.0
|
||||
# 6 s per device: devices 1-14 fit inside 80 s (78 s spent), the 15th finds the deadline
|
||||
# gone, and its sibling is skipped with it.
|
||||
assert len(clock.calls) == 14, [name for name, _ in clock.calls]
|
||||
assert clock.calls[-1][1] == pytest.approx(scaled - 6.0 * 13)
|
||||
budget_lines = [e for e in snap["errors"] if "budget" in e]
|
||||
assert len(budget_lines) == 1, snap["errors"]
|
||||
assert budget_lines[0] == (f"smart: pass budget of {scaled:.0f}s exhausted, "
|
||||
f"2 disk(s) skipped")
|
||||
assert f"{host_agent.SMARTCTL_BUDGET_SECONDS:.0f}s" not in budget_lines[0]
|
||||
by_name = {device["name"]: device for device in snap["disk_io"]["devices"]}
|
||||
assert [by_name[name]["smart"] is None for name in SHELF_DISKS[:14]] == [False] * 14
|
||||
assert [by_name[name]["smart"] is None for name in SHELF_DISKS[14:]] == [True, True]
|
||||
|
||||
|
||||
def test_unexecutable_smartctl_degrades_like_a_missing_one(tmp_path, monkeypatch) -> None:
|
||||
root = tmp_path / "root"
|
||||
write_fake_disk(root, "nvme0n1")
|
||||
@@ -606,6 +764,9 @@ def agent_server():
|
||||
def start(token: str | None = None) -> tuple[str, int]:
|
||||
host_agent.Handler.collector = host_agent.Collector("/")
|
||||
host_agent.Handler.token = token
|
||||
# Cleared on purpose: `Handler` is process-wide state, so a token-mode server started
|
||||
# after a TLS test in this file must not inherit that test's bundle.
|
||||
host_agent.Handler.bundle = None
|
||||
httpd = ThreadingHTTPServer(("127.0.0.1", 0), host_agent.Handler)
|
||||
servers.append(httpd)
|
||||
threading.Thread(target=httpd.serve_forever, daemon=True).start()
|
||||
@@ -687,8 +848,10 @@ def test_agent_has_no_write_path(agent_server) -> None:
|
||||
assert excinfo.value.code in (405, 501)
|
||||
|
||||
|
||||
def test_refuses_to_bind_off_loopback_without_a_token() -> None:
|
||||
def test_refuses_to_bind_off_loopback_without_a_token(monkeypatch) -> None:
|
||||
"""0.0.0.0 with no token would hand host inventory to the whole LAN."""
|
||||
monkeypatch.delenv("AGENT_TOKEN", raising=False)
|
||||
monkeypatch.delenv("AGENT_CERT_BUNDLE", raising=False)
|
||||
assert host_agent.main(["--host", "0.0.0.0", "--port", "0"]) == 2
|
||||
|
||||
|
||||
@@ -712,9 +875,10 @@ def test_agent_is_dependency_free() -> None:
|
||||
top = {name.split(".")[0] for name in imports}
|
||||
third_party = top - set(sys.stdlib_module_names) - {"host_agent"}
|
||||
assert not third_party, f"agent grew a non-stdlib import: {third_party}"
|
||||
assert top <= {"__future__", "argparse", "collections", "dataclasses", "http", "json",
|
||||
"os", "pathlib", "re", "socket", "subprocess", "threading", "time",
|
||||
"typing"}, f"agent import footprint grew: {sorted(top)}"
|
||||
assert top <= {"__future__", "argparse", "collections", "contextlib", "dataclasses", "http",
|
||||
"json", "os", "pathlib", "re", "shutil", "socket", "ssl", "subprocess",
|
||||
"tempfile", "threading", "time", "typing"}, (
|
||||
f"agent import footprint grew: {sorted(top)}")
|
||||
|
||||
|
||||
# ---- how the token is supplied ------------------------------------------- #
|
||||
@@ -746,6 +910,477 @@ def test_no_token_at_all_is_allowed_on_loopback(tmp_path) -> None:
|
||||
assert (token, error) == (None, None)
|
||||
|
||||
|
||||
def test_main_refuses_to_bind_off_loopback_without_a_token(tmp_path) -> None:
|
||||
def test_main_refuses_to_bind_off_loopback_without_a_token(tmp_path, monkeypatch) -> None:
|
||||
monkeypatch.delenv("AGENT_TOKEN", raising=False)
|
||||
monkeypatch.delenv("AGENT_CERT_BUNDLE", raising=False)
|
||||
code = host_agent.main(["--host", "0.0.0.0", "--token-file", str(tmp_path / "missing")])
|
||||
assert code == 2, "the agent must not start exposed and unauthenticated"
|
||||
|
||||
|
||||
# ---- TLS mode: one installed bundle, and its client certificate is the auth --- #
|
||||
#
|
||||
# Everything below runs a real `ssl` handshake against the real handler, because the claims
|
||||
# here are handshake claims: "optional at the handshake, enforced by the handler" is exactly
|
||||
# the kind of thing that reads fine and then makes `/health` unprobeable. The certificates come
|
||||
# from `dashboards.certs` over a dict store, so the agent is tested against the artifact the
|
||||
# dashboard actually signs rather than against a second generator that could drift from it.
|
||||
|
||||
|
||||
#: The agent module is loaded by `importlib` (it is one file with no importable package), so
|
||||
#: pyright sees `host_agent` as an untyped module and rejects `host_agent.TlsBundle` in a type
|
||||
#: position. This alias is the honest spelling: the object is that class, and the tests below
|
||||
#: assert `isinstance` where the claim actually matters.
|
||||
Bundle = Any
|
||||
|
||||
|
||||
class DictMeta:
|
||||
"""The three-method `certs.MetaStore` slice, backed by a dict (the `test_certs.py` idiom)."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.rows: dict[str, str] = {}
|
||||
|
||||
def get_meta(self, key: str) -> str | None:
|
||||
return self.rows.get(key)
|
||||
|
||||
def set_meta(self, key: str, value: str) -> None:
|
||||
self.rows[key] = value
|
||||
|
||||
def delete_meta(self, key: str) -> int:
|
||||
return 1 if self.rows.pop(key, None) is not None else 0
|
||||
|
||||
|
||||
def sign_client_leaf(state: certs.CertsState, cn: str, where: Path) -> SimpleNamespace:
|
||||
"""A second client identity under the dashboard's own root, so it verifies and can be revoked.
|
||||
|
||||
`certs.py` mints exactly one client certificate on purpose (the dashboard has one identity),
|
||||
so the revoked-peer and wrong-authority cases need a leaf signed by hand. The serial comes
|
||||
back because the denylist names a serial, not a name.
|
||||
"""
|
||||
root_key = cast(EllipticCurvePrivateKey,
|
||||
load_pem_private_key(state.root_key.encode("utf-8"), password=None))
|
||||
root_cert = x509.load_pem_x509_certificate(state.root_cert.encode("utf-8"))
|
||||
key = ec.generate_private_key(ec.SECP256R1())
|
||||
moment = datetime.now(UTC)
|
||||
cert = (
|
||||
x509.CertificateBuilder()
|
||||
.subject_name(x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, cn)]))
|
||||
.issuer_name(root_cert.subject)
|
||||
.public_key(key.public_key())
|
||||
.serial_number(x509.random_serial_number())
|
||||
.not_valid_before(moment - timedelta(days=1))
|
||||
.not_valid_after(moment + timedelta(days=365))
|
||||
.add_extension(x509.BasicConstraints(ca=False, path_length=None), critical=True)
|
||||
.sign(root_key, SHA256())
|
||||
)
|
||||
cert_path = where / f"{cn}-cert.pem"
|
||||
key_path = where / f"{cn}-key.pem"
|
||||
cert_path.write_bytes(cert.public_bytes(Encoding.PEM))
|
||||
key_path.write_bytes(key.private_bytes(Encoding.PEM, PrivateFormat.PKCS8, NoEncryption()))
|
||||
return SimpleNamespace(cert=cert_path, key=key_path, serial=cert.serial_number)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def tls_fleet(tmp_path_factory: pytest.TempPathFactory) -> SimpleNamespace:
|
||||
"""A dashboard CA, its client pair, one issued agent bundle, and one foreign authority."""
|
||||
base = tmp_path_factory.mktemp("agent-tls")
|
||||
store = DictMeta()
|
||||
state = certs.ensure_certs(store)
|
||||
state = certs.issue_agent_cert(store, state, "localhost", ["127.0.0.1", "localhost"])
|
||||
entry = state.agent("localhost")
|
||||
assert entry is not None
|
||||
for name in ("dashboard", "foreign"):
|
||||
(base / name).mkdir()
|
||||
dashboard = certs.write_material(state, base / "dashboard")
|
||||
foreign = certs.write_material(certs.ensure_certs(DictMeta()), base / "foreign")
|
||||
return SimpleNamespace(
|
||||
root_cert=state.root_cert,
|
||||
bundle=entry.bundle,
|
||||
bundle_rev=entry.bundle_rev,
|
||||
client_cert=str(dashboard.client_cert),
|
||||
client_key=str(dashboard.client_key),
|
||||
foreign_cert=str(foreign.client_cert),
|
||||
foreign_key=str(foreign.client_key),
|
||||
stolen=sign_client_leaf(state, "stolen", base),
|
||||
)
|
||||
|
||||
|
||||
def write_bundle(tmp_path: Path, text: str) -> Path:
|
||||
"""An installed bundle: one PEM file, exactly as `install-agent.sh` will place it."""
|
||||
path = tmp_path / "dashboards-agent.certs"
|
||||
path.write_text(text, encoding="ascii")
|
||||
return path
|
||||
|
||||
|
||||
def load_bundle(tmp_path: Path, text: str) -> Bundle:
|
||||
path = write_bundle(tmp_path, text)
|
||||
bundle, error = host_agent.TlsBundle.load(str(path))
|
||||
assert error is None and bundle is not None
|
||||
return bundle
|
||||
|
||||
|
||||
def client_context(ca: str, cert: str | None = None, key: str | None = None) -> ssl.SSLContext:
|
||||
"""A client that trusts `ca`, and presents a client certificate only when given one."""
|
||||
context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
|
||||
context.load_verify_locations(cadata=ca)
|
||||
if cert and key:
|
||||
context.load_cert_chain(cert, key)
|
||||
return context
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def tls_agent_server():
|
||||
"""The real handler on a TLS-wrapped loopback port; yields the base URL."""
|
||||
servers: list[ThreadingHTTPServer] = []
|
||||
|
||||
def start(bundle: Bundle, token: str | None = None) -> str:
|
||||
host_agent.Handler.collector = host_agent.Collector("/", smart=False)
|
||||
host_agent.Handler.token = token
|
||||
host_agent.Handler.bundle = bundle
|
||||
httpd = ThreadingHTTPServer(("127.0.0.1", 0), host_agent.Handler)
|
||||
httpd.socket = bundle.context.wrap_socket(httpd.socket, server_side=True)
|
||||
servers.append(httpd)
|
||||
threading.Thread(target=httpd.serve_forever, daemon=True).start()
|
||||
return f"https://127.0.0.1:{httpd.server_address[1]}"
|
||||
|
||||
yield start
|
||||
for httpd in servers:
|
||||
httpd.shutdown()
|
||||
httpd.server_close()
|
||||
host_agent.Handler.bundle = None
|
||||
|
||||
|
||||
def _json_or_raw(body: bytes) -> dict:
|
||||
"""JSON when the handler answered with JSON; the raw text for the status page."""
|
||||
try:
|
||||
return json.loads(body)
|
||||
except json.JSONDecodeError:
|
||||
return {"raw": body.decode(errors="replace")}
|
||||
|
||||
|
||||
def _https_get(url: str, context: ssl.SSLContext) -> tuple[int, dict]:
|
||||
request = urllib.request.Request(url)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=15, context=context) as response:
|
||||
return response.status, _json_or_raw(response.read())
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, _json_or_raw(error.read())
|
||||
|
||||
|
||||
def test_the_agent_and_the_dashboard_agree_on_the_bundle_lines() -> None:
|
||||
"""The agent parses a PEM by text with stdlib only, so the two literals are the contract."""
|
||||
assert host_agent.BUNDLE_REV_LINE == certs.BUNDLE_REV_LINE
|
||||
assert host_agent.REVOKED_LINE == certs.REVOKED_LINE
|
||||
|
||||
|
||||
def test_the_bundle_lines_are_read_as_written() -> None:
|
||||
text = "# BUNDLE-REV 12\n# REVOKED 5\n# REVOKED 77\n"
|
||||
assert host_agent.bundle_revision(text) == 12
|
||||
assert host_agent.bundle_revoked(text) == frozenset({"5", "77"})
|
||||
assert host_agent.bundle_revision("nothing at all\n") is None
|
||||
assert host_agent.bundle_revision("# BUNDLE-REV not-a-number\n") is None
|
||||
assert host_agent.bundle_revoked("# REVOKED\n") == frozenset()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("revoked, peer_hex, expected", [
|
||||
# the shape a real `x509.random_serial_number()` serial arrives in, both spellings
|
||||
("57989923439447977599989997426575336710275559875",
|
||||
"A285BAC84EB9C5FC282EF7CDB8D962F38BF71C3", True),
|
||||
("12648430", "C0FFEE", True),
|
||||
("14", "0E", True),
|
||||
("0E", "0E", True),
|
||||
("15", "0E", False),
|
||||
("0", "0E", False),
|
||||
])
|
||||
def test_the_denylist_matches_a_serial_whichever_way_it_is_written(revoked: str,
|
||||
peer_hex: str,
|
||||
expected: bool) -> None:
|
||||
"""`getpeercert()` says hex, `certs.py` writes decimal, and the agent must meet in the middle.
|
||||
|
||||
An agent comparing one spelling to the other accepts a revoked client forever: the operator
|
||||
clicked revoke, `/certs` says revoked, and the fleet keeps trusting the stolen identity.
|
||||
"""
|
||||
cert = {"serialNumber": peer_hex} if peer_hex else {}
|
||||
assert host_agent.serial_is_revoked(cert, frozenset({revoked})) is expected
|
||||
|
||||
|
||||
def test_a_certificate_with_no_serial_is_never_treated_as_revoked() -> None:
|
||||
assert host_agent.serial_is_revoked({}, frozenset({"1", "0"})) is False
|
||||
|
||||
|
||||
def test_resolve_bundle_reports_the_revision_and_the_denylist_of_a_real_bundle(
|
||||
tls_fleet: SimpleNamespace, tmp_path: Path) -> None:
|
||||
text = tls_fleet.bundle + f"{host_agent.REVOKED_LINE}{tls_fleet.stolen.serial}\n"
|
||||
context, revision, revoked, error = host_agent.resolve_bundle(str(write_bundle(tmp_path, text)))
|
||||
assert error is None
|
||||
assert context is not None and isinstance(context, ssl.SSLContext)
|
||||
assert revision == tls_fleet.bundle_rev
|
||||
assert revoked == frozenset({str(tls_fleet.stolen.serial)})
|
||||
|
||||
|
||||
@pytest.mark.parametrize("shape", ["missing", "empty", "no-key", "no-revision"])
|
||||
def test_a_bundle_that_cannot_be_loaded_is_an_error_naming_the_file(
|
||||
tls_fleet: SimpleNamespace, tmp_path: Path, shape: str) -> None:
|
||||
"""Every failure is a message the operator can act on, and none of them is "no TLS today"."""
|
||||
if shape == "missing":
|
||||
path = tmp_path / "absent.certs"
|
||||
elif shape == "empty":
|
||||
path = write_bundle(tmp_path, " \n")
|
||||
elif shape == "no-key":
|
||||
body = [block for block in re.findall(
|
||||
r"-----BEGIN [A-Z0-9 ]+-----.*?-----END [A-Z0-9 ]+-----\n", tls_fleet.bundle, re.S)
|
||||
if block.startswith("-----BEGIN CERTIFICATE")]
|
||||
path = write_bundle(tmp_path, "# BUNDLE-REV 1\n" + "".join(body))
|
||||
else:
|
||||
path = write_bundle(tmp_path, tls_fleet.bundle.replace("# BUNDLE-REV 1\n", ""))
|
||||
context, revision, revoked, error = host_agent.resolve_bundle(str(path))
|
||||
assert context is None and revision == 0 and revoked == frozenset()
|
||||
assert error is not None and ("absent" in error if shape == "missing" else str(path) in error)
|
||||
if shape == "no-revision":
|
||||
assert "BUNDLE-REV" in error
|
||||
|
||||
|
||||
def test_the_unpacked_pems_are_gone_once_the_context_has_them(
|
||||
tls_fleet: SimpleNamespace, tmp_path: Path, monkeypatch) -> None:
|
||||
"""The unit's RuntimeDirectory is used when it exists, and holds nothing afterwards.
|
||||
|
||||
`ssl` loads a chain from paths, so the key is written; leaving it there means the private key
|
||||
outlives the process on a disk an operator did not choose, and `kill -9` has no cleanup path.
|
||||
"""
|
||||
runtime = tmp_path / "run"
|
||||
runtime.mkdir()
|
||||
monkeypatch.setattr(host_agent, "BUNDLE_RUNTIME_DIR", runtime)
|
||||
context, _revision, _revoked, error = host_agent.resolve_bundle(str(write_bundle(tmp_path,
|
||||
tls_fleet.bundle)))
|
||||
assert error is None and context is not None
|
||||
assert list(runtime.iterdir()) == [], "the private key must not stay on disk once loaded"
|
||||
|
||||
|
||||
def test_a_missing_runtime_directory_falls_back_to_a_temporary_one(
|
||||
tls_fleet: SimpleNamespace, tmp_path: Path, monkeypatch) -> None:
|
||||
monkeypatch.setattr(host_agent, "BUNDLE_RUNTIME_DIR", tmp_path / "not-mounted")
|
||||
context, revision, _revoked, error = host_agent.resolve_bundle(str(write_bundle(tmp_path,
|
||||
tls_fleet.bundle)))
|
||||
assert error is None and context is not None
|
||||
assert revision == tls_fleet.bundle_rev
|
||||
|
||||
|
||||
def test_a_client_certificate_signed_by_the_root_reads_the_snapshot(
|
||||
tls_fleet: SimpleNamespace, tls_agent_server, tmp_path: Path) -> None:
|
||||
"""The whole point of the phase, plus the two fields task 03 reads.
|
||||
|
||||
`bundle_rev` is how `/certs` tells "this host runs an old bundle" from "this host is current",
|
||||
so it is the number the bundle carried, not a value the agent made up.
|
||||
"""
|
||||
base = tls_agent_server(load_bundle(tmp_path, tls_fleet.bundle))
|
||||
status, body = _https_get(base + "/snapshot",
|
||||
client_context(tls_fleet.root_cert, tls_fleet.client_cert,
|
||||
tls_fleet.client_key))
|
||||
assert status == 200
|
||||
assert body["ok"] is True and body["host"]["hostname"]
|
||||
assert body["tls"] is True
|
||||
assert body["bundle_rev"] == tls_fleet.bundle_rev
|
||||
for path in ("/info", "/schema", "/"):
|
||||
status, _body = _https_get(base + path, client_context(tls_fleet.root_cert,
|
||||
tls_fleet.client_cert,
|
||||
tls_fleet.client_key))
|
||||
assert status == 200, path
|
||||
|
||||
|
||||
def test_health_is_probeable_with_the_root_ca_alone_while_every_other_path_is_not(
|
||||
tls_fleet: SimpleNamespace, tls_agent_server, tmp_path: Path) -> None:
|
||||
"""The `CERT_OPTIONAL` assumption: a probe with no client certificate must still work.
|
||||
|
||||
`install-agent.sh --status` and the systemd probe hold the CA and were never handed a client
|
||||
certificate; `CERT_REQUIRED` would make every TLS install look dead. Everything that carries
|
||||
metrics is 403 — including the status page, which repeats hostname, kernel and CPU model.
|
||||
"""
|
||||
base = tls_agent_server(load_bundle(tmp_path, tls_fleet.bundle))
|
||||
probe = client_context(tls_fleet.root_cert)
|
||||
status, body = _https_get(base + "/health", probe)
|
||||
assert status == 200 and body["ok"] is True
|
||||
for path in ("/snapshot", "/info", "/schema", "/", "/nope"):
|
||||
refused, detail = _https_get(base + path, probe)
|
||||
assert refused == 403, path
|
||||
assert detail["detail"] == "client certificate required", path
|
||||
|
||||
|
||||
def test_a_client_signed_by_another_authority_is_refused(
|
||||
tls_fleet: SimpleNamespace, tls_agent_server, tmp_path: Path) -> None:
|
||||
"""A certificate that does not chain to the bundle's root never reaches a handler.
|
||||
|
||||
The server verifies at the handshake, so the answer is an aborted connection rather than an
|
||||
HTTP code — the assertion is only ever `not 200` because which alert arrives is the peer
|
||||
OpenSSL's choice, not ours.
|
||||
"""
|
||||
base = tls_agent_server(load_bundle(tmp_path, tls_fleet.bundle))
|
||||
stranger = client_context(tls_fleet.root_cert, tls_fleet.foreign_cert, tls_fleet.foreign_key)
|
||||
with pytest.raises((OSError, http.client.HTTPException)) as caught:
|
||||
_https_get(base + "/snapshot", stranger)
|
||||
assert not isinstance(caught.value, urllib.error.HTTPError)
|
||||
|
||||
|
||||
def test_a_plaintext_request_to_a_tls_agent_gets_nothing(
|
||||
tls_fleet: SimpleNamespace, tls_agent_server, tmp_path: Path) -> None:
|
||||
"""A port probe cannot read the answer, and does not get a helpful error either."""
|
||||
base = tls_agent_server(load_bundle(tmp_path, tls_fleet.bundle))
|
||||
port = int(base.rsplit(":", 1)[-1])
|
||||
with socket.create_connection(("127.0.0.1", port), timeout=15) as raw:
|
||||
raw.sendall(b"GET /snapshot HTTP/1.1\r\nHost: x\r\n\r\n")
|
||||
raw.settimeout(5)
|
||||
try:
|
||||
payload = raw.recv(200)
|
||||
except OSError:
|
||||
payload = b""
|
||||
assert b"hostname" not in payload
|
||||
|
||||
|
||||
def test_a_revoked_client_certificate_is_refused_and_the_bundle_is_re_read(
|
||||
tls_fleet: SimpleNamespace, tls_agent_server, tmp_path: Path) -> None:
|
||||
"""The two halves: the denylist bites, and re-deploying the bundle takes effect no later
|
||||
than the next request — no restart, which is the only reason the lines live in the file."""
|
||||
path = write_bundle(tmp_path, tls_fleet.bundle
|
||||
+ f"{host_agent.REVOKED_LINE}{tls_fleet.stolen.serial}\n")
|
||||
bundle, error = host_agent.TlsBundle.load(str(path))
|
||||
assert error is None and bundle is not None
|
||||
base = tls_agent_server(bundle)
|
||||
stolen = client_context(tls_fleet.root_cert, str(tls_fleet.stolen.cert), str(tls_fleet.stolen.key))
|
||||
|
||||
status, body = _https_get(base + "/snapshot", stolen)
|
||||
assert status == 403
|
||||
assert body["detail"] == "client certificate revoked"
|
||||
# A withdrawn identity is refused even on the open path: the exemption is for a probe that
|
||||
# holds the CA and no certificate, not for a certificate the operator has just taken away.
|
||||
assert _https_get(base + "/health", stolen)[0] == 403
|
||||
# The dashboard's own identity is untouched — the list names a serial, not a host.
|
||||
assert _https_get(base + "/snapshot", client_context(tls_fleet.root_cert, tls_fleet.client_cert,
|
||||
tls_fleet.client_key))[0] == 200
|
||||
|
||||
path.write_text(tls_fleet.bundle, encoding="ascii") # a redeploy: the line is gone
|
||||
status, body = _https_get(base + "/snapshot", stolen)
|
||||
assert status == 200, "the denylist is re-read when the file changes, without a restart"
|
||||
|
||||
|
||||
def test_a_renamed_denylist_entry_is_re_read_on_the_next_request(
|
||||
tls_fleet: SimpleNamespace, tmp_path: Path) -> None:
|
||||
"""An unreadable bundle keeps the list this agent already had rather than forget it."""
|
||||
path = write_bundle(tmp_path, tls_fleet.bundle
|
||||
+ f"{host_agent.REVOKED_LINE}{tls_fleet.stolen.serial}\n")
|
||||
bundle, error = host_agent.TlsBundle.load(str(path))
|
||||
assert error is None and bundle is not None
|
||||
assert bundle.revoked_serials() == frozenset({str(tls_fleet.stolen.serial)})
|
||||
path.unlink()
|
||||
assert bundle.revoked_serials() == frozenset({str(tls_fleet.stolen.serial)})
|
||||
|
||||
|
||||
def test_the_token_is_not_read_on_the_tls_path(
|
||||
tls_fleet: SimpleNamespace, tls_agent_server, tmp_path: Path) -> None:
|
||||
"""Bundle plus token: the handshake is the auth, so a stolen token cannot outlive a revoke.
|
||||
|
||||
Checking the bearer on the TLS path too would not be defence in depth — it would be a second
|
||||
credential that keeps working after the operator revoked the certificate, which is the one
|
||||
thing the denylist exists to prevent.
|
||||
"""
|
||||
base = tls_agent_server(load_bundle(tmp_path, tls_fleet.bundle), token="s3cret")
|
||||
status, _body = _https_get(base + "/snapshot",
|
||||
client_context(tls_fleet.root_cert, tls_fleet.client_cert,
|
||||
tls_fleet.client_key))
|
||||
assert status == 200
|
||||
|
||||
|
||||
def test_token_mode_reports_no_tls_fields(agent_server) -> None:
|
||||
"""Token mode is byte-identical: no `tls`, no `bundle_rev`, no new field that an agent
|
||||
installed before phase 15 does not have."""
|
||||
assert host_agent.Handler.bundle is None, "the token-mode fixture must not inherit a bundle"
|
||||
base, _port = agent_server(token="s3cret")
|
||||
_status, body = _get(base + "/snapshot", {"Authorization": "Bearer s3cret"})
|
||||
assert body["ok"] is True
|
||||
assert "tls" not in body and "bundle_rev" not in body
|
||||
|
||||
|
||||
def test_serve_wraps_the_socket_and_says_tls_and_the_revision(
|
||||
tls_fleet: SimpleNamespace, tmp_path: Path, capsys, monkeypatch) -> None:
|
||||
"""`serve()` is the function the unit runs, so its start line is what an operator reads.
|
||||
|
||||
The server is recorded rather than guessed at a port: the test needs the ephemeral port the
|
||||
OS chose *and* a way to stop `serve_forever`, which is the same object.
|
||||
"""
|
||||
created: list[ThreadingHTTPServer] = []
|
||||
|
||||
class Recording(ThreadingHTTPServer):
|
||||
def __init__(self, *args: object, **kwargs: object) -> None:
|
||||
super().__init__(*args, **kwargs) # type: ignore[arg-type]
|
||||
created.append(self)
|
||||
|
||||
monkeypatch.setattr(host_agent, "ThreadingHTTPServer", Recording)
|
||||
bundle = load_bundle(tmp_path, tls_fleet.bundle)
|
||||
thread = threading.Thread(target=host_agent.serve,
|
||||
args=("127.0.0.1", 0, host_agent.Collector("/", smart=False),
|
||||
None, bundle),
|
||||
daemon=True)
|
||||
thread.start()
|
||||
deadline = time.monotonic() + 15
|
||||
while not created and time.monotonic() < deadline:
|
||||
time.sleep(0.01)
|
||||
assert created, "serve() never built its server"
|
||||
httpd = created[0]
|
||||
port = httpd.server_address[1]
|
||||
status, _body = _https_get(f"https://127.0.0.1:{port}/health",
|
||||
client_context(tls_fleet.root_cert))
|
||||
assert status == 200
|
||||
httpd.shutdown()
|
||||
thread.join(timeout=15)
|
||||
printed = capsys.readouterr().out
|
||||
assert "tls (client cert required, rev " in printed
|
||||
# the scheme is the whole claim here (the port in the line is the requested one, which is 0
|
||||
# when the OS picks, exactly as in token mode)
|
||||
assert "[agent] listening on https://127.0.0.1" in printed
|
||||
assert "token" not in printed, "a TLS agent does not read a token"
|
||||
|
||||
|
||||
def test_a_bundle_allows_the_off_loopback_bind_without_a_token(
|
||||
tls_fleet: SimpleNamespace, tmp_path: Path, monkeypatch) -> None:
|
||||
"""The client certificate *is* the protection, so the exit-2 refusal does not apply."""
|
||||
served: list[tuple] = []
|
||||
monkeypatch.setattr(host_agent, "serve",
|
||||
lambda *args: served.append(args))
|
||||
path = write_bundle(tmp_path, tls_fleet.bundle)
|
||||
assert host_agent.main(["--host", "0.0.0.0", "--port", "0", "--cert-bundle", str(path)]) == 0
|
||||
assert served, "a bundle-configured agent must start"
|
||||
host_arg, _port, _collector, token_arg, bundle_arg = served[0]
|
||||
assert (host_arg, token_arg) == ("0.0.0.0", None)
|
||||
assert isinstance(bundle_arg, host_agent.TlsBundle)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("extra", [[], ["--cert-bundle", ""]])
|
||||
def test_off_loopback_without_a_token_or_a_bundle_is_refused_and_names_both(
|
||||
tmp_path: Path, capsys, monkeypatch, extra: list[str]) -> None:
|
||||
monkeypatch.delenv("AGENT_TOKEN", raising=False)
|
||||
monkeypatch.delenv("AGENT_CERT_BUNDLE", raising=False)
|
||||
args = ["--host", "0.0.0.0", "--port", "0", *extra]
|
||||
assert host_agent.main(args) == 2
|
||||
printed = capsys.readouterr().out
|
||||
assert "--token" in printed and "--cert-bundle" in printed
|
||||
|
||||
|
||||
def test_an_unreadable_bundle_is_fatal_at_startup(tmp_path: Path, capsys) -> None:
|
||||
"""Exit 2 naming the file: a degraded-to-plain-HTTP agent is the answer that is worse."""
|
||||
missing = tmp_path / "absent.certs"
|
||||
assert host_agent.main(["--host", "127.0.0.1", "--port", "0",
|
||||
"--cert-bundle", str(missing)]) == 2
|
||||
assert str(missing) in capsys.readouterr().out
|
||||
|
||||
|
||||
def schema_documents(field: str) -> bool:
|
||||
"""Is this field in the SCHEMA block? `GET /schema` answers with that text, self-documenting
|
||||
an install for whoever has to debug it, so a field the payload carries belongs there."""
|
||||
doc = host_agent.__doc__ or ""
|
||||
schema = doc.split("SCHEMA (GET /snapshot)", 1)[-1].split("Deployment:", 1)[0]
|
||||
return re.search(rf"^\s+{field}\s", schema, re.M) is not None
|
||||
|
||||
|
||||
def test_the_agent_version_and_schema_document_the_new_fields() -> None:
|
||||
"""1.1.0 because the payload grew, and the docstring is what `GET /schema` answers."""
|
||||
assert host_agent.AGENT_VERSION == "1.1.0"
|
||||
assert schema_documents("tls") and schema_documents("bundle_rev")
|
||||
assert "--cert-bundle" in (host_agent.__doc__ or "")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user