services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 15_agent_mtls

Browse Source 
All green. Report:

**Verified / implemented (final pass of `15_agent_mtls`)**
- Wrote `data/logs/verify-15-tls-e2e.py`: a real `host_agent.py --cert-bundle` + a real dashboard process over a scratch DB (`HOST_TARGETS=loop=https://…`) — **27/27 PASS**, transcript `verify-15-tls-e2e.out`, notes `data/logs/NOTES-tls-rollout.md`.
- Read-only checks against the running dev store: 5 `certs.*` rows byte-identical across a restart; served bundle == the `meta` row (1430 B); 0 events containing `-----BEGIN`; unsigned `/certs`→`303 ?next=`, unsigned `/api/certs`→`401`, signed-in `200`; screenshots `data/logs/15-final-{certs,system}.png`.
- Docs (step 6, full review): README — added *Protecting an agent, host by host* (the rollout task-07 left out), a `tests/test_certs.py`/TLS-matrix/sandbox-installer paragraph, and fixed two stale “the one write endpoint in the app” claims (`NUT_ALLOW_ACTIONS`, Battery-test buttons). AGENTS.md — test count 1424→1427, bundle-fetch command in *Commands*, a Testing-policy bullet on structurally pinning a key-holding module, fleet transport bullet (token mode is now the compatibility path), gap 11 final state.

**Tests / lint (full suite 100% green)**
- `uv run pytest` → **1427 passed, 15 skipped**, exit 0 · `bash .agents/validate.sh` → **exit 0** (coverage **98.94%** vs 94% floor, `certs.py` 100%; ruff “All checks passed”; pyright 0 errors; `node --check` clean; hygiene clean)
- `uv run pytest -m live --live` → **11 passed / 0 failed** · `./scripts/dev-server.sh status` → all 9 sources `ok`
- `uv pip list --outdated` → empty after bumping `fastapi 0.152.0 → 0.153.0` (`uv.lock` only); full suite re-run green.

**Completion criteria**
- Fresh-DB root CA, idempotent: ✅ (tests + live store hashes) · agent TLS matrix incl. wrong-CA/certless 403/`/health` CA-only/revoked 403 + live denylist re-read: ✅
- `https` `host:`/`disks:` `ok` + revoked says “agent certificate revoked” with no dial: ✅ on loopback (real agent + real dashboard) and spy-pinned hermetically; **not** on the fleet (below)
- Signed-in `/certs` + rotate/revoke/root-rotate + byte-identical bundle download: ✅ · installer bundle/unit/`--status` over TLS: ✅ (0640, see deviation)
- `pytest`/`validate.sh`/`--outdated`: ✅ · token-mode byte-identical, engine/UPS untouched, public surface unchanged (route walk): ✅
- **Live fleet TLS (deskwork+gateway flip): NOT done** — no live changes. It needs root on hosts without the installer grant (`deskwork`), has no both-ways-working window (`http://` dashboard vs https-only agent), and would deploy the agent from this not-yet-committed tree. Exact procedure recorded in README + `data/logs/NOTES-tls-rollout.md`.

**Deviations**: bundle mode `0640 root:dashboards-agent` instead of the task file’s `0600` (the agent re-reads its own bundle for the denylist; 0600 breaks the unit — already recorded in AGENTS.md); the loopback dashboard substitutes for the fleet flip. Incidental: a stray `pkill` of mine killed the dev server mid-pass; restarted and verified healthy.

**Next pending phase**: `.agents/phases/todo/16_agent_smart_pass_budget` (then `17_installer_device_filter`).
This commit is contained in:
ducoterra
2026-10-20 10:02:16 -04:00
parent 569ca25fdb
commit cffbac5771
72 changed files with 7168 additions and 216 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_api.py
+732 -21
View File
@@ -7,6 +7,8 @@ asset the network can fail to deliver.
"""
from __future__ import annotations
import ipaddress
import json
import re
import time
from collections.abc import Iterator
@@ -16,13 +18,15 @@ from typing import Any
import httpx
import pytest
from cryptography import x509
from cryptography.x509.oid import NameOID
from fastapi import HTTPException
from fastapi.testclient import TestClient
import store_perf
from auth_client import sign_in
from auth_contract import ADMIN_PASSWORD, FAST_PBKDF2
from dashboards import alerts, auth
from dashboards import alerts, auth, certs
from dashboards.api import MIN_PASSWORD_LENGTH, RANGE_SECONDS, parse_range
from dashboards.app import create_app
from dashboards.collectors import UpstreamError
@@ -61,7 +65,11 @@ PAGES = ("/", "/system", "/disks", "/llm", "/ups")
#: tab, a fixed tab order — which this page deliberately does not have. `MARKUP_PAGES` is the
#: list for the checks that are about the bytes rather than the nav.
SETTINGS_PAGE = "/settings"
MARKUP_PAGES = (*PAGES, SETTINGS_PAGE)
#: `/certs` is the second admin page (phase 15 task 05), kept out of `PAGES` for exactly the same
#: reason: it is not a tab. It is in `MARKUP_PAGES` and `PAGE_SCRIPTS`, so the CDN, `.sev-`,
#: flavour-text and orphan-`id` audits cover it like the five dashboards.
CERTS_PAGE = "/certs"
MARKUP_PAGES = (*PAGES, SETTINGS_PAGE, CERTS_PAGE)
@contextmanager
@@ -105,6 +113,25 @@ def ups_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Iterator[Test
yield c
def wait_for_source_error(client: TestClient, key: str, fragment: str,
*, timeout: float = 15.0) -> str:
"""Wait for one poll loop to record an error saying `fragment`, and hand back the sentence.
The poll loops run in the app's event loop, so an assertion about what a source said has to
wait for a poll rather than read the state at import time. The failure names the last error
seen, because "it never said that" and "it said something else entirely" are different bugs.
"""
deadline = time.time() + timeout
error = ""
while time.time() < deadline:
rows = {row["key"]: row for row in client.get("/api/health").json()["sources"]}
error = str((rows.get(key) or {}).get("error") or "")
if fragment in error:
return error
time.sleep(0.05)
pytest.fail(f"{key} never reported {fragment!r} (last error: {error!r})")
def wait_for_payload(client: TestClient, path: str, *, timeout: float = 5.0) -> dict[str, Any]:
deadline = time.time() + timeout
while time.time() < deadline:
@@ -721,6 +748,49 @@ def test_a_dead_host_never_writes_its_token_into_the_feed(tmp_path: Path) -> Non
store.close()
def test_a_revoked_agent_fails_both_of_its_sources_without_a_dial(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The chain end to end, wiring included: revoke it, and the next poll stops talking to it.
`collectors.agent_tls_map` is what `build_hub` hands the jobs, so this is the proof that a
revocation reaches a *running* app through its own `meta` table — no restart, and no second
process telling the first. The spy replaces `httpx.AsyncClient.get` on the class, so it
catches the shared client and the per-dial TLS client alike: what must not happen is any
request at all, because a dial that then gets refused still put a socket on a host the
operator struck out.
"""
db = str(tmp_path / "revoked-agent.sqlite3")
store = SampleStore(db)
settings = Settings(db_path=db, admin_password=ADMIN_PASSWORD,
pbkdf2_iterations=FAST_PBKDF2,
host_interval=0.05, disks_interval=0.05,
host_targets=(Target(name="deskwork", url="https://127.0.0.1:1"),))
with TestClient(create_app(settings, store)) as client:
sign_in(client)
# Before the revoke the same dead host fails as an unreachable upstream. That sentence is
# the control: it is what a dial produces, so the one after it has to mean something else.
wait_for_source_error(client, "host:deskwork", "unreachable")
state = certs.issue_agent_cert(store, certs.ensure_certs(store), "deskwork",
["127.0.0.1"])
certs.revoke(store, state, "deskwork")
calls: list[str] = []
async def spy_get(self: Any, url: Any, **kwargs: Any) -> Any:
calls.append(str(url))
raise AssertionError(f"a revoked agent was dialled at {url}")
monkeypatch.setattr(httpx.AsyncClient, "get", spy_get)
for key in ("host:deskwork", "disks:deskwork"):
error = wait_for_source_error(client, key, "agent certificate revoked")
assert "unreachable" not in error, "the refusal replaced the dial's sentence"
assert calls == [], "the gate is in front of the network, not behind it"
# and the overview says the operator revoked it rather than that the box went down
messages = [row["message"] for row in client.get("/api/summary").json()["alerts"]]
assert any("certificate revoked" in text for text in messages), messages
store.close()
def test_engine_urls_are_redacted_the_same_way_hosts_are(tmp_path: Path) -> None:
"""`GUFO_BASE_URL`/`VLLM_BASE_URL` go to the browser verbatim in `/api/config`, and unlike
`HOST_TARGETS` nothing parses a userinfo token out of them into a header — so a URL that
@@ -926,25 +996,176 @@ def test_the_settings_page_refuses_a_caller_with_no_session(tmp_path: Path) -> N
store.close()
def test_every_dashboard_page_links_to_settings(client) -> None:
"""The `⚙ Account` link is the only way to the session page, so it is on every dashboard.
def test_every_page_links_to_both_admin_pages(client) -> None:
"""`⚙ Account` and `⚷ Certs` are the only way to the two admin pages, so they are everywhere.
It sits in `.masthead-right` before the theme toggle and **outside** `.tabs`, because the tab
They sit in `.masthead-right` before the theme toggle and **outside** `.tabs`, because the tab
strip's structure — five dashboards, one of them current — is pinned by
`test_the_disks_tab_is_in_the_masthead_of_every_page` and a sixth entry with no
`aria-current` would break what that marker means.
Neither page links to itself, which is the `/settings` precedent from phase 11 and the reason
the two loops below are over different tuples: an `⚷ Certs` link on `/certs` is a no-op the
operator learns to ignore, and ignored chrome is worse than none.
"""
link = '<a class="ghost account" href="/settings"'
for path in PAGES:
account = '<a class="ghost account" href="/settings"'
certs_link = '<a class="ghost account" href="/certs"'
for path in (*PAGES, SETTINGS_PAGE):
html = client.get(path).text
assert link in html, f"{path} has no route to the account page"
assert certs_link in html, f"{path} has no route to the certificate authority"
assert certs_link in html[:html.index('id="theme-toggle"')], \
f"{path} put the Certs link after the theme toggle"
tabs = html[html.index('<nav class="tabs"'):html.index("</nav>")]
assert "/settings" not in tabs, f"{path} made Account a dashboard tab"
assert html.index(link) < html.index('id="theme-toggle"'), \
f"{path} put the account link after the theme toggle"
# the stylesheet styles the link with the button's own rule, not a copy of it
css = client.get("/static/dash.css").text
assert "a.ghost" in css
assert "/certs" not in tabs, f"{path} made Certs a dashboard tab"
for path in (*PAGES, CERTS_PAGE):
html = client.get(path).text
assert account in html, f"{path} has no route to the account page"
assert account in html[:html.index('id="theme-toggle"')], \
f"{path} put the Account link after the theme toggle"
assert account not in client.get(SETTINGS_PAGE).text, "/settings links to itself"
assert certs_link not in client.get(CERTS_PAGE).text, "/certs links to itself"
def test_the_certs_page_is_a_page_and_not_a_sixth_dashboard(client) -> None:
"""`/certs` borrows the dashboard chrome and none of the dashboard meaning (task 15.05).
Five tabs with nothing current, no `#nav-status` (a CA page has no fleet reachability to
report), and the same head and stylesheet as every other page — this is the
`test_the_settings_page_is_not_one_of_the_dashboards` assertion for the second admin page,
and it is what a sixth tab in the strip would fail on.
"""
html = client.get(CERTS_PAGE).text
assert html.count('aria-current="page"') == 0
tabs = re.findall(r'<a href="([^"]+)"[^>]*aria-current="page"', html)
assert tabs == [], f"/certs marked a tab current: {tabs}"
assert 'id="nav-status"' not in html, "/certs grew fleet dots it has nothing to fill"
for path in PAGES:
assert f'href="{path}"' in html, f"/certs cannot navigate to {path}"
assert '/static/dash.css' in html
def test_the_certs_page_carries_the_authority_and_the_agents_table(client) -> None:
"""The three blocks the page is: the root, the client line, and one row per host.
Pinned on the served bytes rather than on a browser: the root and client panels and the table
are markup, and the two rows of the table body are built from `GET /api/certs` by
`page-certs.js` — so the per-row action labels are asserted on the served **script**, the way
`/ups` pins its command buttons. `tests/test_api.py` has no JS runner (a locked anchor), and a
label that shipped in neither file would be an invisible button either way.
"""
html = client.get(CERTS_PAGE).text
js = client.get("/static/page-certs.js").text
assert "Root CA" in html and 'id="root-serial"' in html
assert 'id="root-expires"' in html and 'id="root-left"' in html
assert "Rotate root CA" in html, "the CA-wide action disappeared"
assert "Dashboard client" in html and 'id="client-serial"' in html
assert 'id="client-expires"' in html, "/certs lost the client expiry"
assert 'id="agents-table"' in html and "<tbody></tbody>" in html
assert 'id="agents-empty"' in html and "hidden" in html
for label in ("Rotate", "Revoke", "bundle"):
assert f"'{label}'" in js, f"the {label} action lost its label"
for route in ("/rotate", "/revoke", "/bundle"):
assert route in js, f"the page no longer reaches {route}"
assert "/api/certs/root/rotate" in js
def test_the_certs_page_prints_the_names_the_authority_mints(client) -> None:
"""The CN of each certificate is markup, so the literal is pinned to `certs.py`.
`MIN_PASSWORD_LENGTH`'s idiom: there is no build step to share a Python constant with a page,
so a page that restates one says it in markup and a test compares the two. Printing a CN the
module did not sign with would be the page inventing a fact about a stored certificate.
"""
html = client.get(CERTS_PAGE).text
assert f">{certs.ROOT_CN}<" in html, "the root panel's CN is not what certs.py signs"
assert f">{certs.CLIENT_CN}<" in html, "the client line's CN is not what certs.py signs"
def test_the_certs_page_reads_only_fields_the_cert_api_publishes(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""End to end, in both directions: the page's column table and `GET /api/certs` are one shape.
The served script's `COLUMNS` entries name every payload key the page indexes, so this is the
`SLOT_FIELDS` device on the surface where a rename is worst: a `deployed_bundle_rev` that
quietly became `agent_bundle_rev` would print `n/a` beside a healthy fleet forever.
"""
script = "/static/page-certs.js"
with build_client(tmp_path, monkeypatch, host_targets=cert_hosts(3),
host_interval=3600.0) as client:
js = client.get(script).text
names: set[str] = set()
for listed in re.findall(r"fields: \[([^\]]*)\]", js):
names.update(re.findall(r"'([^']+)'", listed))
assert names == CERT_AGENT_KEYS, sorted(names ^ CERT_AGENT_KEYS)
client.post("/api/certs/deskwork/rotate")
for row in cert_agent_rows(client):
assert len(row) == 8
for name in names:
assert name in row, f"the page indexes `{name}` and the API does not publish it"
# …and the two authority blocks: the page prints a serial and an expiry for each, which is
# exactly the key set the API answers there.
body = cert_document(client)
for block in ("root", "client"):
assert set(body[block]) == CERT_TIMESERIAL_KEYS
for name in CERT_TIMESERIAL_KEYS:
assert f".{name}" in js, f"the page stopped reading `{name}`"
def test_the_certs_page_never_colours_a_certificate(client) -> None:
"""No severity vocabulary reaches this page at all — an expiry is a date, not a verdict.
`/api/certs` sends no `severity` anywhere (task 04, and `severity.py` rates no certificate),
so a page that coloured an expiring cert would be inventing a threshold in JavaScript. The one
colour on the page is the payload's `status` word through a word→class map, and `unissued`
maps to no class at all — a certificate that does not exist must not borrow green.
"""
js = _code(client.get("/static/page-certs.js").text)
assert "severity" not in js, "/certs grew a verdict the API does not send"
assert "severityFrom" not in js and "severityFor" not in js
table = js[js.index("const STATUS_CHIP = {"):js.index("};")]
assert "active: 'ok'" in table and "revoked: 'fail'" in table
assert "unissued: ''" in table, "an unissued host would be coloured"
assert "not_after" not in table and "serial" not in table, "an expiry became a verdict"
def test_the_certs_page_asks_before_it_writes_and_offers_nothing_the_api_refuses(
client) -> None:
"""Every write is confirmed, and no row offers a control its own payload rules out.
`confirm()` and not the typed `DANGER` word: that gate belongs to commands that cut power,
and this page has none — the worst outcome here is a host that is not dialled until somebody
installs a bundle. The guards are the API's, restated: a row outside `HOST_TARGETS` gets no
buttons at all (`404 {"configured": false}` is all three routes' first line), a never-issued
host has nothing to revoke and no bundle to download, and the bundle download is a plain
`GET` link because a session cookie is all a signed `GET` needs.
"""
js = _code(client.get("/static/page-certs.js").text)
assert "window.confirm(" in js
assert "DANGER" not in js, "the power-cut word migrated to a page with no power cut"
assert "entry.configured" in js and "not in HOST_TARGETS" in js
assert "entry.status !== 'unissued'" in js, "a row could revoke nothing"
assert "entry.serial" in js
assert "bundle.href = '/api/certs/'" in js
assert "method: 'POST'" in js
# …and the refusal is text, never markup: these are server sentences
assert "innerHTML" not in js
def test_the_certs_page_prints_the_restart_sentence_the_api_spoke(client) -> None:
"""`restart_required` prints the route's `hint`, not a sentence the page wrote for itself.
The restart fact lives in `api.py` (the CA and the client pair are startup material, so the
table is new before the socket is). A page copy of it is the version that quietly stops
saying "and every bundle needs re-installing".
"""
js = client.get("/static/page-certs.js").text
assert "restart_required" in js
assert "body.hint" in js
assert "restart the dashboard" not in js.lower(), "the page invented the restart sentence"
def test_the_password_floor_the_form_checks_is_the_one_the_api_enforces(client) -> None:
@@ -1000,7 +1221,7 @@ def test_a_rotated_secret_clears_the_cached_session_every_page_uses(client) -> N
ASSETS = ("/static/dash.css", "/static/app.js", "/static/charts.js",
"/static/page-index.js", "/static/page-disks.js", "/static/page-system.js",
"/static/page-llm.js", "/static/page-ups.js", "/static/page-login.js",
"/static/page-settings.js")
"/static/page-settings.js", "/static/page-certs.js")
def _css(client: TestClient) -> str:
@@ -2894,6 +3115,9 @@ PAGE_SCRIPTS = {
# `/settings` is audited like a dashboard even though it is not one: an id nothing reads is
# dead markup whatever page it sits on, and this page has five new ones.
SETTINGS_PAGE: "page-settings.js",
# `/certs` too (phase 15 task 05): an authority panel and an agents table are markup, and
# every id in them is written by this page's own script.
CERTS_PAGE: "page-certs.js",
}
# `/login` is deliberately **not** in this table even though it is audited (phase 11): a
# signed-in client asking for it gets a 303 to the dashboard, so this helper's own
@@ -3459,14 +3683,19 @@ def test_the_disks_surface_is_gets_only_and_the_write_stays_the_ups_one(
assert response.status_code == 405, f"{verb} /api/disks/snapshot was accepted"
writes = sorted(str(route.path) for route in routes
if "POST" in (getattr(route, "methods", None) or set()))
# Invariant 1 is about the fleet, and phase 11 did not touch the fleet: the three `/auth/`
# writes create, re-credential, and clear a session in this process and dial nothing. They
# are named here instead of filtered out of sight, because the failure this assertion catches
# is a POST arriving quietly — a list the reader recognises is worth more than a short one.
upstream = [path for path in writes if not path.startswith("/api/auth/")]
# Invariant 1 is about the fleet. The `/auth/` writes create, re-credential, and clear a
# session in this process; the `/certs/` writes (phase 15) mint, reissue, and revoke a
# certificate in this process's `meta` table. Neither dials anything — and a revocation's whole
# effect is that the *next poll* refuses to dial — so they are named here rather than filtered
# out of sight, because the failure this assertion catches is a POST arriving quietly.
authority = [path for path in writes if path.startswith("/api/certs/")]
session = [path for path in writes if path.startswith("/api/auth/")]
upstream = [path for path in writes
if not path.startswith(("/api/auth/", "/api/certs/"))]
assert upstream == ["/api/ups/command"], "one write endpoint, four gates, unchanged"
assert session == ["/api/auth/login", "/api/auth/logout", "/api/auth/password"]
assert authority == ["/api/certs/root/rotate", "/api/certs/{name}/revoke",
"/api/certs/{name}/rotate"], "the CA surface grew a write nobody named"
# …and nothing anywhere grew the verbs this app has no use for, session included.
assert not sorted(path for route in routes
for path in [str(route.path)]
@@ -3478,7 +3707,8 @@ def test_the_disks_surface_is_gets_only_and_the_write_stays_the_ups_one(
and not str(route.path).startswith("/api/"))
# `/settings` (phase 11) is a fifth dashboard-shaped **page** and it grew no verb: a page
# route is a GET, so the write surface this test guards stays the one UPS command.
assert pages == ["/", "/disks", "/llm", "/login", "/settings", "/system", "/ups"]
assert pages == ["/", "/certs", "/disks", "/llm", "/login", "/settings", "/system",
"/ups"]
def test_config_prints_the_disks_cadence_the_page_explains_itself_with(
@@ -4160,6 +4390,487 @@ def test_samples_written_by_the_poller_are_queryable(tmp_path: Path,
store.close()
# --------------------------------------------------------------------------- #
# The certificate authority surface (phase 15 task 04)
#
# These routes mint, re-sign, and revoke the identities every agent in the fleet is dialled with,
# so the assertions are about **stored rows and wire effects** rather than about JSON being shaped
# nicely: a read that issued a certificate, a typo that wrote a row, a revocation that did not
# reach the collector, and a key that left `meta` are the four ways this surface can be wrong while
# every status code still reads 200.
#
# The fleet here is loopback ports nobody listens on. That is deliberate: nothing on this surface
# is allowed to need a live agent, so a refused connection is both the whole environment and the
# control that says "a dial was attempted, and it was not this".
# --------------------------------------------------------------------------- #
#: Three configured hosts, none of them reachable, on the scheme a protected exporter is dialled
#: by. The URL host is also what a rotated certificate's SAN must carry, so these are the SANs the
#: assertions below read.
DEAD_TARGETS = (Target(name="deskwork", url="https://127.0.0.1:1"),
Target(name="driveripper", url="https://127.0.0.1:2"),
Target(name="gateway", url="https://127.0.0.1:3"))
#: The whole `GET /api/certs` document and one agent row, pinned as a table so a panel that
#: retires can never shrink the payload (`/api/disks`' idiom) and task 05's page has a contract.
CERT_DOCUMENT_KEYS = {"root", "client", "agents"}
CERT_TIMESERIAL_KEYS = {"serial", "not_after"}
CERT_AGENT_KEYS = {"name", "configured", "serial", "not_after", "status", "bundle_rev",
"deployed_bundle_rev", "san"}
def cert_document(client: TestClient) -> dict[str, Any]:
"""`GET /api/certs`, with the status asserted: a refusal here would pass a `.get()` chain."""
response = client.get("/api/certs")
assert response.status_code == 200, response.text
body = response.json()
assert set(body) == CERT_DOCUMENT_KEYS, "the CA document grew or lost a top-level key"
for block in ("root", "client"):
assert set(body[block]) == CERT_TIMESERIAL_KEYS, block
assert int(body[block]["serial"]) > 0, "a serial is the decimal text of a positive int"
assert body[block]["not_after"] > time.time(), "a stored CA may not already be expired"
return body
def cert_agent_rows(client: TestClient, name: str | None = None) -> list[dict[str, Any]]:
"""The agent rows, every one of them shaped, optionally just one host's."""
rows = cert_document(client)["agents"]
for row in rows:
assert set(row) == CERT_AGENT_KEYS, sorted(row)
if name is None:
return rows
return [row for row in rows if row["name"] == name]
def cert_context(client: TestClient) -> Any:
"""The app's `AppContext`.
`TestClient.app` is typed as the ASGI callable it wraps, so the one cast lives here instead of
one `# type: ignore` at every assertion that needs the store or a job.
"""
return client.app.state.ctx # type: ignore[attr-defined]
def cert_meta(store: SampleStore) -> dict[str, str]:
"""Every `certs.*` row, for the before/after comparisons that prove a route wrote nothing.
Coalesced to `""` because the comparison is "did any byte of these five rows move", and an
absent row is the empty answer to that question rather than a second kind of nothing.
"""
return {key: store.get_meta(key) or "" for key in certs.META_KEYS}
def revoked_serials(bundle: str) -> list[str]:
"""The denylist of a bundle, as the agent reads it: the `# REVOKED <serial>` lines."""
return [line.removeprefix(certs.REVOKED_LINE).strip()
for line in bundle.splitlines() if line.startswith(certs.REVOKED_LINE)]
#: One PEM block out of a bundle, by text — the same read task 02's agent does with stdlib.
_PEM_BLOCK = re.compile(r"-----BEGIN [A-Z0-9 ]+-----.*?-----END [A-Z0-9 ]+-----\n", re.DOTALL)
def bundle_certs(bundle: str) -> list[x509.Certificate]:
"""The two certificates of a bundle, in the order the format writes them.
`[0]` is the agent's own server certificate and `[1]` the root it chains to (`certs.py`'s
`_bundle_text` order: metadata lines, agent cert, agent key, root cert). Reading them out of the
served bytes rather than trusting the response's echo is what makes "did the rotation reach the
artifact?" a checked claim.
"""
blocks = [block for block in _PEM_BLOCK.findall(bundle) if "CERTIFICATE" in block]
assert len(blocks) == 2, f"a bundle is two certificates and a key, got {len(blocks)}"
return [x509.load_pem_x509_certificate(block.encode()) for block in blocks]
def bundle_of(client: TestClient, name: str) -> str:
"""The served bundle, asserted to still be a bundle (2 certs, 1 key) before it is compared."""
response = client.get(f"/api/certs/{name}/bundle")
assert response.status_code == 200, response.text
assert response.headers["content-type"].startswith("application/x-pem-file")
assert "no-store" in response.headers["cache-control"], "a private key may not be cached"
blocks = re.findall(r"-----BEGIN [A-Z0-9 ]+-----", response.text)
assert blocks.count("-----BEGIN CERTIFICATE-----") == 2, blocks
assert blocks.count("-----BEGIN PRIVATE KEY-----") == 1, blocks
return response.text
def stored_bundle(store: SampleStore, name: str) -> str:
"""The bundle exactly as the `certs.agents` row holds it."""
row: dict[str, Any] = json.loads(store.get_meta(certs.META_AGENTS) or "{}")
return str(row[name]["bundle"])
def cert_events(client: TestClient) -> list[dict[str, Any]]:
return client.get("/api/events", params={"source": "certs"}).json()["events"]
def cert_hosts(count: int) -> tuple[Target, ...]:
return DEAD_TARGETS[:count]
@pytest.mark.parametrize("hosts", [0, 1, 3], ids=["none", "one", "three"])
def test_the_cert_list_is_the_stored_table_for_zero_one_and_three_hosts(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, hosts: int) -> None:
"""The list works with no fleet at all, and says *unissued* rather than omitting a host.
The CA exists whether or not `HOST_TARGETS` does — a fresh database boots with a root and a
client certificate — so this is not the `/api/disks` "not configured" surface and does not 404.
A configured host with nothing issued still gets a row (invariant 3: a machine that is deployed
and not yet installed is not an absent machine), with `None` and not `0` where there is no
certificate (rule 2).
"""
with build_client(tmp_path, monkeypatch, host_targets=cert_hosts(hosts),
host_interval=3600.0) as client:
rows = cert_agent_rows(client)
assert len(rows) == hosts, "one row per configured host, and no more"
assert [row["name"] for row in rows] == sorted(row["name"] for row in rows), (
"one order in the payload and one in the table")
for row in rows:
assert row["configured"] is True
assert row["status"] == "unissued", "nothing has been signed for it yet"
assert row["serial"] is None and row["bundle_rev"] is None, "no invented zeros"
assert row["deployed_bundle_rev"] is None, "nobody has answered, so nobody reported"
assert row["san"] == [], "no names on record, not a certificate with no names"
def test_the_cert_list_carries_the_rev_the_agent_reported(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""`deployed_bundle_rev` is what the host said about itself, not what the CA hopes.
Task 05's "stale bundle" sentence is a comparison of two numbers that come from two different
places: the row this dashboard issued and the `agent` block of the last `system` payload. Both
halves are pinned here, including the direction that matters — issued 2 against deployed 1 is
the operator's "install the new bundle" case, and it has to be visible rather than smoothed
over.
"""
with build_client(tmp_path, monkeypatch, host_targets=cert_hosts(1),
host_interval=3600.0) as client:
assert client.post("/api/certs/deskwork/rotate").status_code == 200
job = cert_context(client).hub.jobs["host:deskwork"]
assert cert_agent_rows(client, "deskwork")[0]["deployed_bundle_rev"] is None, (
"a host that has never answered reports nothing")
job.state.payload = {"name": "deskwork", "agent": {"tls": True, "bundle_rev": 1}}
assert cert_agent_rows(client, "deskwork")[0]["deployed_bundle_rev"] == 1
assert client.post("/api/certs/deskwork/rotate").json()["bundle_rev"] == 2
row = cert_agent_rows(client, "deskwork")[0]
assert (row["bundle_rev"], row["deployed_bundle_rev"]) == (2, 1), "the stale pair, both"
# A token-mode agent publishes no `agent.tls` block at all, and that is `n/a`, not rev 0.
job.state.payload = {"name": "deskwork", "agent": {"version": "1.0.0"}}
assert cert_agent_rows(client, "deskwork")[0]["deployed_bundle_rev"] is None
def test_an_entry_for_a_host_that_left_the_config_is_listed_and_not_writable(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""A row issued for a machine that has gone is a key still sitting in the table.
Listing it is the honest answer (`/certs` is where you learn you still hold one); refusing to
rotate or revoke it is the same invariant from the other side — `HOST_TARGETS` is what this
dashboard dials, and a name it no longer dials is not a name to install a bundle on. The row is
written straight to the store because that is the only way this state arises: a host removed
from the config, never a route that created it.
"""
with build_client(tmp_path, monkeypatch, host_targets=cert_hosts(1),
host_interval=3600.0) as client:
store: SampleStore = cert_context(client).store
certs.issue_agent_cert(store, certs.read_state(store), "old-host", ["10.9.8.7"])
rows = cert_agent_rows(client)
assert [row["name"] for row in rows] == ["deskwork", "old-host"]
ghost = rows[1]
assert ghost["configured"] is False, "listed, but this dashboard does not dial it"
assert ghost["status"] == certs.ACTIVE and ghost["bundle_rev"] == 1
assert ghost["deployed_bundle_rev"] is None, "no job means no report, not rev 0"
assert client.post("/api/certs/old-host/rotate").status_code == 404
assert client.post("/api/certs/old-host/revoke").status_code == 404
assert cert_events(client) == [], "a refusal is not an audit line"
def test_a_cert_route_for_an_unknown_name_dials_nothing_and_writes_nothing(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The invariant-3 gate on the per-host routes, taken before anything can have a side effect.
`not-installed` is refused in the first line of each route, so no client is built, no store row
is written, and no key is minted. All three are measured: the two network spies raise, `set_meta`
is watched for any `certs.*` write, and the five rows are compared byte for byte. A typo that
minted a certificate would be invisible on every status code and permanent in the table.
"""
with build_client(tmp_path, monkeypatch, host_targets=cert_hosts(1),
host_interval=3600.0) as client:
store: SampleStore = cert_context(client).store
calls: list[str] = []
written: list[str] = []
async def spy_get(self: Any, url: Any, **kwargs: Any) -> Any:
calls.append(f"GET {url}")
raise AssertionError(f"an unconfigured cert name dialled {url}")
def spy_fetch(*args: Any, **kwargs: Any) -> Any:
calls.append("fetch_snapshot")
raise AssertionError("an unconfigured cert name fetched an agent snapshot")
real_set_meta = SampleStore.set_meta
def spy_set_meta(self: SampleStore, key: str, value: str) -> None:
written.append(key)
real_set_meta(self, key, value)
monkeypatch.setattr(httpx.AsyncClient, "get", spy_get)
monkeypatch.setattr("dashboards.collectors.fetch_snapshot", spy_fetch)
monkeypatch.setattr(SampleStore, "set_meta", spy_set_meta)
before = cert_meta(store)
for response in (client.get("/api/certs/not-configured/bundle"),
client.post("/api/certs/not-configured/rotate"),
client.post("/api/certs/not-configured/revoke")):
assert response.status_code == 404, response.text
body = response.json()
assert body["configured"] is False
assert "HOST_TARGETS" in body["detail"], "the sentence names the fix"
assert calls == [], "the gate is in front of the network, not behind it"
assert [key for key in written if key.startswith("certs.")] == [], "a write happened"
assert cert_meta(store) == before, "the stored CA and agent table did not move"
assert cert_events(client) == [], "and no audit line was written for a name that is not here"
# …and the same route still works for the host that *is* configured, so the refusal above
# is the gate and not a broken router.
assert client.post("/api/certs/deskwork/rotate").status_code == 200
def test_rotating_a_host_mints_the_bundle_it_asks_for(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Rotate is also *issue*: first call is `bundle_rev` 1, and the SAN is the config's address.
The SAN assertion is the one a rollout fails on silently — a certificate that does not name the
address the dashboard dials is a handshake error and a red source, not a warning — so it is read
out of the served bundle's own certificate rather than out of the response's echo.
"""
with build_client(tmp_path, monkeypatch, host_targets=cert_hosts(1),
host_interval=3600.0) as client:
store: SampleStore = cert_context(client).store
body = client.post("/api/certs/deskwork/rotate").json()
assert body["ok"] is True and body["status"] == certs.ACTIVE
assert body["bundle_rev"] == 1 and body["serial"] and body["not_after"] > time.time()
assert body["san"] == ["127.0.0.1", "deskwork"], "the dialled address and the name"
cert = bundle_certs(bundle_of(client, "deskwork"))[0]
names = cert.extensions.get_extension_for_class(
x509.SubjectAlternativeName).value.get_values_for_type(x509.IPAddress)
assert names == [ipaddress.ip_address("127.0.0.1")], "an IP SAN, or an IP dial fails"
assert cert.subject == x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "deskwork")])
events = cert_events(client)
assert len(events) == 1 and events[0]["kind"] == "cert-rotated"
assert events[0]["host"] == "deskwork" and events[0]["message"] == (
"agent certificate rotated for deskwork")
assert events[0]["data"] == {"bundle_rev": 1}
second = client.post("/api/certs/deskwork/rotate").json()
assert second["bundle_rev"] == 2 and second["serial"] != body["serial"]
# The new identity is not on the denylist: rotation replaces an identity, it does not strike
# one out, and a bundle that listed its own serial would refuse the client it ships.
bundle = bundle_of(client, "deskwork")
assert revoked_serials(bundle) == [], revoked_serials(bundle)
assert second["serial"] not in bundle.splitlines(), "no denylist line for the live serial"
assert stored_bundle(store, "deskwork") == bundle, "served bytes are the stored bytes"
def test_revoking_a_host_marks_the_table_and_the_next_poll_refuses_without_a_dial(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The API half of the revocation chain: this route is what task 03's refusal reads.
The control is the sentence from *before* the click (`unreachable`, which is what a dial to a
dead port says), and after it the poll has to say `agent certificate revoked` with a spy on
`httpx.AsyncClient.get` proving no socket was opened. The stored denylist is the other half: an
agent holding a bundle that omits the serial keeps accepting the client it was given.
"""
with build_client(tmp_path, monkeypatch, host_targets=cert_hosts(1),
host_interval=0.05, disks_interval=0.05) as client:
store: SampleStore = cert_context(client).store
wait_for_source_error(client, "host:deskwork", "unreachable")
serial = client.post("/api/certs/deskwork/rotate").json()["serial"]
body = client.post("/api/certs/deskwork/revoke").json()
assert body["ok"] is True and body["status"] == certs.REVOKED
assert body["serial"] == serial, "revocation marks the identity it has, it does not replace"
assert body["hint"] and "next poll" in body["hint"]
calls: list[str] = []
async def spy_get(self: Any, url: Any, **kwargs: Any) -> Any:
calls.append(str(url))
raise AssertionError(f"a revoked agent was dialled at {url}")
monkeypatch.setattr(httpx.AsyncClient, "get", spy_get)
for key in ("host:deskwork", "disks:deskwork"):
error = wait_for_source_error(client, key, "agent certificate revoked")
assert "unreachable" not in error
assert calls == []
assert revoked_serials(bundle_of(client, "deskwork")) == [serial]
events = cert_events(client)
assert [row["kind"] for row in events] == ["cert-revoked", "cert-rotated"]
assert events[0]["message"] == "agent certificate revoked for deskwork"
assert certs.agent_status(store, "deskwork") == certs.REVOKED, (
"the collector's own read agrees with the row the route wrote")
def test_root_rotation_reissues_every_host_and_answers_restart_required(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""One action re-arms the fleet — and says the part it cannot do from inside the process.
`restart_required` is not decoration: the CA and the client pair are startup material, so the
table is new immediately while the sockets keep the old identity. The page (task 05) prints
that sentence because the API answered it, and the alternative is an operator who rotated a
root and watches a fleet that still chains to the previous one.
"""
with build_client(tmp_path, monkeypatch, host_targets=cert_hosts(3),
host_interval=3600.0) as client:
store: SampleStore = cert_context(client).store
for target in DEAD_TARGETS:
assert client.post(f"/api/certs/{target.name}/rotate").status_code == 200
before = cert_document(client)
rows_before = {row["name"]: row for row in before["agents"]}
root_pem_before = store.get_meta(certs.META_ROOT_CERT) or ""
key_before = store.get_meta(certs.META_ROOT_KEY) or ""
assert root_pem_before and key_before, "both halves of the CA exist before the rotation"
body = client.post("/api/certs/root/rotate").json()
assert body["ok"] is True and body["restart_required"] is True
assert body["agents"] == 3
assert body["root"]["serial"] != before["root"]["serial"], "a fresh root"
after = cert_document(client)
assert after["root"] == body["root"], "the list reports what the rotation wrote"
assert after["client"]["serial"] != before["client"]["serial"], "and a fresh client leaf"
for row in after["agents"]:
was = rows_before[row["name"]]
assert row["bundle_rev"] == was["bundle_rev"] + 1, "every host reissued, rev bumped"
assert row["serial"] != was["serial"] and row["status"] == certs.ACTIVE
bundle = bundle_of(client, row["name"])
leaf, root = bundle_certs(bundle)
assert leaf.serial_number == int(row["serial"]), "the served bundle is the listed one"
assert root.serial_number == int(after["root"]["serial"]), (
"and it chains to the authority the list now reports")
assert revoked_serials(bundle) == [], "the denylist resets with the generation"
assert store.get_meta(certs.META_ROOT_KEY) != key_before
assert store.get_meta(certs.META_ROOT_CERT) != root_pem_before
assert root_pem_before not in "".join(cert_meta(store).values()), (
"the old root is gone")
events = cert_events(client)
assert events[0]["kind"] == "cert-root-rotated"
assert events[0]["message"] == (
"root certificate rotated — every agent certificate reissued, restart required")
assert events[0]["data"] == {"agents": 3}
def test_the_bundle_route_names_a_file_and_changes_when_the_identity_does(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Byte-identical to the stored row, an `attachment`, and different bytes after a reissue.
This is the installer's input (task 06), so the claim is about bytes and not about a bundle
that merely re-verifies: `install-agent.sh` is handed the file and compares nothing else.
"""
with build_client(tmp_path, monkeypatch, host_targets=cert_hosts(1),
host_interval=3600.0) as client:
store: SampleStore = cert_context(client).store
missing = client.get("/api/certs/deskwork/bundle")
assert missing.status_code == 404, "nothing issued yet, so nothing to download"
body = missing.json()
assert body["configured"] is True and body["issued"] is False, "not the same 404"
assert "rotate" in body["detail"], "and the answer names the way to get one"
first = client.post("/api/certs/deskwork/rotate").json()["serial"]
response = client.get("/api/certs/deskwork/bundle")
assert response.headers["content-disposition"] == (
'attachment; filename="agent-deskwork.bundle"')
assert response.text == stored_bundle(store, "deskwork")
client.post("/api/certs/deskwork/rotate")
second = client.post("/api/certs/deskwork/rotate").json()["serial"]
assert second != first
updated = client.get("/api/certs/deskwork/bundle")
assert updated.text != response.text, "a reissue changes the served bytes"
assert updated.text == stored_bundle(store, "deskwork")
def test_no_certificate_material_leaves_meta_through_the_cert_surface(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""invariant 6 on the API that owns the keys: serials and dates go out, PEMs do not.
Every write here is exercised first, so the grep is over a table that has been rotated and
revoked rather than a pristine one, and it covers both the payloads a browser reads and the
stored event rows those payloads are built from — an event row is kept forever, which is what
makes a PEM in a message permanent rather than transient.
"""
with build_client(tmp_path, monkeypatch, host_targets=cert_hosts(2),
host_interval=3600.0) as client:
store: SampleStore = cert_context(client).store
client.post("/api/certs/deskwork/rotate")
client.post("/api/certs/driveripper/rotate")
client.post("/api/certs/driveripper/revoke")
client.post("/api/certs/root/rotate")
secrets = [value for value in cert_meta(store).values() if value]
assert any("PRIVATE KEY" in value for value in secrets), "the table really does hold keys"
for path in ("/api/certs", "/api/health", "/api/events", "/api/summary", "/api/config"):
text = client.get(path).text
for marker in ("BEGIN CERTIFICATE", "PRIVATE KEY", "BUNDLE-REV"):
assert marker not in text, f"{path} served {marker}"
assert client.get("/api/health").json()["db"], "health still answers, keys or no keys"
for event in store.recent_events(50, source="certs"):
assert "BEGIN" not in event.message and "PRIVATE KEY" not in event.message
assert not any(secret in json.dumps(event.as_json()) for secret in secrets)
def test_a_host_named_root_cannot_be_rotated_past_the_certificate_authority(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""`/api/certs/root/rotate` and `/api/certs/{name}/rotate` are one URL for a host called root.
The CA route is registered first, so it wins — which means the wrong button would rotate the
whole authority instead of one host's leaf. Refusing with the reason is the only honest answer:
the alternative is a fleet-wide outage behind a click that looked like routine maintenance.
"""
with build_client(tmp_path, monkeypatch,
host_targets=(Target(name="root", url="https://127.0.0.1:1"),),
host_interval=3600.0) as client:
store: SampleStore = cert_context(client).store
before = cert_document(client)["root"]["serial"]
refused = client.post("/api/certs/root/rotate")
assert refused.status_code == 409, refused.text
assert "rename" in refused.json()["detail"], "the sentence says what to do"
assert cert_document(client)["root"]["serial"] == before, "and the CA did not move"
assert cert_events(client) == []
assert cert_meta(store)[certs.META_AGENTS] == "{}", "nothing was issued for the name either"
def test_a_read_of_the_cert_surface_issues_nothing_and_writes_nothing(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""A GET that mints a certificate is the surprise this route must never be.
`certs.ensure_certs` is the bootstrapping read and `certs.read_state` is the request-time one;
making `ensure_certs` explode is what proves the difference rather than asserting it in prose,
and the meta rows are compared byte for byte across three reads.
"""
with build_client(tmp_path, monkeypatch, host_targets=cert_hosts(1),
host_interval=3600.0) as client:
store: SampleStore = cert_context(client).store
before = cert_meta(store)
def explode(*args: Any, **kwargs: Any) -> Any:
raise AssertionError("a GET tried to bootstrap the certificate authority")
monkeypatch.setattr(certs, "ensure_certs", explode)
assert cert_agent_rows(client)
assert cert_document(client)["root"]["serial"] == str(
x509.load_pem_x509_certificate(
str(before[certs.META_ROOT_CERT]).encode()).serial_number)
# A bundle that has not been issued is still a 404 rather than a mint, on a read too.
assert client.get("/api/certs/deskwork/bundle").status_code == 404
assert cert_meta(store) == before, "three reads, no write"
assert cert_events(client) == []
# --------------------------------------------------------------------------- #
# phase 14 task 04: the endpoint budgets
#