services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 15_agent_mtls

Browse Source 
All green. Report:

**Verified / implemented (final pass of `15_agent_mtls`)**
- Wrote `data/logs/verify-15-tls-e2e.py`: a real `host_agent.py --cert-bundle` + a real dashboard process over a scratch DB (`HOST_TARGETS=loop=https://…`) — **27/27 PASS**, transcript `verify-15-tls-e2e.out`, notes `data/logs/NOTES-tls-rollout.md`.
- Read-only checks against the running dev store: 5 `certs.*` rows byte-identical across a restart; served bundle == the `meta` row (1430 B); 0 events containing `-----BEGIN`; unsigned `/certs`→`303 ?next=`, unsigned `/api/certs`→`401`, signed-in `200`; screenshots `data/logs/15-final-{certs,system}.png`.
- Docs (step 6, full review): README — added *Protecting an agent, host by host* (the rollout task-07 left out), a `tests/test_certs.py`/TLS-matrix/sandbox-installer paragraph, and fixed two stale “the one write endpoint in the app” claims (`NUT_ALLOW_ACTIONS`, Battery-test buttons). AGENTS.md — test count 1424→1427, bundle-fetch command in *Commands*, a Testing-policy bullet on structurally pinning a key-holding module, fleet transport bullet (token mode is now the compatibility path), gap 11 final state.

**Tests / lint (full suite 100% green)**
- `uv run pytest` → **1427 passed, 15 skipped**, exit 0 · `bash .agents/validate.sh` → **exit 0** (coverage **98.94%** vs 94% floor, `certs.py` 100%; ruff “All checks passed”; pyright 0 errors; `node --check` clean; hygiene clean)
- `uv run pytest -m live --live` → **11 passed / 0 failed** · `./scripts/dev-server.sh status` → all 9 sources `ok`
- `uv pip list --outdated` → empty after bumping `fastapi 0.152.0 → 0.153.0` (`uv.lock` only); full suite re-run green.

**Completion criteria**
- Fresh-DB root CA, idempotent: ✅ (tests + live store hashes) · agent TLS matrix incl. wrong-CA/certless 403/`/health` CA-only/revoked 403 + live denylist re-read: ✅
- `https` `host:`/`disks:` `ok` + revoked says “agent certificate revoked” with no dial: ✅ on loopback (real agent + real dashboard) and spy-pinned hermetically; **not** on the fleet (below)
- Signed-in `/certs` + rotate/revoke/root-rotate + byte-identical bundle download: ✅ · installer bundle/unit/`--status` over TLS: ✅ (0640, see deviation)
- `pytest`/`validate.sh`/`--outdated`: ✅ · token-mode byte-identical, engine/UPS untouched, public surface unchanged (route walk): ✅
- **Live fleet TLS (deskwork+gateway flip): NOT done** — no live changes. It needs root on hosts without the installer grant (`deskwork`), has no both-ways-working window (`http://` dashboard vs https-only agent), and would deploy the agent from this not-yet-committed tree. Exact procedure recorded in README + `data/logs/NOTES-tls-rollout.md`.

**Deviations**: bundle mode `0640 root:dashboards-agent` instead of the task file’s `0600` (the agent re-reads its own bundle for the denylist; 0600 breaks the unit — already recorded in AGENTS.md); the loopback dashboard substitutes for the fleet flip. Incidental: a stray `pkill` of mine killed the dev server mid-pass; restarted and verified healthy.

**Next pending phase**: `.agents/phases/todo/16_agent_smart_pass_budget` (then `17_installer_device_filter`).
This commit is contained in:
ducoterra
2026-10-20 10:02:16 -04:00
parent 569ca25fdb
commit cffbac5771
72 changed files with 7168 additions and 216 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+282 -20
View File
@@ -34,9 +34,11 @@ History is a first-class feature, not a sparkline afterthought: every number wor
is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`.
None of it is public. One admin signs in at `/login` (the only page an anonymous visitor
reaches), and `/settings` — `⚙ Account` in the masthead — is where that session is read, ended,
and re-credentialled. Every page above, `/settings`, and every `/api` route but the four named
in invariant 12 require the session.
reaches), and the two admin pages live behind it: `/settings` — `⚙ Account` in the masthead —
where that session is read, ended and re-credentialled, and `/certs` — `⚷ Certs`, next to it —
where the fleet's certificate authority is listed, rotated, revoked and downloaded. Every page
above, `/settings`, `/certs`, and every `/api` route but the four named in invariant 12 require
the session.
## Non-negotiable rules
@@ -94,6 +96,12 @@ page still renders.
startup and the empty-new-password refusal) carry **fixed** operator text with nothing
interpolated, not even a prefix of a hash; `tests/test_auth.py` pins that structurally, so
the day someone adds `logger.warning(f"bad verifier {v}")` a test fails instead of a log line.
**A private key is the same case** (phase 15): the root CA key and the dashboard's client key
live only in `meta`, `certs.py` has no `logging` import and one fixed-text raise naming the rows
to delete (pinned structurally by `tests/test_certs.py`, which also asserts the refusal quotes
no line of the stored key), the root key is never written to the temp dir the client material
lives in, and a bundle carries the root *certificate* — never the key — because an agent only
has to verify its clients and a stolen agent install must not be a stolen CA.
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
`sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
@@ -173,9 +181,31 @@ Responsibilities, one line each:
- `agent/host_agent.py` — stdlib-only (3.10+), reads `/proc`, `/sys/block`, `/sys/class/drm`,
`statvfs`; serves `GET /snapshot` + `GET /info` + `GET /schema` + `GET /health`, plus an
HTML status page at `GET /`. Refuses to bind off-loopback without a token (exit 2).
Counters stay counters: the dashboard computes
HTML status page at `GET /`. Refuses to bind off-loopback without a token **or** a certificate
bundle (exit 2). Counters stay counters: the dashboard computes
rates so the agent never remembers who polled it last.
**TLS mode (phase 15 task 02, `AGENT_VERSION` 1.1.0):** `--cert-bundle`/`AGENT_CERT_BUNDLE`
loads the one PEM the dashboard signs (`resolve_bundle`, the `resolve_token` idiom: server cert
+ server key + root CA + `# BUNDLE-REV <n>` + `# REVOKED <serial>` lines, grepped by text
because the agent has no `cryptography`), unpacks it into the unit's `RuntimeDirectory` when
there is one else a temp dir, builds a `PROTOCOL_TLS_SERVER` context with
`verify_mode = CERT_OPTIONAL`, and **deletes the PEMs once loaded** (the key then lives only in
this process, so a `kill -9` has no cleanup to forget). `CERT_OPTIONAL` is the assumption that
keeps `/health` probeable: the *handler* enforces, refusing a peer with no verified certificate
403 `client certificate required` on every path but `/health`, and refusing a serial on the
bundle's denylist 403 `client certificate revoked` on **every** path including `/health` (the
exemption is for a probe holding the CA and no certificate, not for an identity the operator
withdrew). The denylist is re-read when the bundle's `(mtime_ns, size)` stamp changes — that is
the whole reason the `# REVOKED` lines live in the bundle — while a new certificate needs a
restart, because `ssl` cannot swap a chain under a live socket. Serial matching accepts the hex
`getpeercert()` renders *and* the decimal `certs.py` writes, because comparing one spelling to
the other accepts a revoked client forever. On the TLS path the bearer token is **not** read (the
handshake is the auth, and a second credential that outlives a revocation defeats the list),
`/snapshot` gains additive `tls: true` + `bundle_rev` only in TLS mode, and token mode is
byte-identical to before. Task 03 has since flipped the collector (see the `collectors.py`
bullet), so those two fields are how the dashboard answers "is this host running the bundle
`/certs` issued?" — `sources/host.py` and `sources/disks.py` carry them into both payloads as
`agent.tls` / `agent.bundle_rev`, present only when the agent published them.
SMART (phase 09): `smart()` runs `smartctl -i -A -H` per whole `/sys/block` disk
(`parse_smart_output` covers ATA + NVMe; NVMe media errors map to
`offline_uncorrectable`, hex-packed vendor raws are never decoded, and an attribute row is
@@ -217,7 +247,31 @@ Responsibilities, one line each:
Events are stored under the **dashboard** name (`system`/`disks`/`llm`/`ups`), not the sample
source (`host`) — a page filtering `/api/events` by `host` gets an empty feed forever
(found on `/system` during phase 03 browser verification).
- `collectors.py` — the fetch half. `NutMonitor` owns per-UPS transition bookkeeping (its
- `collectors.py` — the fetch half. `AgentTls` (phase 15 task 03) is what makes an `https://`
`HOST_TARGETS` entry a **certificate-authenticated** dial: `verify` and `cert` are the paths
`certs.write_material` wrote at startup, and the dial is verified against this app's own root
and presents this app's own client certificate. Two things about it are load-bearing. (a) The
credential travels in a **client built per dial**, because `httpx` resolves TLS material when a
client is *constructed* (there is no per-request `verify`/`cert`, and `verify=<str>`/`cert=` are
deprecated anyway — the supported form is an `ssl.SSLContext`, which is also the honest shape:
trust in one named CA, hostname checks on) while one shared client dials the plain agents and
both engines — a `verify` on *that* would make every engine read depend on a CA nobody issued
their cert with. (b) `status` is a **callable** doing a fresh `certs.agents` read, so a
revocation clicked in the browser bites on the next poll with no restart, and a revoked host is
refused **before any socket opens** — invariant 3's shape: a source the app knows it must not
talk to says so without a network round trip, and the poller's stale/last-good semantics do the
rest. `refusal_detail()` quotes the agent's own 403 sentence (`client certificate required` /
`client certificate revoked`) because those are two different operator actions, and the
certificate wording is used only for a dial where a certificate was presented — a 401/403 from a
token-mode host keeps the token/HTTP wording, or the operator is sent to `/certs` for a token
problem. `agent_tls_map()` builds one entry per configured host (the CA and client pair are
shared — the dashboard has one identity — only the revocation answer is per name, and an absent
entry is *active*, never-issued being a different fact from revoked); `build_hub` takes that map
and hands the **same** entry to the `host:` and `disks:` jobs of one host, so a revocation
fails both sources with one sentence and neither of them dials. `fetch_snapshot()` keeps
`http://` targets on the shared client with their bearer token, unchanged from before the phase,
so a rollout is per host.
`NutMonitor` owns per-UPS transition bookkeeping (its
self-test event takes its severity from `nut.self_test_outcome()` and not its own word list,
so a passed test is never an amber line in the feed while the card beside it reads green).
Its per-UPS tolerance for a failed `LIST CMD`/`LIST RW` is for a *complete* error reply only:
@@ -237,7 +291,10 @@ Responsibilities, one line each:
smartmontools or device read access is unconfigured, not broken (invariant 3); they stay
in `agent.errors` for the `/disks` page to explain.
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card
(`gpu.busy_pct@<index>`).
(`gpu.busy_pct@<index>`). `agent_tls_fields()` is the phase-15 passthrough: a TLS agent's
`tls`/`bundle_rev` ride into the payload's existing `agent` block, and a token-mode snapshot
grows nothing (no `tls: false`, no `bundle_rev: null` — rule 2), so the fields are how the API
distinguishes a stale bundle from an agent that never had one.
- `sources/disks.py` — the same snapshot projected onto **physical** drives for `/disks`
(phase 09): one row per whole-`/sys/block` disk (never a partition, never a mount), the
agent's `smart` block carried through verbatim, `health` the drive's own word normalised to
@@ -351,6 +408,50 @@ Responsibilities, one line each:
`DASH_SESSION_HOURS`. Its two raises, like the loader's, carry
fixed text — `tests/test_auth.py` pins the file's raise sites and its one logging call
structurally, so no future edit can interpolate a credential into either.
- `certs.py` — the fleet's certificate authority as the same shape of module (phase 15): a pure
lifecycle over the three-method `MetaStore` protocol, no clock it cannot be handed, no `logging`,
and one fixed-text `RuntimeError` (`tests/test_certs.py` pins all three structurally, the way
`tests/test_auth.py` pins `auth.py`). Five `meta` rows and nothing else:
`certs.root.key`/`certs.root.cert` (self-signed `CN=dashboards-root`, 10 years),
`certs.client.key`/`certs.client.cert` (`CN=dashboards`, 1 year — the identity every agent will
check), and `certs.agents`, one JSON object mapping a `HOST_TARGETS` name to
`{serial, not_after, status, bundle_rev, bundle, san}`. **A stored row always wins:** the root is
generated on the first boot of a database and never on a later one, because a CA that rotates
with the process untrusts every bundle already installed on a host — and a newly generated root
takes its client certificate with it, since a leaf signed by the dead root is a credential
nothing can verify. `issue_agent_cert`/`reissue_agent_cert` (the API's "rotate") sign a fresh
`CN=<host>` leaf whose SAN is the addresses that host is dialled by — the name is always in the
list, and an IP goes in as an `iPAddress` or hostname verification of an IP dial fails — bump
`bundle_rev`, and reset `status` to `active`. `revoke` marks the host and **re-seals every
bundle**, because the `# REVOKED` denylist lives inside each of them and an agent that cannot see
a serial keeps accepting that client; there is no "unrevoke", and revoking a never-issued host
still records the revocation (what a revoke enforces is the dashboard refusing to dial). The
**bundle** is the install artifact — agent cert, agent key, the root *cert* (never the root key,
asserted by header count and by parsing the one key against both keypairs), `# BUNDLE-REV <n>`,
then the denylist — stored verbatim so what an operator downloads is byte-identical to what the
row holds. `rotate_root` mints a new root plus a new client and reissues every agent
(`bundle_rev` +1 each, denylist reset with the generation); the running process keeps the
material it loaded at startup, which is the honest `restart_required` task 04 answers with.
Every certificate carries `SubjectKeyIdentifier` and every leaf an `AuthorityKeyIdentifier`
naming it, non-critical: OpenSSL's **strict** mode refuses a leaf with no AKI
(`CERTIFICATE_VERIFY_FAILED: Missing Authority Key Identifier`) and Python's
`ssl.create_default_context()` — the context the dashboard's own https dial builds — turns that
mode on. The agent's server-side check is not strict, so nothing on the agent side notices;
`tests/test_certs.py` pins the identifiers instead, and a bundle recorded without them is one a
strict client rejects. `write_material` puts the CA and the client pair in a `0700` temp dir with
`0600` files for the dialing — the root key is never written to disk, so its only copy stays the
`meta` row — and the lifespan removes the directory on shutdown. Unreadable PEM, a key that does
not match its certificate, or a leaf the stored root did not sign is a **fail-closed startup** (the
`ctx.auth` rule), and the message is the `sqlite3 "<db>" "DELETE FROM meta WHERE key LIKE
'certs.%'"` reset. `cryptography` is this module's dependency and the app's only new one;
`agent/host_agent.py` stays stdlib and loads the PEMs with `ssl`. There are **two reads** and the
difference is a side effect: `ensure_certs` is the startup one (it mints what the table is missing,
which is why it is what `app.build_certs` calls and nothing else may), and `read_state` is the
request-time one task 04's `GET /api/certs` uses — same validation, same fixed-text refusal, never
a write, so a page refresh cannot bootstrap an authority the operator deleted. The material is
read with (task 03): `collectors.agent_tls_map` turns `AppContext.certs` into the per-host dial
pair, and because the revocation answer is `agent_status()` called per poll, a revoke in this
module is what stops the dial — no restart, no cache, no second list to keep in sync.
- `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric,
`classify()` → `ok`/`warn`/`fail`/`None`, plus `attach()`/`with_severity()` to put the verdict
beside a value. No measurement, no verdict: `None` in → `None` out, never a green pass for
@@ -452,9 +553,33 @@ Responsibilities, one line each:
- `api.py` — GETs everywhere plus the one POST that dials an upstream, and the `auth_router()`
that creates, describes, **re-credentials**, and ends a session (`POST /api/auth/login` public;
`GET /api/auth/status`, `POST /api/auth/logout` and `POST /api/auth/password` protected). Those
three are the app's only other writes and they touch this process and its `meta` table and dial
nothing — invariant 1 is about the fleet, and `tests/test_api.py` pins the two lists separately
three, and the `certs_router()` of phase 15 (`GET /api/certs`, `GET /api/certs/<name>/bundle`,
`POST /api/certs/<name>/rotate`, `POST /api/certs/<name>/revoke`,
`POST /api/certs/root/rotate`), touch this process and its `meta` table and dial
nothing — invariant 1 is about the fleet, and `tests/test_api.py` pins the three lists separately
rather than widening the one that names `/api/ups/command`.
The **`/api/certs` surface** (phase 15 task 04) is the CA's own admin half. The list reads the
**stored** state through `certs.read_state` — the request-time twin of `ensure_certs`, which
mints and must only ever be called at startup, because a `GET` that bootstrapped a root CA would
replace the authority nobody asked it to replace — and its rows are the union of `HOST_TARGETS`
and `certs.agents`, so a configured-but-uninstalled host appears as `status: "unissued"` with
nulls (invariant 3, rule 2) and an entry whose host left the config is listed but not writable.
`deployed_bundle_rev` comes from that host's `system` payload `agent` block (task 03's
passthrough), never from what the CA hopes. `bundle_rev` is what was issued; the two numbers side
by side are the "install the new bundle" sentence, and neither carries a `severity` — an expiry
and an identity counter are not graded numbers. A `<name>` outside `HOST_TARGETS` is refused in
the route's first line with `404 {"configured": false}` **before** any signing, dial, or store
write (spy-pinned both ways, and the same gate makes `GET …/bundle` answer
`{"configured": true, "issued": false}` for a host that is configured and has nothing issued —
two different facts, two different bodies). The bundle download is the stored row **verbatim** as
`application/x-pem-file` with an `attachment` filename, because it is `install-agent.sh`'s input
and it carries that host's private key (session-gated, `no-store`). `POST /api/certs/root/rotate`
answers `restart_required: true` rather than pretending a hot-swap: the CA and the client pair are
startup material, so the table is new at once while the process keeps dialling with the old one.
Every write logs one fixed-text `certs`-source event naming host and action, and serials travel as
**decimal strings** — a 64-bit certificate serial through a JS number is a serial a page cannot
match against a bundle's `# REVOKED` line (invariant 6: no PEM reaches an event, a health record,
or any payload but the bundle route's).
The login route is where a *sign-in* password is compared: it
throttles before it hashes, refuses an absent/empty/non-object body with `422` so `""` never
reaches `verify_password`, answers every failure with the one sentence `wrong password`, and
@@ -490,8 +615,9 @@ Responsibilities, one line each:
no `disks` job in the hub at all (invariant 3), and `tests/test_api.py` proves it with
spies on `httpx.AsyncClient.get` and `collectors.fetch_snapshot`. The new surface grows no
write of its own — a fourth dashboard must not quietly become a second one, so the app's
only POST stays `/api/ups/command` (invariant 1, pinned by enumerating every served route's
verbs).
only POST that **dials an upstream** stays `/api/ups/command` (invariant 1, pinned by enumerating
every served route's verbs and sorting the writes into fleet / session / certificate-authority
lists — the third is phase 15's, and it writes `meta` rows and nothing else).
All four `history` routes reach the store through one door, `api._history_batch()` →
`store.history_many()` (phase 14 task 02). That is a cost change and nothing else: the series
list is host-major and metric-minor in request order, each entry the same
@@ -531,7 +657,16 @@ Responsibilities, one line each:
a `RuntimeError` from it leaves the process non-zero with the operator's fix in the output.
That failure is never caught here: a dashboard that cannot say who its admin is does not start
the loops either (`__main__.py` passes `SETTINGS` to `create_app()` explicitly so the served
app is configured by the same object the bind address came from).
app is configured by the same object the bind address came from). `build_certs(store)` is the
same rule for phase 15: one helper, called **before** `build_hub` in the owned-store
(`build_context`) and injected-store (lifespan) branches, whose `CertMaterial` (the CA + client
paths in a private temp dir) hangs off `AppContext.certs` and whose
directory the lifespan's `finally` removes. What hands that material to the loops is
`agent_tls_map(material, store, targets)` → `build_hub(..., tls=...)`, called in **both**
branches the same way: no material (or no targets) means no TLS map, which is the plain
`http://`/token path rather than an error, and one map covers both of a host's jobs. A corrupt stored certificate raises out of the same
place — a dashboard that cannot present its own client certificate would otherwise spend the
fleet's TLS history reporting an upstream problem it caused itself.
- `web/` — five dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`) plus the login page
(`/login`, `pages/login.html` + `static/page-login.js`), one stylesheet, `charts.js`. No
framework. `/login` renders the masthead's **brand link and theme toggle** and nothing else
@@ -555,7 +690,7 @@ Responsibilities, one line each:
sixth dashboard**: same `<head>` and masthead, the five tabs with **no** `aria-current` (it is
not one of them), no status dots, and it is reached from the `⚙ Account` link in
`.masthead-right` of every dashboard — outside `.tabs`, so the pinned tab structure is untouched
(`tests/test_api.py::test_every_dashboard_page_links_to_settings`). It prints
(`tests/test_api.py::test_every_page_links_to_both_admin_pages`). It prints
`/api/auth/status` (name, expiry as a local timestamp *and* as time left), signs out through
`POST /api/auth/logout`, and changes the password through `POST /api/auth/password` after
checking locally what it can judge itself (non-empty current, ≥ 8 new characters —
@@ -567,7 +702,30 @@ Responsibilities, one line each:
happened" call. So `app.js` exports **`resetAuth()`**: it clears the memoised `authStatus` and its
in-flight promise, and must be called after a password change (the secret rotated, so the cached
`csrf` is now invalid — every later write, the UPS buttons included, 403s until a reload) and on
any `401` bounce. `app.js`'s masthead status dots are one per dashboard job —
any `401` bounce.
`/certs` (`pages/certs.html` + `static/page-certs.js`) is the second of those admin pages, and it
is the same shape for the same reasons: five tabs with **no** `aria-current`, no `#nav-status`,
and the `⚷ Certs` masthead link beside `⚙ Account` (neither admin page links to itself, which is
why the pinned two-link audit loops over two different tuples). Everything on it is
`GET /api/certs`: the Root CA block, the display-only Dashboard client line (that certificate is
minted with the root and has no route of its own), and one row per host with its status chip,
serial, expiry (local timestamp *and* `Dash.duration` left), issued-vs-deployed `bundle_rev`,
`san` list, and the three controls. Two rules it exists to keep. (1) **No colour the API did not
send**: `/api/certs` publishes no `severity` — an expiry is a date and a revision count is not a
graded number — so the page has no `severity` token at all, the one chip class map is
`status` word → `ok|fail` with `unissued` mapping to *nothing* (a certificate that does not exist
never borrows green), and a behind bundle is a neutral `stale` chip. (2) **No control the API
would refuse**: a row with `configured: false` prints why (`not in HOST_TARGETS`) instead of
three buttons that all answer `404`, a never-issued host has no **Revoke** and no **bundle** link
(that route answers `issued: false`), and the download is a plain `GET` link because a session
cookie is all a signed `GET` needs. Writes are `confirm()`d, not typed-`DANGER`d — that gate is
for cutting power and this page cannot — and the outcome line prints the route's own `hint`, so
`restart_required`'s restart sentence lives in `api.py` and nowhere else. Its `COLUMNS` table
names every payload key it indexes, and `tests/test_api.py` resolves that table against a live
`GET /api/certs` row in both directions (the `SLOT_FIELDS` device), so a rename on either side
fails a test instead of printing `n/a` beside a healthy fleet. The two CNs are markup, and pinned
to `certs.ROOT_CN`/`certs.CLIENT_CN` the way `/settings` pins its password floor.
`app.js`'s masthead status dots are one per dashboard job —
including `disks`, because the same agent can answer the 10-second utilisation poll and
have a wedged SMART pass, and one dot for both would be right about neither. `app.js` owns the
colour vocabulary: `Dash.severityFrom(cellOrSeverity)` turns a payload verdict into an
@@ -634,6 +792,11 @@ That scheme is the history API's key — treat it as a public interface.
`/api/health` today reads **all nine sources `ok`**: `host:driveripper`, `host:deskwork`,
`host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `nut`, `llm:gufo`,
`llm:vllm`.
**Since phase 15 that bearer-token transport is the compatibility path, not the designed one:**
the agent can serve `https://` from an installed bundle and the dashboard dials it with its own
client certificate, but no bundle is installed on any host here, so `HOST_TARGETS` stays
`http://` + userinfo until somebody runs the rollout in README's *Protecting an agent, host by
host* (gap 11 says what that needs and why flipping the URL first turns a host red).
Phase 10's proof that the old drop was not this repo's firewall stands as history in
`data/logs/NOTES-verify.md` — do not re-add a tunnel on the strength of it, and re-run the
probe rather than trusting either note.
@@ -784,9 +947,9 @@ set -a; . ./.env.dev; set +a # dev dials the fleet the prod way, no
./scripts/dev-server.sh status # per-source ok/DOWN — unauthenticated: /api/health
./scripts/dev-server.sh logs
uv run pytest # hermetic: 1267 passed + 15 skipped (11 live,
uv run pytest # hermetic: 1427 passed + 15 skipped (11 live,
# 4 store_perf), no network
uv run pytest --cov # coverage gate (98.75%; floor 94%)
uv run pytest --cov # coverage gate (98.94%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
uv run pytest -m store_perf --store-perf # opt-in: the store hot-path timings over the seeded
# store, priced against `store_perf.BUDGETS` (the cached
@@ -804,6 +967,10 @@ uv run python scripts/store_perf_probe.py # ... as a report (data/logs/store-per
# store read-only, DASH_PERF_STORE_DIR moves the cache, and
# every pass that *writes* runs on a scratch copy so the
# seeded baseline never moves under the next measurement
# the bundle for one host: the stored bytes (that host's private key included) and
# `install-agent.sh --cert-bundle`'s only input — signed-in GET, see README's rollout
# curl -s -H "Authorization: Bearer $SESSION" -o /tmp/agent-<host>.bundle \
# http://127.0.0.1:8090/api/certs/<host>/bundle
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh # re-record ground truth (agents + UPS + gufo /slots)
SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh
@@ -820,11 +987,23 @@ bash .agents/validate.sh # the phase gate: the four checks abov
Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on `0.0.0.0:8090`,
SQLite under `/var/lib/dashboards` — put a volume there) and `deploy/install-agent.sh`
per monitored host (systemd unit, own user, `--bind`/`--port`/`--token`/`--rotate-token`/
`--status`/`--uninstall`; it stores the token `0640 root:dashboards-agent` — the unit hands the
agent the *path* and the agent opens it as its own user, so that user must be able to read it and
no other account may — never puts a token on argv, and runs under `ProtectSystem=strict` with
`--cert-bundle`/`--status`/`--uninstall`; it stores the token `0640 root:dashboards-agent` — the
unit hands the agent the *path* and the agent opens it as its own user, so that user must be able
to read it and no other account may, and a `--cert-bundle` file is installed to
`/etc/dashboards-agent.certs` at that same mode after a shape check that runs before anything is
touched and never dials the dashboard (the agent re-reads the bundle live for the denylist, so
0600 would fail the unit exactly like it fails the token file; the task file's `0600` is the
intent — *root and the agent's own user only* — and 0640 is the mode that achieves it) — never
puts a token on argv, and runs under `ProtectSystem=strict` with
`DevicePolicy=strict` reopening only the disk nodes plus the `/dev/null` family, because strict
without `/dev/null` kills the unit at spawn (`status=208/STDIN`). No root shell? The one sudoers
without `/dev/null` kills the unit at spawn (`status=208/STDIN`); a bundle-installed unit also
gets `--cert-bundle` plus `RuntimeDirectory=dashboards-agent` (mode 0700 — the bundle unpacks its
PEM blocks there and the agent deletes the key once the SSL context holds it), keeps the
`--token-file` line for a host-by-host rollout, and `--status` then probes `https` on the
bundle's CA with **no** client certificate via a `python3` `ssl` one-liner — task 02's
`CERT_OPTIONAL` + `/health` exemption is what makes that probeable, and there is no external TLS
binary. A re-run without the flag leaves an installed bundle in place, and `--uninstall` removes
it with the token. No root shell? The one sudoers
line that lets `ops` run the installer is `deploy/agent-sudoers`; run the script without root and
it prints that line with its own resolved path.)
The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other
@@ -927,6 +1106,21 @@ branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only
verbatim, the four config rows, the `DASH_COOKIE_SECURE=1` deployment note, the three auth
gaps). No server starts and no request is made: a doc that stops naming the reset command is a
locked-out admin, and that is worth a test even though it asserts nothing about Python.
- **A module that holds a private key is pinned structurally, not only behaviourally**
(phase 15). `tests/test_certs.py` reads `src/dashboards/certs.py` as text and fails if it grows a
`logging` import, a raise carrying an interpolated value, or an import from the rest of the
package — the same three pins `tests/test_auth.py` holds over `auth.py` — because this app turns
log lines and error strings into `event` rows and `/api/health` fields that a browser reads, and a
stored PEM is a key. Its behavioural half runs the whole lifecycle over a dict `MetaStore` (five
rows on first boot, no byte changed on the second, rotate/revoke/root-rotate and their denylists,
a bundle that carries the agent key and never the root key), and one test completes a real
client-certificate handshake through the material `write_material` lays down. The other two halves
run real code rather than a model: `tests/test_agent.py` drives the real `host_agent` handler over
a TLS-wrapped loopback port (the handshake matrix, the live denylist re-read, token mode unchanged)
and `tests/test_deploy_scripts.py` runs `install-agent.sh` in a sandbox whose `install`/`chown` are
faked, which is how a `0640` mode, a byte-identical bundle and the https `--status` probe are proven
without a root shell or a host. **A `0600` in a task file is not a fact about this repo**: the agent
opens its own bundle and re-reads it for the denylist, so the group bit is load-bearing.
- **A cost claim is pinned as a shape and a control, never as a wall clock** (phase 14). The default
run keeps the seeded-store tests hermetic and asserts **shape and determinism** — counts,
retention windows, the statements a plan is taken from, idempotence — while the timings live behind
@@ -1545,6 +1739,70 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
`/proc/devices` lists NVMe as `blkext`) — the template change is its own phase.
`tests/fixtures/smartctl_driveripper_*.txt` (captured via `sudo`) remain the ground truth
for the drives' shape.
11. **The CA is durable, the collector dials with it, and no bundle is installed with one**
(phase 15 tasks 01-05, landed 2026-10-20). `certs.py` mints the CA and the dashboard's client
certificate into `meta` on the first boot of a database — verified against the running dev
store: the five `certs.*` rows exist
and did not change across two restarts — signs per-host server certificates into
`certs.agents`, and does reissue, revocation, and root rotation. Task 02 gave
`agent/host_agent.py` its TLS mode (see the agent bullet above: one installed bundle, the
per-path client-cert check, the live `# REVOKED` denylist, `bundle_rev` in the snapshot) and
task 03 gave the collector the matching half: an `https://` `HOST_TARGETS` entry is fetched
with this app's client certificate and verified against its own root, and a host marked
`revoked` fails its `host:` and `disks:` jobs with `agent certificate revoked` **before any
socket opens** — spy-pinned in `tests/test_collectors.py` and again through the real
`build_hub` wiring in `tests/test_api.py`, where the revocation is written into the same `meta`
table the running app reads. So both halves are proven against each other with no network and
no hardware. What is still missing is an **installed** bundle: `HOST_TARGETS` is `http://` for
every host here and no agent runs `--cert-bundle`, so deleting `certs.*` costs nothing today
and everything once bundles are placed (which is what the reset message warns about). Proving
the dial also found one defect in task 01's output: the leaves carried no
`AuthorityKeyIdentifier`, and OpenSSL's strict mode — which is what Python's
`ssl.create_default_context()` turns on, and that is the context this dial builds — refuses
such a chain with `CERTIFICATE_VERIFY_FAILED: Missing Authority Key Identifier`. Fixed in
`certs.py` (SKI on the root and every leaf, AKI on every leaf, non-critical), pinned in
`tests/test_certs.py`, and the reason a bundle recorded before it must not be installed. Task
04 put the same lifecycle behind the gate: `/api/certs` lists the **stored** authority (through
`certs.read_state`, so a page refresh cannot mint anything) with one row per configured host and
per existing entry, `GET /api/certs/<name>/bundle` serves the stored bytes as an `attachment`,
and the three writes rotate, revoke, and rotate the root with `restart_required: true` said out
loud — each leaving one fixed-text `certs`-source event and no PEM outside `meta` (pinned by a
grep of the stored rows and of every payload a browser reads). Task 05 put the page on top of it:
`/certs` (`⚷ Certs` in every masthead, including `/settings`) renders that document — the root
CN/serial/expiry, the display-only client line, one row per host with its status chip and
issued-vs-deployed bundle rev and its **Rotate** / **Revoke** / **bundle** controls — and keeps
two rules the API already keeps: no colour the payload did not send (no `severity` token on the
page at all; `unissued` is a grey chip, a behind bundle a neutral `stale` one) and no control
the API would refuse (no buttons on a `configured: false` row, no revoke or download on a
never-issued host, the download a plain session `GET`). Writes are `confirm()`d rather than
typed-`DANGER`d, and the restart sentence is the route's own `hint`. Task 06 has since landed
the installer half: `install-agent.sh --cert-bundle <file>` validates the bundle by the
agent's own text shape *before* `need_root` (a bad file says "wrong file", not "add a sudoers
line"; exit 2), installs it `0640 root:dashboards-agent` byte-identically — the task file's
literal `0600` cannot work, because the agent opens the bundle as itself and re-reads the
denylist live (the exact incident the token file's comment records), and the task's own
intent is "only root and the agent's own service user may read it", which is what 0640
achieves — re-runs without the flag leave the installed bundle in place, the unit gains
`--cert-bundle` + `RuntimeDirectory=dashboards-agent` (`0700`) from one conditional branch
while keeping the token line, `--status` probes `https` with the bundle's CA and no client
certificate (`python3` `ssl`, never an external TLS binary; an unreadable bundle says *run
--status as root* instead of pretending), `--uninstall` sweeps the bundle with the token, and
`deploy/agent-sudoers` gained no line (the bundle travels through the same root invocation).
`tests/test_deploy_scripts.py` pins all of it, including the probe run end-to-end against a
real in-process TLS agent: CA-only reads `/health`, a foreign root is refused, token mode
keeps the plain probe. The final pass proved the composition rather than the halves: a real
`host_agent.py --cert-bundle` on loopback and a real dashboard process over a scratch database
(`HOST_TARGETS=loop=https://127.0.0.1:<port>`) answer 27 checks — `host:loop`/`disks:loop` `ok`
through the certificate dial, `agent.tls`/`agent.bundle_rev` in both snapshots, the bundle
download byte-identical to the `meta` row, an unsigned `/certs` a `303` with `?next=`, a revoke
followed by `agent certificate revoked` on both jobs with no dial, the dashboard's own client
identity untouched by a *host* revoke (the list is serials, not hosts), and a revoked **client**
serial refused 403 by the running agent on `/health` too the moment the re-sealed bundle is in
place (`data/logs/verify-15-tls-e2e.py`, transcript `verify-15-tls-e2e.out`). What is left of
the phase is the live rollout (07): the fleet's agents still run token mode, so `HOST_TARGETS`
stays `http://` here until somebody installs a bundle on a host — which needs root on that host
(`ops` has no installer grant on `deskwork`, and `driveripper`'s unit must not be rewritten, gap
10), and flipping the URL before the bundle is installed turns that host red by construction.
## Locked decisions (and why)
@@ -1560,6 +1818,10 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
explicit gating.
- **In-house Prometheus parser.** Only two upstreams use the format; keeps `prometheus_client`
out of the dependency set.
- **`cryptography` for X.509, dashboard-side only** (phase 15). Minting a CA and signing leaves is
not something to hand-roll, and there is exactly one place in this app that does it
(`certs.py`). The agent gets no new dependency: it loads the PEMs the dashboard signed with stdlib
`ssl`, which is why the bundle is plain text and why the agent still runs on a host's Python 3.10.
- **Port 8090**, inherited from the `llm_monitor` prototype this started from: 8000 is vLLM
and 8731 is gufo in this fleet.
- **`.env` is a seed, never an override** — container/systemd env must win.