phase: 15_agent_mtls
All green. Report:
**Verified / implemented (final pass of `15_agent_mtls`)**
- Wrote `data/logs/verify-15-tls-e2e.py`: a real `host_agent.py --cert-bundle` + a real dashboard process over a scratch DB (`HOST_TARGETS=loop=https://…`) — **27/27 PASS**, transcript `verify-15-tls-e2e.out`, notes `data/logs/NOTES-tls-rollout.md`.
- Read-only checks against the running dev store: 5 `certs.*` rows byte-identical across a restart; served bundle == the `meta` row (1430 B); 0 events containing `-----BEGIN`; unsigned `/certs`→`303 ?next=`, unsigned `/api/certs`→`401`, signed-in `200`; screenshots `data/logs/15-final-{certs,system}.png`.
- Docs (step 6, full review): README — added *Protecting an agent, host by host* (the rollout task-07 left out), a `tests/test_certs.py`/TLS-matrix/sandbox-installer paragraph, and fixed two stale “the one write endpoint in the app” claims (`NUT_ALLOW_ACTIONS`, Battery-test buttons). AGENTS.md — test count 1424→1427, bundle-fetch command in *Commands*, a Testing-policy bullet on structurally pinning a key-holding module, fleet transport bullet (token mode is now the compatibility path), gap 11 final state.
**Tests / lint (full suite 100% green)**
- `uv run pytest` → **1427 passed, 15 skipped**, exit 0 · `bash .agents/validate.sh` → **exit 0** (coverage **98.94%** vs 94% floor, `certs.py` 100%; ruff “All checks passed”; pyright 0 errors; `node --check` clean; hygiene clean)
- `uv run pytest -m live --live` → **11 passed / 0 failed** · `./scripts/dev-server.sh status` → all 9 sources `ok`
- `uv pip list --outdated` → empty after bumping `fastapi 0.152.0 → 0.153.0` (`uv.lock` only); full suite re-run green.
**Completion criteria**
- Fresh-DB root CA, idempotent: ✅ (tests + live store hashes) · agent TLS matrix incl. wrong-CA/certless 403/`/health` CA-only/revoked 403 + live denylist re-read: ✅
- `https` `host:`/`disks:` `ok` + revoked says “agent certificate revoked” with no dial: ✅ on loopback (real agent + real dashboard) and spy-pinned hermetically; **not** on the fleet (below)
- Signed-in `/certs` + rotate/revoke/root-rotate + byte-identical bundle download: ✅ · installer bundle/unit/`--status` over TLS: ✅ (0640, see deviation)
- `pytest`/`validate.sh`/`--outdated`: ✅ · token-mode byte-identical, engine/UPS untouched, public surface unchanged (route walk): ✅
- **Live fleet TLS (deskwork+gateway flip): NOT done** — no live changes. It needs root on hosts without the installer grant (`deskwork`), has no both-ways-working window (`http://` dashboard vs https-only agent), and would deploy the agent from this not-yet-committed tree. Exact procedure recorded in README + `data/logs/NOTES-tls-rollout.md`.
**Deviations**: bundle mode `0640 root:dashboards-agent` instead of the task file’s `0600` (the agent re-reads its own bundle for the denylist; 0600 breaks the unit — already recorded in AGENTS.md); the loopback dashboard substitutes for the fleet flip. Incidental: a stray `pkill` of mine killed the dev server mid-pass; restarted and verified healthy.
**Next pending phase**: `.agents/phases/todo/16_agent_smart_pass_budget` (then `17_installer_device_filter`).
This commit is contained in:
@@ -0,0 +1,44 @@
|
||||
# Phase 15 — Protected exporter web servers, auth by certificate
|
||||
|
||||
**Source:** `TODO.md` L3 — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||
**Story:** n/a
|
||||
**Context:** `agent/host_agent.py` (the stdlib-only exporter web server; today: a bearer token via `--token`/`--token-file`, refuses an off-loopback bind without one, `ThreadingHTTPServer`, the `resolve_token` idiom), `src/dashboards/config.py` (`Target`/`parse_targets` — token in URL userinfo, `https://` URLs already accepted; **no change needed**), `src/dashboards/collectors.py` (`fetch_snapshot`, the one `UpstreamError`, redacted display URLs, the one shared `httpx.AsyncClient` from the lifespan), `src/dashboards/app.py` (the `PAGES` dict with the settings-page comment, `build_hub`/`build_context`, the lifespan that builds `ctx.auth`), `src/dashboards/api.py` (`AppContext`, the gated-write + event pattern), `src/dashboards/store.py` (`get_meta`/`set_meta`/`delete_meta` — the only durable key/value state, home of `auth.*`), `deploy/install-agent.sh` (the token-file idiom `0640 root:agent-user`, the `ProtectSystem=strict` unit with no `RuntimeDirectory`, the `--status` probe), `tests/test_agent.py` (the in-process agent harness), `tests/test_collectors.py`, `tests/test_deploy_scripts.py`, `tests/test_api.py` (the route walk, the settings-page pin, the orphan-id audit), `src/dashboards/web/pages/settings.html` + `src/dashboards/web/static/page-settings.js` (the "a page, not a dashboard" pattern the new `/certs` admin page follows)
|
||||
|
||||
## Objective
|
||||
|
||||
Every exporter (host agent) web server is protected end-to-end by certificates the dashboard itself generates and manages: the dashboard holds a root CA in the `meta` table, signs its own client cert and each agent's server cert, the agent serves TLS with a per-path client-cert check (only the dashboard can read it), the dashboard dials `https://` with its client cert, and a new `/certs` admin page lists every cert with its expiry and status and triggers rotation, revocation, and bundle re-delivery. The operator's stated follow-up — firewalling upsd 3051 and talking to the agent only — is the motivation, not part of this phase. The agent's bearer-token mode stays a compatibility mode; the dashboard's own login is a separate auth domain and is untouched.
|
||||
|
||||
**Scope (operator-confirmed):** "exporters" = the host agents this repo installs. gufo / vLLM / upsd are third-party services this repo does not deploy; their TLS (e.g. upsd `STARTTLS`) is a separate operator concern.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `14_store_performance` (todo) — **shared-file, run-order note**: 14 rewrites the hot paths in `store.py`/`api.py`; 15 only *adds* meta keys and new routes/pages. Apply on top of the file state at the time it runs; never restore a line the other phase owns.
|
||||
- `11_admin_login` (complete) — the session gate (invariant 12) for the new routes, the gated-write + CSRF + event pattern for the new `POST /api/certs/*`, the `meta` helpers, and the settings-page pattern `/certs` follows.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_cert_management.md` — `src/dashboards/certs.py`: the dashboard-managed root CA, client cert, per-agent server certs, revocation and reissue, all in `meta`; the `cryptography` dependency (dashboard-side only).
|
||||
2. `02_agent_tls.md` — `host_agent.py` TLS mode: one installed `--cert-bundle`, the per-path client-cert check, the bundle revocation list, `bundle_rev` in the snapshot; token mode unchanged.
|
||||
3. `03_collector_tls.md` — the dashboard dials `https://` agent URLs with its client cert, verifies against its root, refuses a revoked agent without dialing; error sentences stay redacted.
|
||||
4. `04_cert_api.md` — the gated `/api/certs` surface: list, per-host bundle, rotate, revoke, root-rotate; fixed-text events, no keys out of `meta`.
|
||||
5. `05_certs_admin_page.md` — the `/certs` admin page (the settings pattern): every cert with its expiry and status, issued vs deployed bundle rev, rotate/revoke/root-rotate actions, bundle download.
|
||||
6. `06_deploy_cert_bundle.md` — `install-agent.sh --cert-bundle` installs `/etc/dashboards-agent.certs` (`0600 root:agent-user`), the unit gains the flag + `RuntimeDirectory`, `--status` probes over TLS; the script pins move with it.
|
||||
7. `07_live_rollout.md` — the two live agents flip to TLS, `HOST_TARGETS` goes `https://`, the live suite + health table + screenshot prove it; the `driveripper` install procedure is documented.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: `tests/test_certs.py` (new) — the whole lifecycle against a dict `MetaStore`, exactly how `tests/test_auth.py` does it.
|
||||
- Integration: `tests/test_agent.py` (the TLS handshake matrix against the in-process agent), `tests/test_collectors.py` (the `https` dial, the revoked refusal with a no-dial spy, the error sentences), `tests/test_api.py` (the `/api/certs` shapes, the gate/CSRF on the new writes, the served `/certs` page pins), `tests/test_deploy_scripts.py` (the new flags/files/permissions).
|
||||
- Live: task 07 (AGENTS.md invariant 10 — the dev server stays up; screenshots in `data/logs/`).
|
||||
- Coverage: **>90%** on new/modified code; the 94% gate holds. `uv pip list --outdated` empty (the `cryptography` dependency — invariant 8).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] a fresh database boots with a `meta` root CA, generated once and idempotent across restarts (pinned in `tests/test_certs.py`)
|
||||
- [ ] an agent run with a bundle answers `/snapshot` only to a client cert signed by its root; a wrong-CA or certless client is refused; a revoked serial gets the 403; `/health` stays probeable with the CA only
|
||||
- [ ] with `https://` `HOST_TARGETS`, the dev server's `host:`/`disks:` jobs are `ok`, and a revoked host says "certificate revoked" without any dial (spy-pinned)
|
||||
- [ ] signed-in `GET /certs` lists every cert with expiry and status; rotate/revoke/root-rotate work through the gated API; the bundle download is byte-identical to the stored bundle
|
||||
- [ ] the installer places the bundle `0600 root:agent-user`, the unit comes up in TLS mode, `--status` is green over TLS
|
||||
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green, `uv pip list --outdated` empty
|
||||
- [ ] live: deskwork + gateway in TLS mode, `https://` dials, live suite green, screenshots in `data/logs/`
|
||||
- [ ] no behaviour change in completed phases: the token-mode agent is byte-identical, the engine/UPS polling is untouched, the public surface is unchanged (the route walk)
|
||||
@@ -0,0 +1,43 @@
|
||||
# Task 01 — The dashboard owns a root CA and every cert it signs
|
||||
|
||||
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||
|
||||
## Objective
|
||||
|
||||
The certificate lifecycle becomes a pure, testable module: the dashboard generates a root CA once (idempotent across restarts), signs its own client cert and each agent's server cert, and supports reissue and revocation — all state in the `meta` table, exactly how `auth.py` keeps the verifier and the session secret.
|
||||
|
||||
## Work
|
||||
|
||||
1. `pyproject.toml` — add `cryptography` to `dependencies` (dashboard-side only; the agent stays stdlib-only) and run `uv lock`. Invariant 8: `uv pip list --outdated` empty afterwards.
|
||||
- ASSUMPTION: `cryptography` is the only new dependency and it stays on the dashboard side; the agent (task 02) gains no third-party code — it loads the PEMs with stdlib `ssl`.
|
||||
2. `src/dashboards/certs.py` (new) — the whole lifecycle as pure functions over the three-method `MetaStore` protocol (the same protocol `auth.load_auth_state` uses, so this module imports nothing else from the package, and its tests use a dict):
|
||||
- `ensure_certs(store, *, now: float | None = None)` → a frozen `CertsState` carrying the PEM text (`root_key`, `root_cert`, `client_key`, `client_cert`) plus the root/client `serial` + `not_after`. Reads the `meta` rows first; only generates and persists when missing (first boot); an existing row wins and is **never regenerated on restart** — regenerating on every start would rotate the CA and invalidate every installed agent bundle. `now` is injected (the module never reads a clock itself — the `auth.py` rule).
|
||||
- meta keys (one concern per key, the `auth.*` precedent): `certs.root.key`, `certs.root.cert`, `certs.client.key`, `certs.client.cert`, `certs.agents` (one JSON object mapping `host name` → `{serial, not_after, status, bundle_rev, bundle, san}` — the bundle text is stored, so what the agent holds is byte-identical to what the API serves).
|
||||
- Shapes: root CA self-signed, CN `dashboards-root`, **10 years**; client cert signed by the root, CN `dashboards`, **1 year**; agent server certs signed by the root, CN = the host name, **1 year**, `SAN` = the address(es) the dashboard dials for that host.
|
||||
- ASSUMPTION: 10 y root / 1 y leaf lifetimes. No env seed for the root CA (unlike the admin password, the root key is a secret only the app needs to know, so it is auto-generated on first boot and merely persisted).
|
||||
- `issue_agent_cert(store, state, name, san: list[str])` / `reissue_agent_cert(store, state, name, san)` — a fresh serial, `bundle_rev` +1 (1 on first issue), `status: "active"`, a fresh `bundle` (below). Idempotent per call — never issued on a read.
|
||||
- `rotate_root(store, state)` — a fresh root CA + fresh client cert + every existing agent entry reissued (each `bundle_rev` +1): one action re-arms the whole fleet.
|
||||
- `revoke(store, state, name)` — `status: "revoked"` (the serial stays on the denylist until a reissue gives the host a fresh identity; there is no "unrevoke" — reissue is the fix).
|
||||
- the **bundle** (the install artifact, the TODO's "root certificate installed with each exporter"): one PEM text = the agent's server cert + the agent's key + the root CA *cert* (never the root key) + a `# BUNDLE-REV <n>` line + a `# REVOKED <serial>` line (one per revoked serial of the *current* root generation — a root rotation resets the list, since every cert is reissued anyway).
|
||||
- ASSUMPTION: one client cert for the whole dashboard (the agent verifies "signed by our root", not a per-process identity); one agent entry per `HOST_TARGETS` name.
|
||||
- invariant 6, the `auth.py` rule: no `logging` and no raise with anything interpolated — a malformed stored PEM is a fixed-text `RuntimeError` telling the operator to delete the `certs.*` meta keys (the shape of the auth reset), and nothing else.
|
||||
3. `src/dashboards/app.py` (+ the `AppContext` in `src/dashboards/api.py`) — the startup path builds `ctx.certs` (an `ensure_certs` read, the one-time bootstrap included) **before** `build_hub`, in both the owned-store (`build_context`) and the injected-store (lifespan) branches, through **one** helper so the two wiring sites cannot drift (the `build_hub` rule in AGENTS.md); the PEMs are written once to a private temp dir (mode `0600`, lives as long as the process) and the *paths* are what `AppContext.certs` carries for task 03's dialing.
|
||||
- A `certs` whose stored PEM is malformed (a truncated `meta` row) is a fail-closed startup — a dashboard that cannot present its own credential must not start the loops (the `ctx.auth` rule, the one deliberate raise).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- `tests/test_certs.py` (new) — a dict `MetaStore` (the `tests/test_auth.py` idiom):
|
||||
- the first `ensure_certs` call persists exactly the five `certs.*` keys; the second is byte-identical (no regeneration — pin the `certs.root.cert` value);
|
||||
- the parsed chain (`cryptography.x509` loaders) verifies: client → root, agent → root; the SAN list appears in the agent cert; the expiries match the assumed lifetimes (injected `now`);
|
||||
- `reissue_agent_cert` bumps `serial` and `bundle_rev` and sets `status` active; `revoke` flips the status and puts the serial on the bundle's `# REVOKED` line; a reissue after a revoke gives a fresh serial that is *not* on the list;
|
||||
- `rotate_root` reissues every agent entry (each `bundle_rev` bumped), the bundle carries the new root cert, and no stored value still contains the old root key;
|
||||
- the bundle contains the agent cert + key + root cert + `# BUNDLE-REV` + the `# REVOKED` lines and **never** the root key (assert via the PEM header and a parsed-key mismatch);
|
||||
- a malformed stored PEM → the fixed-text `RuntimeError` naming the meta keys, nothing interpolated.
|
||||
- Coverage: **>90%** on this task's new code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest tests/test_certs.py` green; the full suite green
|
||||
- [ ] `uv pip list --outdated` empty; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||
- [ ] a fresh store boots with the `certs.*` rows present; a second boot changes no byte of `certs.root.cert`
|
||||
- [ ] no behaviour change in completed work: `auth.*` keys, the public surface, and the poller wiring are untouched
|
||||
@@ -0,0 +1,37 @@
|
||||
# Task 02 — The agent serves a protected TLS web server from one installed bundle
|
||||
|
||||
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||
|
||||
## Objective
|
||||
|
||||
`host_agent.py` gains a TLS mode that makes the web server protected by certificate: one installed bundle holds the server cert + key, the root CA, a bundle revision, and a revocation list; the agent requires a client cert signed by the root on every path except `/health` (mirroring the token-mode `/health` exemption, so an installer probe can reach `/health` with the CA only), and it reports the bundle revision it loaded so the dashboard can tell a stale bundle from a fresh one. The token mode is byte-identical to today.
|
||||
|
||||
## Work
|
||||
|
||||
1. `agent/host_agent.py` — stdlib only (an `ssl` import; no new third-party dependency):
|
||||
- a new flag `--cert-bundle <file>` (env `AGENT_CERT_BUNDLE`): one PEM text containing the server cert, the server key, the root CA cert, a `# BUNDLE-REV <n>` line, and `# REVOKED <serial>` lines. `resolve_bundle(path) -> (context, bundle_rev, revoked: frozenset[str], error)` is a pure function (the `resolve_token` idiom): parse the PEM blocks by text, extract the three blocks, write them to a temp dir for `load_cert_chain` (into `/run/dashboards-agent` when present and writable — the unit's `RuntimeDirectory`, task 06 — else `tempfile.mkdtemp`), and build the context.
|
||||
- The TLS context: `ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)`, `load_cert_chain`, `load_verify_locations(ca)`, `verify_mode = ssl.CERT_OPTIONAL` — the handshake is optional and the **handler** enforces it: on every path except `/health`, `self.connection.getpeercert()` must carry a verified certificate (subject present), else `403 {"ok": false, "detail": "client certificate required"}`; that cert's serial must not be in the bundle's `# REVOKED` set, else `403 {"ok": false, "detail": "client certificate revoked"}`. The denylist file is re-read on mtime change per request (a small file; a redeployed bundle takes effect without a restart — the reason the `# REVOKED` lines live in the bundle at all).
|
||||
- The startup gate: off-loopback with neither a token nor a bundle → the existing exit-2 refusal (its message now names the bundle too); a bundle present → the off-loopback bind is allowed (the client cert is the protection); a token plus a bundle → the token check is skipped on the TLS path (the handshake is the auth).
|
||||
- The snapshot payload gains two **additive** top-level fields in TLS mode: `tls: true` and `bundle_rev: <n>` (absent in token mode — an additive field never changes an existing shape); the `serve()` start line says `tls (client cert required, rev <n>)`.
|
||||
- `AGENT_VERSION` 1.0.0 → 1.1.0; the docstring's deployment paragraph gains one line for the TLS mode (no flavour text — the AGENTS.md audit rule).
|
||||
- ASSUMPTION: `CERT_OPTIONAL` + the per-path check (not `CERT_REQUIRED`) is what keeps `/health` probeable with the CA only — the installer's `--status` and the systemd probe (task 06) cannot present a client cert the installer does not hold, and a probe that cannot health-check is a dead install.
|
||||
- ASSUMPTION: the agent never generates or renews certificates (stdlib cannot mint x509); the bundle is the artifact the dashboard signs and the installer places.
|
||||
2. `tests/test_agent.py` — the existing in-process harness (`serve()` in a thread on `127.0.0.1`, `urllib`/`ssl` clients), with a test CA + leafs generated by `src/dashboards/certs.py` over a dict store (no second generator):
|
||||
- a client with a matching client cert → `/snapshot` 200 with `tls: true` and `bundle_rev` equal to the bundle's;
|
||||
- a client with no cert → the handshake fails or a 403 on a non-`/health` path; `/health` still 200 with a CA-only context (the probe path);
|
||||
- a client signed by a *different* CA → refused;
|
||||
- a client whose serial is in the bundle's `# REVOKED` list → 403 "revoked"; rewrite the bundle file without the line (mtime change) → the same client is accepted on the next request (the denylist re-read);
|
||||
- a token-mode agent (no bundle): the existing tests unchanged and green (byte-identical behaviour — no `tls`/`bundle_rev` fields);
|
||||
- `resolve_bundle` error paths: a missing file, an empty file, a bundle missing the key block, a bundle missing the `# BUNDLE-REV` line (an error — a revless bundle is not one this app signed; ASSUMPTION: the `# BUNDLE-REV` line is mandatory).
|
||||
3. `deploy/` is untouched here (task 06); the non-systemd fallback (`tempfile.mkdtemp`) keeps dev and tests working.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- the `tests/test_agent.py` matrix above; the existing token-mode tests stay green unchanged.
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] a TLS-mode agent on a loopback port answers the full handshake matrix (tests) and reports `bundle_rev`
|
||||
- [ ] a token-mode agent is byte-identical to today (existing tests green, no new fields present)
|
||||
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||
@@ -0,0 +1,35 @@
|
||||
# Task 03 — The dashboard dials agents over https with its own client cert
|
||||
|
||||
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||
|
||||
## Objective
|
||||
|
||||
An `https://` `HOST_TARGETS` entry is now a certificate-authenticated dial: the dashboard presents the client cert task 01 gives it, verifies the agent against its own root CA, refuses a *revoked* agent before dialing (invariant 3's shape — a source it knows is dead says so without a network), and passes the agent's `tls`/`bundle_rev` through the `system` and `disks` payloads so task 05's page can compare the issued rev against the deployed one. Engines and token-mode agents are untouched.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/config.py` — **no change** (an `https://` URL already parses; a TLS host carries no userinfo, and the userinfo token stays available for token-mode hosts).
|
||||
2. `src/dashboards/collectors.py` —
|
||||
- a small frozen `AgentTls` dataclass: `verify: str` (the CA path), `cert: tuple[str, str]` (the client cert + key paths), `status: Callable[[], str]` — the `status` callable returns `"active"`/`"revoked"` from a **fresh** `get_meta("certs.agents")` read (a small-table `SELECT`, cheap at poll cadence, and phase 14's work keeps it off the hot paths; reading it live is what makes a runtime revocation take effect on the next poll without a restart).
|
||||
- `fetch_snapshot(client, target, *, timeout=None, tls: AgentTls | None = None)`: when `tls` is given **and** `target.url` is `https://`, the request passes `verify=<ca path>` and `cert=(cert path, key path)` to `client.get` (httpx per-request overrides — the shared client dials both plain and TLS upstreams, so a per-client `verify` would poison the engine dials).
|
||||
- Revocation before any network: `tls is not None` and `tls.status() == "revoked"` → `UpstreamError(f"{target.name}: agent certificate revoked")` **before** `client.get` — the poller's stale/last-good semantics then do the rest (invariant 4).
|
||||
- A `403` from an agent gets its own honest sentence (the existing `401` branch keeps the token-mode wording): `f"{target.name}: agent refused the client certificate ({detail})"` with the `detail` from the agent's JSON (`client certificate required` / `client certificate revoked`), the redacted `display_url` (invariant 6) like every message in this function.
|
||||
- `fetch_host_snapshot`/`fetch_disk_snapshot` pass the `tls` through; the per-host `AgentTls` is plumbed from `AppContext.certs` into the `host_jobs`/`disks_jobs` lambdas (a new optional `tls` parameter on both `*_jobs`, `build_hub` takes it — one call site per branch so the production and injected wiring cannot drift, the `build_hub` rule in AGENTS.md). `status` reads from the store the jobs already hold.
|
||||
3. `src/dashboards/app.py` (+ the `AppContext` in `api.py`) — `AppContext.certs` (task 01) exposes the temp-dir paths (`ca`, `client_cert`, `client_key`); `build_context` and the lifespan's injected branch build the per-host `AgentTls` objects **before** `build_hub`.
|
||||
4. `src/dashboards/sources/host.py` + `src/dashboards/sources/disks.py` — pass the agent's `tls`/`bundle_rev` snapshot fields through into the per-host payload (additive, the `degraded` idiom) so `/api/system/snapshot` and `/api/disks/snapshot` carry the deployed rev. **No** `severity` for a rev mismatch — a count is not a graded number (the fleet-header rule), the page colours nothing for it.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- `tests/test_collectors.py` — an in-process TLS agent (task 02's harness, in a thread) + `AgentTls` material generated by `certs.ensure_certs` over a dict store:
|
||||
- an `https` target with `tls` → the 200 body; a missing client-cert file and a wrong-CA failure still surface as `UpstreamError` with the redacted URL;
|
||||
- a revoked status (dict store with `status: "revoked"`) → the error is raised and a spy on `client.get` (the `tests/test_api.py` spy idiom) proves **no dial happened**;
|
||||
- an `http` token-mode target: the existing tests unchanged;
|
||||
- the 403 sentence: a scripted agent 403 (a `tests/conftest.py` fake) → the `detail` appears in the message, the URL redacted;
|
||||
- the payload passthrough: a snapshot with `tls: true, bundle_rev: 3` → both the `system` and the `disks` payloads carry it.
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest` green; the engine/UPS/token-host paths are byte-identical (existing pins hold)
|
||||
- [ ] a revoked host fails its `host:` and `disks:` jobs with "certificate revoked" and no network dial (spy-pinned)
|
||||
- [ ] `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||
@@ -0,0 +1,36 @@
|
||||
# Task 04 — The gated /api/certs surface: list, bundle, rotate, revoke, root-rotate
|
||||
|
||||
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||
|
||||
## Objective
|
||||
|
||||
The API half of "the collector manages the certs": a signed-in operator can list every cert with its expiry and status, download the exact bundle an agent should hold, and trigger rotation / revocation / root rotation. The routes dial nothing (invariant 1's shape), are gated like every other write (invariant 12), and write fixed-text events — a key never leaves `meta` (invariant 6).
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/api.py` — the new routes (the middleware gates them by default; the route-walk test covers them for free):
|
||||
- `GET /api/certs` — `{root: {serial, not_after}, client: {serial, not_after}, agents: [{name, serial, not_after, status, bundle_rev, deployed_bundle_rev, san}]}`. `deployed_bundle_rev` is the `bundle_rev` from the host's last `system` job payload (the same source `alerts.py` reads job states from); a host that never answered → `null`. Reads never issue (no side effects on a read — a `GET` that mints a cert is a surprise).
|
||||
- `GET /api/certs/<name>/bundle` — the stored bundle text, `Content-Type: application/x-pem-file`, `Content-Disposition: attachment; filename="agent-<name>.bundle"` — byte-identical to what `certs.py` stored (the installer's input, task 06/07).
|
||||
- `POST /api/certs/<name>/rotate` — a reissue (`certs.reissue_agent_cert`, the SANs taken from the configured `HOST_TARGETS` URL host(s) + the name); a `cert-rotated` event (a fixed message naming the host — no PEM, no key).
|
||||
- `POST /api/certs/<name>/revoke` — `certs.revoke`; a `cert-revoked` event. The next poll refuses without dialing (task 03) — that is the enforcement; the event is the audit line.
|
||||
- `POST /api/certs/root/rotate` — `certs.rotate_root`; a `cert-root-rotated` event; the response carries `{"ok": true, "restart_required": true}` — the client cert and CA are startup material, so a new root is fully in force at the next process start (the page, task 05, prints that; the honest constraint, not a hidden one).
|
||||
- ASSUMPTION: a root rotation requires a dashboard restart to be fully in force (the client cert + CA load at startup); the API says so instead of pretending a hot-swap that does not exist.
|
||||
- A `<name>` that is not a configured `HOST_TARGETS` name → `404 {"configured": false}` **without** issuing, dialing, or touching the store (invariant 3, the `/api/disks` gate shape, spy-pinned in `tests/test_api.py`).
|
||||
- Every `POST` is a gated write: session + `Origin`/`Sec-Fetch-Site` + the cookie's `X-CSRF-Token` (the `POST /api/ups/command` test idiom — a tokenless write records nothing on the wire, with the token it does).
|
||||
- invariant 6: no `meta` value (a key PEM) may appear in an `event` row, a `/api/health` field, or an error `detail`; the events carry the host name + the action in fixed text.
|
||||
2. `tests/test_api.py` — the signed-in harness (`tests/auth_client.py::sign_in`):
|
||||
- `GET /api/certs` with zero / one / three configured hosts: the key paths pinned (the `/api/disks` pin idiom — a panel that retires can never shrink the payload);
|
||||
- an unknown `<name>`: the `404` and the spies on `httpx.AsyncClient.get` and `collectors.fetch_snapshot` (the `tests/test_api.py` spy idiom) prove no network and no store write (a `meta` row count before/after);
|
||||
- `POST rotate` → `bundle_rev` +1, a `cert-rotated` event row, the `meta` value changed, and the old serial is not on the new bundle's denylist;
|
||||
- `POST revoke` → the status, the event, and (through task 03's path) the next `fetch_snapshot` for that host refuses without a dial (the spy);
|
||||
- `POST root/rotate` → a fresh root serial, every agent reissued, `restart_required: true` in the body;
|
||||
- `GET bundle` → the body byte-identical to the stored `meta` bundle, and a reissue changes the served bytes;
|
||||
- the unsigned gate: the route-walk test (existing, automatic) plus one explicit unsigned `303`/`401` on `GET /api/certs` and a `POST` refusal (the `tests/test_auth_api.py` idiom).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] the routes are signed-in + CSRF-gated (the wire test); the unsigned shape pinned
|
||||
- [ ] an unknown name 404s with no dial and no store write (spy-pinned)
|
||||
- [ ] rotate / revoke / root-rotate each leave a fixed-text event row and no key in the `event`/`health` tables (a test greps the stored events for a PEM header and finds none)
|
||||
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||
@@ -0,0 +1,35 @@
|
||||
# Task 05 — The /certs admin page: every cert, its expiry, rotate, revoke, re-deliver
|
||||
|
||||
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter." (operator addition: an admin page to see generated certs and expirations, trigger rotation, and revoke certs)
|
||||
|
||||
## Objective
|
||||
|
||||
A signed-in operator sees every cert the dashboard has generated — the root CA, its own client, and one per host — with its expiry and status, can rotate / revoke / root-rotate through task 04's API, and can download the exact bundle an agent should hold. The page follows `settings.html`'s precedent exactly: a page and not a dashboard — in `PAGES` (so it is a route), not in `.tabs`, linked from `.masthead-right`, with no `aria-current` on the tabs and no status dots.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/web/pages/certs.html` + `src/dashboards/web/static/page-certs.js` (the `page-settings.js` idiom: the same `<head>` + masthead, `Dash.fetchJSON`, errors printed through `textContent`, a `confirm()` for destructive actions — the typed `DANGER` word is reserved for power cuts):
|
||||
- the **Root CA** block: the CN, the serial, the expiry (a local timestamp **and** the time left, the settings session-display idiom), and the **Rotate root CA** button → `POST /api/certs/root/rotate` with a confirm dialog; the response's `restart_required` prints "restart the dashboard to activate the new root" (the API's own sentence, not an invented one).
|
||||
- the **Agents** table: one row per configured `HOST_TARGETS` host (the `GET /api/certs` `agents[]`): the name, the serial, the `status` chip (`active` → `ok`, `revoked` → `fail` — a string-to-class map, the page never compares a number), the expiry (local timestamp + time left), the **issued** `bundle_rev` against the **deployed** `bundle_rev` (`n/a` when the agent never reported — a `null` verdict, uncoloured; issued > deployed → a neutral `stale` note, uncoloured — a count is not a graded number, the fleet-header rule), and the `san` list in mono.
|
||||
- per-row actions: **Rotate** (a confirm → `POST …/rotate`), **Revoke** (a confirm that says it stops the dashboard trusting that host until a reissue and a re-delivered bundle — a fixed sentence from the page, not the API), and **bundle** (a plain `<a href="/api/certs/<name>/bundle">` — a signed-in `GET` needs no CSRF).
|
||||
- a client-cert line (the serial + the expiry) — the one cert the agents check. It is **display-only**: no revoke button (a revoked client cert is a dashboard that reaches no agent — a footgun the page should not make easy); its rotation happens with the root.
|
||||
- ASSUMPTION: the client cert is display-only on the page (no revoke control); revoking it is deliberately not offered from the UI.
|
||||
2. `src/dashboards/app.py` — `PAGES` gains `"/certs": "certs.html"` (the settings-page comment at the `PAGES` dict extends by one line of the same shape); a `Certs` link (a `⚷` glyph is acceptable) in `.masthead-right` of every page including `/settings`, next to `⚙ Account` — **outside** `.tabs` (the settings precedent, `tests/test_api.py::test_every_dashboard_page_links_to_settings` pins that structure; extend the pin, do not break it).
|
||||
3. `src/dashboards/web/static/dash.css` — reuse existing vocabulary only (`.chip`, the table/`.hero`-family styles present in `settings.html`); **no** new `:root` tokens (the `tests/test_api.py` pin fails a CSS rule that reaches for an undeclared token).
|
||||
4. `tests/test_api.py` — the served-page pins (the settings-pin idiom):
|
||||
- the `Certs` masthead link on every dashboard page and on `/settings` (extend `test_every_dashboard_page_links_to_settings` into a two-link pin, or a sibling test);
|
||||
- the served `/certs` HTML: the root block, the agents table container, the three action labels; the page's own tab structure carries **no** `aria-current` on any of the five (the settings pin, copied);
|
||||
- the automatic audits cover the rest: every `id` the page keeps is referenced by `page-certs.js` (the orphan-id audit), no `Dash.severityFor` named (the existing pin), the deleted-flavour-text audit, the `no-store`/gate shape from the route walk.
|
||||
5. the live verification screenshot is task 07 (invariant 10).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- the served-HTML pins above; the JS is verified by the live screenshot (the repo's UI test policy — no JS test runner).
|
||||
- Coverage: **>90%** on the modified Python (`app.py`'s `PAGES`/route code).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] signed-in `GET /certs` serves the page; unsigned is a `303` to `/login?next=/certs` (the query included — `auth.safe_next`)
|
||||
- [ ] the masthead pin green (every page links to both settings and certs)
|
||||
- [ ] the orphan-id + flavour-text + `severityFor` audits green on the new page
|
||||
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||
@@ -0,0 +1,32 @@
|
||||
# Task 06 — The installer places the bundle; the unit runs TLS; --status probes over TLS
|
||||
|
||||
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||
|
||||
## Objective
|
||||
|
||||
`deploy/install-agent.sh` can install the certificate bundle the same way it already installs the token — one file, the right owner and mode, the unit flag, a `--status` that proves the TLS path end-to-end — and `tests/test_deploy_scripts.py` moves its pins so a regression is a red test, not a dead agent.
|
||||
|
||||
## Work
|
||||
|
||||
1. `deploy/install-agent.sh` —
|
||||
- a new flag `--cert-bundle <file>` (the operator already fetched it from `GET /api/certs/<name>/bundle` — task 04's surface; the script does not dial the dashboard, per its existing no-dial rule): validate the bundle before touching anything (a `python3` one-liner: three PEM blocks — server cert, key, CA cert — and a `# BUNDLE-REV` line; a bad bundle refuses with the reason, the `resolve_bundle` idiom).
|
||||
- install to `/etc/dashboards-agent.certs` as `0600 root:$AGENT_USER` (the token-file idiom, with one line of the same comment explaining why the mode matters — the bundle holds the agent's key, so only root and the agent's own service user may read it); on re-run, replace it only if the flag is given (an existing bundle survives an unrelated re-run — the token-reuse idiom).
|
||||
- the unit: when a bundle is installed, `ExecStart` gains `--cert-bundle /etc/dashboards-agent.certs` (the `--token-file` line stays — task 02's token check is skipped in TLS mode, so the harmless dual mode eases a rollout) and `RuntimeDirectory=dashboards-agent` (the unit's `ProtectSystem=strict` leaves the agent no writable file but this — the extracted PEM blocks live there; `PrivateTmp` already exists, so a non-systemd dev run falls back to `tempfile`).
|
||||
- the `--status` probe: when the unit runs with a bundle, probe `https://127.0.0.1:$PORT/health` with a `python3 -c` `ssl` context that `load_verify_locations`-es the bundle's CA block and presents **no** client cert (task 02's `CERT_OPTIONAL` + the `/health` exemption is what makes this possible — the installer holds no client cert); a token-mode install keeps the plain `/health` probe. The probe is `python3` (the unit's own interpreter), never a new binary dependency (no `openssl` — one less thing to install on a host).
|
||||
2. `deploy/agent-sudoers` — **no** new line (the bundle travels through the same root installer invocation as the token file).
|
||||
3. `tests/test_deploy_scripts.py` — the new pins (the file's existing idiom: read the script, run it unprivileged so it exits before touching anything, and assert on the lines):
|
||||
- `--cert-bundle` in the usage + the `/etc/dashboards-agent.certs` path with the `0600` and `root:$AGENT_USER` `chown` (assert the `install -m`/`chown` line);
|
||||
- a `RuntimeDirectory=dashboards-agent` line in the unit when a bundle is installed (assert the conditional that emits it);
|
||||
- the `--status` probe branch: the `python3` `ssl` one-liner for TLS, the plain probe for token mode;
|
||||
- the existing pins unchanged (the sudoers-line identity, phase 10's no-tunnel name/port).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- the `tests/test_deploy_scripts.py` pins above; the real-host smoke is task 07 (invariant 10 keeps the dev server up).
|
||||
- Coverage: the script is shell — the pins are its tests (the file's stated policy: no shell test framework; a subprocess buys nothing more).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] an unprivileged `--cert-bundle /nonexistent` run refuses with the reason and touches nothing
|
||||
- [ ] the pins green: the bundle path/mode, the unit flag + `RuntimeDirectory`, the TLS `--status` probe
|
||||
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||
@@ -0,0 +1,29 @@
|
||||
# Task 07 — The fleet runs TLS: install, flip, verify, document
|
||||
|
||||
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||
|
||||
## Objective
|
||||
|
||||
The two live agents (deskwork, gateway) present the dashboard-issued server cert and require the dashboard's client cert; the dashboard dials `https://`; the health table, the live suite, and a `/certs` screenshot prove it — and the still-missing `driveripper` install is documented to carry a bundle from day one.
|
||||
|
||||
## Work
|
||||
|
||||
1. **Fleet (operator root actions, the `deploy/agent-sudoers` grant):** for `deskwork` and `gateway`: `GET /api/certs/<name>/bundle` (a signed-in `curl`), `scp` the bundle to the host, `sudo ./deploy/install-agent.sh --cert-bundle <file>`, `systemctl restart dashboards-agent`, `./deploy/install-agent.sh --status` green over TLS. Record the exact commands + output in `data/logs/NOTES-agent-install.md` (an append to the file's existing per-host record; local evidence, not tracked, the AGENTS.md rule).
|
||||
2. **The flip:** `.env.dev` — the `HOST_TARGETS` entries become `https://<same address>:9120` (no userinfo — the cert is the auth; the token goes away for a TLS host), `./scripts/dev-server.sh restart`, `./scripts/dev-server.sh status` shows `host:deskwork ok`, `host:gateway ok`, `disks:deskwork ok`, `disks:gateway ok` (the two `driveripper` jobs stay `DOWN` — unchanged, the designed degradation).
|
||||
3. **Proof (invariant 10):** `uv run pytest -m live --live` — the host/disks tests over the `https` path; a `/certs` screenshot (the table showing both rows `active`, issued rev == deployed rev) **and** a `/system` screenshot (the host rows not dimmed) in `data/logs/`; a `data/logs/NOTES-verify.md`-style note: the `curl` codes (unsigned `/api/certs` refused, signed-in 200), one line for a wrong-CA client's handshake failure (the redacted URL only).
|
||||
4. **The `driveripper` procedure:** an append to `data/logs/NOTES-agent-install.md` — when the agent is finally installed (the outstanding operator action, the AGENTS.md fleet section), install the bundle in the same invocation (`sudo ./deploy/install-agent.sh --bind … --cert-bundle …`) and point `HOST_TARGETS` at `https://` from day one; a token-only install after this phase is a host the `https`-dialing dashboard cannot reach.
|
||||
5. **Docs (the AGENTS.md definition of done):** the AGENTS.md fleet section (the transport fact: the agent is now mTLS, the bundle file, the `/certs` page) + the commands section (the bundle-fetch line); a short "Protecting the agent" section in `README.md` (the operator's how-to: fetch the bundle, install, flip the URL — three sentences, no flavour).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- the live suite is read-only (the repo's rule) — no bundle mutations on the hardware; the revocation/rotation proof happens on a dev-box agent, not the fleet's.
|
||||
- invariant 10: the dev server stays up throughout; screenshots in `data/logs/`.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `./scripts/dev-server.sh status`: `host:deskwork ok`, `host:gateway ok`, `disks:deskwork ok`, `disks:gateway ok` over the `https://` dials
|
||||
- [ ] `uv run pytest -m live --live` green for the host/disks paths
|
||||
- [ ] the `/certs` + `/system` screenshots in `data/logs/` show `active`, issued rev == deployed rev, the host rows not dimmed
|
||||
- [ ] the `data/logs/` note records the exact commands + the `curl` codes + the wrong-CA failure line
|
||||
- [ ] AGENTS.md + `README.md` updated (the definition of done)
|
||||
- [ ] `bash .agents/validate.sh` green
|
||||
Reference in New Issue
Block a user