chore(agent): phase 15_agent_mtls from TODO.md (protected exporter web servers)
The single TODO item (exporters as protected web servers, certs as auth, root certificate generated and managed by the collector) is now phase 15: cert management in meta, agent TLS mode, https collector dials, gated /api/certs surface, /certs admin page, installer + live rollout. LLM-Generated: true
This commit is contained in:
@@ -0,0 +1,44 @@
|
|||||||
|
# Phase 15 — Protected exporter web servers, auth by certificate
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** `agent/host_agent.py` (the stdlib-only exporter web server; today: a bearer token via `--token`/`--token-file`, refuses an off-loopback bind without one, `ThreadingHTTPServer`, the `resolve_token` idiom), `src/dashboards/config.py` (`Target`/`parse_targets` — token in URL userinfo, `https://` URLs already accepted; **no change needed**), `src/dashboards/collectors.py` (`fetch_snapshot`, the one `UpstreamError`, redacted display URLs, the one shared `httpx.AsyncClient` from the lifespan), `src/dashboards/app.py` (the `PAGES` dict with the settings-page comment, `build_hub`/`build_context`, the lifespan that builds `ctx.auth`), `src/dashboards/api.py` (`AppContext`, the gated-write + event pattern), `src/dashboards/store.py` (`get_meta`/`set_meta`/`delete_meta` — the only durable key/value state, home of `auth.*`), `deploy/install-agent.sh` (the token-file idiom `0640 root:agent-user`, the `ProtectSystem=strict` unit with no `RuntimeDirectory`, the `--status` probe), `tests/test_agent.py` (the in-process agent harness), `tests/test_collectors.py`, `tests/test_deploy_scripts.py`, `tests/test_api.py` (the route walk, the settings-page pin, the orphan-id audit), `src/dashboards/web/pages/settings.html` + `src/dashboards/web/static/page-settings.js` (the "a page, not a dashboard" pattern the new `/certs` admin page follows)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Every exporter (host agent) web server is protected end-to-end by certificates the dashboard itself generates and manages: the dashboard holds a root CA in the `meta` table, signs its own client cert and each agent's server cert, the agent serves TLS with a per-path client-cert check (only the dashboard can read it), the dashboard dials `https://` with its client cert, and a new `/certs` admin page lists every cert with its expiry and status and triggers rotation, revocation, and bundle re-delivery. The operator's stated follow-up — firewalling upsd 3051 and talking to the agent only — is the motivation, not part of this phase. The agent's bearer-token mode stays a compatibility mode; the dashboard's own login is a separate auth domain and is untouched.
|
||||||
|
|
||||||
|
**Scope (operator-confirmed):** "exporters" = the host agents this repo installs. gufo / vLLM / upsd are third-party services this repo does not deploy; their TLS (e.g. upsd `STARTTLS`) is a separate operator concern.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
- `14_store_performance` (todo) — **shared-file, run-order note**: 14 rewrites the hot paths in `store.py`/`api.py`; 15 only *adds* meta keys and new routes/pages. Apply on top of the file state at the time it runs; never restore a line the other phase owns.
|
||||||
|
- `11_admin_login` (complete) — the session gate (invariant 12) for the new routes, the gated-write + CSRF + event pattern for the new `POST /api/certs/*`, the `meta` helpers, and the settings-page pattern `/certs` follows.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
|
||||||
|
1. `01_cert_management.md` — `src/dashboards/certs.py`: the dashboard-managed root CA, client cert, per-agent server certs, revocation and reissue, all in `meta`; the `cryptography` dependency (dashboard-side only).
|
||||||
|
2. `02_agent_tls.md` — `host_agent.py` TLS mode: one installed `--cert-bundle`, the per-path client-cert check, the bundle revocation list, `bundle_rev` in the snapshot; token mode unchanged.
|
||||||
|
3. `03_collector_tls.md` — the dashboard dials `https://` agent URLs with its client cert, verifies against its root, refuses a revoked agent without dialing; error sentences stay redacted.
|
||||||
|
4. `04_cert_api.md` — the gated `/api/certs` surface: list, per-host bundle, rotate, revoke, root-rotate; fixed-text events, no keys out of `meta`.
|
||||||
|
5. `05_certs_admin_page.md` — the `/certs` admin page (the settings pattern): every cert with its expiry and status, issued vs deployed bundle rev, rotate/revoke/root-rotate actions, bundle download.
|
||||||
|
6. `06_deploy_cert_bundle.md` — `install-agent.sh --cert-bundle` installs `/etc/dashboards-agent.certs` (`0600 root:agent-user`), the unit gains the flag + `RuntimeDirectory`, `--status` probes over TLS; the script pins move with it.
|
||||||
|
7. `07_live_rollout.md` — the two live agents flip to TLS, `HOST_TARGETS` goes `https://`, the live suite + health table + screenshot prove it; the `driveripper` install procedure is documented.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Unit: `tests/test_certs.py` (new) — the whole lifecycle against a dict `MetaStore`, exactly how `tests/test_auth.py` does it.
|
||||||
|
- Integration: `tests/test_agent.py` (the TLS handshake matrix against the in-process agent), `tests/test_collectors.py` (the `https` dial, the revoked refusal with a no-dial spy, the error sentences), `tests/test_api.py` (the `/api/certs` shapes, the gate/CSRF on the new writes, the served `/certs` page pins), `tests/test_deploy_scripts.py` (the new flags/files/permissions).
|
||||||
|
- Live: task 07 (AGENTS.md invariant 10 — the dev server stays up; screenshots in `data/logs/`).
|
||||||
|
- Coverage: **>90%** on new/modified code; the 94% gate holds. `uv pip list --outdated` empty (the `cryptography` dependency — invariant 8).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] a fresh database boots with a `meta` root CA, generated once and idempotent across restarts (pinned in `tests/test_certs.py`)
|
||||||
|
- [ ] an agent run with a bundle answers `/snapshot` only to a client cert signed by its root; a wrong-CA or certless client is refused; a revoked serial gets the 403; `/health` stays probeable with the CA only
|
||||||
|
- [ ] with `https://` `HOST_TARGETS`, the dev server's `host:`/`disks:` jobs are `ok`, and a revoked host says "certificate revoked" without any dial (spy-pinned)
|
||||||
|
- [ ] signed-in `GET /certs` lists every cert with expiry and status; rotate/revoke/root-rotate work through the gated API; the bundle download is byte-identical to the stored bundle
|
||||||
|
- [ ] the installer places the bundle `0600 root:agent-user`, the unit comes up in TLS mode, `--status` is green over TLS
|
||||||
|
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green, `uv pip list --outdated` empty
|
||||||
|
- [ ] live: deskwork + gateway in TLS mode, `https://` dials, live suite green, screenshots in `data/logs/`
|
||||||
|
- [ ] no behaviour change in completed phases: the token-mode agent is byte-identical, the engine/UPS polling is untouched, the public surface is unchanged (the route walk)
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
# Task 01 — The dashboard owns a root CA and every cert it signs
|
||||||
|
|
||||||
|
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The certificate lifecycle becomes a pure, testable module: the dashboard generates a root CA once (idempotent across restarts), signs its own client cert and each agent's server cert, and supports reissue and revocation — all state in the `meta` table, exactly how `auth.py` keeps the verifier and the session secret.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `pyproject.toml` — add `cryptography` to `dependencies` (dashboard-side only; the agent stays stdlib-only) and run `uv lock`. Invariant 8: `uv pip list --outdated` empty afterwards.
|
||||||
|
- ASSUMPTION: `cryptography` is the only new dependency and it stays on the dashboard side; the agent (task 02) gains no third-party code — it loads the PEMs with stdlib `ssl`.
|
||||||
|
2. `src/dashboards/certs.py` (new) — the whole lifecycle as pure functions over the three-method `MetaStore` protocol (the same protocol `auth.load_auth_state` uses, so this module imports nothing else from the package, and its tests use a dict):
|
||||||
|
- `ensure_certs(store, *, now: float | None = None)` → a frozen `CertsState` carrying the PEM text (`root_key`, `root_cert`, `client_key`, `client_cert`) plus the root/client `serial` + `not_after`. Reads the `meta` rows first; only generates and persists when missing (first boot); an existing row wins and is **never regenerated on restart** — regenerating on every start would rotate the CA and invalidate every installed agent bundle. `now` is injected (the module never reads a clock itself — the `auth.py` rule).
|
||||||
|
- meta keys (one concern per key, the `auth.*` precedent): `certs.root.key`, `certs.root.cert`, `certs.client.key`, `certs.client.cert`, `certs.agents` (one JSON object mapping `host name` → `{serial, not_after, status, bundle_rev, bundle, san}` — the bundle text is stored, so what the agent holds is byte-identical to what the API serves).
|
||||||
|
- Shapes: root CA self-signed, CN `dashboards-root`, **10 years**; client cert signed by the root, CN `dashboards`, **1 year**; agent server certs signed by the root, CN = the host name, **1 year**, `SAN` = the address(es) the dashboard dials for that host.
|
||||||
|
- ASSUMPTION: 10 y root / 1 y leaf lifetimes. No env seed for the root CA (unlike the admin password, the root key is a secret only the app needs to know, so it is auto-generated on first boot and merely persisted).
|
||||||
|
- `issue_agent_cert(store, state, name, san: list[str])` / `reissue_agent_cert(store, state, name, san)` — a fresh serial, `bundle_rev` +1 (1 on first issue), `status: "active"`, a fresh `bundle` (below). Idempotent per call — never issued on a read.
|
||||||
|
- `rotate_root(store, state)` — a fresh root CA + fresh client cert + every existing agent entry reissued (each `bundle_rev` +1): one action re-arms the whole fleet.
|
||||||
|
- `revoke(store, state, name)` — `status: "revoked"` (the serial stays on the denylist until a reissue gives the host a fresh identity; there is no "unrevoke" — reissue is the fix).
|
||||||
|
- the **bundle** (the install artifact, the TODO's "root certificate installed with each exporter"): one PEM text = the agent's server cert + the agent's key + the root CA *cert* (never the root key) + a `# BUNDLE-REV <n>` line + a `# REVOKED <serial>` line (one per revoked serial of the *current* root generation — a root rotation resets the list, since every cert is reissued anyway).
|
||||||
|
- ASSUMPTION: one client cert for the whole dashboard (the agent verifies "signed by our root", not a per-process identity); one agent entry per `HOST_TARGETS` name.
|
||||||
|
- invariant 6, the `auth.py` rule: no `logging` and no raise with anything interpolated — a malformed stored PEM is a fixed-text `RuntimeError` telling the operator to delete the `certs.*` meta keys (the shape of the auth reset), and nothing else.
|
||||||
|
3. `src/dashboards/app.py` (+ the `AppContext` in `src/dashboards/api.py`) — the startup path builds `ctx.certs` (an `ensure_certs` read, the one-time bootstrap included) **before** `build_hub`, in both the owned-store (`build_context`) and the injected-store (lifespan) branches, through **one** helper so the two wiring sites cannot drift (the `build_hub` rule in AGENTS.md); the PEMs are written once to a private temp dir (mode `0600`, lives as long as the process) and the *paths* are what `AppContext.certs` carries for task 03's dialing.
|
||||||
|
- A `certs` whose stored PEM is malformed (a truncated `meta` row) is a fail-closed startup — a dashboard that cannot present its own credential must not start the loops (the `ctx.auth` rule, the one deliberate raise).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- `tests/test_certs.py` (new) — a dict `MetaStore` (the `tests/test_auth.py` idiom):
|
||||||
|
- the first `ensure_certs` call persists exactly the five `certs.*` keys; the second is byte-identical (no regeneration — pin the `certs.root.cert` value);
|
||||||
|
- the parsed chain (`cryptography.x509` loaders) verifies: client → root, agent → root; the SAN list appears in the agent cert; the expiries match the assumed lifetimes (injected `now`);
|
||||||
|
- `reissue_agent_cert` bumps `serial` and `bundle_rev` and sets `status` active; `revoke` flips the status and puts the serial on the bundle's `# REVOKED` line; a reissue after a revoke gives a fresh serial that is *not* on the list;
|
||||||
|
- `rotate_root` reissues every agent entry (each `bundle_rev` bumped), the bundle carries the new root cert, and no stored value still contains the old root key;
|
||||||
|
- the bundle contains the agent cert + key + root cert + `# BUNDLE-REV` + the `# REVOKED` lines and **never** the root key (assert via the PEM header and a parsed-key mismatch);
|
||||||
|
- a malformed stored PEM → the fixed-text `RuntimeError` naming the meta keys, nothing interpolated.
|
||||||
|
- Coverage: **>90%** on this task's new code.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `uv run pytest tests/test_certs.py` green; the full suite green
|
||||||
|
- [ ] `uv pip list --outdated` empty; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||||
|
- [ ] a fresh store boots with the `certs.*` rows present; a second boot changes no byte of `certs.root.cert`
|
||||||
|
- [ ] no behaviour change in completed work: `auth.*` keys, the public surface, and the poller wiring are untouched
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
# Task 02 — The agent serves a protected TLS web server from one installed bundle
|
||||||
|
|
||||||
|
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`host_agent.py` gains a TLS mode that makes the web server protected by certificate: one installed bundle holds the server cert + key, the root CA, a bundle revision, and a revocation list; the agent requires a client cert signed by the root on every path except `/health` (mirroring the token-mode `/health` exemption, so an installer probe can reach `/health` with the CA only), and it reports the bundle revision it loaded so the dashboard can tell a stale bundle from a fresh one. The token mode is byte-identical to today.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `agent/host_agent.py` — stdlib only (an `ssl` import; no new third-party dependency):
|
||||||
|
- a new flag `--cert-bundle <file>` (env `AGENT_CERT_BUNDLE`): one PEM text containing the server cert, the server key, the root CA cert, a `# BUNDLE-REV <n>` line, and `# REVOKED <serial>` lines. `resolve_bundle(path) -> (context, bundle_rev, revoked: frozenset[str], error)` is a pure function (the `resolve_token` idiom): parse the PEM blocks by text, extract the three blocks, write them to a temp dir for `load_cert_chain` (into `/run/dashboards-agent` when present and writable — the unit's `RuntimeDirectory`, task 06 — else `tempfile.mkdtemp`), and build the context.
|
||||||
|
- The TLS context: `ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)`, `load_cert_chain`, `load_verify_locations(ca)`, `verify_mode = ssl.CERT_OPTIONAL` — the handshake is optional and the **handler** enforces it: on every path except `/health`, `self.connection.getpeercert()` must carry a verified certificate (subject present), else `403 {"ok": false, "detail": "client certificate required"}`; that cert's serial must not be in the bundle's `# REVOKED` set, else `403 {"ok": false, "detail": "client certificate revoked"}`. The denylist file is re-read on mtime change per request (a small file; a redeployed bundle takes effect without a restart — the reason the `# REVOKED` lines live in the bundle at all).
|
||||||
|
- The startup gate: off-loopback with neither a token nor a bundle → the existing exit-2 refusal (its message now names the bundle too); a bundle present → the off-loopback bind is allowed (the client cert is the protection); a token plus a bundle → the token check is skipped on the TLS path (the handshake is the auth).
|
||||||
|
- The snapshot payload gains two **additive** top-level fields in TLS mode: `tls: true` and `bundle_rev: <n>` (absent in token mode — an additive field never changes an existing shape); the `serve()` start line says `tls (client cert required, rev <n>)`.
|
||||||
|
- `AGENT_VERSION` 1.0.0 → 1.1.0; the docstring's deployment paragraph gains one line for the TLS mode (no flavour text — the AGENTS.md audit rule).
|
||||||
|
- ASSUMPTION: `CERT_OPTIONAL` + the per-path check (not `CERT_REQUIRED`) is what keeps `/health` probeable with the CA only — the installer's `--status` and the systemd probe (task 06) cannot present a client cert the installer does not hold, and a probe that cannot health-check is a dead install.
|
||||||
|
- ASSUMPTION: the agent never generates or renews certificates (stdlib cannot mint x509); the bundle is the artifact the dashboard signs and the installer places.
|
||||||
|
2. `tests/test_agent.py` — the existing in-process harness (`serve()` in a thread on `127.0.0.1`, `urllib`/`ssl` clients), with a test CA + leafs generated by `src/dashboards/certs.py` over a dict store (no second generator):
|
||||||
|
- a client with a matching client cert → `/snapshot` 200 with `tls: true` and `bundle_rev` equal to the bundle's;
|
||||||
|
- a client with no cert → the handshake fails or a 403 on a non-`/health` path; `/health` still 200 with a CA-only context (the probe path);
|
||||||
|
- a client signed by a *different* CA → refused;
|
||||||
|
- a client whose serial is in the bundle's `# REVOKED` list → 403 "revoked"; rewrite the bundle file without the line (mtime change) → the same client is accepted on the next request (the denylist re-read);
|
||||||
|
- a token-mode agent (no bundle): the existing tests unchanged and green (byte-identical behaviour — no `tls`/`bundle_rev` fields);
|
||||||
|
- `resolve_bundle` error paths: a missing file, an empty file, a bundle missing the key block, a bundle missing the `# BUNDLE-REV` line (an error — a revless bundle is not one this app signed; ASSUMPTION: the `# BUNDLE-REV` line is mandatory).
|
||||||
|
3. `deploy/` is untouched here (task 06); the non-systemd fallback (`tempfile.mkdtemp`) keeps dev and tests working.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- the `tests/test_agent.py` matrix above; the existing token-mode tests stay green unchanged.
|
||||||
|
- Coverage: **>90%** on this task's new/modified code.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] a TLS-mode agent on a loopback port answers the full handshake matrix (tests) and reports `bundle_rev`
|
||||||
|
- [ ] a token-mode agent is byte-identical to today (existing tests green, no new fields present)
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Task 03 — The dashboard dials agents over https with its own client cert
|
||||||
|
|
||||||
|
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
An `https://` `HOST_TARGETS` entry is now a certificate-authenticated dial: the dashboard presents the client cert task 01 gives it, verifies the agent against its own root CA, refuses a *revoked* agent before dialing (invariant 3's shape — a source it knows is dead says so without a network), and passes the agent's `tls`/`bundle_rev` through the `system` and `disks` payloads so task 05's page can compare the issued rev against the deployed one. Engines and token-mode agents are untouched.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/config.py` — **no change** (an `https://` URL already parses; a TLS host carries no userinfo, and the userinfo token stays available for token-mode hosts).
|
||||||
|
2. `src/dashboards/collectors.py` —
|
||||||
|
- a small frozen `AgentTls` dataclass: `verify: str` (the CA path), `cert: tuple[str, str]` (the client cert + key paths), `status: Callable[[], str]` — the `status` callable returns `"active"`/`"revoked"` from a **fresh** `get_meta("certs.agents")` read (a small-table `SELECT`, cheap at poll cadence, and phase 14's work keeps it off the hot paths; reading it live is what makes a runtime revocation take effect on the next poll without a restart).
|
||||||
|
- `fetch_snapshot(client, target, *, timeout=None, tls: AgentTls | None = None)`: when `tls` is given **and** `target.url` is `https://`, the request passes `verify=<ca path>` and `cert=(cert path, key path)` to `client.get` (httpx per-request overrides — the shared client dials both plain and TLS upstreams, so a per-client `verify` would poison the engine dials).
|
||||||
|
- Revocation before any network: `tls is not None` and `tls.status() == "revoked"` → `UpstreamError(f"{target.name}: agent certificate revoked")` **before** `client.get` — the poller's stale/last-good semantics then do the rest (invariant 4).
|
||||||
|
- A `403` from an agent gets its own honest sentence (the existing `401` branch keeps the token-mode wording): `f"{target.name}: agent refused the client certificate ({detail})"` with the `detail` from the agent's JSON (`client certificate required` / `client certificate revoked`), the redacted `display_url` (invariant 6) like every message in this function.
|
||||||
|
- `fetch_host_snapshot`/`fetch_disk_snapshot` pass the `tls` through; the per-host `AgentTls` is plumbed from `AppContext.certs` into the `host_jobs`/`disks_jobs` lambdas (a new optional `tls` parameter on both `*_jobs`, `build_hub` takes it — one call site per branch so the production and injected wiring cannot drift, the `build_hub` rule in AGENTS.md). `status` reads from the store the jobs already hold.
|
||||||
|
3. `src/dashboards/app.py` (+ the `AppContext` in `api.py`) — `AppContext.certs` (task 01) exposes the temp-dir paths (`ca`, `client_cert`, `client_key`); `build_context` and the lifespan's injected branch build the per-host `AgentTls` objects **before** `build_hub`.
|
||||||
|
4. `src/dashboards/sources/host.py` + `src/dashboards/sources/disks.py` — pass the agent's `tls`/`bundle_rev` snapshot fields through into the per-host payload (additive, the `degraded` idiom) so `/api/system/snapshot` and `/api/disks/snapshot` carry the deployed rev. **No** `severity` for a rev mismatch — a count is not a graded number (the fleet-header rule), the page colours nothing for it.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- `tests/test_collectors.py` — an in-process TLS agent (task 02's harness, in a thread) + `AgentTls` material generated by `certs.ensure_certs` over a dict store:
|
||||||
|
- an `https` target with `tls` → the 200 body; a missing client-cert file and a wrong-CA failure still surface as `UpstreamError` with the redacted URL;
|
||||||
|
- a revoked status (dict store with `status: "revoked"`) → the error is raised and a spy on `client.get` (the `tests/test_api.py` spy idiom) proves **no dial happened**;
|
||||||
|
- an `http` token-mode target: the existing tests unchanged;
|
||||||
|
- the 403 sentence: a scripted agent 403 (a `tests/conftest.py` fake) → the `detail` appears in the message, the URL redacted;
|
||||||
|
- the payload passthrough: a snapshot with `tls: true, bundle_rev: 3` → both the `system` and the `disks` payloads carry it.
|
||||||
|
- Coverage: **>90%** on this task's new/modified code.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `uv run pytest` green; the engine/UPS/token-host paths are byte-identical (existing pins hold)
|
||||||
|
- [ ] a revoked host fails its `host:` and `disks:` jobs with "certificate revoked" and no network dial (spy-pinned)
|
||||||
|
- [ ] `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# Task 04 — The gated /api/certs surface: list, bundle, rotate, revoke, root-rotate
|
||||||
|
|
||||||
|
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The API half of "the collector manages the certs": a signed-in operator can list every cert with its expiry and status, download the exact bundle an agent should hold, and trigger rotation / revocation / root rotation. The routes dial nothing (invariant 1's shape), are gated like every other write (invariant 12), and write fixed-text events — a key never leaves `meta` (invariant 6).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/api.py` — the new routes (the middleware gates them by default; the route-walk test covers them for free):
|
||||||
|
- `GET /api/certs` — `{root: {serial, not_after}, client: {serial, not_after}, agents: [{name, serial, not_after, status, bundle_rev, deployed_bundle_rev, san}]}`. `deployed_bundle_rev` is the `bundle_rev` from the host's last `system` job payload (the same source `alerts.py` reads job states from); a host that never answered → `null`. Reads never issue (no side effects on a read — a `GET` that mints a cert is a surprise).
|
||||||
|
- `GET /api/certs/<name>/bundle` — the stored bundle text, `Content-Type: application/x-pem-file`, `Content-Disposition: attachment; filename="agent-<name>.bundle"` — byte-identical to what `certs.py` stored (the installer's input, task 06/07).
|
||||||
|
- `POST /api/certs/<name>/rotate` — a reissue (`certs.reissue_agent_cert`, the SANs taken from the configured `HOST_TARGETS` URL host(s) + the name); a `cert-rotated` event (a fixed message naming the host — no PEM, no key).
|
||||||
|
- `POST /api/certs/<name>/revoke` — `certs.revoke`; a `cert-revoked` event. The next poll refuses without dialing (task 03) — that is the enforcement; the event is the audit line.
|
||||||
|
- `POST /api/certs/root/rotate` — `certs.rotate_root`; a `cert-root-rotated` event; the response carries `{"ok": true, "restart_required": true}` — the client cert and CA are startup material, so a new root is fully in force at the next process start (the page, task 05, prints that; the honest constraint, not a hidden one).
|
||||||
|
- ASSUMPTION: a root rotation requires a dashboard restart to be fully in force (the client cert + CA load at startup); the API says so instead of pretending a hot-swap that does not exist.
|
||||||
|
- A `<name>` that is not a configured `HOST_TARGETS` name → `404 {"configured": false}` **without** issuing, dialing, or touching the store (invariant 3, the `/api/disks` gate shape, spy-pinned in `tests/test_api.py`).
|
||||||
|
- Every `POST` is a gated write: session + `Origin`/`Sec-Fetch-Site` + the cookie's `X-CSRF-Token` (the `POST /api/ups/command` test idiom — a tokenless write records nothing on the wire, with the token it does).
|
||||||
|
- invariant 6: no `meta` value (a key PEM) may appear in an `event` row, a `/api/health` field, or an error `detail`; the events carry the host name + the action in fixed text.
|
||||||
|
2. `tests/test_api.py` — the signed-in harness (`tests/auth_client.py::sign_in`):
|
||||||
|
- `GET /api/certs` with zero / one / three configured hosts: the key paths pinned (the `/api/disks` pin idiom — a panel that retires can never shrink the payload);
|
||||||
|
- an unknown `<name>`: the `404` and the spies on `httpx.AsyncClient.get` and `collectors.fetch_snapshot` (the `tests/test_api.py` spy idiom) prove no network and no store write (a `meta` row count before/after);
|
||||||
|
- `POST rotate` → `bundle_rev` +1, a `cert-rotated` event row, the `meta` value changed, and the old serial is not on the new bundle's denylist;
|
||||||
|
- `POST revoke` → the status, the event, and (through task 03's path) the next `fetch_snapshot` for that host refuses without a dial (the spy);
|
||||||
|
- `POST root/rotate` → a fresh root serial, every agent reissued, `restart_required: true` in the body;
|
||||||
|
- `GET bundle` → the body byte-identical to the stored `meta` bundle, and a reissue changes the served bytes;
|
||||||
|
- the unsigned gate: the route-walk test (existing, automatic) plus one explicit unsigned `303`/`401` on `GET /api/certs` and a `POST` refusal (the `tests/test_auth_api.py` idiom).
|
||||||
|
- Coverage: **>90%** on this task's new/modified code.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] the routes are signed-in + CSRF-gated (the wire test); the unsigned shape pinned
|
||||||
|
- [ ] an unknown name 404s with no dial and no store write (spy-pinned)
|
||||||
|
- [ ] rotate / revoke / root-rotate each leave a fixed-text event row and no key in the `event`/`health` tables (a test greps the stored events for a PEM header and finds none)
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Task 05 — The /certs admin page: every cert, its expiry, rotate, revoke, re-deliver
|
||||||
|
|
||||||
|
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter." (operator addition: an admin page to see generated certs and expirations, trigger rotation, and revoke certs)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
A signed-in operator sees every cert the dashboard has generated — the root CA, its own client, and one per host — with its expiry and status, can rotate / revoke / root-rotate through task 04's API, and can download the exact bundle an agent should hold. The page follows `settings.html`'s precedent exactly: a page and not a dashboard — in `PAGES` (so it is a route), not in `.tabs`, linked from `.masthead-right`, with no `aria-current` on the tabs and no status dots.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/pages/certs.html` + `src/dashboards/web/static/page-certs.js` (the `page-settings.js` idiom: the same `<head>` + masthead, `Dash.fetchJSON`, errors printed through `textContent`, a `confirm()` for destructive actions — the typed `DANGER` word is reserved for power cuts):
|
||||||
|
- the **Root CA** block: the CN, the serial, the expiry (a local timestamp **and** the time left, the settings session-display idiom), and the **Rotate root CA** button → `POST /api/certs/root/rotate` with a confirm dialog; the response's `restart_required` prints "restart the dashboard to activate the new root" (the API's own sentence, not an invented one).
|
||||||
|
- the **Agents** table: one row per configured `HOST_TARGETS` host (the `GET /api/certs` `agents[]`): the name, the serial, the `status` chip (`active` → `ok`, `revoked` → `fail` — a string-to-class map, the page never compares a number), the expiry (local timestamp + time left), the **issued** `bundle_rev` against the **deployed** `bundle_rev` (`n/a` when the agent never reported — a `null` verdict, uncoloured; issued > deployed → a neutral `stale` note, uncoloured — a count is not a graded number, the fleet-header rule), and the `san` list in mono.
|
||||||
|
- per-row actions: **Rotate** (a confirm → `POST …/rotate`), **Revoke** (a confirm that says it stops the dashboard trusting that host until a reissue and a re-delivered bundle — a fixed sentence from the page, not the API), and **bundle** (a plain `<a href="/api/certs/<name>/bundle">` — a signed-in `GET` needs no CSRF).
|
||||||
|
- a client-cert line (the serial + the expiry) — the one cert the agents check. It is **display-only**: no revoke button (a revoked client cert is a dashboard that reaches no agent — a footgun the page should not make easy); its rotation happens with the root.
|
||||||
|
- ASSUMPTION: the client cert is display-only on the page (no revoke control); revoking it is deliberately not offered from the UI.
|
||||||
|
2. `src/dashboards/app.py` — `PAGES` gains `"/certs": "certs.html"` (the settings-page comment at the `PAGES` dict extends by one line of the same shape); a `Certs` link (a `⚷` glyph is acceptable) in `.masthead-right` of every page including `/settings`, next to `⚙ Account` — **outside** `.tabs` (the settings precedent, `tests/test_api.py::test_every_dashboard_page_links_to_settings` pins that structure; extend the pin, do not break it).
|
||||||
|
3. `src/dashboards/web/static/dash.css` — reuse existing vocabulary only (`.chip`, the table/`.hero`-family styles present in `settings.html`); **no** new `:root` tokens (the `tests/test_api.py` pin fails a CSS rule that reaches for an undeclared token).
|
||||||
|
4. `tests/test_api.py` — the served-page pins (the settings-pin idiom):
|
||||||
|
- the `Certs` masthead link on every dashboard page and on `/settings` (extend `test_every_dashboard_page_links_to_settings` into a two-link pin, or a sibling test);
|
||||||
|
- the served `/certs` HTML: the root block, the agents table container, the three action labels; the page's own tab structure carries **no** `aria-current` on any of the five (the settings pin, copied);
|
||||||
|
- the automatic audits cover the rest: every `id` the page keeps is referenced by `page-certs.js` (the orphan-id audit), no `Dash.severityFor` named (the existing pin), the deleted-flavour-text audit, the `no-store`/gate shape from the route walk.
|
||||||
|
5. the live verification screenshot is task 07 (invariant 10).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- the served-HTML pins above; the JS is verified by the live screenshot (the repo's UI test policy — no JS test runner).
|
||||||
|
- Coverage: **>90%** on the modified Python (`app.py`'s `PAGES`/route code).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] signed-in `GET /certs` serves the page; unsigned is a `303` to `/login?next=/certs` (the query included — `auth.safe_next`)
|
||||||
|
- [ ] the masthead pin green (every page links to both settings and certs)
|
||||||
|
- [ ] the orphan-id + flavour-text + `severityFor` audits green on the new page
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 06 — The installer places the bundle; the unit runs TLS; --status probes over TLS
|
||||||
|
|
||||||
|
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`deploy/install-agent.sh` can install the certificate bundle the same way it already installs the token — one file, the right owner and mode, the unit flag, a `--status` that proves the TLS path end-to-end — and `tests/test_deploy_scripts.py` moves its pins so a regression is a red test, not a dead agent.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `deploy/install-agent.sh` —
|
||||||
|
- a new flag `--cert-bundle <file>` (the operator already fetched it from `GET /api/certs/<name>/bundle` — task 04's surface; the script does not dial the dashboard, per its existing no-dial rule): validate the bundle before touching anything (a `python3` one-liner: three PEM blocks — server cert, key, CA cert — and a `# BUNDLE-REV` line; a bad bundle refuses with the reason, the `resolve_bundle` idiom).
|
||||||
|
- install to `/etc/dashboards-agent.certs` as `0600 root:$AGENT_USER` (the token-file idiom, with one line of the same comment explaining why the mode matters — the bundle holds the agent's key, so only root and the agent's own service user may read it); on re-run, replace it only if the flag is given (an existing bundle survives an unrelated re-run — the token-reuse idiom).
|
||||||
|
- the unit: when a bundle is installed, `ExecStart` gains `--cert-bundle /etc/dashboards-agent.certs` (the `--token-file` line stays — task 02's token check is skipped in TLS mode, so the harmless dual mode eases a rollout) and `RuntimeDirectory=dashboards-agent` (the unit's `ProtectSystem=strict` leaves the agent no writable file but this — the extracted PEM blocks live there; `PrivateTmp` already exists, so a non-systemd dev run falls back to `tempfile`).
|
||||||
|
- the `--status` probe: when the unit runs with a bundle, probe `https://127.0.0.1:$PORT/health` with a `python3 -c` `ssl` context that `load_verify_locations`-es the bundle's CA block and presents **no** client cert (task 02's `CERT_OPTIONAL` + the `/health` exemption is what makes this possible — the installer holds no client cert); a token-mode install keeps the plain `/health` probe. The probe is `python3` (the unit's own interpreter), never a new binary dependency (no `openssl` — one less thing to install on a host).
|
||||||
|
2. `deploy/agent-sudoers` — **no** new line (the bundle travels through the same root installer invocation as the token file).
|
||||||
|
3. `tests/test_deploy_scripts.py` — the new pins (the file's existing idiom: read the script, run it unprivileged so it exits before touching anything, and assert on the lines):
|
||||||
|
- `--cert-bundle` in the usage + the `/etc/dashboards-agent.certs` path with the `0600` and `root:$AGENT_USER` `chown` (assert the `install -m`/`chown` line);
|
||||||
|
- a `RuntimeDirectory=dashboards-agent` line in the unit when a bundle is installed (assert the conditional that emits it);
|
||||||
|
- the `--status` probe branch: the `python3` `ssl` one-liner for TLS, the plain probe for token mode;
|
||||||
|
- the existing pins unchanged (the sudoers-line identity, phase 10's no-tunnel name/port).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- the `tests/test_deploy_scripts.py` pins above; the real-host smoke is task 07 (invariant 10 keeps the dev server up).
|
||||||
|
- Coverage: the script is shell — the pins are its tests (the file's stated policy: no shell test framework; a subprocess buys nothing more).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] an unprivileged `--cert-bundle /nonexistent` run refuses with the reason and touches nothing
|
||||||
|
- [ ] the pins green: the bundle path/mode, the unit flag + `RuntimeDirectory`, the TLS `--status` probe
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# Task 07 — The fleet runs TLS: install, flip, verify, document
|
||||||
|
|
||||||
|
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The two live agents (deskwork, gateway) present the dashboard-issued server cert and require the dashboard's client cert; the dashboard dials `https://`; the health table, the live suite, and a `/certs` screenshot prove it — and the still-missing `driveripper` install is documented to carry a bundle from day one.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. **Fleet (operator root actions, the `deploy/agent-sudoers` grant):** for `deskwork` and `gateway`: `GET /api/certs/<name>/bundle` (a signed-in `curl`), `scp` the bundle to the host, `sudo ./deploy/install-agent.sh --cert-bundle <file>`, `systemctl restart dashboards-agent`, `./deploy/install-agent.sh --status` green over TLS. Record the exact commands + output in `data/logs/NOTES-agent-install.md` (an append to the file's existing per-host record; local evidence, not tracked, the AGENTS.md rule).
|
||||||
|
2. **The flip:** `.env.dev` — the `HOST_TARGETS` entries become `https://<same address>:9120` (no userinfo — the cert is the auth; the token goes away for a TLS host), `./scripts/dev-server.sh restart`, `./scripts/dev-server.sh status` shows `host:deskwork ok`, `host:gateway ok`, `disks:deskwork ok`, `disks:gateway ok` (the two `driveripper` jobs stay `DOWN` — unchanged, the designed degradation).
|
||||||
|
3. **Proof (invariant 10):** `uv run pytest -m live --live` — the host/disks tests over the `https` path; a `/certs` screenshot (the table showing both rows `active`, issued rev == deployed rev) **and** a `/system` screenshot (the host rows not dimmed) in `data/logs/`; a `data/logs/NOTES-verify.md`-style note: the `curl` codes (unsigned `/api/certs` refused, signed-in 200), one line for a wrong-CA client's handshake failure (the redacted URL only).
|
||||||
|
4. **The `driveripper` procedure:** an append to `data/logs/NOTES-agent-install.md` — when the agent is finally installed (the outstanding operator action, the AGENTS.md fleet section), install the bundle in the same invocation (`sudo ./deploy/install-agent.sh --bind … --cert-bundle …`) and point `HOST_TARGETS` at `https://` from day one; a token-only install after this phase is a host the `https`-dialing dashboard cannot reach.
|
||||||
|
5. **Docs (the AGENTS.md definition of done):** the AGENTS.md fleet section (the transport fact: the agent is now mTLS, the bundle file, the `/certs` page) + the commands section (the bundle-fetch line); a short "Protecting the agent" section in `README.md` (the operator's how-to: fetch the bundle, install, flip the URL — three sentences, no flavour).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- the live suite is read-only (the repo's rule) — no bundle mutations on the hardware; the revocation/rotation proof happens on a dev-box agent, not the fleet's.
|
||||||
|
- invariant 10: the dev server stays up throughout; screenshots in `data/logs/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `./scripts/dev-server.sh status`: `host:deskwork ok`, `host:gateway ok`, `disks:deskwork ok`, `disks:gateway ok` over the `https://` dials
|
||||||
|
- [ ] `uv run pytest -m live --live` green for the host/disks paths
|
||||||
|
- [ ] the `/certs` + `/system` screenshots in `data/logs/` show `active`, issued rev == deployed rev, the host rows not dimmed
|
||||||
|
- [ ] the `data/logs/` note records the exact commands + the `curl` codes + the wrong-CA failure line
|
||||||
|
- [ ] AGENTS.md + `README.md` updated (the definition of done)
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
@@ -1,3 +1 @@
|
|||||||
# TODO
|
# TODO
|
||||||
|
|
||||||
- [ ] For privileged actions (UPS commands) the exporter should be installed with password protection and that password must be used to register it with the collector. Effectively the exporters become protected web servers. Additionally, the collector should generate and manage a root certificate that needs to be installed with each exporter. In fact, just use the certs as auth rather than a password.
|
|
||||||
|
|||||||
Reference in New Issue
Block a user