services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 33_gateway_route_switching

Browse Source 
Phase 33 completion pass is done. Final report:

## Phase 33 — completion report

**All 8 task files complete** (`.agents/phases/complete/33_gateway_route_switching/`), full gate green, docs verified.

### Work in this pass

1. **`tests/test_routectl.py` added** (17 tests, all green) — exercises the *real* helper binary unprivileged against a fake tree (`--root`), the real agent parser, the captured gateway fixtures, and stubbed `nginx`/`haproxy`/`systemctl` on `PATH`. Pins the full protocol (locate → state → dry-run → temp-file/fsync/`os.replace` write → checker → restore on checker failure → reload-never-restart → re-read proof → sidecar record), all three route shapes (haproxy `server`, nginx `proxy_pass`, nginx stream `map` via its sink-value id), `drift` / unknown-id / `map-default` refusals, `list`/`verify`, and the `--serve` peer gate (`SO_PEERCRED` uid check, one JSON line per request, exact body shape).

2. **Real helper defect fixed** — `deploy/dashboards-routectl.py`: `main()`'s early-return paths returned `({"ok": False, …}, EXIT_*)` tuples, so `sys.exit()` printed a Python repr instead of a JSON line (usage errors, agent-import refusal). Now prints `json.dumps(…)` and returns only the code, keeping the one-JSON-line contract.

3. **Documentation consistency fixes** (task 08 criterion 6):
   - `README.md`: garbled verdict-colour sentence on the switch ("green and to the position") → "in the **on** position and tinted `--gate-on`" (the existence-pair contract).
   - `README.md`: audit sentence now names all three event kinds — `route-disabled` / `route-enabled` / **`route-denied`** (the code stores one per refused call; dry runs store none).
   - `page-gateway.js`: replaced the stale pre-33 module docstring ("the switch is not wired") with the two-arm `GATEWAY_ALLOW_WRITES` contract.
   - `AGENTS.md`: every phase-33 claim re-verified against code (the ten gates, the 35 s/30 s budget pair, `gateway.writes_enabled`, `/api/summary`'s `disabled`, the 1.4.0 agent bullet, the gap list) — no changes needed.

### Verification
- `uv run pytest --cov`: **1970 passed**, 18 skipped — coverage **98.74%** (floor 94%)
- `ruff` clean · `pyright` 0 errors · `bash .agents/validate.sh` **exit 0** (incl. `node --check` on every page script)
- Dev server running; all 10 sources `ok`; bundles exported

### Live infrastructure
**No live changes made.** The live drill remains unexecuted per `data/logs/NOTES-33-switching.md` (no operator approval; checker binaries unavailable in the podman proxy units). All routes stay enabled, config hashes unchanged, and the sandbox rehearsal + backup tree are in place for the operator's own `DANGER` call. All changes are in the working tree only.
This commit is contained in:
ducoterra
2026-11-01 11:50:51 -05:00
parent 0271387053
commit 86c8bf4992
91 changed files with 7568 additions and 519 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+109 -34
View File
@@ -28,24 +28,31 @@ file — plus an overview page:
| `/disks` | Are any physical disks failing, and which — health, temp, wear | the same agents, `smartctl -i -A -H` per whole drive |
| `/llm` | Tokens/s, active sessions, queue depth, KV pressure, latency | gufo `/metrics` + `/slots`, vLLM `/metrics` |
| `/ups` | Battery health, runtime left, power events, test buttons | NUT `upsd` (our own wire client; port **3051**, not 3493) |
| `/gateway` | Which domains on which port, and where each goes | the same agent's `haproxy` + `nginx` sections joined to `/proc/net/tcp{,6}` on the one `GATEWAY_HOST` host |
| `/gateway` | Which domains on which port, and where each goes | the same agent's `haproxy` + `nginx` sections joined to `/proc/net/tcp{,6}` on the one `GATEWAY_HOST` host (the page needs agent **≥ 1.2.0**; the route switch's `#off` marker read is **1.4.0**) |
History is a first-class feature: every number worth graphing is stored, rolled up
(raw → 1m → 1h) and served from `/api/*/history`.
None of it is public. One admin signs in at `/login` (the only page an anonymous visitor
reaches); `/settings` (⚙ Account) and `/certs` (⚷ Certs) live behind it. Every page and
every `/api` route requires the session except the public entries in rule 12.
every `/api` route requires the session except the public entries in invariant 12.
## Non-negotiable rules
Breaking one is a bug even if the page still renders.
1. **Read-only by construction, with exactly one exception.** Nothing writes to an upstream
except `POST /api/ups/command`, gated four times (env allow, not `DASH_READ_ONLY`, the
upsd account actually having the command, a UI confirmation — typed `DANGER` for
anything that can cut power) and re-checked on every request; power-cutting commands are
refused outright while on battery. Hiding a button is not a safety control.
1. **Read-only by construction, with exactly two exceptions, and both are gated on purpose.**
Nothing writes to an upstream except `POST /api/ups/command` and
`POST /api/gateway/route`, and each is re-checked on every request by its own gates:
the UPS one is gated four times (env allow, not `DASH_READ_ONLY`, the upsd account
actually having the command, a UI confirmation — typed `DANGER` for anything that can
cut power) and power-cutting commands are refused outright while on battery; the gateway
one is gated ten (not `DASH_READ_ONLY`, `GATEWAY_ALLOW_WRITES` — off by default — a
per-address write throttle, a hand-parsed body, the route's state in the current
payload, the freshness of the read end to end, and a typed `danger="DANGER"` for
**both** verbs — a `dry_run` preview asks nothing and writes nothing — for anything
that can cut a route through the fleet's only ingress), and it dials once through
the same revocation gate the reads take. Hiding a button is not a safety control.
2. **Never invent a number.** A value the upstream does not publish renders as `n/a`, never
as `0`. Every derived value carries its `source` string and `derived: true`; the rate
estimator is keyed by the *resolved* series name **plus its label set** and refuses a
@@ -108,9 +115,9 @@ Breaking one is a bug even if the page still renders.
session's `X-CSRF-Token`. A process whose lifespan never produced `ctx.auth` answers
`503` for everything, public paths included. `tests/test_auth_api.py` walks the
**served** route table so a route that forgets the gate fails a test instead of
shipping — the middleware guards by default. Nothing may be added to `/api/health`
without re-deciding that: it is public, and a new field there is a field the whole
internet reads.
shipping — the middleware guards by default. `/api/health` is a contract with
deployment, which means **nothing may be added to it without re-deciding that**: it is
public, and a new field there is a field the whole internet reads.
## Architecture
@@ -118,7 +125,7 @@ Breaking one is a bug even if the page still renders.
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
│ GET /snapshot │ GET /metrics │ LIST UPS / VAR / CMD
▼ ▼ ▼
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command and post_route
│ parse + Δcounter/Δt
▼
sources/host.py · disks.py · llm.py · nut.py · gateway.py poller.py (task per source)
@@ -144,10 +151,25 @@ One line each — the load-bearing behaviour is in the bullet, the rest is in th
error line per non-`true` arm — the missing one saying **not found**, the pinned wording
the dashboard reads to answer "is it installed?" — and `--no-proxy` / `--no-resolve`
remove their keys in silence. Nothing in a section can fail `/snapshot` or `/health`.
Counters stay counters: the dashboard computes rates.
Counters stay counters: the dashboard computes rates. **The one write (1.4.0, gateway
hosts):** a single `POST /route`, refused without a verified, non-revoked client
certificate **and** the agent's own `--allow-writes` flag (certificate first, in that
order), forwarding exactly `{id, action, dry_run}` to the socket-activated root helper
`dashboards-routectl` and answering its JSON verbatim — **one connection, one
request, one answer, no retry**, because a retry of a write is a second reload. The
privilege is the helper's socket, not the agent's: the unit keeps
`NoNewPrivileges=true` + `ProtectSystem=strict` byte-identical with and without
`--with-routectl`, and no sudoers line is involved. Every `GET` path is untouched —
`#off` markers are read on the normal proxy sections (the marker's own grammar is the
agent's: `ROUTE_MARKER` / `split_marker` / `config_line_index` — `disabled` on
`server`/`proxy_pass`/`map` nodes; a marked line stays in the document).
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
env var always wins. Safe defaults except the sources, which default to unset. The
gateway source is enabled only when `GATEWAY_HOST` names a `HOST_TARGETS` entry.
gateway source is enabled only when `GATEWAY_HOST` names a `HOST_TARGETS` entry
(default `gateway`). The write's two settings are beside it: `GATEWAY_ALLOW_WRITES`
(default **off**, the NUT flag's truthy rule) and `GATEWAY_WRITE_TIMEOUT_SECONDS`
(default 35 s, deliberately longer than the agent's 30 s peer timeout so the dashboard
never times out first); the read's 10 s budget stays the read's alone.
- `poller.py` — job loop, health records, stale/last-good semantics, transition events.
`disks:<host>` is a source of its own (300 s / 30 s) and `gateway` is **one** job (30 s
/ 10 s) that exists only when configured, because the page is about the single ingress;
@@ -160,7 +182,11 @@ One line each — the load-bearing behaviour is in the bullet, the rest is in th
- `collectors.py` — the fetch half. `AgentTls` makes an `https://` `HOST_TARGETS` entry a
certificate-authenticated dial (per-dial client, this app's client cert verified against
its own root; the revocation answer is a fresh read per poll, so a `/certs` revoke bites
on the next poll with no restart, before any socket opens). Failure messages name the
on the next poll with no restart, before any socket opens). `post_route` is the write's
dial beside `fetch_snapshot`: the same per-dial client and the same revocation check
before any socket opens, but **`Authorization` is never sent on the certificate path** —
the bundle is the credential there, and the token travels only on the plain token path.
The answer is the helper's own object, verbatim. Failure messages name the
**redacted** URL. Agent `errors` become `degraded` **except** the add-on lines
(`ADDON_ERROR_PREFIXES`: `smart`, `haproxy`, `nginx`, `sockets`, `resolver`) — a host
without smartctl or a proxy config is unconfigured, not broken (rule 3); those lines
@@ -179,8 +205,15 @@ One line each — the load-bearing behaviour is in the bullet, the rest is in th
credited **only through an address the gateway's own `resolver` section published** —
the module never calls a resolver itself (spy-pinned). Pipes are `null`, never `0`, when
unmeasured. `warnings()` judges the two config files against themselves (seven kinds,
each `{id, kind, subject, message}`) and never joins a warning to a row. No `severity`
anywhere in the payload (a busy proxy is the job).
each `{id, kind, subject, message}`) and never joins a warning to a row. Each row's
`enabled` carries the config's own state (the 1.4.0 `#off` marker, or a map entry
pointed at the helper's sink); a merge that would mix states splits into two rows
keyed by `enabled` (a union of selectors with one disabled line is two facts; the
`#off` suffix keeps the halves distinct); the marker-caused warning shapes — an
emptied backend, a targetless route, a sink-pointed entry — warn nothing, while the
same shape from a hand-deleted line still warns; the switch-off tally is published
as `/api/summary`'s `disabled` field. No `severity` anywhere in the payload (a busy
proxy is the job).
- `sources/llm.py` — gufo + vLLM field maps, measured Δ rates **diffed per label set**,
prefill speed, the session list from gufo `/slots`. A refused estimator window is still
an answer: the cell keeps its `source` and an honest `Δ<window>` beside a `null`.
@@ -225,11 +258,18 @@ One line each — the load-bearing behaviour is in the bullet, the rest is in th
the startup read (it mints; nothing else may call it); `read_state` is the request-time
read (same validation, never a write). `cryptography` is the app's only new dependency,
used here alone; the agent stays stdlib.
- `api.py` — GETs everywhere plus the one POST that dials an upstream. Every source's
- `api.py` — GETs everywhere plus two POSTs that dial an upstream (`/api/ups/command` and
`/api/gateway/route`, invariant 1's two exceptions). Every source's
configured gate runs **before** anything that could reach the network (spy-pinned: no
client, no fetch). `/api/config` is a redaction boundary. All five history routes go
through one door (`history_many`). `/api/ups/command` stays the app's only upstream
write (pinned by enumerating every served route's verbs).
client, no fetch). `/api/config` is a redaction boundary and serves
`gateway.writes_enabled` so the page can render the honest arm. All five history routes go
through one door (`history_many`). The upstream write set is exactly those two (pinned by
enumerating every served route's verbs; a third fails by name). The route write's gates
(invariant 1) run in a pinned order — read-only, flag, per-address throttle, body
shape, the id's state in the current payload, end-to-end freshness, typed `DANGER` for
both verbs — and every answered call stores one audit event under its own kind
(`route-disabled`/`route-enabled`/`route-denied`; a dry run stores none) in a fixed
shape: the id, the action, the helper's words — **never the config text**.
- `app.py` — assembly and the **session gate** middleware (invariant 12), answering in
this order: no `AuthState` → `503` for everything; no session and no such route → the
router's `404`; public → through (the origin check still applies to a public write);
@@ -247,8 +287,10 @@ One line each — the load-bearing behaviour is in the bullet, the rest is in th
Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
`gpu.busy_pct@0`, `ups.charge_pct`), `@stream` for per-device series. `sample.source` is
the dashboard family — `host`, `llm`, `ups`, `disks` (the SMART series) and `gateway`
(the connection gauges) — so `GET /api/metrics?source=…` answers "what does *this page*
have history for". Treat the scheme as a public interface of the history API.
(the connection gauges: `gateway.in_conns@<listen-port>`, `gateway.out_conns@<host>:<port>`
— the port belongs in the outbound name because two services on one host are two facts) —
so `GET /api/metrics?source=…` answers "what does *this page* have history for".
Treat the scheme as a public interface of the history API.
## The fleet (real environment)
@@ -256,19 +298,43 @@ have history for". Treat the scheme as a public interface of the history API.
| ------------- | ------------- | ------------------------------------------------------------------------- |
| `driveripper` | `driveripper-ops` | Threadripper 24c/48t + Radeon AI PRO R9700, vLLM `:8000`, **upsd**, the dashboards. 16 whole disks; the agent unit runs `User=root` with `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` (kernel 7.2's unprivileged-passthrough gate; an empty bounding set is capability-less, `CapEff 0`) |
| `deskwork` | `deskwork-ops` | Strix Halo (Radeon 8060S iGPU), gufo `:8731` |
| `gateway` | `gateway-ops` | the router, the fleet's ingress: haproxy (`/etc/haproxy/haproxy.cfg`, six frontends → six backends) + nginx (`/etc/nginx/nginx.conf` plus `conf.d/` vhosts and `stream.d/`'s `map` on `:443`); runs agent **1.3.0** |
| `gateway` | `gateway-ops` | the router, the fleet's ingress: haproxy (`/etc/haproxy/haproxy.cfg`, six frontends → six backends) + nginx (`/etc/nginx/nginx.conf` plus `conf.d/` vhosts and `stream.d/`'s `map` on `:443`); runs agent **1.4.0** with the `dashboards-routectl` helper units |
- **Transport is direct dial; the fleet is mTLS.** `HOST_TARGETS` are `https://` with no
userinfo — the certificate bundle is the credential (the token in a URL is the
compatibility path for a host nobody has bundled). All three hosts run the installed
`dashboards-agent` systemd service, installed by `deploy/install-agent.sh` under the
sudo grant. **Re-stage `/home/ops/agent-install/` first — the grant names a path, not a
checkout —** and a re-run whose rendered unit differs from the installed one prints the
diff and refuses without `--force`. `deskwork`/`driveripper` still run **1.1.0**: the
1.2.0/1.3.0 sections are purely additive (a 1.1.0 snapshot has none of the keys, which
`sources/gateway.py` answers as `supported: false`), so they move at their next re-stage.
sudo grant — registration is the install: `--fetch-bundle <url> --cert-name <name>
--session <token>` pulls the host's own bundle from this dashboard during the install, so a
host is never registered onto a token URL because a `curl` was forgotten. **Re-stage
`/home/ops/agent-install/` first — the grant names a path, not a checkout —** and a re-run
whose rendered unit differs from the installed one prints the diff and refuses without
`--force`. `deskwork`/`driveripper` still run **1.1.0**: the
1.2.0/1.3.0/1.4.0 sections are purely additive (a 1.1.0 snapshot has none of the keys,
which `sources/gateway.py` answers as `supported: false`), so they move at their next
re-stage.
A TLS agent serves https only — rolling a host back means removing the bundle lines too,
not just the URL.
not just the URL. **The gateway's write (1.4.0) installs beside the agent, never into
it**: `install-agent.sh --allow-writes --with-routectl` adds one argument to the agent
unit's `ExecStart` (`--allow-writes` — a diff that shows a security directive moving
is a stop, not a `--force`) and installs `dashboards-routectl.py` (0755, root,
`/usr/local/sbin/`) plus the two units **`dashboards-routectl.socket`** (`0600`, owned
by the agent's user) and **`dashboards-routectl@.service`** (root, one instance per
connection, `ReadWritePaths` = `/etc/haproxy` + `/etc/nginx/conf.d` +
`/etc/nginx/stream.d` only), verifying the agent's uid can connect before reporting
success. **No sudoers line is involved** — the grant is a file mode systemd enforces,
and `--with-routectl` alone leaves the agent's unit byte-identical; the helper edits
one `#off` route (the marker carries the line's original text after its `was:`, a
per-route state sidecar records who last changed it and the file's hash, an `enable`
restores only an unedited line — an edited one is a `drift` refusal, no byte moves —
and a switched-off `map` entry points at the helper's own sink `server`), validates
with the proxy's own checker (restore on failure) and **reloads, never restarts**.
`--without-routectl` removes it, printing the still-off routes first;
`deploy/gateway-routectl-README.md` is the operator's page. The live **drill has not
run**: the install is verified end to end (task 07), but this fleet's proxies run in
podman containers, so the host has no `nginx`/`haproxy` binary for the checker —
`data/logs/NOTES-33-switching.md` records it, and a real `DANGER` is still the
operator's call.
- **NUT 2.8.5 on driveripper: configs in `/etc/ups/` (not `/etc/nut/`), upsd on port
3051 (not 3493 — nothing listens on 3493).** NUT 2.8 has no `ACL` directive;
per-source restriction is a **firewall** rule, never an `upsd.conf` line. Two UPSes:
@@ -367,7 +433,12 @@ operator's how-to for both.
configured routes branching off; its only colour pair is existence
(`--gate-on`/`--gate-off`) — no verdict colour anywhere on the page; its filter box is
an **eye, not a query** (it hides rows whose own payload text does not contain what was
typed; no DNS, no re-ranking).
typed; no DNS, no re-ranking). Each row's switch runs on `GATEWAY_ALLOW_WRITES`: off
is a `aria-disabled` reading with the flag's own sentence (no hover, no listener, no
wire); on is a real control whose press is the dry-run diff → typed `DANGER` → one
send, and the row's colour is the payload's word from the next poll — never the
click's — with `off` reading `--gate-off` end to end (row, spine, the dialog's diff
tints) and the diff scrolling in its own `<pre>`.
- Every chart names itself for assistive tech (`role="img"` + `aria-label`); every chart
on every page has one, pinned in `tests/test_api.py`.
@@ -376,9 +447,9 @@ operator's how-to for both.
1. The UPS gates are the control: the upsd account has `instcmds = all`, and NUT has no
call for a user's privileges — `enabled: true` cannot mean "may send", so the risk
tiers and typed `DANGER` are load-bearing, not decoration.
2. One admin, one factor: no TOTP/WebAuthn, no per-user accounts; a logout ends only
2. One admin, one factor — no second factor and no per-user accounts; a logout ends only
this browser's cookie, and the only revocation is a password change (which rotates the
session secret). The login throttle is per-process and keys on the socket peer address
session secret). The throttle is per-process and keys on the socket peer address
(behind nginx every login shares one bucket; a restart clears it).
3. GPU coverage is AMD only — an NVIDIA host would report no GPU at all.
4. UPS history depth: events and numeric series are stored; long-window capacity fade has
@@ -392,8 +463,12 @@ operator's how-to for both.
7. The fleet trusts **this** deployment's CA — a second deployment with its own
`certs.*` rows cannot dial these agents until it carries the same root or re-issues
and re-installs every bundle.
8. The `/gateway` route switches are built but **not wired** — a route write path is
phase 33's decision and amends invariant 1; until then the knobs stay `aria-disabled`.
8. Route switching is closed, and its honest limits are the gap list's:
`GATEWAY_ALLOW_WRITES` is off by default (a fresh deployment sees no switch arm at
all); the switch is one admin (gap 2) — whoever holds the session can switch a route
— with no per-host kill switch and no scheduler; and there is no runtime `disable
server` state: the marker is a config line, so a hand reload or a reboot does not
undo it (a runtime switch would be quietly undone by the next reload).
## Locked decisions (and why)