phase: 33_gateway_route_switching
Phase 33 completion pass is done. Final report:
## Phase 33 — completion report
**All 8 task files complete** (`.agents/phases/complete/33_gateway_route_switching/`), full gate green, docs verified.
### Work in this pass
1. **`tests/test_routectl.py` added** (17 tests, all green) — exercises the *real* helper binary unprivileged against a fake tree (`--root`), the real agent parser, the captured gateway fixtures, and stubbed `nginx`/`haproxy`/`systemctl` on `PATH`. Pins the full protocol (locate → state → dry-run → temp-file/fsync/`os.replace` write → checker → restore on checker failure → reload-never-restart → re-read proof → sidecar record), all three route shapes (haproxy `server`, nginx `proxy_pass`, nginx stream `map` via its sink-value id), `drift` / unknown-id / `map-default` refusals, `list`/`verify`, and the `--serve` peer gate (`SO_PEERCRED` uid check, one JSON line per request, exact body shape).
2. **Real helper defect fixed** — `deploy/dashboards-routectl.py`: `main()`'s early-return paths returned `({"ok": False, …}, EXIT_*)` tuples, so `sys.exit()` printed a Python repr instead of a JSON line (usage errors, agent-import refusal). Now prints `json.dumps(…)` and returns only the code, keeping the one-JSON-line contract.
3. **Documentation consistency fixes** (task 08 criterion 6):
- `README.md`: garbled verdict-colour sentence on the switch ("green and to the position") → "in the **on** position and tinted `--gate-on`" (the existence-pair contract).
- `README.md`: audit sentence now names all three event kinds — `route-disabled` / `route-enabled` / **`route-denied`** (the code stores one per refused call; dry runs store none).
- `page-gateway.js`: replaced the stale pre-33 module docstring ("the switch is not wired") with the two-arm `GATEWAY_ALLOW_WRITES` contract.
- `AGENTS.md`: every phase-33 claim re-verified against code (the ten gates, the 35 s/30 s budget pair, `gateway.writes_enabled`, `/api/summary`'s `disabled`, the 1.4.0 agent bullet, the gap list) — no changes needed.
### Verification
- `uv run pytest --cov`: **1970 passed**, 18 skipped — coverage **98.74%** (floor 94%)
- `ruff` clean · `pyright` 0 errors · `bash .agents/validate.sh` **exit 0** (incl. `node --check` on every page script)
- Dev server running; all 10 sources `ok`; bundles exported
### Live infrastructure
**No live changes made.** The live drill remains unexecuted per `data/logs/NOTES-33-switching.md` (no operator approval; checker binaries unavailable in the podman proxy units). All routes stay enabled, config hashes unchanged, and the sandbox rehearsal + backup tree are in place for the operator's own `DANGER` call. All changes are in the working tree only.
This commit is contained in:
@@ -10,7 +10,7 @@ file:
|
||||
| `/disks` | Are any of my **physical** disks failing, and which one — health, temperature, power-on hours, wear, sector alarms | the same agents, running `smartctl -i -A -H` per whole drive (never a partition or a mount); a host whose agent has no `smartctl` is not a host on this page |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics` — read-only |
|
||||
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
|
||||
| `/gateway` | Which domains are coming in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`): a trunk per bound port with its configured routes branching off it, each line reading the config's own **inbound** word (`server_name`, `map` key, `backend` name) on line 1 over the counterpart it routes to on line 2 (`→ host:port`), plus its live connection count — and one filter box above the map that hides the routes whose own labels do not contain what you typed, asking nothing of the server | the agent on **one** host, `GATEWAY_HOST` (default `gateway`), reading its `haproxy.cfg` + nginx tree and `/proc/net/tcp{,6}` — so it needs **agent 1.2.0**, plus **1.3.0**'s `resolver` section to put a number on a route whose target the config writes as a *name* (without it those lines are drawn and dashed), and the whole page is *unconfigured* (never red) unless `GATEWAY_HOST` names a `HOST_TARGETS` entry |
|
||||
| `/gateway` | Which domains are coming in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`): a trunk per bound port with its configured routes branching off it, each line reading the config's own **inbound** word (`server_name`, `map` key, `backend` name) on line 1 over the counterpart it routes to on line 2 (`→ host:port`), plus its live connection count — and one filter box above the map that hides the routes whose own labels do not contain what you typed, asking nothing of the server | the agent on **one** host, `GATEWAY_HOST` (default `gateway`), reading its `haproxy.cfg` + nginx tree and `/proc/net/tcp{,6}` — so it needs **agent 1.2.0**, plus **1.3.0**'s `resolver` section to put a number on a route whose target the config writes as a *name* (without it those lines are drawn and dashed), plus **1.4.0**'s `#off` marker read to show a route the switch has turned off **or back on** (without it an off route is drawn on, because the marker is a comment a 1.3.0 agent does not read) — an off route stays drawn and counted, its row simply reads `enabled: false` (its id gains a `#off` suffix), and the config-integrity warnings the marker *caused* (a backend emptied of its last `server`, a map entry pointed at the switch's sink) are read as a chosen state while the same shape from a hand-deleted line is still a warning — and the whole page is *unconfigured* (never red) unless `GATEWAY_HOST` names a `HOST_TARGETS` entry |
|
||||
|
||||
No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
|
||||
runtime; every asset is served by this app.
|
||||
@@ -111,8 +111,12 @@ sudo deploy/install-agent.sh --fetch-bundle https://dashboards.example.com \
|
||||
sudo deploy/install-agent.sh --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN
|
||||
# ...or a kernel-gated host (SMART passthrough, below)
|
||||
sudo deploy/install-agent.sh --force # ...only when it must overwrite a unit it would change
|
||||
sudo deploy/install-agent.sh --with-routectl # ...plus the route-switch helper (the gateway, below)
|
||||
sudo deploy/install-agent.sh --allow-writes --with-routectl # the gateway's full write install:
|
||||
# the flag on the agent unit + the helper beside it
|
||||
deploy/install-agent.sh --status # unit state and /health (https when TLS)
|
||||
sudo deploy/install-agent.sh --uninstall
|
||||
sudo deploy/install-agent.sh --without-routectl # remove helper + units, keep config and state
|
||||
```
|
||||
|
||||
Re-running the installer is the silent, idempotent path it has always been **as long as the unit
|
||||
@@ -127,9 +131,30 @@ No root shell on the monitored host? `deploy/agent-sudoers` is the single sudoer
|
||||
lets the fleet's unprivileged `ops` account run the installer; run it without root and it
|
||||
prints that exact line with the script's own resolved path instead of saying "run as root".
|
||||
|
||||
It is read-only by construction: there is no code path that writes, and the HTTP server
|
||||
only answers `GET`. Counters are reported as counters — the dashboard computes rates over
|
||||
a labelled window, so the agent never has to remember who polled it last.
|
||||
It is read-only by construction **except the one write below**: every `GET` path answers
|
||||
a read, and the HTTP server's one other path is the gated `POST /route`. Counters are
|
||||
reported as counters — the dashboard computes rates over a labelled window, so the agent
|
||||
never has to remember who polled it last.
|
||||
|
||||
**The one write (1.4.0, the gateway).** 1.4.0 adds a single `POST /route`, reachable only
|
||||
with a verified client certificate and the agent's own `--allow-writes` flag, and it dials
|
||||
a **socket-activated root service** rather than giving the agent any privilege. The full
|
||||
write install is `--allow-writes --with-routectl`: the first adds exactly one argument to
|
||||
the agent unit's `ExecStart` (a re-run diff that shows a security directive moving is a
|
||||
stop, not a `--force`), and the second installs `deploy/dashboards-routectl.py` to
|
||||
`/usr/local/sbin/` (0755, root) and the two units **`dashboards-routectl.socket`** (a
|
||||
`0600` stream socket at `/run/dashboards-routectl.sock`, owned by the agent's user) and
|
||||
**`dashboards-routectl@.service`** (the helper, root, one instance per connection, `StandardInput=socket`,
|
||||
`ReadWritePaths` limited to `/etc/haproxy`, `/etc/nginx/conf.d` and `/etc/nginx/stream.d`), then
|
||||
verifies that the agent's own uid can connect before reporting success. **No sudoers line is
|
||||
involved**: the grant is a file mode systemd enforces, the helper's `SO_PEERCRED` check is
|
||||
the second look, and with `--with-routectl` alone the rendered `dashboards-agent.service` is
|
||||
byte-identical. The helper
|
||||
edits exactly one route (a `#off <name>` marker), validates the file with the proxy's own
|
||||
checker before it means anything, restores the previous bytes on a checker failure, and
|
||||
**reloads, never restarts**; `--without-routectl` removes it again, printing the list of
|
||||
still-off routes first. `deploy/gateway-routectl-README.md` is the operator's page for all
|
||||
of it.
|
||||
|
||||
Per-disk `smart` health is a best-effort add-on on the whole-disk `/sys/block` list (no
|
||||
partitions): the agent runs `smartctl -i -A -H` per physical disk — 5 s bound per device,
|
||||
@@ -271,8 +296,9 @@ a documentation one (RFC 5737 / RFC 3849), because a global IPv6 on Linux is EUI
|
||||
visitor's address is not ours to publish. Every port, state, column and width in those tables is
|
||||
verbatim, and those files are volatile by nature — a proxy in service has a different connection set
|
||||
every second — so the tests assert shape and the ports the configs claim, never a particular row.
|
||||
`AGENT_VERSION` is **1.3.0**, the version that adds the `resolver` section on top of the three
|
||||
phase 28 shipped; `GET /schema` names all four, each with its `available` flag, and `--no-proxy`
|
||||
`AGENT_VERSION` is **1.4.0** — the `resolver` section (1.3.0) and the `#off` marker read
|
||||
plus the one write (below) on top of the three phase 28 shipped; `GET /schema` names all
|
||||
four, each with its `available` flag, and `--no-proxy`
|
||||
and `--no-resolve` are listed there too. A host still running 1.2.0 answers with a snapshot that
|
||||
simply has no `resolver` key — which is why the version spread costs nothing: a section an agent
|
||||
did not collect is absence, and the gateway page counts the address-written routes it has and draws
|
||||
@@ -714,10 +740,12 @@ curl -s -H "Authorization: Bearer $SESSION" -X POST -H 'content-type: applicatio
|
||||
-d '{"ups":"apc","command":"test.panel.start"}' http://127.0.0.1:8090/api/ups/command
|
||||
```
|
||||
|
||||
That is the app's one write endpoint **that dials anything** (the `/api/auth/*` and `/api/certs/*`
|
||||
writes change only this process and its SQLite `meta` table — invariant 1 is about the fleet), and
|
||||
the bearer header changes nothing about its gates: a command that can cut output still needs
|
||||
`"confirm":"DANGER"`, and is still refused while the UPS is on battery.
|
||||
That is the UPS half of the app's two write endpoints **that dial anything** (the
|
||||
`/api/auth/*` and `/api/certs/*` writes change only this process and its SQLite `meta` table —
|
||||
invariant 1 is about the fleet; the other half is the `/gateway` route switch,
|
||||
`GATEWAY_ALLOW_WRITES`-gated), and the bearer header changes nothing about its gates:
|
||||
a command that can cut output still needs `"confirm":"DANGER"`, and is still refused
|
||||
while the UPS is on battery.
|
||||
|
||||
### Lockout
|
||||
|
||||
@@ -754,7 +782,9 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
|
||||
| `GUFO_API_KEY` / `VLLM_API_KEY` | only if an engine is behind a bearer token; they never leave the server |
|
||||
| `NUT_HOST` / `NUT_PORT` | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT |
|
||||
| `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) |
|
||||
| `NUT_ALLOW_ACTIONS` | `1` enables the one write endpoint that dials **anything** — the `/api/auth/*` and `/api/certs/*` writes touch only this process and its `meta` table |
|
||||
| `NUT_ALLOW_ACTIONS` | `1` enables the UPS write endpoint that dials **anything** — the `/api/auth/*` and `/api/certs/*` writes touch only this process and its `meta` table |
|
||||
| `GATEWAY_ALLOW_WRITES` | `1` enables the second write endpoint, the `/gateway` route switch (`POST /api/gateway/route`); off by default, the same truthy rule as `NUT_ALLOW_ACTIONS`, and the two writes are two decisions — one flag does not carry the other |
|
||||
| `GATEWAY_WRITE_TIMEOUT_SECONDS` | the route write's own budget (default 35 s), deliberately longer than the agent's 30 s peer timeout so the dashboard never times out first and leaves you guessing whether the reload happened; the read's 10 s stays the read's own |
|
||||
| `NUT_SIMULATE` | `1` replaces upsd with a **labelled** simulator (development only) |
|
||||
| `DASH_READ_ONLY` | `1` disables every mutating endpoint regardless of other flags |
|
||||
| `DASH_DB` | SQLite path (default `data/dashboards.sqlite3`) |
|
||||
@@ -1218,17 +1248,23 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
|
||||
behind a collapsed **Charts** toggle like `/system`'s, one `/api/gateway/history` request per
|
||||
panel per refresh. With no `GATEWAY_HOST` the page is the setup sentence rather than a red panel,
|
||||
and an ingress running an agent older than **1.2.0** says so in the agent's own words with every
|
||||
count `n/a` and nothing coloured. Each line's head carries the sketched **switch** — a real
|
||||
`role="switch"` button, green and to the position because the route is in the config, named in full
|
||||
for a screen reader (`endpoint, reached by <selector>, in the nginx config, on`) — and it is
|
||||
**unwired on purpose**: `aria-disabled` with a tooltip that says so, and this app writes to no
|
||||
upstream, so a route is disabled by editing `haproxy.cfg` or an nginx `server` block and by nothing
|
||||
in a browser. Giving either of those two dashes or knobs a real backing store is deliberately **not**
|
||||
documented here as a thing you can have: name-based liveness needs the agent to report resolved
|
||||
targets (a future agent version), and switching a route needs a second gated write path in front of
|
||||
the proxy configs, which is a change to this app's read-only promise rather than a feature. What runs
|
||||
today is **agent 1.3.0** — name resolution on the gateway, counts on every declared route, and no
|
||||
switching.
|
||||
count `n/a` and nothing coloured. Each line's head carries the **switch** — a real
|
||||
`role="switch"` button, in the **on** position and tinted `--gate-on` because the route is in the config, named in full
|
||||
for a screen reader (`endpoint, reached by <selector>, in the nginx config, on`) — and it has two
|
||||
arms on `GATEWAY_ALLOW_WRITES`. With the flag off — the default — it is **unwired on
|
||||
purpose**: `aria-disabled` with a tooltip that says so, and a route changes only by
|
||||
editing `haproxy.cfg` or an nginx `server` block, never in a browser. With
|
||||
`GATEWAY_ALLOW_WRITES=1` the same control is live:
|
||||
activation opens one confirm dialog that first dials the helper as a dry run (`dry_run: true` —
|
||||
it validates and answers with the one-line diff without writing a byte), then asks the operator
|
||||
to type `DANGER`, then sends exactly once. The helper edits the route in its config file,
|
||||
validates the file with the proxy's own checker and reloads that proxy — nothing is restarted —
|
||||
and the switch flips the moment the helper answers `applied: true`, with the row re-read from
|
||||
the next gateway poll. The write takes the read's own revocation gate on the same dial, and its
|
||||
gates (read-only, the flag, a per-address throttle, the route's state in the current payload, the
|
||||
read's freshness, the typed `DANGER` for both verbs) all run before anything reaches the host.
|
||||
What runs today is **agent 1.4.0** — name resolution on the gateway, counts on every declared
|
||||
route, the `#off` marker read, and the switch when the flag is set.
|
||||
The four event feeds (`/system`, `/disks`, `/llm`, `/ups`) are a **paged window** over the `event`
|
||||
table rather than a printout of it: the list is a bounded box that scrolls, 50 events to a
|
||||
page, with `1–50 of 200` between `‹ prev` and `next ›` (disabled at the ends) underneath.
|
||||
@@ -1444,9 +1480,10 @@ advertising a window no poll ever measured.
|
||||
|
||||
## Battery-test buttons
|
||||
|
||||
Instant commands are the only endpoint in the application that changes anything **outside this
|
||||
process** (`POST /api/ups/command`; the session and certificate-authority writes change only this
|
||||
app's own rows), and they are gated four times: `NUT_ALLOW_ACTIONS=1`, not `DASH_READ_ONLY`, an `upsd.users`
|
||||
Instant commands are one of exactly two endpoints in the application that change anything
|
||||
**outside this process** (the other is the route switch, in the next section; the session
|
||||
and certificate-authority writes change only this app's own rows), and they are gated
|
||||
four times: `NUT_ALLOW_ACTIONS=1`, not `DASH_READ_ONLY`, an `upsd.users`
|
||||
account that actually has the command, and a confirmation in the UI — destructive ones
|
||||
(deep test, calibration, `load.off`) require typing `DANGER`. Two more sit in front of those
|
||||
(since the login landed): a session, and this session's CSRF token when the write arrived on the
|
||||
@@ -1493,6 +1530,58 @@ and an unauthenticated `LIST VAR` from any routed source is the normal state of
|
||||
`NUT_SIMULATE=1` stands in for upsd during
|
||||
development; everything it produces is badged `SIMULATED`, and it is never the default.
|
||||
|
||||
## Route switching
|
||||
|
||||
`/gateway` can change its own subject: each row's switch turns a route **off** and, the
|
||||
other way around, **back on**. Off is the helper's `#off` marker on the config line
|
||||
(a switched-off `map` entry is pointed at the helper's own sink `server` instead); on is
|
||||
the marker removed and the line's original text restored — but only if the line is
|
||||
unedited since (an edited line is a `drift` refusal and no byte moves). The switch is
|
||||
**a decision of its own**, off by default, and it is the application's second write path
|
||||
(the first is the UPS command buttons above).
|
||||
|
||||
### Enabling it
|
||||
|
||||
The write is the flag `GATEWAY_ALLOW_WRITES` (the same truthy rule as
|
||||
`NUT_ALLOW_ACTIONS`, and one flag does not carry the other) and it only exists on a
|
||||
host the operator installed `--allow-writes --with-routectl` on — the certificate-first
|
||||
gate on the agent and the socket-activated root helper beside it (*The host agent*
|
||||
section above). The install is the agent unit's one extra argument plus the helper and
|
||||
its two units, `dashboards-routectl.socket` and `dashboards-routectl@.service`; to undo
|
||||
everything, `sudo ./deploy/install-agent.sh --without-routectl` removes the helper and
|
||||
its units, **leaves the proxy configs and the state directory alone**, and prints the
|
||||
still-off routes first — those are yours to switch back on. With the flag off, every
|
||||
switch is a `disabled` reading with the flag's own sentence, and a click does nothing:
|
||||
no wire is touched, no request is sent.
|
||||
|
||||
### Switching a route
|
||||
|
||||
The press runs the flow the operator can see: the **dry-run diff first** — the
|
||||
helper's own word for what the change will do, before the operator is asked for the
|
||||
word — then the confirmation that carries the diff and a **typed `DANGER`** (both verbs
|
||||
ask for it: turning a route back on is a write to the fleet's only ingress, just as
|
||||
turning it off is), then **exactly one send** — one connection, one request, one
|
||||
answer, **no retry** (a retry of a write is a second reload). The answer's words are
|
||||
printed verbatim: `applied: true` flips the switch and the row's colour follows the
|
||||
**payload's** word on the next gateway poll — never the click's; `applied: false`
|
||||
flips nothing. Every call stores one audit event — `route-disabled`, `route-enabled`, or `route-denied`
|
||||
for the gates' and the helper's refusals (a dry run stores none: it changed nothing) —
|
||||
so the audit trail is one line per call with who and when — and a switch **outages the
|
||||
route on purpose: it will appear in the event feed**.
|
||||
|
||||
The honest limits: the switch is one admin (the sign-in's one factor, no more), with no
|
||||
per-host kill switch and no scheduler; and there is no runtime `disable server` state —
|
||||
the marker is a config line, so a hand reload or a reboot does not undo it (a runtime
|
||||
switch would be quietly undone by the next reload).
|
||||
|
||||
When the dashboard is the thing that is broken at 03:00, the recovery is on the box:
|
||||
`/usr/local/sbin/dashboards-routectl.py` (its operator page is
|
||||
`deploy/gateway-routectl-README.md`). `sudo /usr/local/sbin/dashboards-routectl.py list`
|
||||
says what is off (the state sidecar), `sudo … verify` checks the sidecar against the
|
||||
files (drift is reported, never repaired), and `sudo … enable --id <id>` puts one route
|
||||
back on through the same protocol — the write, the proxy's own checker, restore on a
|
||||
checker failure, the reload (never a restart), the state record.
|
||||
|
||||
## Deployment
|
||||
|
||||
`Containerfile` is a two-stage build: `uv.lock` is resolved in the builder (exactly the
|
||||
|
||||
Reference in New Issue
Block a user