dashboards/.agents/phases/complete/33_gateway_route_switching/00_phase.md

Phase 33 — The switch is wired: turning one route off, and back on, without ever weakening the box

Source: operator request, 2026-10-23 — "Also write the phase to disable and enable services. I want to be able to turn off and on connections as needed in the event of security concerns or upgrades." + the two decisions taken with them in that conversation: enforcement is a config edit, validated and reloaded, by a narrow root helper on the gateway (so a disabled route stays disabled across a reload and a reboot), and one switch per connection line, as already drawn. Story: n/a Context: deploy/install-agent.sh (the unit template, whose NoNewPrivileges=true / ProtectSystem=strict / SystemCallFilter=@system-service / empty CapabilityBoundingSet this phase does not touch, the --force re-run guard, deploy/agent-sudoers as the precedent for a path-pinned grant), agent/host_agent.py (1.3.0 → 1.4.0: do_GET-only Handler, _certificate_allows, the haproxy/nginx parsers that must learn the disabled marker), src/dashboards/sources/gateway.py (connections[], config_connections, warnings()' seven kinds), src/dashboards/api.py (/api/ups/command's four-gate shape and the certs_router()'s audit-event rule), src/dashboards/collectors.py (AgentTls, fetch_gateway_snapshot, the revoke-before-dial rule), src/dashboards/config.py (GATEWAY_* block), src/dashboards/web/static/page-gateway.js (buildSwitch, routeInConfig, renderLeaf), tests/test_api.py (the verb-enumeration pins), tests/test_auth_api.py (the served-route walk), tests/test_deploy_scripts.py (the fake-root sandbox the helper's tests extend).

Objective

/gateway's switch becomes a control: disable brain.reeseapps.com and the nginx stream map entry stops routing it, persistently, with the config validated and the proxy reloaded and the page showing that exact route red until it is re-enabled. One new write route, POST /api/gateway/route, gated seven times, with the write itself performed off-box by a root helper on the gateway that the agent reaches over a systemd-activated unix socket — because the agent's own unit forbids privilege (NoNewPrivileges=true) and this phase is not going to weaken the sandbox of a read-only collector to give it a root shell.

This amends AGENTS.md invariant 1 from one exception to two, and that is the phase's real risk: a bug here is an outage on the fleet's only ingress. Everything below exists to make the blast radius one line of one file, and to make the app refuse rather than improvise whenever it cannot see what it is doing.

Scope decisions taken with the operator (binding for this phase)

#	Decision	Consequence
1	Invariant 1 becomes "two exceptions".	POST /api/ups/command and POST /api/gateway/route. Both: session, CSRF/origin on a cookie write, an env gate (NUT_ALLOW_ACTIONS / GATEWAY_ALLOW_WRITES, default off), a typed DANGER, and every gate re-checked server-side on every request. PLAN.md's anchor table records a read-only app, so this phase's overview carries the sign-off request; the phase never edits PLAN.md.
2	DANGER is required for both verbs.	Enabling is not the safe direction: applying either change re-validates the file and reloads the proxy, and a reload touches every route on the box. A word that means "I know this affects the ingress" has to mean it both ways.
3	The agent keeps its sandbox; the privilege lives outside it.	No sudo from the agent, no setuid, no change to NoNewPrivileges/ProtectSystem/SystemCallFilter. The write peer is dashboards-routectl.socket + dashboards-routectl@.service (systemd socket activation, Accept=yes, User=root, SocketUser=dashboards-agent, SocketGroup=dashboards-agent, SocketMode=0600): the agent connects to a socket it is filemode-permitted to connect to, and no other uid on the box can even open it, and the helper proves the peer uid with SO_PEERCRED. Access control is a file mode plus the mTLS client certificate that already authorises the agent's other paths — not a capability the agent process can be tricked into.
4	Only an id crosses a wire. Never a path, never a line, never text.	The body is {id, action, danger?, dry_run?} where id is the payload's own connections[].id. The helper re-parses the configs itself, maps that id to the one line it declared, and refuses anything it cannot map. There is no file, no line, no pattern, no command anywhere in the contract, and the argv to the helper is fixed with no shell.
5	A route is off in the way that actually stops traffic for its proxy.	haproxy: the server line is commented with a # dashboards:off … was: <verbatim original> marker. nginx vhost: the proxy_pass line is commented the same way. nginx stream map entry: commenting it would fall through to the map's default (127.0.0.1:4443) and mis-route the domain instead of blocking it, so the helper rewrites that entry's value to a sink (127.0.0.1:44450) and keeps the original in the marker; the sink is a server block the helper maintains in /etc/nginx/stream.d/00-dashboards-sink.conf, which the existing stream { include /etc/nginx/stream.d/*.conf; } glob already loads — so no hand edit of the main config is ever required. The map's default entry is refused: it is the fallback for every unmapped name, not a service.
6	Validate, restore on failure, reload — never restart.	Write the new bytes, run nginx -t / haproxy -c -f …; on failure put the previous bytes back, do not reload, and answer applied: false, validated: false, restored: true. On success systemctl reload <unit> — reload, because a restart cuts every Minecraft session and every in-flight upload on an ingress that is doing its job. A reload failure is reported with the unit's own status and the file is left in the applied state (the config is correct; the reload is the operator's next action, said out loud).
7	Self-describing marker + drift refusal, not silent trust.	The disabled line carries the original verbatim after was:, so a human can restore it by eye and the helper can restore it mechanically. A sidecar /var/lib/dashboards-routectl/state.json (0600, root) records actor, timestamp and the sha256 of what was written. Enabling restores only if the current line still matches what the helper wrote; otherwise it refuses and reports drift — a file an operator hand-edited is theirs to fix, not the app's to guess at.
8	Refuse rather than act blind.	The write is refused when: writes are not enabled; the id is not in the current payload; the route is already in that state (changed: false, no reload); either config section is not available: true; the gateway job is ok: false or stale beyond 3× its interval; the host:<GATEWAY_HOST> system job is not ok; the agent is revoked (refused before any socket, the existing rule); the throttle is tripped. That is the UPS "refused while on battery" rule, translated: do not rewrite a config you cannot currently read end to end.
9	off finally means something the payload said.	The agent's parsers learn the marker and report the marked server/proxy_pass/map entry with disabled: true, so the route stays in the graph as a declaration and connections[].enabled becomes false. routeInConfig() stops being a literal, the off styling phase 30 built becomes reachable, and warnings() learns the marker: a backend emptied by a marker is not haproxy_backend_empty (the operator chose that; the page shows it red), while a backend with no server line at all still is.
10	A live route's count is never stamped.	Disabling does not write 0 into the payload. The sockets drain on the reload and the next poll reports what /proc says. Invariant: no code path in this phase writes a count.
11	The audit is one event, fixed in shape, with no config text in it.	gateway-source events route-disabled / route-enabled / route-denied: verb, id, the session's actor name, applied/validated/reloaded/drift, and the proxy's own one-line error when there was one. The unified diff is returned to the caller's response (it is what the confirm dialog shows) and never stored — events are served to a browser and /api/events is a feed, not a config archive.
12	The overview is told, because "something is switched off" is not a page-only fact.	/api/summary's gateway block gains one field, disabled (an int count of enabled is false rows, null in the arms that did not measure) — the thirteenth key, restated in the pinned key set with its reason. A notifier has to learn about a deliberate outage without scraping a page, and it prints nothing at 0 (the warnings rule: a permanent 0 switched off is ink that trains the eye).
13	One route, one file, one reload per request.	A request touches exactly one config file and reloads exactly one proxy. There is no batch verb, no "disable all to this host", no dry_run that writes. If a security incident needs five routes off, that is five reviewed requests, each with its own event.

Out of scope: editing anything a marker cannot express (adding/removing vhosts, certificates, bind lines, the map's default), haproxy runtime disable server via a stats socket (a runtime-only state that a reload silently undoes — decision 5 is persistent instead), disabling an entire host or port, per-user authorisation (one admin, gap 6), a scheduler ("off at 02:00"), and any change to the agent's unit security directives or to another host's agent (gateway alone runs 1.4.0; deskwork/driveripper's 1.1.0 agents have no routes and are unaffected).

Dependencies

• 32_gateway_resolved_target_liveness (todo, must complete first) — agent 1.3.0's resolver section and the counted name routes are what a switched-off line reads against: off + a draining count to 0 is the proof the switch worked, and without resolution a disabled route and a never-counted route look identical.
• 31_gateway_route_readability (todo) — the row that names the inbound domain, the green end-to-end route, and the off styling this phase finally reaches.
• 30_gateway_connection_map (complete) — connections[].id as the vocabulary, the switch markup and its a11y contract, routeInConfig() as the named seam.
• 15_agent_mtls / 24_dev_mtls_rollout (complete) — the verified-client-certificate gate the new POST path inherits, and the revoke-before-dial rule this phase's collector reuses.
• 11_admin_login (complete) — session, CSRF, origin checks, LoginThrottle (the route:<address> bucket is its third key shape).
• Operator actions, not repo work: the dashboards-agent-to-helper socket grant lands through the units (no sudoers line needed), but installing units on a host and running the first live drill are the operator's — and the drill stops unless they are present. PLAN.md's anchor sign-off for invariant 1 is theirs too.

Tasks

1. 01_the_root_helper_edits_exactly_one_route.md — deploy/dashboards-routectl.py: show/list/diff/disable/enable/verify, the marker with was:, the sink file it manages, validate-restore-reload, drift refusal, --serve over a systemd-passed socket, --root for tests.
2. 02_the_privileged_peer_is_a_socket_not_a_sudo_line.md — the two systemd units + install-agent.sh --with-routectl; the proof that the agent's NoNewPrivileges=true/ProtectSystem=strict unit is unchanged and can still reach the socket.
3. 03_the_agent_accepts_one_write.md — agent 1.4.0: POST /route behind the verified client certificate and --allow-writes, body validation, one fixed peer call, plus the parsers reporting disabled for a marked line; /schema and the additivity control.
4. 04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.md — connections[].enabled, off reachable in the payload, warnings() marker-aware, stub/skeleton parity, the one new /api/summary field (gateway.disabled), and no count ever written.
5. 05_the_api_gates_the_write_ten_times.md — GATEWAY_ALLOW_WRITES, DASH_READ_ONLY, collectors.post_route, POST /api/gateway/route, the ten-gate order, the throttle, the events, /api/config.gateway.writes_enabled, and the verb-enumeration + served-route pins restated (this is where invariant 1 is amended in AGENTS.md).
6. 06_the_switch_obeys.md — page-gateway.js + gateway.html + dash.css: DANGER confirm with the diff from a dry_run first, patch from the response, routeInConfig() reading enabled, the writes-disabled arm that keeps the switch honest, footer paragraph 3.
7. 07_rehearse_then_drill_on_the_live_ingress.md — the fake-tree rehearsal, then the live install on gateway and a drill on the one named-safe route, restore, verify, data/logs/NOTES-33-switching.md. Stops, correctly, without the operator present.
8. 08_docs_and_live_verify.md — AGENTS.md (invariant 1, the agent bullet, sources/gateway.py, api.py, collectors.py, config.py's block, the /gateway UI paragraph, the fleet record, gap list) + README.md (including the operator's sudo/units steps), the browser pass, the mobile audit, the full gate.

Testing & Quality

• The helper is tested unprivileged, against a fake tree, with the proxy commands stubbed. tests/test_routectl.py runs the real helper in a tmp_path tree (--root) with nginx/haproxy/systemctl faked on PATH, and pins: one line changed and nothing else in the file (byte-diff assertion); was: restores verbatim; a failing validator restores the exact previous bytes and reloads nothing; default refused; an unknown id refused; a second disable changed: false; the sink file created on first use, left alone if an operator's file is there, removed only when it wrote it and nothing is disabled; drift refused rather than restored; SO_PEERCRED refused for a non-dashboards-agent uid in the fake-peer test; no shell=True anywhere (grep pin, the installer's precedent).
• The agent's write path is proven against a fake peer, not a real one. tests/test_agent.py runs the real handler over a TLS loopback port (the phase-15 harness) plus a unix-socket stub: POST /route without a client certificate is 403 client certificate required; with --allow-writes absent it is 403 writes are not enabled on this agent; a body with an unexpected key, a bad id, or a missing danger is refused before the peer is touched (asserted by the stub logging nothing); GET behaviour is byte-identical to 1.3.0 (the additivity control re-run).
• The API gates are proven twice, as the UPS ones are. tests/test_api.py walks every gate with spies: writes-disabled refuses without dialling; a stale gateway job refuses; a revoked agent refuses without a socket; DANGER is required for disable and enable; a right request produces one peer call, one event, and the response's applied/validated/reloaded triple. tests/test_auth_api.py's served-route walk demands a session for /api/gateway/route, and the CSRF pin is extended to it on the wire (a scripted helper records nothing without the header, once with it) — a 403 after the write is an outage with an error message attached.
• Live, opt-in, and reversible: -m live --live gains test_the_switch_round_trip_on_one_named_route which is skipped unless --allow-live-writes is passed as well, names the route in the skip reason, and always restores in a finally. The default live suite stays read-only (AGENTS.md's rule survives this phase; the drill belongs to task 07 with the operator watching).
• Coverage >90% on deploy/dashboards-routectl.py, agent/host_agent.py, sources/gateway.py, collectors.py, api.py; the whole suite 100%; bash .agents/validate.sh green.

Documentation & Consistency

• The final pass reviews README.md and AGENTS.md in full against the project as it will stand after this phase and fixes every inconsistency found, including pre-existing ones. This phase's specific list: invariant 1 rewritten to two exceptions with the seven gates named (and the sentence "hiding a button is not a safety control" kept, because it is the reason the switches are gated server-side rather than greyed in JS); the agent bullet's POST /route and the disabled marker; the sources/gateway.py bullet's enabled/off rules and the marker-aware warnings(); the api.py bullet's write lists (fleet writes are now two); the collectors.py bullet (post_route, the revocation check it shares); config.py's GATEWAY_ALLOW_WRITES row and README's configuration table; the /gateway UI-conventions paragraph (the switch is wired, what off looks like, what the writes-disabled arm says); the fleet record (gateway 1.4.0, its unit's security directives unchanged, the two new systemd units, NOTES-33-switching.md); and the gap list (name-route switching closed, plus the honest limits: one admin, no scheduler, no per-host kill).
• README.md gains the operator's install steps for the helper units and the exact --with-routectl invocation, plus the recovery command for the state file (systemctl status dashboards-routectl.socket, routectl list, routectl verify).
• Tasks update both files immediately when their own changes are user-visible.

Completion Criteria

• With GATEWAY_ALLOW_WRITES=1, /gateway disables brain.reeseapps.com: the confirm dialog shows the diff from a dry_run, asks for DANGER, and the row turns red (off) on the poll after the reload while its live count drains to 0 on its own — no code wrote that zero.
• ssh gateway-ops 'sudo /usr/local/sbin/dashboards-routectl.py list' names exactly that route; verify reports no drift; the marked line contains the original verbatim after was:; the map entry points at the sink and stream.d/00-dashboards-sink.conf exists and is owned by the helper's header.
• Enabling restores the original bytes exactly (byte-compare against a pre-drill copy), the row turns green, and list is empty afterwards. The live ingress ends the phase in the state it started in.
• Every gate refuses with its own sentence and without touching a peer: writes disabled, no session, no CSRF, wrong Origin, missing DANGER, unknown id, already-in-state, section unreadable, gateway job stale, host:gateway not ok, agent revoked. Each is a test, and each test asserts the stub peer logged nothing.
• The agent's unit is unchanged: systemctl cat dashboards-agent still shows NoNewPrivileges=true, ProtectSystem=strict, SystemCallFilter=@system-service, and the same sha256 the phase recorded before installing the units — the sandbox was not traded for the feature.
• A failing validator (nginx -t stubbed to exit 1) restores the previous bytes, reloads nothing, answers applied: false, validated: false, restored: true, and the page's line never turned off.
• POST inventory: the app now has exactly two upstream writes (/api/ups/command, /api/gateway/route) and the enumeration pin lists them as such; AGENTS.md invariant 1 says two and names both gates; /api/health gained nothing.
• uv run pytest (with --cov ≥ 94), uv run ruff check src tests agent deploy, uv run pyright, node --check, bash .agents/validate.sh all green; uv run pytest -m live --live green without --allow-live-writes (the round-trip test skipped by default, its reason naming the route it would touch).
• data/logs/33-*.png (switch off in both themes, the DANGER dialog with its diff, the writes-disabled arm, 360 px) and data/logs/33-*.out; data/logs/27-task02-mobile-audit.cjs exits 0 with 0 FAIL / 0 FINDING; data/logs/NOTES-33-switching.md records the drill, the hashes and the restore.
• README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass), including the operator's steps for undoing all of this (--without-routectl, remove the units, re-enable every route).
• No behaviour change in completed work: the other five dashboards, /settings, /certs, /login, the UPS command path and its four gates, and every non-gateway source behave exactly as phase 32 left them.