diff --git a/.agents/phases/todo/33_gateway_route_switching/00_phase.md b/.agents/phases/complete/33_gateway_route_switching/00_phase.md similarity index 100% rename from .agents/phases/todo/33_gateway_route_switching/00_phase.md rename to .agents/phases/complete/33_gateway_route_switching/00_phase.md diff --git a/.agents/phases/todo/33_gateway_route_switching/01_the_root_helper_edits_exactly_one_route.md b/.agents/phases/complete/33_gateway_route_switching/01_the_root_helper_edits_exactly_one_route.md similarity index 100% rename from .agents/phases/todo/33_gateway_route_switching/01_the_root_helper_edits_exactly_one_route.md rename to .agents/phases/complete/33_gateway_route_switching/01_the_root_helper_edits_exactly_one_route.md diff --git a/.agents/phases/todo/33_gateway_route_switching/02_the_privileged_peer_is_a_socket_not_a_sudo_line.md b/.agents/phases/complete/33_gateway_route_switching/02_the_privileged_peer_is_a_socket_not_a_sudo_line.md similarity index 100% rename from .agents/phases/todo/33_gateway_route_switching/02_the_privileged_peer_is_a_socket_not_a_sudo_line.md rename to .agents/phases/complete/33_gateway_route_switching/02_the_privileged_peer_is_a_socket_not_a_sudo_line.md diff --git a/.agents/phases/todo/33_gateway_route_switching/03_the_agent_accepts_one_write.md b/.agents/phases/complete/33_gateway_route_switching/03_the_agent_accepts_one_write.md similarity index 100% rename from .agents/phases/todo/33_gateway_route_switching/03_the_agent_accepts_one_write.md rename to .agents/phases/complete/33_gateway_route_switching/03_the_agent_accepts_one_write.md diff --git a/.agents/phases/todo/33_gateway_route_switching/04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.md b/.agents/phases/complete/33_gateway_route_switching/04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.md similarity index 100% rename from .agents/phases/todo/33_gateway_route_switching/04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.md rename to .agents/phases/complete/33_gateway_route_switching/04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.md diff --git a/.agents/phases/todo/33_gateway_route_switching/05_the_api_gates_the_write_ten_times.md b/.agents/phases/complete/33_gateway_route_switching/05_the_api_gates_the_write_ten_times.md similarity index 100% rename from .agents/phases/todo/33_gateway_route_switching/05_the_api_gates_the_write_ten_times.md rename to .agents/phases/complete/33_gateway_route_switching/05_the_api_gates_the_write_ten_times.md diff --git a/.agents/phases/todo/33_gateway_route_switching/06_the_switch_obeys.md b/.agents/phases/complete/33_gateway_route_switching/06_the_switch_obeys.md similarity index 100% rename from .agents/phases/todo/33_gateway_route_switching/06_the_switch_obeys.md rename to .agents/phases/complete/33_gateway_route_switching/06_the_switch_obeys.md diff --git a/.agents/phases/todo/33_gateway_route_switching/07_rehearse_then_drill_on_the_live_ingress.md b/.agents/phases/complete/33_gateway_route_switching/07_rehearse_then_drill_on_the_live_ingress.md similarity index 100% rename from .agents/phases/todo/33_gateway_route_switching/07_rehearse_then_drill_on_the_live_ingress.md rename to .agents/phases/complete/33_gateway_route_switching/07_rehearse_then_drill_on_the_live_ingress.md diff --git a/.agents/phases/todo/33_gateway_route_switching/08_docs_and_live_verify.md b/.agents/phases/complete/33_gateway_route_switching/08_docs_and_live_verify.md similarity index 100% rename from .agents/phases/todo/33_gateway_route_switching/08_docs_and_live_verify.md rename to .agents/phases/complete/33_gateway_route_switching/08_docs_and_live_verify.md diff --git a/.agents/phases/todo/34_radiance_llm_engine/00_phase.md b/.agents/phases/todo/34_radiance_llm_engine/00_phase.md new file mode 100644 index 0000000..1f5d896 --- /dev/null +++ b/.agents/phases/todo/34_radiance_llm_engine/00_phase.md @@ -0,0 +1,54 @@ +# Phase 34 — Radiance is a first-class engine: a third `/llm` panel that says what its /metrics and /stats publish, nothing invented + +**Source:** operator request, 2026-10-24 — "add support for a new llm engine 'radiance' which is running on driveripper now. Check it out at http://driveripper.reeselink.com:8000/ and come up with a plan." + the two decisions taken with them in that conversation: **radiance replaced vLLM at driveripper:8000** (the dev config's `VLLM_BASE_URL` points at the engine that is now radiance, so the dev config moves to `RADIANCE_BASE_URL`; vLLM *code* stays — the API shape is additive-only and other deployments may run vLLM), and **the per-position spec-acceptance curve is in** (seven positions, each its own acceptance ratio). +**Story:** n/a +**Context:** `src/dashboards/sources/llm.py` (the gufo/vLLM field maps, the rule-2 throughput precedence, `HISTORY_METRICS`, `_sessions()`, `fleet_totals()`, `SEVERITY_CELLS`), `src/dashboards/config.py` (the `*_BASE_URL`/`*_API_KEY` engine settings, `any_source_enabled`), `src/dashboards/collectors.py` (`fetch_engine`, the best-effort `/slots` fetch, `llm_jobs`), `src/dashboards/api.py` (the `llm_router` 404 gate, `/api/config`'s llm block, the summary's llm block, the history defaults), `src/dashboards/alerts.py` (the cell-driven `_engine_alerts`), `src/dashboards/severity.py` (the only threshold table), `src/dashboards/web/static/page-llm.js` + `pages/llm.html` (the `only:` scoping, strip cells, slot grid), `tests/test_llm_source.py` / `tests/test_collectors.py` / `tests/test_api.py` / `tests/test_config.py` / `tests/test_live_fleet.py`, `scripts/capture-fixtures.sh`, `.env.dev` / `.env.dev.example`, `README.md`. + +## Objective + +The dashboard treats radiance (a vLLM-derived engine, v1.15.1, model `juggernaut`, on `driveripper:8000`) as a **first-class third engine** — `llm:radiance` — with its own field map pinned by a verbatim capture, its native windowed throughput gauges winning per rule 2, its `/stats` endpoint as the session source (radiance's answer to gufo's `/slots`), and a radiance strip on `/llm` (queue cap, version/uptime, per-card VRAM, the 7-bar spec-acceptance curve) — while gufo and vLLM keep their exact payloads, history names, and behaviour. + +## Scope decisions taken with the operator (binding for this phase) + +| # | Decision | Consequence | +|---|----------|-------------| +| 1 | **Radiance replaced vLLM at driveripper:8000.** | `.env.dev`'s `VLLM_BASE_URL` line becomes `RADIANCE_BASE_URL` (same address). The vLLM engine **code and env var stay** — `VLLM_BASE_URL` still configures a vLLM engine for other deployments; only the dev fleet's config moves. `AGENTS.md`'s fleet table says `vLLM :8000` for driveripper; that is the one stale fact the final pass fixes. | +| 2 | **The per-position spec-acceptance curve is in.** | `radiance:draft_position_acceptance_ratio{position="0".."6"}` renders as a 7-bar hand-rolled SVG on the radiance strip, `role="img"` + `aria-label`, pinned in `tests/test_api.py` beside every other chart. Snapshot-only: the seven per-position ratios do **not** get history series (a bounded store, rule 9); the overall `draft_acceptance_ratio` does. | +| 3 | **Engine internals the operator cannot act on are out.** | `link`/`passes`/`steps*`/`ckpt*`/`hit_cuts`/`released`/`kv_stream*`/`lookup*`/`linear*`/`attn_cached`/`client_disconnect`/`stream_overflow` stay out of the payload (flavour text is not helpful information). What the payload carries: native + measured throughputs, KV usage + max tokens/bytes, queue depth/cap/admission-rejected, preemptions, the prefix hit-rate gauge, overall + per-position spec acceptance, the decode-step gauges, `requests[]` sessions, per-card VRAM, `model`/`version`/`uptime_s`/`startup_s`, lifetime `tokens_total`. | +| 4 | **No new thresholds, no new write path.** | The only rated radiance cell is KV pressure, via the **existing** `llm.kv_cache_usage_pct` row (radiance publishes the same 0-1 ratio the vLLM map already feeds it). Spec acceptance is an efficiency number, not a verdict — uncoloured. No radiance endpoint accepts writes; the app's only upstream write stays `POST /api/ups/command` (invariant 1 unchanged). No new dependency, no new `/api` route — the served-route walk and the public-surface pin are untouched. | + +## Dependencies + +- `23_llm_rate_responsiveness` (complete) — the per-cadence `RateEstimator` and the refused-window → `n/a` + `Δ` semantics this phase reuses for the measured Δ of radiance's vLLM-shaped counters. +- `13_llm_page_and_decode_rate` (complete) — the engine-panel/strip shape, the `only:` scoping, the served-byte pins this phase extends. +- `24_dev_mtls_rollout` (complete) — the direct-dial fleet; radiance is dialled the same plain way the other engines are (no token, no certificate — the operator's address answers unauthenticated). +- Independent of `33_gateway_route_switching` (todo, in progress) — orthogonal module; no shared file besides `api.py`'s already-present llm routes, which this phase does not touch. + +## Tasks + +1. `01_capture_radiance_ground_truth.md` — `capture-fixtures.sh` records radiance's `/metrics`, `/stats` and `/v1/models` verbatim into `tests/fixtures/`, and the fixture name-set is pinned hermetically. +2. `02_config_and_the_engine_registry.md` — `RADIANCE_BASE_URL` / `RADIANCE_API_KEY`, one `llm_engines()` registry replacing the five hardcoded gufo/vllm enumeration sites, the dev config moves to radiance. +3. `03_the_radiance_field_map.md` — `RADIANCE_SERIES`/`RADIANCE_HISTOGRAMS`, `parse_radiance_metrics()`, the radiance `_sessions()` arm, the additive history series, `fleet_totals`' seams copy. +4. `04_the_radiance_job.md` — `fetch_engine` learns the best-effort `/stats` fetch, the `llm:radiance` job. +5. `05_the_llm_page_learns_radiance.md` — the radiance strip (queue cap, version/uptime, per-card VRAM, the 7-bar spec curve), the `only:` scopes, `requests[]` rows, the empty-state text. +6. `06_docs_and_live_verify.md` — the live read-only test, the fixture re-capture, README/AGENTS.md pass, the live-data screenshots. + +## Testing & Quality +- Unit/integration: the radiance parse against the verbatim fixtures (name-set drift pin, rule-2 precedence in both directions, absent → `n/a`, the refused-window arm, severity pass-through, the per-position shape, both session arms, `fleet_totals` with a radiance payload), the collector's best-effort `/stats` (a dead `/stats` keeps the `/metrics` poll healthy — invariant 4), the registry (one place lists the engines), the API surface (`/api/config.llm.radiance`, the 404 gate, the history defaults, the summary), and the live read-only test. +- Coverage: **>90%** on new/modified code; the whole suite 100%; `ruff` + `pyright` clean. +- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's — pre-existing failures included) and achieves **>90%** code coverage on new/modified code. +- Every UI change is verified in a real browser against live data, screenshot in `data/logs/` (AGENTS.md rule 10) — with gufo and radiance both answering, the cross-engine fleet block finally renders on the dev fleet, which is the phase's own proof that the sum is honest. + +## Documentation & Consistency +- The final pass reviews `README.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**. For `AGENTS.md` it fixes only *stale statements* (a dead command, a changed rule, an outdated fact) and **never adds or expands its content** (a file that states its own update policy, e.g. "updated rarely, if ever", wins). +- Tasks update `README.md` immediately when their own changes are user-visible (setup, commands, configuration, features, conventions), and `AGENTS.md` only when a statement in it became stale. + +## Completion Criteria +- [ ] `RADIANCE_BASE_URL` set and `GUFO_BASE_URL` set: `curl -s http://127.0.0.1:8090/api/llm/snapshot | jq .engines | jq 'map(.engine)'` lists `gufo`, `radiance` (and `vllm` only if configured), and `fleet.engines_live` is 2; `/api/config.llm.radiance.enabled` is `true`. +- [ ] `RADIANCE_BASE_URL` unset: `/api/llm/snapshot` answers `404 {"configured": false}` with no dial (spy-pinned), and no `llm:radiance` job exists in the hub. +- [ ] `tests/fixtures/metrics_radiance.txt` and `tests/fixtures/stats_radiance.json` exist, are verbatim captures, and a hermetic test fails if the fixture's `radiance:*`/`vllm:*` name set drifts from `RADIANCE_SERIES`/`RADIANCE_HISTOGRAMS`. +- [ ] The radiance panel on `/llm` shows the native `Δ`-labelled throughputs when the gauge is absent, the KV bar with its severity colour, the 7-bar spec curve with its `aria-label`, the `requests[]` rows when a request is live and the counts-otherwise arm at rest — screenshot in `data/logs/`. +- [ ] `uv run pytest -m live --live` includes a radiance test that passes read-only against the running engine. +- [ ] full test suite **100% passing** (every test in the project, not just this phase's), coverage >90%; `ruff` + `pyright` clean; `bash .agents/validate.sh` green. +- [ ] README.md is accurate and consistent with the whole project; AGENTS.md has no stale statements (verified in the final pass — the `vLLM :8000` fleet fact included). +- [ ] no behaviour change in completed work: the gufo and vLLM payloads, history names, alerts, and served pages are byte-identical where this phase does not touch them (the suite's existing pins untouched). diff --git a/.agents/phases/todo/34_radiance_llm_engine/01_capture_radiance_ground_truth.md b/.agents/phases/todo/34_radiance_llm_engine/01_capture_radiance_ground_truth.md new file mode 100644 index 0000000..77b7bd3 --- /dev/null +++ b/.agents/phases/todo/34_radiance_llm_engine/01_capture_radiance_ground_truth.md @@ -0,0 +1,34 @@ +# Task 01 — Ground truth first: radiance's /metrics, /stats and /v1/models recorded verbatim + +**Phase:** `34_radiance_llm_engine` · **Story:** n/a + +## Objective + +`scripts/capture-fixtures.sh` records radiance's three endpoints verbatim into `tests/fixtures/`, driven by the same env the app reads (AGENTS.md rule 7: field maps are captured from the running engine, never trusted to a README or a memory of an upstream), and a hermetic test pins the recorded `radiance:*`/`vLLM-compat` name set so a metric rename fails the suite before the dashboard does. + +## Work + +1. `.env.dev` — the engine block: `VLLM_BASE_URL=http://driveripper.reeselink.com:8000` becomes `RADIANCE_BASE_URL=http://driveripper.reeselink.com:8000` with a comment line (the operator's decision of 2026-10-24: radiance replaced vLLM at that address; the vLLM engine code stays for other deployments). `.env.dev` is gitignored — this edit is what makes the live capture below dial the right thing. +2. `scripts/capture-fixtures.sh` — the engine section learns radiance, the same way it reads the engine URLs (nothing hard-coded, so a capture can never come from a different topology than the dashboard polls): + - `[ -n "${RADIANCE_BASE_URL:-}" ] && ENGINES+=("radiance:${RADIANCE_BASE_URL%/}/metrics")` (the existing `--with-llm` gate stays the only switch). + - The radiance entry captures **three** files, not one: `metrics_radiance.txt` (the `/metrics` body, verbatim), `stats_radiance.json` (the `/stats` body, verbatim JSON), `models_radiance.json` (the `/v1/models` body — small, and the model's `capabilities`/`max_model_len` are the only place the engine says what it can do). Each gets the same verbatim-header comment the other fixtures carry: recorded by the script, volatile values move between captures, the contract asserts **shape and name set, never the numbers** (the `nut_*.vars` precedent). + - The `requests[]` shape: the capture at rest records `requests: []` (the engine was idle at first contact). The header comment says so, and the script prints a hint line (`requests[] was empty — re-run with a request in flight to record its shape`) when it sees `in_flight == 0` in the stats body. Task 03's session arm must handle the empty shape; if the operator runs a re-capture with live traffic before task 03 lands, the fixture carries the real rows and task 03's defensive arm is tested against both. + - `--with-llm`'s `ENGINES` loop currently appends `:/metrics` only; restructure it so each engine entry is ` ` and the loop writes the `.txt` metrics file plus, for radiance only, the two JSON sidecars (gufo keeps its separate `/slots` pass exactly as today). +3. `tests/test_llm_source.py` — a fixture-loader following the existing skip pattern (`tests/conftest.py` skips a contract test when its fixture is missing, naming the capture command in the skip message — the new test skips the same way). With the fixtures present it asserts, against `metrics_radiance.txt`: + - the sorted set of series names in the file equals the sorted set `RADIANCE_SERIES` names + `RADIANCE_HISTOGRAMS` names that the fixture actually carries (the drift pin; a renamed `radiance:*` series fails this test, the way the gufo/vLLM captures are pinned); + - every name in the map that the fixture does not carry is listed in a module-level `RADIANCE_OPTIONAL` set (names that exist on other radiance builds), so "absent here" and "absent everywhere" are different things in the test (rule 2's absence arms need to be tested against real absence, not a typo). + - a second loader for `stats_radiance.json`: the top-level key set is pinned as a frozenset (shape, not values), `cards[]` is a non-empty list of dicts with `index`/`name`/`vram_used`/`vram_total`, `requests` is a list, and `draft_positions` is a list of dicts with `reached`/`accepted`/`rate`. + +## Testing & Quality +- Unit: the fixture-loader + name-set + shape tests above (they are the unit — the fixtures are the input contract). +- Coverage: **>90%** on this task's new/modified code (the script is shell; its behaviour is pinned by `tests/test_deploy_scripts.py`'s existing capture-script tests — extend one of them to assert the radiance branch exists and writes the three files when `RADIANCE_BASE_URL` is set, using the script's existing sandbox/fake-curl pattern). +- The live run itself (operator's machine, this dev fleet): `set -a; . ./.env.dev; set +a && ./scripts/capture-fixtures.sh --with-llm` produces the three files with the recorded headers. + +## Completion Criteria +- [ ] `./scripts/capture-fixtures.sh --with-llm` (with `.env.dev` sourced) writes `tests/fixtures/metrics_radiance.txt`, `stats_radiance.json`, `models_radiance.json`, each with a verbatim header; the `requests[]`-empty hint printed at rest. +- [ ] `tests/fixtures/metrics_radiance.txt` carries both families on one file: `grep -c '^radiance:' tests/fixtures/metrics_radiance.txt` > 0 and `grep -c '^vllm:' tests/fixtures/metrics_radiance.txt` > 0 (the vLLM-derived shape is part of the contract). +- [ ] The hermetic name-set/shape tests pass (and skip with the capture command named when the fixtures are absent). +- [ ] `tests/test_deploy_scripts.py`'s capture-script pin covers the radiance branch. +- [ ] full test suite **100% passing** (every test in the project, not just this task's). +- [ ] README.md updated for anything user-visible this task changes (the capture script's `--with-llm` line now records three engine endpoints); AGENTS.md touched only if a statement in it became stale (the phase's final pass re-checks both) +- [ ] no behavior change in completed work (gufo's `/slots` pass and the agent/UPS captures behave exactly as before) diff --git a/.agents/phases/todo/34_radiance_llm_engine/02_config_and_the_engine_registry.md b/.agents/phases/todo/34_radiance_llm_engine/02_config_and_the_engine_registry.md new file mode 100644 index 0000000..5c59498 --- /dev/null +++ b/.agents/phases/todo/34_radiance_llm_engine/02_config_and_the_engine_registry.md @@ -0,0 +1,38 @@ +# Task 02 — One registry: `RADIANCE_BASE_URL` and the five hardcoded engine lists become one function + +**Phase:** `34_radiance_llm_engine` · **Story:** n/a + +## Objective + +Radiance is configured the way the other two engines are (a base URL, an optional API key, unset by default → "not configured", rule 3), and the **five** places that enumerate the gufo/vllm pair by hand — `collectors.llm_jobs`, the `llm_router` 404 gate, the history-route defaults, `/api/config`'s llm block, and the summary's `llm.enabled` — all read one `llm_engines()` registry, so the next engine is a one-line addition and no enumeration site can silently keep a two-engine world. + +## Work + +1. `src/dashboards/config.py` + - Beside the existing engine settings: `radiance_url: str = _base_url("RADIANCE_BASE_URL")` and `radiance_api_key: str = _env("RADIANCE_API_KEY")` — the same shapes, the same comment idiom (unset = "this deployment does not run it", never a red panel). + - `any_source_enabled` gains `or bool(self.radiance_url)`. + - `def llm_engines(self) -> list[tuple[str, str, str, str]]` — ordered `(key, display, url, api_key)` for the three engines, in **stable display order** `gufo, vllm, radiance` (the UI's card order and the history defaults follow it; the order is part of the contract, pinned in a test), with a comment that the list is the single place an engine's presence is decided. A `bool(self.radiance_url)`-style `configured` test lives at the call sites (`url` non-empty), matching how `nut_enabled`/`gateway_enabled` work — no new property for nothing. +2. `src/dashboards/collectors.py` — `llm_jobs` rewrites its two `if` blocks over `settings.llm_engines()`: one `Job(f"llm:{key}", "llm", display, settings.llm_interval, …)` per **configured** engine; `with_slots=True` when `key == "gufo"`, `with_stats=True` when `key == "radiance"` (the flag plumbing lands in task 04; this task introduces the flag on `fetch_engine` as a no-op parameter so `llm_jobs` reads its final shape now). +3. `src/dashboards/api.py` + - The `llm_router` 404 gate (today `if not (ctx.settings.gufo_url or ctx.settings.vllm_url)`) becomes "no configured engine" over the registry — same `404 {"configured": false, "detail": …}` shape, and the `detail` sentence lists all three variable names (`GUFO_BASE_URL` / `VLLM_BASE_URL` / `RADIANCE_BASE_URL`) so the not-configured arm names the fix. + - The `/api/llm/history` defaults (`(("gufo", ctx.settings.gufo_url), ("vllm", ctx.settings.vllm_url))`) become the registry's configured `(display, url)` pairs — the `engines` query-param filtering behaves exactly as today. + - The summary's `llm.enabled` (today `bool(ctx.settings.gufo_url or ctx.settings.vllm_url)`) becomes "any engine configured" over the registry. + - `/api/config`'s llm block gains a third key, **additively**: `"radiance": {"enabled": bool(settings.radiance_url), "base_url": strip_userinfo(settings.radiance_url), "auth": bool(settings.radiance_api_key)}` — the existing `gufo`/`vllm` keys keep their exact names and shape (the API shape is additive-only, LOCKED). +4. `.env.dev.example` — the engines block keeps the gufo line, keeps the vllm line (vLLM is still a supported engine for other deployments), and adds `export RADIANCE_BASE_URL=http://driveripper.example.net:8000` with the same documentation idiom. `README.md`'s env-var table row (`GUFO_BASE_URL / VLLM_BASE_URL …`) gains radiance in the same sentence shape (the "unset means *not deployed here*" clause applies unchanged). +5. `tests/test_config.py` + - `RADIANCE_BASE_URL` unset → `radiance_url == ""`, no job, `any_source_enabled` unchanged by it; set → job present, `any_source_enabled` true. + - `llm_engines()` pins: the exact ordered triple, the display names (`gufo`, `vLLM`, `radiance`), the api-key passthrough (a set `RADIANCE_API_KEY` travels into the tuple), and that an unset engine is **still listed** with an empty url (the registry is the full fleet, configured-ness is a url test — the `parse_targets` idiom). +6. `tests/test_api.py` — `/api/config.llm.radiance` absent-by-default pin (additive key, `enabled: false`); the 404 gate with only `RADIANCE_BASE_URL` set (no 404) and none set (404 + the three-variable detail sentence); the history defaults listing a configured radiance. + +## Testing & Quality +- Unit/integration: the registry pins above; the 404 gate in all three single-engine configurations; the config block's additivity (a diff-pin: the gufo/vllm keys' bytes are unchanged when radiance is added). +- Coverage: **>90%** on `config.py` and the touched `api.py`/`collectors.py` paths; the whole suite 100%; `ruff` + `pyright` clean. + +## Completion Criteria +- [ ] `RADIANCE_BASE_URL=… uv run python -c "from dashboards.config import load; e=[x for x in load().llm_engines() if x[0]=='radiance']; print(e)"` prints the one configured entry; unset, the entry is present with an empty url. +- [ ] `grep -n "gufo_url\|vllm_url" src/dashboards/collectors.py src/dashboards/api.py` shows no engine enumeration outside the registry (the settings fields themselves remain — they are the env surface). +- [ ] The 404 gate and the history defaults answer from the registry in all configurations (spy-pinned: an unconfigured radiance is never dialled — no client, no fetch). +- [ ] `/api/config` serves `llm.radiance` beside byte-identical `llm.gufo`/`llm.vllm`. +- [ ] full test suite **100% passing** (every test in the project, not just this task's). +- [ ] README.md updated for anything user-visible this task changes (the env-var table row); AGENTS.md touched only if a statement in it became stale (the phase's final pass re-checks both) +- [ ] no behavior change in completed work (a gufo-only or vLLM-only deployment's `/api/config`, 404, history and summary bodies are byte-identical to before, modulo the new additive `radiance` key in the config block) diff --git a/.agents/phases/todo/34_radiance_llm_engine/03_the_radiance_field_map.md b/.agents/phases/todo/34_radiance_llm_engine/03_the_radiance_field_map.md new file mode 100644 index 0000000..3418048 --- /dev/null +++ b/.agents/phases/todo/34_radiance_llm_engine/03_the_radiance_field_map.md @@ -0,0 +1,43 @@ +# Task 03 — The radiance field map: rule-2 precedence, `/stats` as the session source, no invented number + +**Phase:** `34_radiance_llm_engine` · **Story:** n/a + +## Objective + +`src/dashboards/sources/llm.py` parses a radiance poll the way it parses gufo and vLLM — a field map **built from the task-01 fixture** (rule 7), the native windowed throughput gauges winning over a measured Δ (rule 2's precedence, the vLLM-native-gauge case), `n/a` where the engine is silent (never `0`), the KV cell rated through the **existing** `llm.kv_cache_usage_pct` severity row, and `/stats` as the session source beside gufo's `/slots` and vLLM's counts-only arm. + +## Work + +1. `src/dashboards/sources/llm.py` — the map, **names taken from `tests/fixtures/metrics_radiance.txt`, not from memory** (the module docstring's capture ritual already says how; the vLLM-compat subset is part of the same file, so one `Prom` parse serves both families): + - `RADIANCE_SERIES` (UI key → (names, agg)), the radiance gauges plus the vLLM-shaped series this build carries: `decode_tokens_per_second_native` (`radiance:decode_tokens_per_second`, avg — the scheduler's recent-window aggregate, the thing that wins), `prompt_tokens_per_second_native` (`radiance:prefill_tokens_per_second`, avg), `num_requests_running` / `num_requests_waiting` (`vllm:…`, sum), `gpu_cache_usage` (`vllm:kv_cache_usage_perc`, max — the same 0-1 ratio the vLLM map feeds `SEVERITY_CELLS`, so `attach_cell_severity` rates it with **no new row**), `kv_cache_max_tokens` / `kv_cache_max_bytes` (`radiance:…`, max), `queue_depth` (`radiance:http_queue_depth`), `admission_rejected_total` (`radiance:http_admission_rejected_total`), `num_preemptions_total` (`vllm:num_preemptions_total`), `prompt_tokens_total` / `generation_tokens_total` (`vllm:…`, sum — the measured-Δ source), `gpu_prefix_cache_hit_rate` (`vllm:gpu_prefix_cache_hit_rate`, avg — a 0-1 gauge; **this build publishes no prefix counters**, so a hits/queries ratio would be an invented number: the gauge is the answer, source named, not derived — the documented asymmetry, the gufo `*_native` note's sibling), `spec_accept_rate` (`radiance:draft_acceptance_ratio`, avg — a 0-1 gauge, **reusing gufo's UI key** so the history name, the fleet block and the UI row all work without a new vocabulary), `decode_step_seconds_recent` / `decode_step_seconds_best` (`radiance:…`, max). + - `RADIANCE_HISTOGRAMS`: `time_to_first_token_seconds` (`vllm:time_to_first_token_seconds`), `time_per_output_token_seconds` (`vllm:time_per_output_token_seconds`, `vllm:inter_token_latency_seconds`), `e2e_request_latency_seconds` (`vllm:e2e_request_latency_seconds`) — the vLLM-shaped histograms the build carries. This build has **no** `request_queue_time_seconds`, so `queue_seconds` is `n/a` for radiance by absence, never back-filled. + - `RADIANCE_OPTIONAL` (task 01's set of map names absent from the fixture — e.g. the `vllm:num_requests_swapped`-class names) so the hermetic test distinguishes "not on this build" from "not anywhere". +2. `parse_radiance_metrics(text, rates, now, stats=None)` — the gufo/vLLM parse functions' idiom, with the **vLLM-side** throughput precedence (native gauge is the aggregate, so it wins): + - `decode_tokens_per_second` / `prompt_tokens_per_second`: the native gauge if present (source = the gauge's name, `derived=False`); else the measured Δ of the token counter **per label set** via the existing `_series_delta` (keyed by resolved name + labels — phase 13's carried-counter rule; a refused window is still an answer: `n/a` beside its honest `Δ`); else `field(None, None)`. **No third estimate** (no `running × 1/tpot`): a gauge + a counter + a guess stacked on one cell is exactly the aggregation rule 2 exists to catch, and the engine's own page shows only the first two. + - `tokens_total` from the two `vllm:` token counters (prompt + generated, as the vLLM path does); `prefix_cache_hit_ratio` from the gauge (0-1, source named); `spec_positions` from the labelled series `radiance:draft_position_acceptance_ratio` via `parsed.by_label(…, "position")` — an **ordered** list of 7 `{position, ratio}` entries (0-1, in the order 0..6 the engine publishes; absent positions stay absent, not zeroed). + - `attach_cell_severity(out)` at the end, unchanged (the `gpu_cache_usage` key carries the verdict; **no** new severity row for spec acceptance — an efficiency number, decision 4). +3. `_sessions()` — the third arm, keyed on the `stats` dict task 04's fetch supplies: + - `stats` present: `running` / `waiting` / `queue_cap` / `requests` come from it (the engine's own summary — the numbers its own page prints). `requests` empty → `detail` = counts + cap, one honest sentence (the vLLM arm's idiom: "radiance reports counts, not sessions: N running of cap C. Per-request detail appears here while requests are in flight."). Non-empty → `list` = the rows verbatim (the fixture's shape is the contract; a row's fields render as-is, task 05), `source` = "radiance /stats". + - `stats` missing (the fetch failed — invariant 4: the poll itself is healthy) → fall back to the `/metrics` gauges `num_requests_running`/`num_requests_waiting` for the counts (they are real numbers, different provenance — the detail says `/stats` did not answer this poll), `queue_cap` `null` (never `0` — rule 2). + - `active` stays `metrics["num_requests_running"]["value"]` in every arm (one definition, the `/metrics` gauge). +4. `engine_result(engine, base_url, text, rates, *, slots=None, stats=None, now=None)` — the parser dispatch becomes the three-way map (gufo / vllm / radiance); a radiance poll attaches `payload["stats"]` = the small additive subset `{model, version, uptime_s, startup_s, total_requests, cards[]}` (each card `+ vram_used_pct` computed `used/total×100`, `derived: true` — the only derived number in the block; the raw bytes travel as the engine published them). The subset is **whitelisted from the fixture's shape**, so a radiance upgrade that renames a `/stats` key degrades to absent, not a crash (invariant 4's poll-body rule). +5. `HISTORY_METRICS` — **additive candidates only** (the existing keys and their candidate order are untouched): `queue_depth` → `("queue_depth",)`, `queue_cap` → `("queue_cap",)` (the `queue_cap` metrics key is the `/stats` passthrough), `decode_step_seconds_recent` → `("decode_step_seconds_recent",)`, `vram_used_pct` → `("vram_used_pct",)` (card 0, the engine's own view of the same card `/system` reads through the agent — different source, different host key, both honest). The existing keys already cover radiance: `decode/prompt_tokens_per_second` (the UI keys the precedence logic fills), `kv_cache_usage` (the `gpu_cache_usage` candidate), `requests_running`/`requests_waiting` (the `num_requests_*` candidates), `tokens_total`, `spec_accept_rate` (the gauge reuses gufo's key). The seven per-position ratios are **snapshot-only** (decision 2: a bounded store, rule 9). +6. `series_names(engine)` — the three-way branch; `fleet_totals(engines)` — the code needs **no change** (the aliases `num_requests_running`/`num_requests_waiting`, the `decode/prompt_tokens_per_second` keys, `tokens_total` and the `gpu_cache_usage` max all already cover a radiance payload); **only** the `seams` sentence is rewritten to state, briefly and honestly, the known asymmetries: gufo's prompt counter excludes cache-hit tokens; the summed throughput is per-engine provenance-labelled (native scheduler window vs measured Δ). It does not claim what radiance's `vllm:prompt_tokens_total` includes or excludes about cached tokens — the engine does not say, and neither does the dashboard (rule 2). + +## Testing & Quality +- Unit/integration in `tests/test_llm_source.py` (fixture-driven, the conftest skip pattern from task 01): + - the fixture parse: every mapped key resolves (shape + source strings), the name set equals the fixture's (task 01's pin), `spec_positions` is the ordered 7, `stats`-arm sessions as specced (empty / non-empty / missing, three fakes), the `payload["stats"]` whitelist (a fixture with a renamed `version` key degrades to absent, the poll still returns), the radiance history-sample names (`llm.queue_depth`, `llm.queue_cap`, `llm.decode_step_seconds_recent`, `llm.vram_used_pct`, …) appear and the gufo/vLLM sample names are untouched. + - rule 2, three arms: gauge present → source is the gauge name, `derived` false; gauge line removed from the fixture copy → measured Δ, `derived` true, the window (or the refused-window `n/a` + `Δ` label, a fresh-estimator fake); both absent → `n/a`, no `0` anywhere; `queue_cap` with `stats` missing → `null`, not `0`. + - the severity pass-through: the `gpu_cache_usage` cell carries a `severity` verdict from the existing table at a fixture value above the warn bound; `spec_accept_rate` carries **no** severity key (decision 4 pinned both ways — an absence-only assertion, so it is asserted on the payload). + - `fleet_totals` with a gufo + radiance payload: the running/waiting sums, the true max KV, `engines_live` 2, the seams copy names the provenance labelling. + - the gufo and vLLM parse functions are byte-behaviour-unchanged (their existing pins untouched). +- Coverage: **>90%** on `sources/llm.py`; the whole suite 100%; `ruff` + `pyright` clean. + +## Completion Criteria +- [ ] `parse_radiance_metrics` against the fixture returns every mapped key with its pinned source string; a series removed from a fixture copy answers `{"value": null}` — never `0`. +- [ ] the three throughput arms are proven in both directions (gauge wins / measured fills / `n/a` with the honest `Δ` label on a refused window). +- [ ] `spec_positions` is the ordered 7 from the fixture; `queue_cap` is `null` (not `0`) when `/stats` is absent; the KV cell carries the severity verdict and the spec cell carries none. +- [ ] `engine_result("radiance", …, stats=…)` carries the whitelisted `stats` block with the one computed `vram_used_pct`; a renamed stats key degrades to absent without raising. +- [ ] full test suite **100% passing** (every test in the project, not just this task's). +- [ ] README.md updated for anything user-visible this task changes (none expected — the payload surface is task 05's); AGENTS.md touched only if a statement in it became stale (the phase's final pass re-checks both) +- [ ] no behavior change in completed work (gufo and vLLM payloads, samples, alerts and pages behave exactly as before — the suite's existing pins untouched) diff --git a/.agents/phases/todo/34_radiance_llm_engine/04_the_radiance_job.md b/.agents/phases/todo/34_radiance_llm_engine/04_the_radiance_job.md new file mode 100644 index 0000000..97c9fda --- /dev/null +++ b/.agents/phases/todo/34_radiance_llm_engine/04_the_radiance_job.md @@ -0,0 +1,33 @@ +# Task 04 — The `llm:radiance` job: a `/stats` sidecar that can fail without failing the poll + +**Phase:** `34_radiance_llm_engine` · **Story:** n/a + +## Objective + +`collectors.fetch_engine` fetches radiance's `/stats` **best-effort** — a dead or partial `/stats` degrades the session arm (task 03's missing-stats fallback) and never takes the `/metrics` poll down (invariant 4: a failed sidecar keeps the last good payload and marks nothing stale, while a dead `/metrics` is the real `UpstreamError`) — and `llm_jobs` (in the task-02 registry shape) builds the `llm:radiance` job with its own `Job` key, display name and the shared LLM cadence/estimator. + +## Work + +1. `src/dashboards/collectors.py` — `fetch_engine(…, with_slots=False, with_stats=False)`: + - The `with_stats` path, beside the existing `/slots` loop: one `GET {base_url}/stats` (no alias — radiance's own page uses exactly that path); on any `httpx.HTTPError` **or** a status ≥ 400 **or** a JSON that is not a dict, `stats` stays `None` and the function returns the metrics result with the missing-stats session arm. No `UpstreamError` is raised from this sidecar, no retry, no second dial — one best-effort read, the `/slots` idiom. + - The `UpstreamError` messages for the **metrics** dial keep the existing `strip_userinfo` redaction (invariant 6) — a `RADIANCE_API_KEY`-free URL never leaks, a tokenised one never leaks its token. + - `llm_jobs` (task 02's registry shape) passes `with_stats=True` for the radiance entry. No new `RateEstimator`: radiance rides the LLM cadence's existing estimator (task: `app.build_hub` is untouched — one estimator per measuring cadence stays the rule). +2. `src/dashboards/app.py` — nothing to change is the check: the `llm` jobs already flow from `llm_jobs` into the hub; confirm `llm:radiance` gets the LLM interval and the LLM-span estimator by the existing wiring (assert in a test, not by comment). +3. `tests/test_collectors.py`: + - a configured `RADIANCE_BASE_URL` (and no gufo/vllm) builds exactly one `llm:radiance` `Job` with `dashboard == "llm"`, display `radiance`, the LLM interval (the registry's shape from task 02, extended); + - a fake server with `/metrics` 200 and `/stats` 500 → the result is a healthy `Result` (payload present, `stats` absent, the session detail names the missing sidecar); `/stats` 200 with a non-JSON body → the same healthy degradation; `/stats` 200 with a valid dict → `stats` travels into the result; + - `/metrics` 500 with `/stats` 200 → the `UpstreamError` is the metrics one (the sidecar never masks the real failure); + - a tokenised `RADIANCE_BASE_URL` (`http://secret@host:8000`) whose metrics dial fails → the `UpstreamError` message contains no `secret` (the redaction pin, the existing engine-fetch precedent extended); + - the job list never dials an unconfigured engine (the existing spy pattern). + +## Testing & Quality +- Unit/integration: the five pins above (healthy / degraded / masked / redacted / undialled). +- Coverage: **>90%** on the touched `collectors.py` paths; the whole suite 100%; `ruff` + `pyright` clean. + +## Completion Criteria +- [ ] `set -a; . ./.env.dev; set +a` + `./scripts/dev-server.sh restart` → `./scripts/dev-server.sh status` shows `llm:radiance` ok (or its real failure) beside the other sources, and `/api/llm/snapshot` (authenticated) carries the radiance payload with the `stats` block. +- [ ] a faked dead `/stats` leaves the poll healthy (the test above, and the page shows the metrics panel with the counts-otherwise session arm). +- [ ] `grep -n "with_stats" src/dashboards/collectors.py` shows the flag feeding `fetch_engine` from `llm_jobs` only. +- [ ] full test suite **100% passing** (every test in the project, not just this task's). +- [ ] README.md updated for anything user-visible this task changes (none expected — no new env surface beyond task 02's); AGENTS.md touched only if a statement in it became stale (the phase's final pass re-checks both) +- [ ] no behavior change in completed work (the gufo `/slots` pass and the vLLM job behave exactly as before — the suite's existing pins untouched) diff --git a/.agents/phases/todo/34_radiance_llm_engine/05_the_llm_page_learns_radiance.md b/.agents/phases/todo/34_radiance_llm_engine/05_the_llm_page_learns_radiance.md new file mode 100644 index 0000000..4705a3d --- /dev/null +++ b/.agents/phases/todo/34_radiance_llm_engine/05_the_llm_page_learns_radiance.md @@ -0,0 +1,39 @@ +# Task 05 — `/llm` learns radiance: the strip, the 7-bar curve, and the honest empty arms + +**Phase:** `34_radiance_llm_engine` · **Story:** n/a + +## Objective + +The radiance card on `/llm` shows the cells its payload carries — the native/measured throughputs with their provenance tags, the KV bar with its severity colour, the radiance strip (queue cap, build, uptime, per-card VRAM, the 7-bar spec-acceptance curve), and the `requests[]` rows when a request is live, the counts-otherwise sentence at rest — with every chart named for assistive tech and **no verdict colour** on any radiance cell except the KV bar (decision 4). + +## Work + +1. `src/dashboards/web/static/page-llm.js` + - The `only:` scopes extend, not new: `ttft`, `tpot`, `queue_mean`, `running` gain `'radiance'` (the build carries the vLLM-shaped histograms and the `num_requests_*` gauges — task 03's map fills the same cell keys); `spec_accept` becomes `['gufo', 'radiance']` (the gauge reuses gufo's key and is the same 0-1 scale, so the existing `%` row renders unchanged). + - New **radiance-only** strip cells (the `STRIP_CELLS` idiom, `only: ['radiance']`, each with a `tip` that tells the operator what the number is for): + - `cap` — "queue cap", the engine's `queue_cap` (the admission ceiling; `null` when `/stats` did not answer → the cell prints `n/a`, never `0`); + - `build` — `stats.version` (e.g. `1.15.1`), tip "The engine's own version string — a radiance upgrade changes it, the panel does not"; + - `uptime` — `stats.uptime_s` through the existing `app.js` duration formatter; + - `vram` — card 0's `vram_used`/`vram_total` as `32.8 / 34.2 GiB` (GiB = 2³⁰, the only conversion on the page, labelled as such in the tip: "the engine's own view of the card; `/system`'s GPU row is the agent's view"), `n/a` when no card answered. + - The **7-bar spec curve** (decision 2): a small hand-rolled SVG on the radiance strip (the `charts.js` idiom, no library, no CDN — rule 5), one bar per position in `spec_positions` order (0..6), height ∝ the 0-1 ratio, drawn in a **neutral** token (the page's existing track/cool colours — never a severity colour, the value is an efficiency number, decision 4), with `role="img"` and an `aria-label` that names the chart and its range (e.g. `Spec-decode draft acceptance by position: 82% at position 0 falling to 65% at position 6`) — the label is **computed from the payload** (the "every chart names itself" rule, pinned below). + - The `requests[]` arm: non-empty `sessions.list` (source "radiance /stats") renders the rows as a compact table beside the counts (the gufo slot-grid idiom, shrunk — radiance rows are the engine's own fields, rendered **as-is**: the fixture's shape is the contract; a field that is a number prints `n/a`-free only when it is a number, anything else prints its string verbatim, no conversion, no invented key). Empty list → the `sessions.detail` sentence (task 03's counts + cap) in the existing detail slot. + - `SLOT_ENGINES` stays `['gufo']` (the slot grid is gufo's); the radiance rows live in the strip, not the slot grid. +2. `src/dashboards/web/pages/llm.html` + - The empty-state paragraph names all three variables: "Set `GUFO_BASE_URL` (llama.cpp / gufo), `VLLM_BASE_URL` and/or `RADIANCE_BASE_URL`." + - The fleet block's `fleet-note`/seams copy is payload-driven (`fleet.seams` from `fleet_totals`, task 03's rewritten sentence) — no new hardcoded copy. +3. `src/dashboards/web/static/dash.css` — only the tokens the 7-bar curve and the `requests` table need (reuse the existing neutral/cool/edge tokens; **no** new colour vocabulary — colour comes only from the payload's severity verdict, which for radiance is the KV bar alone). +4. `tests/test_api.py` — the served-byte pins extend: the radiance strip markup (the four new cells' `data-key`s), the curve's `role="img"` + `aria-label` joined to the existing chart-aria-label pin list (an absent label fails the test), and the requests-table container present-when-the-payload-carries-rows (a fixture-driven payload), absent-at-rest with the detail sentence. +5. Live verification (AGENTS.md rule 10): `./scripts/dev-server.sh restart` with `.env.dev` (gufo + radiance now both configured — the fleet block finally renders: two answering engines), sign in, `/llm`, screenshot the radiance card **at rest** (counts arm, curve, strip) into `data/logs/34-llm-radiance-rest.png`; if a request is in flight during the pass, a second screenshot with the rows visible. + +## Testing & Quality +- Integration: the `test_api.py` pins above (markup, aria-label, both session arms) against fixture-driven payloads. +- Coverage: the UI is proven by the payload-contract pins + the screenshots (the JS-tests-none decision, PLAN.md); Python-side **>90%** on anything touched (none expected); the whole suite 100%; `ruff` + `pyright` clean. + +## Completion Criteria +- [ ] The radiance card renders all task-03 cells; a payload with `stats` missing shows `n/a` on `cap`/`build`/`uptime`/`vram` (never `0`) and the counts-otherwise session sentence. +- [ ] The 7-bar curve is present with its computed `aria-label` (the pin fails on an absent label); no severity-colour token in its markup. +- [ ] The `only:` scopes: the vLLM-shaped rows (TTFT/TPOT/queue-time/running) and `spec accept` render on the radiance card from the fixture payload; no radiance row on the gufo card and no gufo row on the radiance card. +- [ ] `data/logs/34-llm-radiance-rest.png` exists, showing the card against live data with the fleet block rendered (two engines). +- [ ] full test suite **100% passing** (every test in the project, not just this task's). +- [ ] README.md updated for anything user-visible this task changes (the `/llm` page section, if it names the engines it draws); AGENTS.md touched only if a statement in it became stale (the phase's final pass re-checks both) +- [ ] no behavior change in completed work (the gufo and vLLM cards are byte-identical where this task does not touch them — the served-byte pins) diff --git a/.agents/phases/todo/34_radiance_llm_engine/06_docs_and_live_verify.md b/.agents/phases/todo/34_radiance_llm_engine/06_docs_and_live_verify.md new file mode 100644 index 0000000..4ee082a --- /dev/null +++ b/.agents/phases/todo/34_radiance_llm_engine/06_docs_and_live_verify.md @@ -0,0 +1,35 @@ +# Task 06 — Final pass: live read-only proof, fixture re-capture, README/AGENTS.md, the definition of done + +**Phase:** `34_radiance_llm_engine` · **Story:** n/a + +## Objective + +The phase's verification pass: a read-only `live` test that drives the production radiance path against the running engine, the fixture re-capture with a diff read, the documentation pass (README in full; AGENTS.md stale-statements-only), and the definition-of-done checklist run against the dev server. + +## Work + +1. `tests/test_live_fleet.py` — the radiance test (the file's idiom: skipped unless `--live`, each test skips when its source is unset, **live tests must not write** — radiance exposes no write endpoint, so "read-only" is a list of GETs): + - skip when `RADIANCE_BASE_URL` is unset (the source gate, rule 3's live twin); + - dial the engine through the **production** path — `collectors.fetch_engine` with the LLM estimator (never a hand-rolled `client.get`), one poll; + - assert shape, not numbers: the payload's `engine == "radiance"`, the `stats` block's keys (task 03's whitelist), `spec_positions` length 7 when the fixture/running build publishes it, the session arm is one of the two specced arms, every cell that is present is either a number or `None` (never a fake `0` for an absent gauge — the live twin of rule 2), and `unparsed` is empty (a parse error on the live wire is drift, not a tolerance). +2. `./scripts/capture-fixtures.sh --with-llm` (`.env.dev` sourced) — re-record the radiance sidecars (and everything else), **read the diff**: a volatile value moving is expected (the headers say so); a renamed or new `radiance:*` series is a contract change — extend task 01's pin (and `RADIANCE_OPTIONAL` where a name is build-optional), never a silent ignore. If the earlier capture's `requests[]` was empty and a request is in flight now, re-capture so the fixture carries the real row shape, and re-run the session-arm tests against it. +3. `README.md` — the full final-pass review (the phase's documentation mandate): the env-var table (`RADIANCE_BASE_URL` / `RADIANCE_API_KEY` beside the engine rows, the "unset means *not deployed here*" clause), the LLM-engines section (a radiance paragraph in the gufo/vLLM idiom: what it is, the endpoints it serves, the native-gauge/measured-Δ provenance, the `/stats` sidecar), the capture-script section (three radiance endpoints under `--with-llm`), and every other place the engine count or the driveripper fact appears — fixing **every** inconsistency found, including pre-existing ones. +4. `AGENTS.md` — **stale statements only, no expansion** (its own update policy wins): the fleet table's driveripper line `vLLM :8000` → `radiance :8000` (the operator's decision of 2026-10-24); the `/llm` route row's source column (`gufo /metrics + /slots, vLLM /metrics`) gains `radiance /metrics + /stats`; `config.py`'s paragraph names the radiance settings beside the others; `sources/llm.py`'s bullet says the three field maps (gufo, vLLM, radiance) with the rule-2 precedence note already there. Nothing else — no phase narrative, no new rules. +5. The definition-of-done run, in order (AGENTS.md): + - `uv run pytest` green; `uv run pytest --cov` — coverage floor 94 % held, new/modified Python > 90 %; `uv run ruff check src tests agent && uv run pyright` clean; `uv pip list --outdated` empty (no dependency moved — confirm, do not assume). + - `set -a; . ./.env.dev; set +a && ./scripts/dev-server.sh restart` — `/api/health` sane (per-source ok/DOWN; `llm:radiance` answering), `./scripts/dev-server.sh status` read. + - `uv run pytest -m live --live` — the new radiance test passes read-only; the existing live tests (gufo, vLLM-where-configured, upsd, agents) still pass. + - Screenshots in `data/logs/`: the task-05 rest shot (and the in-flight shot if one was possible), and the overview page (`/`) showing the radiance engine's alert row when its KV bar is rated — the cell-driven `_engine_alerts` needs no code, only proof. + - `bash .agents/validate.sh` green. + +## Testing & Quality +- The live test is the integration proof; the re-captured fixtures re-pin the hermetic suite; the documentation pass is checked by reading the two files against the project as it stands. +- Coverage: no new Python beyond the live test; the whole suite 100 %; `ruff` + `pyright` clean. + +## Completion Criteria +- [ ] `uv run pytest -m live --live` includes the passing read-only radiance test (skipped, correctly, when unset — proven by one run without the env var). +- [ ] the fixture re-capture diff is read and either clean or folded into the pins (`RADIANCE_SERIES`/`RADIANCE_OPTIONAL`); the suite re-run is green. +- [ ] README.md: the engine section, env-var table, and capture-script section all name radiance; a full read found no remaining engine-count or driveripper stale statement. +- [ ] AGENTS.md: the three stale statements fixed (fleet line, route row, the two module bullets), nothing added or expanded beyond them. +- [ ] `./scripts/dev-server.sh status` shows `llm:radiance` answering; the `data/logs/` screenshots exist; `bash .agents/validate.sh` is green; the coverage floor held. +- [ ] full test suite **100% passing** (every test in the project, not just this task's) — the phase's "Complete" gate. diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__00_phase.a1.err b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__00_phase.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__00_phase.a1.md b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__00_phase.a1.md new file mode 100644 index 0000000..c270ea1 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__00_phase.a1.md @@ -0,0 +1,25 @@ +Phase 33 completion pass is done. Final report: + +## Phase 33 — completion report + +**All 8 task files complete** (`.agents/phases/complete/33_gateway_route_switching/`), full gate green, docs verified. + +### Work in this pass + +1. **`tests/test_routectl.py` added** (17 tests, all green) — exercises the *real* helper binary unprivileged against a fake tree (`--root`), the real agent parser, the captured gateway fixtures, and stubbed `nginx`/`haproxy`/`systemctl` on `PATH`. Pins the full protocol (locate → state → dry-run → temp-file/fsync/`os.replace` write → checker → restore on checker failure → reload-never-restart → re-read proof → sidecar record), all three route shapes (haproxy `server`, nginx `proxy_pass`, nginx stream `map` via its sink-value id), `drift` / unknown-id / `map-default` refusals, `list`/`verify`, and the `--serve` peer gate (`SO_PEERCRED` uid check, one JSON line per request, exact body shape). + +2. **Real helper defect fixed** — `deploy/dashboards-routectl.py`: `main()`'s early-return paths returned `({"ok": False, …}, EXIT_*)` tuples, so `sys.exit()` printed a Python repr instead of a JSON line (usage errors, agent-import refusal). Now prints `json.dumps(…)` and returns only the code, keeping the one-JSON-line contract. + +3. **Documentation consistency fixes** (task 08 criterion 6): + - `README.md`: garbled verdict-colour sentence on the switch ("green and to the position") → "in the **on** position and tinted `--gate-on`" (the existence-pair contract). + - `README.md`: audit sentence now names all three event kinds — `route-disabled` / `route-enabled` / **`route-denied`** (the code stores one per refused call; dry runs store none). + - `page-gateway.js`: replaced the stale pre-33 module docstring ("the switch is not wired") with the two-arm `GATEWAY_ALLOW_WRITES` contract. + - `AGENTS.md`: every phase-33 claim re-verified against code (the ten gates, the 35 s/30 s budget pair, `gateway.writes_enabled`, `/api/summary`'s `disabled`, the 1.4.0 agent bullet, the gap list) — no changes needed. + +### Verification +- `uv run pytest --cov`: **1970 passed**, 18 skipped — coverage **98.74%** (floor 94%) +- `ruff` clean · `pyright` 0 errors · `bash .agents/validate.sh` **exit 0** (incl. `node --check` on every page script) +- Dev server running; all 10 sources `ok`; bundles exported + +### Live infrastructure +**No live changes made.** The live drill remains unexecuted per `data/logs/NOTES-33-switching.md` (no operator approval; checker binaries unavailable in the podman proxy units). All routes stay enabled, config hashes unchanged, and the sandbox rehearsal + backup tree are in place for the operator's own `DANGER` call. All changes are in the working tree only. diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__00_phase.a1.md.status b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__00_phase.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__00_phase.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__00_phase.a1.validate b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__00_phase.a1.validate new file mode 100644 index 0000000..755e8e2 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__00_phase.a1.validate @@ -0,0 +1,71 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 3%] +........................................................................ [ 7%] +........................................................................ [ 10%] +........................................................................ [ 14%] +........................................................................ [ 18%] +........................................................................ [ 21%] +........................................................................ [ 25%] +........................................................................ [ 28%] +........................................................................ [ 32%] +.........................s.............................................. [ 36%] +........................................................................ [ 39%] +........................................................................ [ 43%] +........................................................................ [ 47%] +........................................................................ [ 50%] +........................................................................ [ 54%] +........................................................................ [ 57%] +........................................................................ [ 61%] +........................................................................ [ 65%] +........................................................................ [ 68%] +........................................................................ [ 72%] +...................................................ssssssssssssss....... [ 76%] +........................................................................ [ 79%] +........................................................................ [ 83%] +........................................................................ [ 86%] +........................................................................ [ 90%] +........................................................................ [ 94%] +..........................................................sss........... [ 97%] +............................................ [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.8-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 32 +src/dashboards/alerts.py 201 0 100% +src/dashboards/api.py 625 9 99% 229, 1273-1275, 1310-1313, 1382, 1390, 1446, 1449 +src/dashboards/app.py 190 0 100% +src/dashboards/auth.py 265 0 100% +src/dashboards/certs.py 292 0 100% +src/dashboards/collectors.py 244 5 98% 386-387, 391-392, 409 +src/dashboards/config.py 128 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183 +src/dashboards/rates.py 73 0 100% +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 134 0 100% +src/dashboards/sources/gateway.py 613 10 98% 339, 588, 601-602, 652, 785, 1089, 1140, 1207, 1569 +src/dashboards/sources/host.py 209 0 100% +src/dashboards/sources/llm.py 178 1 99% 285 +src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 313 2 99% 553-554 +------------------------------------------------------------------ +TOTAL 4369 56 99% +Required test coverage of 94% reached. Total coverage: 98.72% +1970 passed, 18 skipped, 1 warning in 192.42s (0:03:12) +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__01_the_root_helper_edits_exactly_one_route.a1.err b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__01_the_root_helper_edits_exactly_one_route.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__01_the_root_helper_edits_exactly_one_route.a1.md b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__01_the_root_helper_edits_exactly_one_route.a1.md new file mode 100644 index 0000000..ef0c0c6 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__01_the_root_helper_edits_exactly_one_route.a1.md @@ -0,0 +1 @@ +Let me look at the agent's file-reading helpers and the `parse_nginx_config`/`nginx_include_paths` functions. diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__01_the_root_helper_edits_exactly_one_route.a1.md.status b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__01_the_root_helper_edits_exactly_one_route.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__01_the_root_helper_edits_exactly_one_route.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__01_the_root_helper_edits_exactly_one_route.a1.validate b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__01_the_root_helper_edits_exactly_one_route.a1.validate new file mode 100644 index 0000000..0dbd9a1 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__01_the_root_helper_edits_exactly_one_route.a1.validate @@ -0,0 +1,69 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 3%] +........................................................................ [ 7%] +........................................................................ [ 11%] +........................................................................ [ 15%] +........................................................................ [ 19%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 30%] +........................................s............................... [ 34%] +........................................................................ [ 38%] +........................................................................ [ 42%] +........................................................................ [ 46%] +........................................................................ [ 50%] +........................................................................ [ 53%] +........................................................................ [ 57%] +........................................................................ [ 61%] +........................................................................ [ 65%] +........................................................................ [ 69%] +........................................................................ [ 73%] +.........................sssssssssssss.................................. [ 76%] +........................................................................ [ 80%] +........................................................................ [ 84%] +........................................................................ [ 88%] +........................................................................ [ 92%] +........................................................................ [ 96%] +..............sss....................................................... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.8-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 32 +src/dashboards/alerts.py 201 0 100% +src/dashboards/api.py 491 1 99% 220 +src/dashboards/app.py 190 0 100% +src/dashboards/auth.py 265 0 100% +src/dashboards/certs.py 292 0 100% +src/dashboards/collectors.py 216 5 98% 328-329, 333-334, 351 +src/dashboards/config.py 126 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183 +src/dashboards/rates.py 73 0 100% +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 134 0 100% +src/dashboards/sources/gateway.py 584 8 99% 315, 558, 569, 678, 971, 1022, 1089, 1451 +src/dashboards/sources/host.py 209 0 100% +src/dashboards/sources/llm.py 178 1 99% 285 +src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 313 2 99% 553-554 +------------------------------------------------------------------ +TOTAL 4176 46 99% +Required test coverage of 94% reached. Total coverage: 98.90% +1855 passed, 17 skipped, 1 warning in 201.21s (0:03:21) +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__02_the_privileged_peer_is_a_socket_not_a_sudo_line.a1.err b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__02_the_privileged_peer_is_a_socket_not_a_sudo_line.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__02_the_privileged_peer_is_a_socket_not_a_sudo_line.a1.md b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__02_the_privileged_peer_is_a_socket_not_a_sudo_line.a1.md new file mode 100644 index 0000000..e5cc2d3 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__02_the_privileged_peer_is_a_socket_not_a_sudo_line.a1.md @@ -0,0 +1 @@ +Now let me look at the existing deploy-script test harness and the agent's unit template in `install-agent.sh`, plus the agent's main guard. diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__02_the_privileged_peer_is_a_socket_not_a_sudo_line.a1.md.status b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__02_the_privileged_peer_is_a_socket_not_a_sudo_line.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__02_the_privileged_peer_is_a_socket_not_a_sudo_line.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__02_the_privileged_peer_is_a_socket_not_a_sudo_line.a1.validate b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__02_the_privileged_peer_is_a_socket_not_a_sudo_line.a1.validate new file mode 100644 index 0000000..9a87a5d --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__02_the_privileged_peer_is_a_socket_not_a_sudo_line.a1.validate @@ -0,0 +1,69 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 3%] +........................................................................ [ 7%] +........................................................................ [ 11%] +........................................................................ [ 15%] +........................................................................ [ 19%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 30%] +........................................s............................... [ 34%] +........................................................................ [ 38%] +........................................................................ [ 42%] +........................................................................ [ 46%] +........................................................................ [ 50%] +........................................................................ [ 53%] +........................................................................ [ 57%] +........................................................................ [ 61%] +........................................................................ [ 65%] +........................................................................ [ 69%] +........................................................................ [ 73%] +.........................sssssssssssss.................................. [ 76%] +........................................................................ [ 80%] +........................................................................ [ 84%] +........................................................................ [ 88%] +........................................................................ [ 92%] +........................................................................ [ 96%] +..............sss....................................................... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.8-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 32 +src/dashboards/alerts.py 201 0 100% +src/dashboards/api.py 491 1 99% 220 +src/dashboards/app.py 190 0 100% +src/dashboards/auth.py 265 0 100% +src/dashboards/certs.py 292 0 100% +src/dashboards/collectors.py 216 4 98% 329, 333-334, 351 +src/dashboards/config.py 126 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 112 4 96% 126, 163, 180, 183 +src/dashboards/rates.py 73 0 100% +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 134 0 100% +src/dashboards/sources/gateway.py 584 8 99% 315, 558, 569, 678, 971, 1022, 1089, 1451 +src/dashboards/sources/host.py 209 0 100% +src/dashboards/sources/llm.py 178 1 99% 285 +src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 313 2 99% 553-554 +------------------------------------------------------------------ +TOTAL 4176 44 99% +Required test coverage of 94% reached. Total coverage: 98.95% +1855 passed, 17 skipped, 1 warning in 175.99s (0:02:55) +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__03_the_agent_accepts_one_write.a1.err b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__03_the_agent_accepts_one_write.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__03_the_agent_accepts_one_write.a1.md b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__03_the_agent_accepts_one_write.a1.md new file mode 100644 index 0000000..4e9263b --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__03_the_agent_accepts_one_write.a1.md @@ -0,0 +1 @@ +Now let me look at the dashboard side — the `connections[].id` scheme in `sources/gateway.py`, the Handler and `serve`/`main` in the agent, and the phase-32 additivity control script. diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__03_the_agent_accepts_one_write.a1.md.status b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__03_the_agent_accepts_one_write.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__03_the_agent_accepts_one_write.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__03_the_agent_accepts_one_write.a1.validate b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__03_the_agent_accepts_one_write.a1.validate new file mode 100644 index 0000000..7ad6c3b --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__03_the_agent_accepts_one_write.a1.validate @@ -0,0 +1,71 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 3%] +........................................................................ [ 7%] +........................................................................ [ 11%] +........................................................................ [ 15%] +........................................................................ [ 19%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 30%] +........................................s............................... [ 34%] +........................................................................ [ 38%] +........................................................................ [ 42%] +........................................................................ [ 46%] +........................................................................ [ 50%] +........................................................................ [ 53%] +........................................................................ [ 57%] +........................................................................ [ 61%] +........................................................................ [ 65%] +........................................................................ [ 69%] +........................................................................ [ 73%] +.........................sssssssssssss.................................. [ 76%] +........................................................................ [ 80%] +........................................................................ [ 84%] +........................................................................ [ 88%] +........................................................................ [ 92%] +........................................................................ [ 96%] +..............sss....................................................... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.8-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 32 +src/dashboards/alerts.py 201 0 100% +src/dashboards/api.py 491 1 99% 220 +src/dashboards/app.py 190 0 100% +src/dashboards/auth.py 265 0 100% +src/dashboards/certs.py 292 0 100% +src/dashboards/collectors.py 216 5 98% 328-329, 333-334, 351 +src/dashboards/config.py 126 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183 +src/dashboards/rates.py 73 0 100% +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 134 0 100% +src/dashboards/sources/gateway.py 584 8 99% 315, 558, 569, 678, 971, 1022, 1089, 1451 +src/dashboards/sources/host.py 209 0 100% +src/dashboards/sources/llm.py 178 1 99% 285 +src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 313 2 99% 553-554 +------------------------------------------------------------------ +TOTAL 4176 46 99% +Required test coverage of 94% reached. Total coverage: 98.90% +1855 passed, 17 skipped, 1 warning in 174.63s (0:02:54) +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == +advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose): + ruff 0.17.2 0.17.3 wheel diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.a1.err b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.a1.md b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.a1.md new file mode 100644 index 0000000..3f3094c --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.a1.md @@ -0,0 +1,28 @@ +Task 04 is complete. Short report: + +## Task 04 — `enabled` joins the payload, `off` becomes reachable, warnings learn the marker + +**Verdict: done.** All completion criteria met; `bash .agents/validate.sh` **exit 0**. + +### What this run added (the tree carried the source work from the earlier session; I verified every behaviour and added the missing mandated surface) + +- **`tests/conftest.py`** — two new fixture modes, both the *captured* fleet tree plus an edit, run through the real agent `Collector` on a fake `--root`: `marked` (the helper's three edits: `gitea` `server` line, nextcloud vhost `proxy_pass`, `brain` map entry → sink, each with the exact `#off ` trailing comment, plus the sink block in `stream.d/00-dashboards-sink.conf`) and `hand_edited` (same three shapes, markers removed). +- **`tests/test_gateway_source.py`** — the six mandated tests: three-marker fixture → three `enabled: false` rows with selectors/endpoints intact, `warnings == []` with the shapes proven present (suppression, not vacuous); the marker-vs-hand-deleted **pair** (silence ↔ `haproxy_backend_empty` + `nginx_map_route_missing`); sink recognised by block, not a hard-coded port; `0` only when the table answered, `null` + no samples when it didn't; stub/answer key-set parity (stub's `None` = absence of the claim). +- **`tests/test_api.py`** — `disabled` restated in the pinned 13-key set with its reason; summary test gains `marked` (=3) and `hand_edited` (=0) beside `full` (=0); the overview-line test learns `GATEWAY_DISABLED_FIELD` and pins the `> 0` / `Number.isInteger` guards (`0` prints nothing, `null` prints nothing). +- **`page-index.js`** — the `switched off` tally clause beside the `warnings` one, same guard. +- **Docs** — README + AGENTS.md name `enabled`/`off`, the `#off` id suffix, and the marker-aware warnings rule. +- **Rule 8** — `uv pip list --outdated` was non-empty before the gate; updated on purpose (pytest 9.2.0 → needed `asyncio_default_fixture_loop_scope = "function"` in `pyproject.toml`, pydantic 2.14.2, ruff 0.17.3, websockets 17.4). Now empty. + +### Evidence + +- **Gate**: 1877 passed / 17 skipped · coverage 98.86% (floor 94%; `gateway.py` 98%, `api.py` 99% — both >90%) · ruff clean · pyright 0 · `node --check` clean. +- **Live (rule 10)**: dev server restarted; all sources ok; live `/api/summary` serves `gateway: {… 'disabled': 0, 'connections': 33, 'warnings': 0}` (the real fleet has no markers — honest zero, live config untouched); browser pass on the overview page: `gateway 10 listeners · 33 connections · 15 live in · 15 live out` — four figures, **no `· 0 switched off` clause**, exactly the empty-state rule restated for the switch. Screenshot: `data/logs/gateway-switch-04-overview-line.png`. +- Report written to `.agents/reports/33_gateway_route_switching/…04….a2.md`; ntfy sent. + +### Flags (left to the driver/operator) + +1. **Task 05's `complete/` copy is from an aborted run** — its code is **not** in the tree (no `POST /api/gateway/route`, no `GATEWAY_ALLOW_WRITES`, no `collectors.post_route`), and its transcript ends mid-work. 05 needs a clean re-run; its `todo/` copy is still there, which is what the driver looks for. +2. **04's `todo/` copy is orphaned** (identical file in `complete/`; 01–03 follow the move convention). Per the working protocol I didn't touch `todo/` — the driver's move step should remove it. +3. None of the phase-33 work is committed yet. + +Ready for task 05 on your go — or let the pipeline re-run it. diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.a1.md.status b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.a1.validate b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.a1.validate new file mode 100644 index 0000000..d5fbe76 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.a1.validate @@ -0,0 +1,70 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 3%] +........................................................................ [ 7%] +........................................................................ [ 11%] +........................................................................ [ 15%] +........................................................................ [ 19%] +........................................................................ [ 22%] +........................................................................ [ 26%] +........................................................................ [ 30%] +...............................................s........................ [ 34%] +........................................................................ [ 38%] +........................................................................ [ 41%] +........................................................................ [ 45%] +........................................................................ [ 49%] +........................................................................ [ 53%] +........................................................................ [ 57%] +........................................................................ [ 60%] +........................................................................ [ 64%] +........................................................................ [ 68%] +........................................................................ [ 72%] +...............................................sssssssssssss............ [ 76%] +........................................................................ [ 79%] +........................................................................ [ 83%] +........................................................................ [ 87%] +........................................................................ [ 91%] +........................................................................ [ 95%] +....................................sss................................. [ 98%] +...................... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.8-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 32 +src/dashboards/alerts.py 201 0 100% +src/dashboards/api.py 492 1 99% 220 +src/dashboards/app.py 190 0 100% +src/dashboards/auth.py 265 0 100% +src/dashboards/certs.py 292 0 100% +src/dashboards/collectors.py 216 5 98% 328-329, 333-334, 351 +src/dashboards/config.py 126 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183 +src/dashboards/rates.py 73 0 100% +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 134 0 100% +src/dashboards/sources/gateway.py 613 10 98% 339, 588, 601-602, 652, 785, 1089, 1140, 1207, 1569 +src/dashboards/sources/host.py 209 0 100% +src/dashboards/sources/llm.py 178 1 99% 285 +src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 313 2 99% 553-554 +------------------------------------------------------------------ +TOTAL 4206 48 99% +Required test coverage of 94% reached. Total coverage: 98.86% +1877 passed, 17 skipped, 1 warning in 171.02s (0:02:51) +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.a2.md b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.a2.md new file mode 100644 index 0000000..193560d --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.a2.md @@ -0,0 +1,110 @@ +# Task 04 completion report (attempt 2, manual run) + +**Status:** complete — all completion criteria met, `bash .agents/validate.sh` green (exit 0). + +Note: attempt `a1`'s transcript died mid-work (it ends at "Now let me read the current +`warnings()` and `config_connections()` in the source:") yet its `.status` read `ok`; the +`a1` report for task 05 likewise claims tasks 01–04 were unimplemented and was itself +aborted. The tree carried the task-04 source work from the earlier session; this run +verified every mandated behaviour end-to-end and added the missing mandated tests, the +two fixture modes, the overview clause and its pins, and the doc sentences. + +## What this run added (on top of the pre-existing task-04 source work) + +- `tests/conftest.py` — two new `GATEWAY_MODES`, both the **captured** tree plus an edit, + run through the real agent `Collector` with a fake `--root` tree (wire-shape renames + fail tests, never draw empty pipes): + - `marked` — the helper's three edits: the `gitea` `server` line, the nextcloud vhost + `proxy_pass` line and the `brain.reeseapps.com` stream `map` entry (value rewritten + to the sink `127.0.0.1:44450`) each carrying the agent's `#off ` trailing + comment (the whole comment, one name — `split_marker`'s grammar), plus the helper's + sink `server` block in `stream.d/00-dashboards-sink.conf`. + - `hand_edited` — the same three shapes without the markers: the `server` line deleted, + the `proxy_pass` line plain, the map entry's value deleted, no sink file. +- `tests/test_gateway_source.py` — the six mandated tests, all passing: + - `test_a_marker_off_route_is_declared_and_disabled` — the `marked` fixture yields + exactly three `enabled: false` rows, selectors and endpoints intact (`gitea-2222` → + `reeseapps-gitea.reeselink.com:2222`, `nextcloud.reeseapps.com:11000`, + `127.0.0.1:44450`), every other row `true`, `#off` id suffix on every off row, + `warnings == []` with the marker-caused shapes proven present in the raw sections. + - `test_the_source_never_calls_a_route_broken_because_the_operator_disabled_it` / + `test_the_same_shape_without_a_marker_is_still_a_warning` — the pair: marker-caused + shapes warn nothing; the same shapes hand-deleted bring back + `haproxy_backend_empty` (subject `gitea-2222`) and `nginx_map_route_missing` + (subject `targetBackend:brain.reeseapps.com`), with no `#off` row in that arm. + - `test_the_sink_address_is_recognised_not_hard_coded` — a sink block on a different + port (`45999`) still reads as off from the payload's own text; a loopback target + that is *not* the block's port is on. + - `test_a_disabled_line_still_publishes_only_a_measured_count` — table answered and + empty: off rows `0` with a `0.0` sample (measured, never stamped); table absent + (`available: null`): every row `null`, no samples. + - `test_the_stub_and_the_answer_share_one_key_set` — answered rows carry `true`/`false` + (never null, never absent); the stub carries no `enabled`/`disabled` key anywhere + (nobody read the config, so its `None` is the absence of the claim); top-level + parity modulo `unreachable` / `agent_version`. + - plus the extended no-verdict walk (`marked`, `hand_edited` added to the modes). +- `tests/test_api.py` — `GATEWAY_BLOCK_FIELDS` comment restates the thirteenth key with + its reason; `test_the_summary_gateway_block_is_the_payloads_sums...` gains `marked` + (`disabled == 3`) and `hand_edited` (`disabled == 0`, the markers gone → the off state + gone) beside `full`'s `0`; the stub test's `disabled` pin was already in the tree; + `test_the_overview_line_reads_the_gateway_block_it_claims` learns + `GATEWAY_DISABLED_FIELD` (table resolved against the live block) and pins the guard: + printed only `> 0` and `Number.isInteger` — `0` prints nothing, `null` prints nothing. +- `src/dashboards/web/static/page-index.js` — `GATEWAY_DISABLED_FIELD` + (`disabled` / `switched off`) and the second non-zero-guarded clause beside the + `warnings` one. +- `pyproject.toml` — `asyncio_default_fixture_loop_scope = "function"` (pytest 9.2.0 + errors at configure without it; the deprecation's own prescription). +- `README.md` / `AGENTS.md` — the `/gateway` paragraph names `enabled`/`off`, the `#off` + id suffix, and the marker-aware warnings rule (marker-caused shapes warn nothing; the + same shape from a hand-deleted line still warns). Task 08's docs pass re-checks both + project-wide. + +## Dependency maintenance (rule 8) + +`uv pip list --outdated` was non-empty before this run's gate; updated on purpose: +`uv lock --upgrade` + `uv sync` — pydantic 2.14.2, pydantic-core 2.50.2, pytest 9.2.0, +ruff 0.17.3, websockets 17.4 (nodeenv/opentelemetry-api resolved by the same pass). +Now empty. + +## Criteria evidence + +- 3-marker fixture → three `enabled: false` rows, intact, `warnings == []`: the source + test above; the API arm asserts `block["disabled"] == 3`. +- Markers removed → warnings back: the pair above (one test pair, both directions). +- `gateway.disabled`: `3` marked / `0` clean capture / `null` stub (stub test asserts + `block["disabled"] is None`); the overview line prints nothing for `0` — pinned in + the JS test and observed on the live page (below). +- No `severity` key at any depth: the recursive walk passes, extended, unmodified in + meaning. +- Gate: `uv run pytest` 1877 passed / 17 skipped; coverage 98.86% (floor 94%; + `sources/gateway.py` 98%, `api.py` 99%, both >90%); ruff clean; pyright 0; + `node --check` clean; `uv pip list --outdated` empty; `bash .agents/validate.sh` + exit 0. + +## Live verification (rule 10) + +Dev server restarted (`./scripts/dev-server.sh restart`); `/api/health` ok for all +sources; the restarted server's live `/api/summary` gateway block serves the new key: +`{'host': 'gateway', 'listeners': 10, 'connections': 33, 'disabled': 0, 'live_in': 14, +'live_out': 14, 'map_targets': 18, 'warnings': 0}` — `disabled: 0` is the honest live +value (the real fleet carries no `#off` marker; the live config was not touched). +Real-browser pass signed in with `testpass`: the ingress line renders +`gateway 10 listeners · 33 connections · 15 live in · 15 live out` — the four figures +present, no `· 0 switched off` clause (the guard suppresses it at `0`, the +empty-state rule restated for the switch). Screenshot: +`data/logs/gateway-switch-04-overview-line.png`. + +## Flags for the operator (bookkeeping, left to the driver) + +1. `04_...md` now exists **identically in both** `todo/` and `complete/` (the `a1` run + copied it without removing the `todo/` copy; 01–03 follow the move convention). + Per the working protocol I did not touch `todo/`; the driver's move step should + remove the `todo/` copy. +2. `05_...md` also exists in `complete/` from the aborted `a1` run — **task 05's work + is not in the tree** (no `POST /api/gateway/route`, no `GATEWAY_ALLOW_WRITES`, no + `collectors.post_route`), and its `a1` transcript ends mid-work ("Now let me read + the code I'm touching"). Task 05 needs a clean re-run; its `todo/` copy is still + present, which is exactly what the driver looks for. +3. None of the phase-33 file moves or code is committed yet (`git status`: 01–03 moved, + 04/05 in `complete/` untracked, all code changes uncommitted). diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__05_the_api_gates_the_write_ten_times.a1.err b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__05_the_api_gates_the_write_ten_times.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__05_the_api_gates_the_write_ten_times.a1.md b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__05_the_api_gates_the_write_ten_times.a1.md new file mode 100644 index 0000000..c76555c --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__05_the_api_gates_the_write_ten_times.a1.md @@ -0,0 +1,10 @@ +All changes are in the working tree, nothing staged. Task complete — final report: + +**Phase 33 task 05 — done. `POST /api/gateway/route` is the app's second and last upstream write.** +- `config.py`: `GATEWAY_ALLOW_WRITES` (default **off**, NUT truthy rule) + `GATEWAY_WRITE_TIMEOUT_SECONDS` (35 s, outlives the agent's 30 s peer timeout); read budget untouched. +- `collectors.py`: `post_route` — per-dial `AgentTls`, revocation before any socket, no `Authorization` on the certificate path, redacted URLs, helper's object verbatim. +- `api.py`: the route with the ten numbered gates (1–3 middleware; 4 `DASH_READ_ONLY`; 5 flag; 6 `route:
` throttle + `Retry-After`; 7 hand-parsed body 422; 8 payload state 409 `{changed: false}`; 9 freshness 409; 10 `danger="DANGER"` for both verbs); one event per call, no diff/path/config text in any event; `/api/config` serves `gateway.writes_enabled`. +- Tests: surface test rewritten (two fleet writes, fails on a third), all ten gates parametrised with a dial spy, revoked-agent no-socket, no-diff event pin, danger/throttle/read-only/write-timeout pins, wire-id base spelling; `test_auth_api` walk + CSRF on the wire. +- Docs same commit: AGENTS.md invariant 1 now names two gated exceptions (plus the api/collectors/config bullets, gap 8 removed), README + `.env.example` env rows; `DASH_READ_ONLY` greps in both write paths. +- Dev-server curl checks: writes-off answers the `GATEWAY_ALLOW_WRITES` sentence; live 1.3.0 fleet answers the version gate first (correct order). +- `bash .agents/validate.sh` **exit 0**: 1915 passed / 17 skipped, coverage 98.72% (94 floor; `api.py` 99%, `collectors.py` 98%), ruff + pyright clean. diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__05_the_api_gates_the_write_ten_times.a1.md.status b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__05_the_api_gates_the_write_ten_times.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__05_the_api_gates_the_write_ten_times.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__05_the_api_gates_the_write_ten_times.a1.validate b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__05_the_api_gates_the_write_ten_times.a1.validate new file mode 100644 index 0000000..5ce70e9 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__05_the_api_gates_the_write_ten_times.a1.validate @@ -0,0 +1,70 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 3%] +........................................................................ [ 7%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 18%] +........................................................................ [ 22%] +........................................................................ [ 26%] +........................................................................ [ 29%] +.......................................................................s [ 33%] +........................................................................ [ 37%] +........................................................................ [ 40%] +........................................................................ [ 44%] +........................................................................ [ 48%] +........................................................................ [ 52%] +........................................................................ [ 55%] +........................................................................ [ 59%] +........................................................................ [ 63%] +........................................................................ [ 67%] +........................................................................ [ 70%] +........................................................................ [ 74%] +.............sssssssssssss.............................................. [ 78%] +........................................................................ [ 81%] +........................................................................ [ 85%] +........................................................................ [ 89%] +........................................................................ [ 93%] +........................................................................ [ 96%] +..sss....................................................... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.8-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 32 +src/dashboards/alerts.py 201 0 100% +src/dashboards/api.py 622 9 99% 229, 1273-1275, 1310-1313, 1379, 1387, 1436, 1439 +src/dashboards/app.py 190 0 100% +src/dashboards/auth.py 265 0 100% +src/dashboards/certs.py 292 0 100% +src/dashboards/collectors.py 244 4 98% 387, 391-392, 409 +src/dashboards/config.py 128 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 112 4 96% 126, 163, 180, 183 +src/dashboards/rates.py 73 0 100% +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 134 0 100% +src/dashboards/sources/gateway.py 613 10 98% 339, 588, 601-602, 652, 785, 1089, 1140, 1207, 1569 +src/dashboards/sources/host.py 209 0 100% +src/dashboards/sources/llm.py 178 1 99% 285 +src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 313 2 99% 553-554 +------------------------------------------------------------------ +TOTAL 4366 54 99% +Required test coverage of 94% reached. Total coverage: 98.76% +1915 passed, 17 skipped, 1 warning in 173.60s (0:02:53) +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__06_the_switch_obeys.a1.err b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__06_the_switch_obeys.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__06_the_switch_obeys.a1.md b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__06_the_switch_obeys.a1.md new file mode 100644 index 0000000..546e106 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__06_the_switch_obeys.a1.md @@ -0,0 +1,10 @@ +Task 06 complete. Final report: + +**33/06 — the switch obeys `gateway.writes_enabled`** + +- **Two-arm switch** (`page-gateway.js`): the arm is `gateway.writes_enabled` read from `/api/config` (never inferred; a dead fetch leaves the safe arm). Writes-off: `aria-disabled="true"`, `cursor: not-allowed`, no listener, title = the API's refusal byte-for-byte. Writes-on: focusable `role="switch"`, `cursor: pointer`, one click listener, a title naming the config-file edit → validator → reload. Only the switch's own `aria-checked` may be optimistic; the row's word/class/spine stay the payload's. +- **Flow**: click → server `dry_run: true` (Gate 10 now exempts it; it asks nothing, writes nothing, and writes no `route-*` event) → dialog prints the diff verbatim (`+`/`−` tinted from the existence pair, `--gate-on`/`--gate-off`) → typed `DANGER` (wrong word sends nothing) → one send, `aria-busy`, no retry loop. `applied: true` flips the switch early + `refresh()`; `applied: false` prints the helper's word and flips nothing. +- **Pins**: the spec's four named tests added (`…wired_only_when_the_deployment_enables_writes`, `…preview_then_the_word_then_sends_once`, `…painted_from_the_answer_not_the_click`, `…sentence_matches_the_routes_refusal` — the last reads both files via `ast`); hover allowed on the live arm only; `routeInConfig` pin moved to `conn.enabled !== false` with `connections.enabled` in `GATEWAY_PAGE_FIELDS`; dialog/diff pins added. +- **Gate**: `.agents/validate.sh` green — 1919 passed / 17 skipped, coverage 98.72% (floor 94), ruff + pyright clean, `node --check` clean. +- **Browser** (Node Playwright): hermetic writes-on smoke (patched `post_route`, `marked` fixture) — live arm, dry-run diff, wrong word, `DANGER` applied, optimistic flip, refused arm; 360px viewport (no sideways scroll, dialog in view), focus in the confirm field, Esc closes and returns focus; hover tint verified. Live dev server (restarted, all sources ok): off arm exact sentence + `not-allowed`. +- **Docs**: README switch section + agent 1.4.0; AGENTS.md invariant 1 gains the dry-run clause. Screenshots in `data/logs/33-task06-*.png`. diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__06_the_switch_obeys.a1.md.status b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__06_the_switch_obeys.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__06_the_switch_obeys.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__06_the_switch_obeys.a1.validate b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__06_the_switch_obeys.a1.validate new file mode 100644 index 0000000..1880c00 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__06_the_switch_obeys.a1.validate @@ -0,0 +1,70 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 3%] +........................................................................ [ 7%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 18%] +........................................................................ [ 22%] +........................................................................ [ 26%] +........................................................................ [ 29%] +........................................................................ [ 33%] +...s.................................................................... [ 37%] +........................................................................ [ 40%] +........................................................................ [ 44%] +........................................................................ [ 48%] +........................................................................ [ 52%] +........................................................................ [ 55%] +........................................................................ [ 59%] +........................................................................ [ 63%] +........................................................................ [ 66%] +........................................................................ [ 70%] +........................................................................ [ 74%] +.................sssssssssssss.......................................... [ 78%] +........................................................................ [ 81%] +........................................................................ [ 85%] +........................................................................ [ 89%] +........................................................................ [ 92%] +........................................................................ [ 96%] +......sss....................................................... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.8-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 32 +src/dashboards/alerts.py 201 0 100% +src/dashboards/api.py 625 9 99% 229, 1273-1275, 1310-1313, 1382, 1390, 1446, 1449 +src/dashboards/app.py 190 0 100% +src/dashboards/auth.py 265 0 100% +src/dashboards/certs.py 292 0 100% +src/dashboards/collectors.py 244 5 98% 386-387, 391-392, 409 +src/dashboards/config.py 128 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183 +src/dashboards/rates.py 73 0 100% +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 134 0 100% +src/dashboards/sources/gateway.py 613 10 98% 339, 588, 601-602, 652, 785, 1089, 1140, 1207, 1569 +src/dashboards/sources/host.py 209 0 100% +src/dashboards/sources/llm.py 178 1 99% 285 +src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 313 2 99% 553-554 +------------------------------------------------------------------ +TOTAL 4369 56 99% +Required test coverage of 94% reached. Total coverage: 98.72% +1919 passed, 17 skipped, 1 warning in 171.78s (0:02:51) +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a1.err b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a1.md b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a1.md new file mode 100644 index 0000000..25b9160 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a1.md @@ -0,0 +1,13 @@ +No interactive channel is available to confirm operator presence, so per the task protocol I stop after step 3 and report. + +--- + +**Task 07 report — steps 1–3 complete; step 4 (live drill) not started, operator absence** + +1. **Backups** — all four live configs + agent unit hashed to `/root/dashboards-33-backup-20261101-0950`; `diff -r` byte-identical. +2. **Rehearsal** — sandbox tree with real config copies: 3 `--apply` arms (haproxy server, nginx vhost, nginx map→sink) and the `map-default` refusal all behaved; checker-failure restore proven; sandbox left byte-identical. +3. **Install** — 1.4.0 + `--allow-writes` (installer diff: exactly one line), routectl helper/socket/template on the box; two live-kernel bugs found and fixed, both pinned in tests. +4. **Live-kernel fixes (repo + box)** — `Accept=yes` socket activation (systemd 257 rejected `Service=`); `serve()` reads fd 3 then fd 0 with `recv()`; refusals sent on the wire; ucred layout calibrated (this kernel delivers `{pid,uid,gid}`) with a root `setuid` probe so a matching gid on a foreign uid **cannot** open the gate; a `refused()` tuple-inversion that serialised refusals as JSON lists was caught by the live chain and fixed. +5. **Gates proven live** — true uid-994 client passes; `nobody` and uid-1000 get EACCES; root-with-matching-gid is refused by the peer gate; full 4-hop dry run (dashboard TLS dial → agent `POST /route` → socket → helper) returned the helper's verbatim diff (`#off iperf3`), nothing written. +6. **Post-dry-run state** — all four config hashes equal pre-drill, state sidecar empty, zero proxy reloads; full suite/Ruff/Pyright/`validate.sh` green on the final code. +7. **Step 4 blocked** — the off→on drill on a named route awaits an operator present to watch; the ingress is left byte-identical to its pre-work state. diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a1.md.status b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a1.md.status new file mode 100644 index 0000000..2e6b050 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a1.md.status @@ -0,0 +1 @@ +error: compaction failed: Auto-compaction failed: Summarization failed: generation hit the token cap and the summary is incomplete; compaction failed: Auto-compaction… diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a1.validate b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a1.validate new file mode 100644 index 0000000..f735b4d --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a1.validate @@ -0,0 +1,73 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 3%] +........................................................................ [ 7%] +........................................................................ [ 10%] +........................................................................ [ 14%] +........................................................................ [ 18%] +........................................................................ [ 21%] +........................................................................ [ 25%] +........................................................................ [ 29%] +........................................................................ [ 32%] +.........................s.............................................. [ 36%] +........................................................................ [ 40%] +........................................................................ [ 43%] +........................................................................ [ 47%] +........................................................................ [ 51%] +........................................................................ [ 54%] +........................................................................ [ 58%] +........................................................................ [ 62%] +........................................................................ [ 65%] +........................................................................ [ 69%] +........................................................................ [ 73%] +.................................................sssssssssssss.......... [ 76%] +........................................................................ [ 80%] +........................................................................ [ 84%] +........................................................................ [ 87%] +........................................................................ [ 91%] +........................................................................ [ 95%] +......................................sss............................... [ 98%] +........................ [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.8-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 32 +src/dashboards/alerts.py 201 0 100% +src/dashboards/api.py 625 9 99% 229, 1273-1275, 1310-1313, 1382, 1390, 1446, 1449 +src/dashboards/app.py 190 0 100% +src/dashboards/auth.py 265 0 100% +src/dashboards/certs.py 292 0 100% +src/dashboards/collectors.py 244 5 98% 386-387, 391-392, 409 +src/dashboards/config.py 128 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183 +src/dashboards/rates.py 73 0 100% +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 134 0 100% +src/dashboards/sources/gateway.py 613 10 98% 339, 588, 601-602, 652, 785, 1089, 1140, 1207, 1569 +src/dashboards/sources/host.py 209 0 100% +src/dashboards/sources/llm.py 178 1 99% 285 +src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 313 2 99% 553-554 +------------------------------------------------------------------ +TOTAL 4369 56 99% +Required test coverage of 94% reached. Total coverage: 98.72% +1951 passed, 17 skipped, 1 warning in 193.21s (0:03:13) +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == +advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose): + anyio 4.15.1 4.16.0 wheel diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a2.err b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a2.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a2.md b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a2.md new file mode 100644 index 0000000..d90dcc8 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a2.md @@ -0,0 +1,10 @@ +All green. Task 07 re-run is complete: + +1. **Fixed:** anyio 4.15.1 → 4.16.0 (`uv lock` + `uv sync`; `uv pip list --outdated` now empty, rule 8); completed the two missing task-07 deliverables. +2. **Added** the opt-in live test (step 8): `test_the_switch_round_trip_on_one_named_route` — `live`-marked, gated by a separate `--allow-live-writes` flag (new `conftest.py` option), skips by default with a reason naming its route, restores in a `finally`; dials the app's own `collectors.post_route` and proves the round trip on the production read (dry-run diff, helper's `applied/validated/reloaded`, row `enabled` both ways, byte-identical haproxy section pre/post). +3. **Wrote** `data/logs/NOTES-33-switching.md`: first line = *all routes enabled*; before/after hashes identical (drill never wrote); guarded unit diff = exactly ` --allow-writes`; socket `0600 dashboards-agent:dashboards-agent`; three probe transcripts (`list` → `{"ok":true,"state":[]}`, foreign uid → EACCES, wire body gate); journal window (two activations, no reload). +4. **Decisions:** drill stopped at step 4 — (a) no operator said go in-session (the task's own stop condition) and (b) independently, the checker precondition cannot pass: no host `nginx`/`haproxy` binary on the box (podman container units), so a real `--apply` refuses at the checker and restores; the checker fix (container path or host binaries) is a task-08 decision, not made unilaterally. +5. **New finding (recorded, not fixed):** this box's systemd 257 build silently drops the helper unit's `ProtectSystem=read-only` (gperf bug — `off/yes/strict/full` parse clean); the socket gate is intact; candidate `full` + state-dir `ReadWritePaths` fix left for operator review with the unit diff. +6. **Gates:** `validate.sh` exit 0 — **1951 passed, 18 skipped**, coverage **98.72%** (floor 94%), Ruff + Pyright clean, JS syntax + hygiene pass; dev server: 10 sources `ok`. +7. **Live read-only:** `pytest -m live --live` → 13 passed, 1 skipped (the round-trip, skip reason names `haproxy:5201->web.reeselink.com:5201`). +8. **Next pending:** task 08 (docs + live verify) — the drill itself (needs an operator's go plus the checker fix) and the `ProtectSystem` decision. diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a2.md.status b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a2.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a2.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a2.validate b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a2.validate new file mode 100644 index 0000000..34e22df --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__07_rehearse_then_drill_on_the_live_ingress.a2.validate @@ -0,0 +1,71 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 3%] +........................................................................ [ 7%] +........................................................................ [ 10%] +........................................................................ [ 14%] +........................................................................ [ 18%] +........................................................................ [ 21%] +........................................................................ [ 25%] +........................................................................ [ 29%] +........................................................................ [ 32%] +.........................s.............................................. [ 36%] +........................................................................ [ 40%] +........................................................................ [ 43%] +........................................................................ [ 47%] +........................................................................ [ 51%] +........................................................................ [ 54%] +........................................................................ [ 58%] +........................................................................ [ 62%] +........................................................................ [ 65%] +........................................................................ [ 69%] +........................................................................ [ 73%] +.................................................ssssssssssssss......... [ 76%] +........................................................................ [ 80%] +........................................................................ [ 84%] +........................................................................ [ 87%] +........................................................................ [ 91%] +........................................................................ [ 95%] +.......................................sss.............................. [ 98%] +......................... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.8-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 32 +src/dashboards/alerts.py 201 0 100% +src/dashboards/api.py 625 9 99% 229, 1273-1275, 1310-1313, 1382, 1390, 1446, 1449 +src/dashboards/app.py 190 0 100% +src/dashboards/auth.py 265 0 100% +src/dashboards/certs.py 292 0 100% +src/dashboards/collectors.py 244 4 98% 387, 391-392, 409 +src/dashboards/config.py 128 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183 +src/dashboards/rates.py 73 0 100% +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 134 0 100% +src/dashboards/sources/gateway.py 613 10 98% 339, 588, 601-602, 652, 785, 1089, 1140, 1207, 1569 +src/dashboards/sources/host.py 209 0 100% +src/dashboards/sources/llm.py 178 1 99% 285 +src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 313 2 99% 553-554 +------------------------------------------------------------------ +TOTAL 4369 55 99% +Required test coverage of 94% reached. Total coverage: 98.74% +1951 passed, 18 skipped, 1 warning in 189.82s (0:03:09) +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__08_docs_and_live_verify.a1.err b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__08_docs_and_live_verify.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__08_docs_and_live_verify.a1.md b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__08_docs_and_live_verify.a1.md new file mode 100644 index 0000000..6ee7512 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__08_docs_and_live_verify.a1.md @@ -0,0 +1,10 @@ +All green. Final report: + +**Task 08 complete — docs, browser pass, full gate.** + +- **Docs**: `AGENTS.md` (invariant 1's two gated writes, 1.4.0 agent + marker grammar + no-retry rule, routectl deploy surface + units + byte-identical agent unit, `sources/gateway.py` `disabled`/merge/warnings + `/api/summary`, `api.py` gate order + audit events, `post_route` revoke-before-socket, fleet record, UI conventions, gap 8) and `README.md` (Route switching section: Enabling it + undo-everything, Switching a route with the three recovery commands and the deliberate-outage sentence, host-agent write, table row, two config rows, stale "unwired" paragraph fixed, orphaned UPS text restored) — plus `GATEWAY_ALLOW_WRITES` declared (never assigned) in both env templates, pinned by the new `tests/test_auth_docs.py` `WRITE_GATES` test. +- **Browser pass**: `33-switching.cjs` **45/45 checks, exit 0** — both arms, `--gate-off`/`--gate-on` equality end-to-end in dark + light (on-half distinct from `--warn`/`--fail`), paint follows the payload never the click, `applied: false` arm, Escape, reduced-motion still, 360 px zero sideways scroll; 8 screenshots + `33-switching.out`. +- **Mobile audit**: `27-task02-mobile-audit.cjs` re-run → **exit 0, 371 PASS, 0 FAIL, 0 finding** (`33-task08-mobile-audit.out`). +- **Gate**: `validate.sh` **exit 0** (1953 passed, coverage 98.76 % ≥ 94, ruff clean incl. `deploy`, pyright 0, outdated empty); plain `uv run pytest` green; live suite **13 passed + the write round-trip skipped by design** (reason names the route it would touch). +- **Status checks**: dev server (writes off) 10/10 ok unchanged; smoke server (flag on, fake agent) 10/10 ok honest (real mTLS dials — the script now seeds the dev DB's `certs.*` rows read-only, since a fresh DB mints a CA the agents refuse) and a curl write round-trips: dry-run diff → `applied/validated/reloaded` → next payload's row `#off`/`enabled: false`. +- **Live call (conservative, no live changes)**: no real `DANGER` sent. The dry-run rehearsal stands from task 07; the phase's own decision leaves the drill to the operator, and this fleet's proxies run in podman containers so the host lacks the `nginx`/`haproxy` binary the helper's checker needs — a live drill here would refuse by construction (recorded in `NOTES-33-switching.md`). Read-only live verification is green. diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__08_docs_and_live_verify.a1.md.status b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__08_docs_and_live_verify.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__08_docs_and_live_verify.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__08_docs_and_live_verify.a1.validate b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__08_docs_and_live_verify.a1.validate new file mode 100644 index 0000000..33396bd --- /dev/null +++ b/.agents/reports/33_gateway_route_switching/33_gateway_route_switching__08_docs_and_live_verify.a1.validate @@ -0,0 +1,71 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 3%] +........................................................................ [ 7%] +........................................................................ [ 10%] +........................................................................ [ 14%] +........................................................................ [ 18%] +........................................................................ [ 21%] +........................................................................ [ 25%] +........................................................................ [ 29%] +........................................................................ [ 32%] +.........................s.............................................. [ 36%] +........................................................................ [ 40%] +........................................................................ [ 43%] +........................................................................ [ 47%] +........................................................................ [ 51%] +........................................................................ [ 54%] +........................................................................ [ 58%] +........................................................................ [ 62%] +........................................................................ [ 65%] +........................................................................ [ 69%] +........................................................................ [ 73%] +...................................................ssssssssssssss....... [ 76%] +........................................................................ [ 80%] +........................................................................ [ 84%] +........................................................................ [ 87%] +........................................................................ [ 91%] +........................................................................ [ 94%] +.........................................sss............................ [ 98%] +........................... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.8-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 32 +src/dashboards/alerts.py 201 0 100% +src/dashboards/api.py 625 9 99% 229, 1273-1275, 1310-1313, 1382, 1390, 1446, 1449 +src/dashboards/app.py 190 0 100% +src/dashboards/auth.py 265 0 100% +src/dashboards/certs.py 292 0 100% +src/dashboards/collectors.py 244 5 98% 386-387, 391-392, 409 +src/dashboards/config.py 128 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183 +src/dashboards/rates.py 73 0 100% +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 134 0 100% +src/dashboards/sources/gateway.py 613 10 98% 339, 588, 601-602, 652, 785, 1089, 1140, 1207, 1569 +src/dashboards/sources/host.py 209 0 100% +src/dashboards/sources/llm.py 178 1 99% 285 +src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 313 2 99% 553-554 +------------------------------------------------------------------ +TOTAL 4369 56 99% +Required test coverage of 94% reached. Total coverage: 98.72% +1953 passed, 18 skipped, 1 warning in 191.57s (0:03:11) +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.env.dev.example b/.env.dev.example index eddbaff..4e17c30 100644 --- a/.env.dev.example +++ b/.env.dev.example @@ -50,6 +50,12 @@ export HOST_TARGETS="driveripper=https://driveripper.example.net:9120 deskwork=h # else, or to poll faster while developing the page. # export GATEWAY_HOST=gateway # export GATEWAY_INTERVAL_SECONDS=10 +# The route switch's write gate — a decision of its own: off by default (one flag does not +# carry NUT_ALLOW_ACTIONS), and in a dev checkout it is a write to the **real fleet**: it +# authorises the dashboard's session to POST /api/gateway/route through the agent's +# certificate-first gate to the routectl helper, which can mark a route in the gateway's +# config on or off, validate the file with the proxy's own checker, and reload that proxy. +# export GATEWAY_ALLOW_WRITES=1 # The engines are dialled directly on their hostnames, like the agents. export GUFO_BASE_URL=http://deskwork.example.net:8731 diff --git a/.env.example b/.env.example index 05f1813..cf138c5 100644 --- a/.env.example +++ b/.env.example @@ -88,6 +88,12 @@ GATEWAY_HOST=gateway # tables — with headroom because the same handler may be mid-smartctl pass for /disks. GATEWAY_INTERVAL_SECONDS=30 GATEWAY_TIMEOUT_SECONDS=10 +# The route switch is the app's second write path and a decision of its own: off by +# default, the same truthy rule as NUT_ALLOW_ACTIONS, and one flag does not carry the +# other. The write's budget outlives the agent's 30 s peer timeout on purpose, so the +# dashboard never times out first and leaves you guessing whether the reload happened. +# GATEWAY_ALLOW_WRITES=1 +# GATEWAY_WRITE_TIMEOUT_SECONDS=35 # --------------------------------------------------------------------------- # # LLM dashboard: either engine is optional — unset means "not deployed here" @@ -137,8 +143,8 @@ NUT_USER= NUT_PASSWORD= NUT_INTERVAL_SECONDS=5 NUT_TIMEOUT_SECONDS=4 -# Instant commands (battery test, beeper, calibration) are the only write path in the -# app. Requires NUT_USER plus this flag; the UI still confirms every command. +# Instant commands (battery test, beeper, calibration) are one of the app's two write +# paths. Requires NUT_USER plus this flag; the UI still confirms every command. NUT_ALLOW_ACTIONS=0 # NUT_SIMULATE=1 replaces upsd with the labelled simulator (sources/nut_sim.py) so the UPS page # is developable with no battery in the house. Every panel it produces says SIMULATED, and it is diff --git a/AGENTS.md b/AGENTS.md index dcdcec0..5df2ad4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -28,24 +28,31 @@ file — plus an overview page: | `/disks` | Are any physical disks failing, and which — health, temp, wear | the same agents, `smartctl -i -A -H` per whole drive | | `/llm` | Tokens/s, active sessions, queue depth, KV pressure, latency | gufo `/metrics` + `/slots`, vLLM `/metrics` | | `/ups` | Battery health, runtime left, power events, test buttons | NUT `upsd` (our own wire client; port **3051**, not 3493) | -| `/gateway` | Which domains on which port, and where each goes | the same agent's `haproxy` + `nginx` sections joined to `/proc/net/tcp{,6}` on the one `GATEWAY_HOST` host | +| `/gateway` | Which domains on which port, and where each goes | the same agent's `haproxy` + `nginx` sections joined to `/proc/net/tcp{,6}` on the one `GATEWAY_HOST` host (the page needs agent **≥ 1.2.0**; the route switch's `#off` marker read is **1.4.0**) | History is a first-class feature: every number worth graphing is stored, rolled up (raw → 1m → 1h) and served from `/api/*/history`. None of it is public. One admin signs in at `/login` (the only page an anonymous visitor reaches); `/settings` (⚙ Account) and `/certs` (⚷ Certs) live behind it. Every page and -every `/api` route requires the session except the public entries in rule 12. +every `/api` route requires the session except the public entries in invariant 12. ## Non-negotiable rules Breaking one is a bug even if the page still renders. -1. **Read-only by construction, with exactly one exception.** Nothing writes to an upstream - except `POST /api/ups/command`, gated four times (env allow, not `DASH_READ_ONLY`, the - upsd account actually having the command, a UI confirmation — typed `DANGER` for - anything that can cut power) and re-checked on every request; power-cutting commands are - refused outright while on battery. Hiding a button is not a safety control. +1. **Read-only by construction, with exactly two exceptions, and both are gated on purpose.** + Nothing writes to an upstream except `POST /api/ups/command` and + `POST /api/gateway/route`, and each is re-checked on every request by its own gates: + the UPS one is gated four times (env allow, not `DASH_READ_ONLY`, the upsd account + actually having the command, a UI confirmation — typed `DANGER` for anything that can + cut power) and power-cutting commands are refused outright while on battery; the gateway + one is gated ten (not `DASH_READ_ONLY`, `GATEWAY_ALLOW_WRITES` — off by default — a + per-address write throttle, a hand-parsed body, the route's state in the current + payload, the freshness of the read end to end, and a typed `danger="DANGER"` for + **both** verbs — a `dry_run` preview asks nothing and writes nothing — for anything + that can cut a route through the fleet's only ingress), and it dials once through + the same revocation gate the reads take. Hiding a button is not a safety control. 2. **Never invent a number.** A value the upstream does not publish renders as `n/a`, never as `0`. Every derived value carries its `source` string and `derived: true`; the rate estimator is keyed by the *resolved* series name **plus its label set** and refuses a @@ -108,9 +115,9 @@ Breaking one is a bug even if the page still renders. session's `X-CSRF-Token`. A process whose lifespan never produced `ctx.auth` answers `503` for everything, public paths included. `tests/test_auth_api.py` walks the **served** route table so a route that forgets the gate fails a test instead of - shipping — the middleware guards by default. Nothing may be added to `/api/health` - without re-deciding that: it is public, and a new field there is a field the whole - internet reads. + shipping — the middleware guards by default. `/api/health` is a contract with + deployment, which means **nothing may be added to it without re-deciding that**: it is + public, and a new field there is a field the whole internet reads. ## Architecture @@ -118,7 +125,7 @@ Breaking one is a bug even if the page still renders. agent/host_agent.py (each host) gufo / vLLM upsd (NUT) │ GET /snapshot │ GET /metrics │ LIST UPS / VAR / CMD ▼ ▼ ▼ -collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command +collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command and post_route │ parse + Δcounter/Δt ▼ sources/host.py · disks.py · llm.py · nut.py · gateway.py poller.py (task per source) @@ -144,10 +151,25 @@ One line each — the load-bearing behaviour is in the bullet, the rest is in th error line per non-`true` arm — the missing one saying **not found**, the pinned wording the dashboard reads to answer "is it installed?" — and `--no-proxy` / `--no-resolve` remove their keys in silence. Nothing in a section can fail `/snapshot` or `/health`. - Counters stay counters: the dashboard computes rates. + Counters stay counters: the dashboard computes rates. **The one write (1.4.0, gateway + hosts):** a single `POST /route`, refused without a verified, non-revoked client + certificate **and** the agent's own `--allow-writes` flag (certificate first, in that + order), forwarding exactly `{id, action, dry_run}` to the socket-activated root helper + `dashboards-routectl` and answering its JSON verbatim — **one connection, one + request, one answer, no retry**, because a retry of a write is a second reload. The + privilege is the helper's socket, not the agent's: the unit keeps + `NoNewPrivileges=true` + `ProtectSystem=strict` byte-identical with and without + `--with-routectl`, and no sudoers line is involved. Every `GET` path is untouched — + `#off` markers are read on the normal proxy sections (the marker's own grammar is the + agent's: `ROUTE_MARKER` / `split_marker` / `config_line_index` — `disabled` on + `server`/`proxy_pass`/`map` nodes; a marked line stays in the document). - `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real env var always wins. Safe defaults except the sources, which default to unset. The - gateway source is enabled only when `GATEWAY_HOST` names a `HOST_TARGETS` entry. + gateway source is enabled only when `GATEWAY_HOST` names a `HOST_TARGETS` entry + (default `gateway`). The write's two settings are beside it: `GATEWAY_ALLOW_WRITES` + (default **off**, the NUT flag's truthy rule) and `GATEWAY_WRITE_TIMEOUT_SECONDS` + (default 35 s, deliberately longer than the agent's 30 s peer timeout so the dashboard + never times out first); the read's 10 s budget stays the read's alone. - `poller.py` — job loop, health records, stale/last-good semantics, transition events. `disks:` is a source of its own (300 s / 30 s) and `gateway` is **one** job (30 s / 10 s) that exists only when configured, because the page is about the single ingress; @@ -160,7 +182,11 @@ One line each — the load-bearing behaviour is in the bullet, the rest is in th - `collectors.py` — the fetch half. `AgentTls` makes an `https://` `HOST_TARGETS` entry a certificate-authenticated dial (per-dial client, this app's client cert verified against its own root; the revocation answer is a fresh read per poll, so a `/certs` revoke bites - on the next poll with no restart, before any socket opens). Failure messages name the + on the next poll with no restart, before any socket opens). `post_route` is the write's + dial beside `fetch_snapshot`: the same per-dial client and the same revocation check + before any socket opens, but **`Authorization` is never sent on the certificate path** — + the bundle is the credential there, and the token travels only on the plain token path. + The answer is the helper's own object, verbatim. Failure messages name the **redacted** URL. Agent `errors` become `degraded` **except** the add-on lines (`ADDON_ERROR_PREFIXES`: `smart`, `haproxy`, `nginx`, `sockets`, `resolver`) — a host without smartctl or a proxy config is unconfigured, not broken (rule 3); those lines @@ -179,8 +205,15 @@ One line each — the load-bearing behaviour is in the bullet, the rest is in th credited **only through an address the gateway's own `resolver` section published** — the module never calls a resolver itself (spy-pinned). Pipes are `null`, never `0`, when unmeasured. `warnings()` judges the two config files against themselves (seven kinds, - each `{id, kind, subject, message}`) and never joins a warning to a row. No `severity` - anywhere in the payload (a busy proxy is the job). + each `{id, kind, subject, message}`) and never joins a warning to a row. Each row's + `enabled` carries the config's own state (the 1.4.0 `#off` marker, or a map entry + pointed at the helper's sink); a merge that would mix states splits into two rows + keyed by `enabled` (a union of selectors with one disabled line is two facts; the + `#off` suffix keeps the halves distinct); the marker-caused warning shapes — an + emptied backend, a targetless route, a sink-pointed entry — warn nothing, while the + same shape from a hand-deleted line still warns; the switch-off tally is published + as `/api/summary`'s `disabled` field. No `severity` anywhere in the payload (a busy + proxy is the job). - `sources/llm.py` — gufo + vLLM field maps, measured Δ rates **diffed per label set**, prefill speed, the session list from gufo `/slots`. A refused estimator window is still an answer: the cell keeps its `source` and an honest `Δ` beside a `null`. @@ -225,11 +258,18 @@ One line each — the load-bearing behaviour is in the bullet, the rest is in th the startup read (it mints; nothing else may call it); `read_state` is the request-time read (same validation, never a write). `cryptography` is the app's only new dependency, used here alone; the agent stays stdlib. -- `api.py` — GETs everywhere plus the one POST that dials an upstream. Every source's +- `api.py` — GETs everywhere plus two POSTs that dial an upstream (`/api/ups/command` and + `/api/gateway/route`, invariant 1's two exceptions). Every source's configured gate runs **before** anything that could reach the network (spy-pinned: no - client, no fetch). `/api/config` is a redaction boundary. All five history routes go - through one door (`history_many`). `/api/ups/command` stays the app's only upstream - write (pinned by enumerating every served route's verbs). + client, no fetch). `/api/config` is a redaction boundary and serves + `gateway.writes_enabled` so the page can render the honest arm. All five history routes go + through one door (`history_many`). The upstream write set is exactly those two (pinned by + enumerating every served route's verbs; a third fails by name). The route write's gates + (invariant 1) run in a pinned order — read-only, flag, per-address throttle, body + shape, the id's state in the current payload, end-to-end freshness, typed `DANGER` for + both verbs — and every answered call stores one audit event under its own kind + (`route-disabled`/`route-enabled`/`route-denied`; a dry run stores none) in a fixed + shape: the id, the action, the helper's words — **never the config text**. - `app.py` — assembly and the **session gate** middleware (invariant 12), answering in this order: no `AuthState` → `503` for everything; no session and no such route → the router's `404`; public → through (the origin check still applies to a public write); @@ -247,8 +287,10 @@ One line each — the load-bearing behaviour is in the bullet, the rest is in th Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`, `gpu.busy_pct@0`, `ups.charge_pct`), `@stream` for per-device series. `sample.source` is the dashboard family — `host`, `llm`, `ups`, `disks` (the SMART series) and `gateway` -(the connection gauges) — so `GET /api/metrics?source=…` answers "what does *this page* -have history for". Treat the scheme as a public interface of the history API. +(the connection gauges: `gateway.in_conns@`, `gateway.out_conns@:` +— the port belongs in the outbound name because two services on one host are two facts) — +so `GET /api/metrics?source=…` answers "what does *this page* have history for". +Treat the scheme as a public interface of the history API. ## The fleet (real environment) @@ -256,19 +298,43 @@ have history for". Treat the scheme as a public interface of the history API. | ------------- | ------------- | ------------------------------------------------------------------------- | | `driveripper` | `driveripper-ops` | Threadripper 24c/48t + Radeon AI PRO R9700, vLLM `:8000`, **upsd**, the dashboards. 16 whole disks; the agent unit runs `User=root` with `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` (kernel 7.2's unprivileged-passthrough gate; an empty bounding set is capability-less, `CapEff 0`) | | `deskwork` | `deskwork-ops` | Strix Halo (Radeon 8060S iGPU), gufo `:8731` | -| `gateway` | `gateway-ops` | the router, the fleet's ingress: haproxy (`/etc/haproxy/haproxy.cfg`, six frontends → six backends) + nginx (`/etc/nginx/nginx.conf` plus `conf.d/` vhosts and `stream.d/`'s `map` on `:443`); runs agent **1.3.0** | +| `gateway` | `gateway-ops` | the router, the fleet's ingress: haproxy (`/etc/haproxy/haproxy.cfg`, six frontends → six backends) + nginx (`/etc/nginx/nginx.conf` plus `conf.d/` vhosts and `stream.d/`'s `map` on `:443`); runs agent **1.4.0** with the `dashboards-routectl` helper units | - **Transport is direct dial; the fleet is mTLS.** `HOST_TARGETS` are `https://` with no userinfo — the certificate bundle is the credential (the token in a URL is the compatibility path for a host nobody has bundled). All three hosts run the installed `dashboards-agent` systemd service, installed by `deploy/install-agent.sh` under the - sudo grant. **Re-stage `/home/ops/agent-install/` first — the grant names a path, not a - checkout —** and a re-run whose rendered unit differs from the installed one prints the - diff and refuses without `--force`. `deskwork`/`driveripper` still run **1.1.0**: the - 1.2.0/1.3.0 sections are purely additive (a 1.1.0 snapshot has none of the keys, which - `sources/gateway.py` answers as `supported: false`), so they move at their next re-stage. + sudo grant — registration is the install: `--fetch-bundle --cert-name + --session ` pulls the host's own bundle from this dashboard during the install, so a + host is never registered onto a token URL because a `curl` was forgotten. **Re-stage + `/home/ops/agent-install/` first — the grant names a path, not a checkout —** and a re-run + whose rendered unit differs from the installed one prints the diff and refuses without + `--force`. `deskwork`/`driveripper` still run **1.1.0**: the + 1.2.0/1.3.0/1.4.0 sections are purely additive (a 1.1.0 snapshot has none of the keys, + which `sources/gateway.py` answers as `supported: false`), so they move at their next + re-stage. A TLS agent serves https only — rolling a host back means removing the bundle lines too, - not just the URL. + not just the URL. **The gateway's write (1.4.0) installs beside the agent, never into + it**: `install-agent.sh --allow-writes --with-routectl` adds one argument to the agent + unit's `ExecStart` (`--allow-writes` — a diff that shows a security directive moving + is a stop, not a `--force`) and installs `dashboards-routectl.py` (0755, root, + `/usr/local/sbin/`) plus the two units **`dashboards-routectl.socket`** (`0600`, owned + by the agent's user) and **`dashboards-routectl@.service`** (root, one instance per + connection, `ReadWritePaths` = `/etc/haproxy` + `/etc/nginx/conf.d` + + `/etc/nginx/stream.d` only), verifying the agent's uid can connect before reporting + success. **No sudoers line is involved** — the grant is a file mode systemd enforces, + and `--with-routectl` alone leaves the agent's unit byte-identical; the helper edits + one `#off` route (the marker carries the line's original text after its `was:`, a + per-route state sidecar records who last changed it and the file's hash, an `enable` + restores only an unedited line — an edited one is a `drift` refusal, no byte moves — + and a switched-off `map` entry points at the helper's own sink `server`), validates + with the proxy's own checker (restore on failure) and **reloads, never restarts**. + `--without-routectl` removes it, printing the still-off routes first; + `deploy/gateway-routectl-README.md` is the operator's page. The live **drill has not + run**: the install is verified end to end (task 07), but this fleet's proxies run in + podman containers, so the host has no `nginx`/`haproxy` binary for the checker — + `data/logs/NOTES-33-switching.md` records it, and a real `DANGER` is still the + operator's call. - **NUT 2.8.5 on driveripper: configs in `/etc/ups/` (not `/etc/nut/`), upsd on port 3051 (not 3493 — nothing listens on 3493).** NUT 2.8 has no `ACL` directive; per-source restriction is a **firewall** rule, never an `upsd.conf` line. Two UPSes: @@ -367,7 +433,12 @@ operator's how-to for both. configured routes branching off; its only colour pair is existence (`--gate-on`/`--gate-off`) — no verdict colour anywhere on the page; its filter box is an **eye, not a query** (it hides rows whose own payload text does not contain what was - typed; no DNS, no re-ranking). + typed; no DNS, no re-ranking). Each row's switch runs on `GATEWAY_ALLOW_WRITES`: off + is a `aria-disabled` reading with the flag's own sentence (no hover, no listener, no + wire); on is a real control whose press is the dry-run diff → typed `DANGER` → one + send, and the row's colour is the payload's word from the next poll — never the + click's — with `off` reading `--gate-off` end to end (row, spine, the dialog's diff + tints) and the diff scrolling in its own `
`.
 - Every chart names itself for assistive tech (`role="img"` + `aria-label`); every chart
   on every page has one, pinned in `tests/test_api.py`.
 
@@ -376,9 +447,9 @@ operator's how-to for both.
 1. The UPS gates are the control: the upsd account has `instcmds = all`, and NUT has no
    call for a user's privileges — `enabled: true` cannot mean "may send", so the risk
    tiers and typed `DANGER` are load-bearing, not decoration.
-2. One admin, one factor: no TOTP/WebAuthn, no per-user accounts; a logout ends only
+2. One admin, one factor — no second factor and no per-user accounts; a logout ends only
    this browser's cookie, and the only revocation is a password change (which rotates the
-   session secret). The login throttle is per-process and keys on the socket peer address
+   session secret). The throttle is per-process and keys on the socket peer address
    (behind nginx every login shares one bucket; a restart clears it).
 3. GPU coverage is AMD only — an NVIDIA host would report no GPU at all.
 4. UPS history depth: events and numeric series are stored; long-window capacity fade has
@@ -392,8 +463,12 @@ operator's how-to for both.
 7. The fleet trusts **this** deployment's CA — a second deployment with its own
    `certs.*` rows cannot dial these agents until it carries the same root or re-issues
    and re-installs every bundle.
-8. The `/gateway` route switches are built but **not wired** — a route write path is
-   phase 33's decision and amends invariant 1; until then the knobs stay `aria-disabled`.
+8. Route switching is closed, and its honest limits are the gap list's:
+   `GATEWAY_ALLOW_WRITES` is off by default (a fresh deployment sees no switch arm at
+   all); the switch is one admin (gap 2) — whoever holds the session can switch a route
+   — with no per-host kill switch and no scheduler; and there is no runtime `disable
+   server` state: the marker is a config line, so a hand reload or a reboot does not
+   undo it (a runtime switch would be quietly undone by the next reload).
 
 ## Locked decisions (and why)
 
diff --git a/README.md b/README.md
index abc3b72..5f5dc54 100644
--- a/README.md
+++ b/README.md
@@ -10,7 +10,7 @@ file:
 | `/disks`     | Are any of my **physical** disks failing, and which one — health, temperature, power-on hours, wear, sector alarms | the same agents, running `smartctl -i -A -H` per whole drive (never a partition or a mount); a host whose agent has no `smartctl` is not a host on this page |
 | `/llm`       | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics` — read-only |
 | `/ups`       | Battery health, runtime left, power events, and battery-test buttons             | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
-| `/gateway`   | Which domains are coming in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`): a trunk per bound port with its configured routes branching off it, each line reading the config's own **inbound** word (`server_name`, `map` key, `backend` name) on line 1 over the counterpart it routes to on line 2 (`→ host:port`), plus its live connection count — and one filter box above the map that hides the routes whose own labels do not contain what you typed, asking nothing of the server | the agent on **one** host, `GATEWAY_HOST` (default `gateway`), reading its `haproxy.cfg` + nginx tree and `/proc/net/tcp{,6}` — so it needs **agent 1.2.0**, plus **1.3.0**'s `resolver` section to put a number on a route whose target the config writes as a *name* (without it those lines are drawn and dashed), and the whole page is *unconfigured* (never red) unless `GATEWAY_HOST` names a `HOST_TARGETS` entry |
+| `/gateway`   | Which domains are coming in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`): a trunk per bound port with its configured routes branching off it, each line reading the config's own **inbound** word (`server_name`, `map` key, `backend` name) on line 1 over the counterpart it routes to on line 2 (`→ host:port`), plus its live connection count — and one filter box above the map that hides the routes whose own labels do not contain what you typed, asking nothing of the server | the agent on **one** host, `GATEWAY_HOST` (default `gateway`), reading its `haproxy.cfg` + nginx tree and `/proc/net/tcp{,6}` — so it needs **agent 1.2.0**, plus **1.3.0**'s `resolver` section to put a number on a route whose target the config writes as a *name* (without it those lines are drawn and dashed), plus **1.4.0**'s `#off` marker read to show a route the switch has turned off **or back on** (without it an off route is drawn on, because the marker is a comment a 1.3.0 agent does not read) — an off route stays drawn and counted, its row simply reads `enabled: false` (its id gains a `#off` suffix), and the config-integrity warnings the marker *caused* (a backend emptied of its last `server`, a map entry pointed at the switch's sink) are read as a chosen state while the same shape from a hand-deleted line is still a warning — and the whole page is *unconfigured* (never red) unless `GATEWAY_HOST` names a `HOST_TARGETS` entry |
 
 No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
 runtime; every asset is served by this app.
@@ -111,8 +111,12 @@ sudo deploy/install-agent.sh --fetch-bundle https://dashboards.example.com \
 sudo deploy/install-agent.sh --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN
                                             # ...or a kernel-gated host (SMART passthrough, below)
 sudo deploy/install-agent.sh --force        # ...only when it must overwrite a unit it would change
+sudo deploy/install-agent.sh --with-routectl # ...plus the route-switch helper (the gateway, below)
+sudo deploy/install-agent.sh --allow-writes --with-routectl  # the gateway's full write install:
+                                            # the flag on the agent unit + the helper beside it
 deploy/install-agent.sh --status                            # unit state and /health (https when TLS)
 sudo deploy/install-agent.sh --uninstall
+sudo deploy/install-agent.sh --without-routectl   # remove helper + units, keep config and state
 ```
 
 Re-running the installer is the silent, idempotent path it has always been **as long as the unit
@@ -127,9 +131,30 @@ No root shell on the monitored host? `deploy/agent-sudoers` is the single sudoer
 lets the fleet's unprivileged `ops` account run the installer; run it without root and it
 prints that exact line with the script's own resolved path instead of saying "run as root".
 
-It is read-only by construction: there is no code path that writes, and the HTTP server
-only answers `GET`. Counters are reported as counters — the dashboard computes rates over
-a labelled window, so the agent never has to remember who polled it last.
+It is read-only by construction **except the one write below**: every `GET` path answers
+a read, and the HTTP server's one other path is the gated `POST /route`. Counters are
+reported as counters — the dashboard computes rates over a labelled window, so the agent
+never has to remember who polled it last.
+
+**The one write (1.4.0, the gateway).** 1.4.0 adds a single `POST /route`, reachable only
+with a verified client certificate and the agent's own `--allow-writes` flag, and it dials
+a **socket-activated root service** rather than giving the agent any privilege. The full
+write install is `--allow-writes --with-routectl`: the first adds exactly one argument to
+the agent unit's `ExecStart` (a re-run diff that shows a security directive moving is a
+stop, not a `--force`), and the second installs `deploy/dashboards-routectl.py` to
+`/usr/local/sbin/` (0755, root) and the two units **`dashboards-routectl.socket`** (a
+`0600` stream socket at `/run/dashboards-routectl.sock`, owned by the agent's user) and
+**`dashboards-routectl@.service`** (the helper, root, one instance per connection, `StandardInput=socket`,
+`ReadWritePaths` limited to `/etc/haproxy`, `/etc/nginx/conf.d` and `/etc/nginx/stream.d`), then
+verifies that the agent's own uid can connect before reporting success. **No sudoers line is
+involved**: the grant is a file mode systemd enforces, the helper's `SO_PEERCRED` check is
+the second look, and with `--with-routectl` alone the rendered `dashboards-agent.service` is
+byte-identical. The helper
+edits exactly one route (a `#off ` marker), validates the file with the proxy's own
+checker before it means anything, restores the previous bytes on a checker failure, and
+**reloads, never restarts**; `--without-routectl` removes it again, printing the list of
+still-off routes first. `deploy/gateway-routectl-README.md` is the operator's page for all
+of it.
 
 Per-disk `smart` health is a best-effort add-on on the whole-disk `/sys/block` list (no
 partitions): the agent runs `smartctl -i -A -H` per physical disk — 5 s bound per device,
@@ -271,8 +296,9 @@ a documentation one (RFC 5737 / RFC 3849), because a global IPv6 on Linux is EUI
 visitor's address is not ours to publish. Every port, state, column and width in those tables is
 verbatim, and those files are volatile by nature — a proxy in service has a different connection set
 every second — so the tests assert shape and the ports the configs claim, never a particular row.
-`AGENT_VERSION` is **1.3.0**, the version that adds the `resolver` section on top of the three
-phase 28 shipped; `GET /schema` names all four, each with its `available` flag, and `--no-proxy`
+`AGENT_VERSION` is **1.4.0** — the `resolver` section (1.3.0) and the `#off` marker read
+plus the one write (below) on top of the three phase 28 shipped; `GET /schema` names all
+four, each with its `available` flag, and `--no-proxy`
 and `--no-resolve` are listed there too. A host still running 1.2.0 answers with a snapshot that
 simply has no `resolver` key — which is why the version spread costs nothing: a section an agent
 did not collect is absence, and the gateway page counts the address-written routes it has and draws
@@ -714,10 +740,12 @@ curl -s -H "Authorization: Bearer $SESSION" -X POST -H 'content-type: applicatio
      -d '{"ups":"apc","command":"test.panel.start"}' http://127.0.0.1:8090/api/ups/command
 ```
 
-That is the app's one write endpoint **that dials anything** (the `/api/auth/*` and `/api/certs/*`
-writes change only this process and its SQLite `meta` table — invariant 1 is about the fleet), and
-the bearer header changes nothing about its gates: a command that can cut output still needs
-`"confirm":"DANGER"`, and is still refused while the UPS is on battery.
+That is the UPS half of the app's two write endpoints **that dial anything** (the
+`/api/auth/*` and `/api/certs/*` writes change only this process and its SQLite `meta` table —
+invariant 1 is about the fleet; the other half is the `/gateway` route switch,
+`GATEWAY_ALLOW_WRITES`-gated), and the bearer header changes nothing about its gates:
+a command that can cut output still needs `"confirm":"DANGER"`, and is still refused
+while the UPS is on battery.
 
 ### Lockout
 
@@ -754,7 +782,9 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
 | `GUFO_API_KEY` / `VLLM_API_KEY`       | only if an engine is behind a bearer token; they never leave the server |
 | `NUT_HOST` / `NUT_PORT`               | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT |
 | `NUT_USER` / `NUT_PASSWORD`           | only needed for instant commands (battery tests)                |
-| `NUT_ALLOW_ACTIONS`                   | `1` enables the one write endpoint that dials **anything** — the `/api/auth/*` and `/api/certs/*` writes touch only this process and its `meta` table |
+| `NUT_ALLOW_ACTIONS`                   | `1` enables the UPS write endpoint that dials **anything** — the `/api/auth/*` and `/api/certs/*` writes touch only this process and its `meta` table |
+| `GATEWAY_ALLOW_WRITES`                | `1` enables the second write endpoint, the `/gateway` route switch (`POST /api/gateway/route`); off by default, the same truthy rule as `NUT_ALLOW_ACTIONS`, and the two writes are two decisions — one flag does not carry the other |
+| `GATEWAY_WRITE_TIMEOUT_SECONDS`       | the route write's own budget (default 35 s), deliberately longer than the agent's 30 s peer timeout so the dashboard never times out first and leaves you guessing whether the reload happened; the read's 10 s stays the read's own |
 | `NUT_SIMULATE`                        | `1` replaces upsd with a **labelled** simulator (development only) |
 | `DASH_READ_ONLY`                      | `1` disables every mutating endpoint regardless of other flags  |
 | `DASH_DB`                             | SQLite path (default `data/dashboards.sqlite3`)                 |
@@ -1218,17 +1248,23 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
   behind a collapsed **Charts** toggle like `/system`'s, one `/api/gateway/history` request per
   panel per refresh. With no `GATEWAY_HOST` the page is the setup sentence rather than a red panel,
   and an ingress running an agent older than **1.2.0** says so in the agent's own words with every
-  count `n/a` and nothing coloured. Each line's head carries the sketched **switch** — a real
-  `role="switch"` button, green and to the position because the route is in the config, named in full
-  for a screen reader (`endpoint, reached by , in the nginx config, on`) — and it is
-  **unwired on purpose**: `aria-disabled` with a tooltip that says so, and this app writes to no
-  upstream, so a route is disabled by editing `haproxy.cfg` or an nginx `server` block and by nothing
-  in a browser. Giving either of those two dashes or knobs a real backing store is deliberately **not**
-  documented here as a thing you can have: name-based liveness needs the agent to report resolved
-  targets (a future agent version), and switching a route needs a second gated write path in front of
-  the proxy configs, which is a change to this app's read-only promise rather than a feature. What runs
-  today is **agent 1.3.0** — name resolution on the gateway, counts on every declared route, and no
-  switching.
+  count `n/a` and nothing coloured. Each line's head carries the **switch** — a real
+  `role="switch"` button, in the **on** position and tinted `--gate-on` because the route is in the config, named in full
+  for a screen reader (`endpoint, reached by , in the nginx config, on`) — and it has two
+  arms on `GATEWAY_ALLOW_WRITES`. With the flag off — the default — it is **unwired on
+  purpose**: `aria-disabled` with a tooltip that says so, and a route changes only by
+  editing `haproxy.cfg` or an nginx `server` block, never in a browser. With
+  `GATEWAY_ALLOW_WRITES=1` the same control is live:
+  activation opens one confirm dialog that first dials the helper as a dry run (`dry_run: true` —
+  it validates and answers with the one-line diff without writing a byte), then asks the operator
+  to type `DANGER`, then sends exactly once. The helper edits the route in its config file,
+  validates the file with the proxy's own checker and reloads that proxy — nothing is restarted —
+  and the switch flips the moment the helper answers `applied: true`, with the row re-read from
+  the next gateway poll. The write takes the read's own revocation gate on the same dial, and its
+  gates (read-only, the flag, a per-address throttle, the route's state in the current payload, the
+  read's freshness, the typed `DANGER` for both verbs) all run before anything reaches the host.
+  What runs today is **agent 1.4.0** — name resolution on the gateway, counts on every declared
+  route, the `#off` marker read, and the switch when the flag is set.
   The four event feeds (`/system`, `/disks`, `/llm`, `/ups`) are a **paged window** over the `event`
   table rather than a printout of it: the list is a bounded box that scrolls, 50 events to a
   page, with `1–50 of 200` between `‹ prev` and `next ›` (disabled at the ends) underneath.
@@ -1444,9 +1480,10 @@ advertising a window no poll ever measured.
 
 ## Battery-test buttons
 
-Instant commands are the only endpoint in the application that changes anything **outside this
-process** (`POST /api/ups/command`; the session and certificate-authority writes change only this
-app's own rows), and they are gated four times: `NUT_ALLOW_ACTIONS=1`, not `DASH_READ_ONLY`, an `upsd.users`
+Instant commands are one of exactly two endpoints in the application that change anything
+**outside this process** (the other is the route switch, in the next section; the session
+and certificate-authority writes change only this app's own rows), and they are gated
+four times: `NUT_ALLOW_ACTIONS=1`, not `DASH_READ_ONLY`, an `upsd.users`
 account that actually has the command, and a confirmation in the UI — destructive ones
 (deep test, calibration, `load.off`) require typing `DANGER`. Two more sit in front of those
 (since the login landed): a session, and this session's CSRF token when the write arrived on the
@@ -1493,6 +1530,58 @@ and an unauthenticated `LIST VAR` from any routed source is the normal state of
 `NUT_SIMULATE=1` stands in for upsd during
 development; everything it produces is badged `SIMULATED`, and it is never the default.
 
+## Route switching
+
+`/gateway` can change its own subject: each row's switch turns a route **off** and, the
+other way around, **back on**. Off is the helper's `#off` marker on the config line
+(a switched-off `map` entry is pointed at the helper's own sink `server` instead); on is
+the marker removed and the line's original text restored — but only if the line is
+unedited since (an edited line is a `drift` refusal and no byte moves). The switch is
+**a decision of its own**, off by default, and it is the application's second write path
+(the first is the UPS command buttons above).
+
+### Enabling it
+
+The write is the flag `GATEWAY_ALLOW_WRITES` (the same truthy rule as
+`NUT_ALLOW_ACTIONS`, and one flag does not carry the other) and it only exists on a
+host the operator installed `--allow-writes --with-routectl` on — the certificate-first
+gate on the agent and the socket-activated root helper beside it (*The host agent*
+section above). The install is the agent unit's one extra argument plus the helper and
+its two units, `dashboards-routectl.socket` and `dashboards-routectl@.service`; to undo
+everything, `sudo ./deploy/install-agent.sh --without-routectl` removes the helper and
+its units, **leaves the proxy configs and the state directory alone**, and prints the
+still-off routes first — those are yours to switch back on. With the flag off, every
+switch is a `disabled` reading with the flag's own sentence, and a click does nothing:
+no wire is touched, no request is sent.
+
+### Switching a route
+
+The press runs the flow the operator can see: the **dry-run diff first** — the
+helper's own word for what the change will do, before the operator is asked for the
+word — then the confirmation that carries the diff and a **typed `DANGER`** (both verbs
+ask for it: turning a route back on is a write to the fleet's only ingress, just as
+turning it off is), then **exactly one send** — one connection, one request, one
+answer, **no retry** (a retry of a write is a second reload). The answer's words are
+printed verbatim: `applied: true` flips the switch and the row's colour follows the
+**payload's** word on the next gateway poll — never the click's; `applied: false`
+flips nothing. Every call stores one audit event — `route-disabled`, `route-enabled`, or `route-denied`
+for the gates' and the helper's refusals (a dry run stores none: it changed nothing) —
+so the audit trail is one line per call with who and when — and a switch **outages the
+route on purpose: it will appear in the event feed**.
+
+The honest limits: the switch is one admin (the sign-in's one factor, no more), with no
+per-host kill switch and no scheduler; and there is no runtime `disable server` state —
+the marker is a config line, so a hand reload or a reboot does not undo it (a runtime
+switch would be quietly undone by the next reload).
+
+When the dashboard is the thing that is broken at 03:00, the recovery is on the box:
+`/usr/local/sbin/dashboards-routectl.py` (its operator page is
+`deploy/gateway-routectl-README.md`). `sudo /usr/local/sbin/dashboards-routectl.py list`
+says what is off (the state sidecar), `sudo … verify` checks the sidecar against the
+files (drift is reported, never repaired), and `sudo … enable --id ` puts one route
+back on through the same protocol — the write, the proxy's own checker, restore on a
+checker failure, the reload (never a restart), the state record.
+
 ## Deployment
 
 `Containerfile` is a two-stage build: `uv.lock` is resolved in the builder (exactly the
diff --git a/agent/host_agent.py b/agent/host_agent.py
index 3aadfff..27d098e 100644
--- a/agent/host_agent.py
+++ b/agent/host_agent.py
@@ -2,13 +2,19 @@
 """host_agent.py — read-only host metrics agent for the homelab dashboards.
 
 **Stdlib only, one file, Python 3.10+.** Copy it to a host, run it, and the dashboard can
-read that host. It never writes anything anywhere: every number comes from reading
-/proc, /sys and statvfs, and the HTTP server only answers GET.
+read that host. Every number comes from reading /proc, /sys and statvfs, and the HTTP
+server answers GET — with exactly one exception: 1.4.0's `POST /route`, the fleet's one
+narrow write (switching a route's `#off` marker). Its gates are the client certificate and
+`--allow-writes`, and it never touches a config file itself: it dials the socket-activated
+root helper (`--routectl-socket`) once and passes the helper's JSON through verbatim.
 
     GET /health      -> {"ok": true, "hostname": ..., "version": ...}
     GET /info        -> slow-changing identity (os, kernel, cpu model, cores, gpu list)
     GET /snapshot    -> everything, one JSON document (see SCHEMA below)
     GET /schema      -> this docstring's SCHEMA section, for self-documenting deploys
+    POST /route      -> the one write: forward exactly {id, action, dry_run} to the
+                       routectl helper and answer its JSON verbatim (1.4.0; certificate
+                       plus --allow-writes, in that order — a token-mode POST is refused)
 
 Design rules
 ------------
@@ -89,6 +95,20 @@ Design rules
   are simply absent — `null` a pass that could not run), and `--no-resolve` removes the key in
   silence the way `--no-smart` does; `--no-proxy` removes it too, because a box that collects no
   proxy has no declared names to look up.
+* **The one write is a forward, not a file edit.** `POST /route` (1.4.0) is the only non-GET
+  path and the only thing this process ever sends to an upstream. The order is the design:
+  the client certificate (the handshake is the identity — a bearer token never gets this
+  path), then `--allow-writes` (the operator's decision, off by default), then the body
+  (exactly `{id, action, dry_run}`, ≤ 4096 bytes, an `action` of `disable`/`enable`, an
+  `id` of at most 256 characters — a refused body never reaches the peer), then the
+  helper's socket. The dial is `AF_UNIX` with a 30 s budget (the helper may run `nginx -t`
+  and a reload), one newline-delimited request, one line back, and **never a retry — a
+  retry of a write is a second reload**: every failure is a 502 (an unreadable or
+  oversized answer, a timeout, or the errno word) and the helper's JSON is passed through
+  verbatim, refusal included. Without `--routectl-socket` (the flag's value set empty) the
+  answer is `503 no route helper on this host`. The bytes are written by the socket-
+  activated root helper (`deploy/dashboards-routectl.py`); this process opens no config
+  file for writing, so `--allow-writes` on a box without the helper is a 503, not a write.
 * **The state** is a small pid->cpu-time cache so `processes.top_cpu` can answer a
   question /proc cannot (CPU needs two samples), bounded and cleared when stale, plus
   the short-TTL SMART result cache above.
@@ -139,18 +159,27 @@ SCHEMA (GET /snapshot)
                       top_cpu: [{pid, name, cpu_pct}]}
     haproxy          {available: true|false|null, config: str, frontends: [{name, binds:
                       [{addr, port}], default_backend: str|null, backends: [str]}],
-                      backends: [{name, servers: [{name, target, port}]}]}
+                      backends: [{name, servers: [{name, target, port,
+                                                   disabled: bool}]}]}
                       # parsed from --haproxy-config; additive (phase 28) and absent entirely
                       # with --no-proxy. `addr` is the bind's address as written (`:::22` ->
                       # `::`), `port` null when a target names none (never 0), and the
                       # frontend->backend routing is the names themselves: the dashboard
                       # joins them, the agent reports only what the config says.
+                      # `disabled` is the route switch's read side (phase 33, agent 1.4.0):
+                      # true when the line carries the helper's `#off ` marker
+                      # (ROUTE_MARKER), false otherwise. The marker is a comment in both
+                      # configs, so a marked line is still a live line to haproxy/nginx until
+                      # the helper rewrites the value — the agent reports the state, the
+                      # dashboard draws it, and the kernel does not know markers.
     nginx            {available: true|false|null, config: str, files: [str],
                       servers: [{listen: [{addr, port}], server_names: [str], proxy_passes:
                                  [{location: str|null, scheme: str|null,
-                                   target: {host, port}|null, variable: str|null}]}],
-                      upstreams: [{name, servers: [{host, port}]}],
-                      maps: [{variable, entries: [{key, target: {host, port}|null}]}]}
+                                   target: {host, port}|null, variable: str|null,
+                                   disabled: bool}]}],
+                      upstreams: [{name, servers: [{host, port, disabled: bool}]}],
+                      maps: [{variable, entries: [{key, target: {host, port}|null,
+                                                  disabled: bool}]}]}
                       # parsed from --nginx-config plus the files its own `include` lines name
                       # (one level, at most 8 files / 1 MB of included text; `files` is the
                       # main config then every include actually merged); additive (phase 28)
@@ -158,6 +187,10 @@ SCHEMA (GET /snapshot)
                       # written (`[::]:8443` -> `"::"`, a bare `80` -> a null addr, never `*`),
                       # `server_names` keeps the catch-all `_` verbatim, `location` is null for
                       # a proxy_pass written in the server block itself (the stream form), and
+                      # `disabled` (phase 33, agent 1.4.0) is the line's `#off ` marker
+                      # read by the tokenizer, exactly as the haproxy section's: on a
+                      # `proxy_pass`, an `upstream` member line, or a `map` entry line. See
+                      # the haproxy note for what the flag does and does not mean.
                       # a `proxy_pass $var` is served with `variable` (the `$` dropped) and
                       # `target: null`: the map whose variable matches decides per connection,
                       # so both facts travel unmerged and the dashboard joins them.
@@ -213,7 +246,12 @@ other protection is `--cert-bundle`/AGENT_CERT_BUNDLE: one installed PEM (server
 key, the dashboard's root CA, `# BUNDLE-REV ` and `# REVOKED ` lines) turns the
 server into HTTPS that answers every path but `/health` only to a client certificate signed
 by that root, and refuses one on the denylist on every path — the certificate is the auth, so
-no token is read on that path.
+no token is read on that path. The write path (1.4.0, the gateway) is `--allow-writes`
+(default off) plus `--routectl-socket` (default /run/dashboards-routectl.sock, the helper's
+socket; set the flag's value empty for the `503 no route helper on this host` arm): a
+`POST /route` is then answered certificate-first, then flag, then body, then one dial to
+that socket, and the helper's JSON is passed through verbatim — see the design rule above
+for the refusal sentences.
 """
 
 from __future__ import annotations
@@ -237,8 +275,65 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 from pathlib import Path
 from typing import Any, cast
 
-AGENT_VERSION = "1.3.0"
+AGENT_VERSION = "1.4.0"
 SCHEMA_VERSION = 1
+#: POST /route: one newline-delimited JSON request to the routectl helper, one line back.
+#: The budget covers the helper's own checker run (`nginx -t` / `haproxy -c -f`) plus reload.
+ROUTE_TIMEOUT_SECONDS = 30
+#: The largest body the write path accepts, checked before a byte is read off the wire.
+_ROUTE_MAX_BODY = 4096
+#: An answer longer than this is "unreadable" — the helper's status JSON never is.
+_ROUTE_MAX_ANSWER = 131072
+# The route switch's marker, one constant the parser and the write path both read (phase 33).
+# A comment in both haproxy and nginx, so `line, marker = split_marker(line)` on a route line
+# is the whole of the read side: a marked line is still a live line to the proxy until the
+# helper rewrites its value, and this is the state, not a behaviour the agent enforces.
+ROUTE_MARKER = "#off"
+
+
+def split_marker(line: str) -> tuple[str, str | None]:
+    """Split a config line into ``(line, marker)`` for the helper's ``#off `` marker.
+
+    The marker is the **whole** trailing comment (``#...`` through end of line), so the
+    grammar is one and the write path (task 01's ``routectl``) and the parser share it: a
+    line is marked when its comment *is* ``#off `` — a marker with nothing after the
+    word, and nothing but the word, after it. Anything else in the comment is the operator's
+    own note, not a switch, and the line is reported ``disabled: false``. A marked line stays
+    in the document (the helper edits the value, never the line), which is what lets the
+    dashboard tell "declared, off" from "never declared" (that split is the dashboard's
+    ``enabled`` field, phase 33 task 04).
+    """
+    for i, ch in enumerate(line):
+        if ch == "#":
+            marker = line[i:]
+            if marker == ROUTE_MARKER or marker.startswith(ROUTE_MARKER + " "):
+                name = marker[len(ROUTE_MARKER):].strip()
+                if name and " " not in name:
+                    return line[:i].rstrip(), name
+            return line, None
+    return line, None
+
+
+def marker_for(name: str) -> str:
+    """The full marker comment for one route name (the write path's spelling, pinned here so
+    the parser and task 01's helper cannot drift). One name per marker: a marker is a whole
+    comment and the name has no spaces, so the split above is exact."""
+    return f"{ROUTE_MARKER} {name}"
+
+
+def config_line_index(text: str, name: str) -> int:
+    """The 1-based line number of the route line carrying ``name``'s marker, or ``-1``.
+
+    Task 01's helper needs the line to edit; the marker names the route, so the search is
+    the marker itself, never the route's text (a target string appears in many lines and
+    only one of them is the route). `-1` is "not found", which the helper maps to its
+    `not-found` refusal.
+    """
+    marker = marker_for(name)
+    for i, line in enumerate(text.splitlines()):
+        if split_marker(line)[1] == name and marker in line:
+            return i + 1
+    return -1
 PAGE_SIZE = 0  # resolved from sysconf at runtime
 
 # Filesystems that are not storage. Mirrors the spirit of node_exporter's default
@@ -2037,6 +2132,11 @@ def _scan_haproxy_config(text: str) -> tuple[list[dict[str, Any]], list[dict[str
                 if backend is not None and backend not in current_fe["backends"]:
                     current_fe["backends"].append(backend)
         if current_be is not None and directive == "server" and len(words) > 2:
+            # the route switch's read side (phase 33, agent 1.4.0): the marker is a trailing
+            # comment, `#off `, and a marked line is still a live line to haproxy until
+            # the helper rewrites the value — the agent reports the state, the dashboard
+            # draws it (`enabled: false`), and the kernel does not know markers.
+            marked = split_marker(stripped)[1]
             target = _haproxy_addr_port(words[2])
             current_be["servers"].append({
                 "name": words[1],
@@ -2044,6 +2144,7 @@ def _scan_haproxy_config(text: str) -> tuple[list[dict[str, Any]], list[dict[str
                 # port — "absent is never 0", on the one field haproxy makes optional
                 "target": words[2] if target is None else target[0],
                 "port": None if target is None else target[1],
+                "disabled": marked is not None,
             })
     return frontends, backends, recognised
 
@@ -2052,9 +2153,12 @@ def parse_haproxy_config(text: str) -> dict[str, Any]:
     """Parse a haproxy config's **text** into the `haproxy` section's data. Pure, never raises.
 
     `{frontends: [{name, binds: [{addr, port}], default_backend, backends}], backends:
-    [{name, servers: [{name, target, port}]}], recognised: bool}` — the frontend→backend routing
-    is the names themselves (`default_backend` plus every `use_backend`), so the dashboard joins
-    them and this file interprets no rule.
+    [{name, servers: [{name, target, port, disabled}]}], recognised: bool}` — the frontend→
+    backend routing is the names themselves (`default_backend` plus every `use_backend`), so the
+    dashboard joins them and this file interprets no rule. `disabled` is the line's `#off
+    ` marker (phase 33, agent 1.4.0; `ROUTE_MARKER`, `split_marker`), `true` on a marked
+    line, `false` on every other line — the write path (the helper) is the only writer, and a
+    marker with anything but one name after it is the operator's comment, not a switch.
 
     `recognised` answers the separate question "is this text a haproxy config at all" (does its
     top level name a section keyword?) and is deliberately **not** part of the served section:
@@ -2070,7 +2174,7 @@ def parse_haproxy_config(text: str) -> dict[str, Any]:
 
 
 def _nginx_tokens(text: str) -> list[tuple[str, str]]:
-    """Split nginx config text into `("w", word)`, `("{", "")`, `("}", "")`, `(";", "")`.
+    """Split nginx config text into `("w", word)`, `("{", "")`, `("}", "")`, `(";", comment)`.
 
     nginx's grammar is words closed by `;` or opened by `{`, with `#` comments and quoted
     strings that may contain any of those three — the captured gateway config writes
@@ -2083,16 +2187,28 @@ def _nginx_tokens(text: str) -> list[tuple[str, str]]:
     A `#` opens a comment only where a word would have started, which is where nginx's own
     lexer reads one: `proxy_pass http://host/#/app` keeps its fragment, and the
     `# include /etc/nginx/old.conf;` line an operator commented out contributes nothing.
+
+    The one value a token carries is a `;` (or `}`) token's **end-of-line comment**: a comment
+    on the same line, after the statement that closed it, is attached to that token, and every
+    other comment is dropped as before. That is the route switch's read side (phase 33):
+    `server s1 10.1.1.9:443 #off s1` is a live line with a state, and the statement scanner is
+    what hands the comment to the directive. A comment at the start of a line (a commented-out
+    directive) belongs to no statement and is dropped, exactly as today.
     """
     tokens: list[tuple[str, str]] = []
     word: list[str] = []
     started = False        # an empty quoted string is still a word: the `''` key of a map
+    line_terminated = False   # a `;` or `}` has been emitted on this line
     i, n = 0, len(text)
     while i < n:
         char = text[i]
         if char == "#" and not started and (i == 0 or text[i - 1] in " \t\r\n"):
-            while i < n and text[i] != "\n":
-                i += 1
+            end = i
+            while end < n and text[end] != "\n":
+                end += 1
+            if line_terminated and tokens and tokens[-1][0] in ";}":
+                tokens[-1] = (tokens[-1][0], text[i:end])
+            i = end
             continue
         if char in "\"'":
             quote = char
@@ -2116,12 +2232,15 @@ def _nginx_tokens(text: str) -> list[tuple[str, str]]:
                 tokens.append(("w", "".join(word)))
                 word, started = [], False
             tokens.append((char, ""))
+            line_terminated = True
             i += 1
             continue
         if char.isspace():
             if started:
                 tokens.append(("w", "".join(word)))
                 word, started = [], False
+            if char == "\n":
+                line_terminated = False
             i += 1
             continue
         word.append(char)
@@ -2132,15 +2251,28 @@ def _nginx_tokens(text: str) -> list[tuple[str, str]]:
     return tokens
 
 
+def _comment_off(comment: str) -> bool:
+    """Does this end-of-line comment carry the route switch's marker?
+
+    The grammar is `split_marker`'s — the one the write path (task 01's helper) shares —
+    applied to a line whose only comment is the one the tokenizer attached: `#off `
+    with exactly one name after the word. The delegation is the point: the parser and the
+    helper cannot read the same marker differently.
+    """
+    return split_marker(f"line {comment}")[1] is not None
+
+
 def _nginx_statements(text: str) -> list[tuple[Any, ...]]:
     """The brace-block scanner: tokens in, nested statements out. Raises on an unbalanced file.
 
-    A statement is `("d", name, args)` for a directive or `("b", name, args, children)` for a
-    block. That is the whole difference between the two config languages this agent reads:
-    haproxy's structure is its indentation, nginx's is its braces, so nginx gets a scanner
-    instead of a line loop — and with it the two shapes that make the language what it is, a
-    `server { … location { … } … }` nested as deep as it goes and an `upstream` body whose
-    `server` lines are *targets* while an `http` body's `server {` is a *block*.
+    A statement is `("d", name, args, comment)` for a directive — `comment` is the end-of-line
+    comment the tokenizer attached (the route switch's marker lives there, phase 33) or `""` —
+    and `("b", name, args, children)` for a block. That is the whole difference between the two
+    config languages this agent reads: haproxy's structure is its indentation, nginx's is its
+    braces, so nginx gets a scanner instead of a line loop — and with it the two shapes that
+    make the language what it is, a `server { … location { … } … }` nested as deep as it goes
+    and an `upstream` body whose `server` lines are *targets* while an `http` body's `server {`
+    is a *block*.
 
     `ValueError` here means "nginx itself would refuse to start on this file": a stray brace or
     a directive that runs into the end of the file without its `;`. The caller turns that into
@@ -2156,7 +2288,7 @@ def _nginx_statements(text: str) -> list[tuple[Any, ...]]:
             continue
         if kind == ";":
             if words:                       # a bare `;` is whitespace to nginx
-                containers[-1].append(("d", words[0], words[1:]))
+                containers[-1].append(("d", words[0], words[1:], value))
                 words = []
             continue
         if kind == "{":
@@ -2271,6 +2403,10 @@ def _collect_nginx(statements: _NginxStatements, out: dict[str, Any], *, root: b
     """
     for statement in statements:
         kind, name, args = statement[0], statement[1], statement[2]
+        # the end-of-line comment, if the tokenizer attached one: the route switch's marker
+        # (phase 33) lives on the route line itself, and `_comment_off` is its read
+        comment = statement[3] if len(statement) > 3 and isinstance(statement[3], str) else ""
+        marked = _comment_off(comment)
         if kind == "d":
             if name == "include" and args:
                 if root:
@@ -2284,6 +2420,7 @@ def _collect_nginx(statements: _NginxStatements, out: dict[str, Any], *, root: b
                         # keeps its name and a null port rather than losing the line entirely
                         "host": args[0] if found is None else found[0],
                         "port": None if found is None else found[1],
+                        "disabled": marked,
                     })
                 continue
             if map_ctx is not None:
@@ -2297,6 +2434,7 @@ def _collect_nginx(statements: _NginxStatements, out: dict[str, Any], *, root: b
                     # `upgrade` / `close` values are header words, and calling one a host
                     # would be this section inventing a fact to fill a field
                     "target": None if found is None else {"host": found[0], "port": found[1]},
+                    "disabled": marked,
                 })
                 continue
             if server is not None:
@@ -2309,7 +2447,9 @@ def _collect_nginx(statements: _NginxStatements, out: dict[str, Any], *, root: b
                     # "any domain" is what the page says about it, not what the agent decides
                     server["server_names"].extend(args)
                 elif name == "proxy_pass" and args:
-                    server["proxy_passes"].append(_nginx_proxy_pass(args[0], location))
+                    entry = _nginx_proxy_pass(args[0], location)
+                    entry["disabled"] = marked
+                    server["proxy_passes"].append(entry)
             continue
         children = cast("_NginxStatements", statement[3])
         if name == "upstream" and args:
@@ -2367,9 +2507,14 @@ def parse_nginx_config(main_text: str,
     """Parse an nginx main config plus the documents its includes name. Pure, never raises.
 
     `{servers: [{listen: [{addr, port}], server_names: [str], proxy_passes: [{location, scheme,
-    target, variable}]}], upstreams: [{name, servers: [{host, port}]}],
-    maps: [{variable, entries: [{key, target}]}], files: [str], includes: [str], unparsed:
-    [str], recognised: bool, error: str|null, resolver: str|null}`.
+    target, variable, disabled}]}], upstreams: [{name, servers: [{host, port, disabled}]}],
+    maps: [{variable, entries: [{key, target, disabled}]}], files: [str], includes: [str],
+    unparsed: [str], recognised: bool, error: str|null, resolver: str|null}`.
+
+    `disabled` on a `proxy_pass`, an `upstream` member line, or a `map` entry line is the
+    route switch's read side (phase 33, agent 1.4.0): `true` when that line carries the
+    helper's `#off ` marker, `false` otherwise — one grammar (`split_marker`), the same
+    one the haproxy section uses, and the same one the write path (task 01's helper) shares.
 
     Four of those keys are the ones `Collector.nginx` does not serve, the way
     `parse_haproxy_config`'s `recognised` is not: `error` is why the main text could not be
@@ -2979,6 +3124,12 @@ STATUS_PAGE = """
 
 

/snapshot · /info · /health · /schema

+

POST /route — the one write (switch a route's #off marker; 1.4.0). + Refusals, in order: 405 no such write path · 403 writes require a certificate (token + mode) · 403 client certificate required · 403 client certificate revoked · 403 writes + are not enabled on this agent · 422 body refusals · 503 no route helper on this + host · 502 helper unreachable / timed out / unreadable answer. The bytes are written + by the socket-activated root helper, never by this process.

""" @@ -2987,6 +3138,8 @@ class Handler(BaseHTTPRequestHandler): collector: Collector token: str | None = None bundle: TlsBundle | None = None + allow_writes: bool = False + routectl_socket: str | None = None def do_GET(self) -> None: # the BaseHTTPRequestHandler API name — keep it path = self.path.split("?", 1)[0].rstrip("/") or "/" @@ -3032,6 +3185,102 @@ class Handler(BaseHTTPRequestHandler): else: self._send(404, {"ok": False, "detail": f"no such path: {path}"}) + def do_POST(self) -> None: # the BaseHTTPRequestHandler API name — keep it + # The one write (1.4.0). The order is the design, and each refusal is its own + # sentence: the certificate (the handshake is the identity — the token arm is + # refused outright, the denylist is not a read-side rule), then the operator's + # flag, then the body (a refused body never reaches the peer), then the helper's + # socket. Nothing in this method writes a file; the bytes are the helper's. + path = self.path.split("?", 1)[0].rstrip("/") or "/" + if path != "/route": + self._send(405, {"ok": False, "detail": "no such write path"}) + return + if self.bundle is None: + self._send(403, {"ok": False, "detail": "writes require a certificate"}) + return + if not self._certificate_allows(self.bundle, path): + return + if not self.allow_writes: + self._send(403, {"ok": False, "detail": "writes are not enabled on this agent"}) + return + try: + length = int(self.headers.get("content-length", "0")) + except ValueError: + length = -1 + if length < 0 or length > _ROUTE_MAX_BODY: + self._send(422, {"ok": False, + "detail": "body longer than the write path accepts"}) + return + raw = self.rfile.read(length) + try: + body = json.loads(raw.decode("utf-8")) + except (UnicodeDecodeError, ValueError): + self._send(422, {"ok": False, "detail": "body is not a JSON object"}) + return + if not isinstance(body, dict): + self._send(422, {"ok": False, "detail": "body is not a JSON object"}) + return + if not set(body) <= {"id", "action", "dry_run"}: + self._send(422, {"ok": False, "detail": "body carries a key the write path does not take"}) + return + action = body.get("action") + if action not in ("disable", "enable"): + self._send(422, {"ok": False, "detail": "action must be disable or enable"}) + return + rid = body.get("id") + if not isinstance(rid, str) or not rid or len(rid) > 256: + self._send(422, {"ok": False, + "detail": "id must be a string of at most 256 characters"}) + return + if self.routectl_socket is None: + self._send(503, {"ok": False, "detail": "no route helper on this host"}) + return + self._route_peer({"id": rid, "action": action, + "dry_run": bool(body.get("dry_run", False))}) + + def _route_peer(self, payload: dict[str, Any]) -> None: + """Dial the routectl helper once, forward `payload`, pass the one answer through. + + One connection, one request, one answer: a retry of a write is a second reload, + so every failure is a 502 and never a second dial. The helper's JSON travels + verbatim — a refusal it answers stays a refusal, and the agent invents no + success of its own. The log line prints the path and the status only: the + `id` is a hostname, and this process's log lines stay out of its body. + """ + buf = b"" + sock = None + try: + sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + sock.settimeout(ROUTE_TIMEOUT_SECONDS) + sock.connect(self.routectl_socket) + sock.sendall(json.dumps(payload).encode("utf-8") + b"\n") + while not buf.endswith(b"\n"): + chunk = sock.recv(65536) + if not chunk: + break + buf += chunk + if len(buf) > _ROUTE_MAX_ANSWER: + self._send(502, {"ok": False, + "detail": "route helper answered something unreadable"}) + return + except TimeoutError: + self._send(502, {"ok": False, "detail": "route helper timed out"}) + return + except OSError as exc: + word = socket.errorname.get(exc.errno, str(exc.errno)) + self._send(502, {"ok": False, "detail": f"route helper: {word}"}) + return + finally: + if sock is not None: + sock.close() + try: + answer = json.loads(buf.decode("utf-8").strip()) + except (UnicodeDecodeError, ValueError): + self._send(502, {"ok": False, + "detail": "route helper answered something unreadable"}) + return + self._send(200, answer) + def _certificate_allows(self, bundle: TlsBundle, path: str) -> bool: """Does this connection's client certificate authorise `path`? Sends the refusal if not. @@ -3094,17 +3343,21 @@ class Handler(BaseHTTPRequestHandler): def serve(host: str, port: int, collector: Collector, token: str | None, - bundle: TlsBundle | None = None) -> None: + bundle: TlsBundle | None = None, *, allow_writes: bool = False, + routectl_socket: str | None = None) -> None: """Serve forever on host:port — plain HTTP with an optional token, or HTTPS with a bundle. - All three class attributes are assigned on every call: `Handler` is process-wide state, so a - server started without a bundle must not inherit one from a server that had it (that would - answer a plain-HTTP socket with 403 on every path, or worse, accept a client certificate the - operator revoked). + All five class attributes are assigned on every call: `Handler` is process-wide state, so a + server started without a bundle (or without the write flag) must not inherit one from a + server that had it (that would answer a plain-HTTP socket with 403 on every path, or + worse, accept a client certificate the operator revoked — or answer a write to a helper + this process was never told about). """ Handler.collector = collector Handler.token = token Handler.bundle = bundle + Handler.allow_writes = allow_writes + Handler.routectl_socket = routectl_socket httpd = ThreadingHTTPServer((host, port), Handler) if bundle is not None: httpd.socket = bundle.context.wrap_socket(httpd.socket, server_side=True) @@ -3178,6 +3431,17 @@ def main(argv: list[str] | None = None) -> int: f"documented degradation (available=false, one error entry), " f"not a failure") parser.add_argument("--once", action="store_true", help="print one snapshot as JSON and exit") + parser.add_argument("--allow-writes", action="store_true", + help="answer POST /route on this agent (the fleet's one narrow write: " + "switch a route's #off marker). The gates are the client " + "certificate and this flag, in that order; the bytes are written " + "by the socket-activated root helper, never by this process") + parser.add_argument("--routectl-socket", + default=os.environ.get("AGENT_ROUTECTL_SOCKET", + "/run/dashboards-routectl.sock"), + help="the routectl helper's socket that POST /route dials once per " + "request; set the value empty for the " + "`503 no route helper on this host` arm") args = parser.parse_args(argv) collector = Collector(args.root, top_processes=args.top, net_all=args.net_all, @@ -3203,7 +3467,9 @@ def main(argv: list[str] | None = None) -> int: "metrics are sensitive); pass --token, --token-file, --cert-bundle or set " "AGENT_TOKEN / AGENT_CERT_BUNDLE", flush=True) return 2 - serve(args.host, args.port, collector, token, bundle) + serve(args.host, args.port, collector, token, bundle, + allow_writes=args.allow_writes, + routectl_socket=args.routectl_socket or None) return 0 diff --git a/deploy/dashboards-routectl.py b/deploy/dashboards-routectl.py new file mode 100644 index 0000000..7e961c8 --- /dev/null +++ b/deploy/dashboards-routectl.py @@ -0,0 +1,1076 @@ +#!/usr/bin/env python3 +# requires: python >=3.10 +"""dashboards-routectl — the route-switch root helper (phase 33, task 01). + +Turns one `connections[].id` off or back on in one of the gateway's two proxy configs, +validates the file with the proxy's own checker, reloads that one proxy, and refuses +everything else. This process is the phase's only writer: the dashboard's +`POST /api/gateway/route` reaches a config file only through it (via the agent's +`POST /route`, which reaches this helper only through the systemd-activated socket, +task 02). Run by hand on the gateway it is the operator's rehearsal: `diff` shows +what the dashboard would do, and `disable --id ID` without `--apply` is a dry run. + +It never `restart`s anything. A restart severs every live connection through the +fleet's only ingress, and a reload does not, so the only verb this file executes is +`systemctl reload `; a failed reload leaves the applied file in place and says +so in the answer (`reloaded: false` + the unit's status line) — the config is right, +the reload is the operator's next action. + +The one write protocol (the order is the safety property): + +1. re-read the file, index it with the agent's parser, map the `id` to the line + (an unknown id is a refusal that writes nothing); +2. already in the requested state → `{"applied": false, "changed": false}`, no reload; +3. `--apply` absent (or the wire's `dry_run: true`) → answer with the unified `diff` + only — no write, no checker, no reload, no state change; +4. write the new text to a `mkstemp` in the same directory, `fsync`, `os.replace`; +5. run the proxy's own checker (`nginx -t` / `haproxy -c -f `); on failure + `os.replace` the previous bytes back and answer + `{"applied": false, "validated": false, "restored": true}` with the checker's own + first line — and reload nothing; +6. `systemctl reload `; a failure is `reloaded: false` with the status line; +7. re-read and re-index the file to prove the marker state is live; record the state. + +Per-kind edit (decision 5; a fourth arm is a new decision, not a fall-through): + +* a haproxy `server` line is commented with the marker (`#off `): the backend + stops receiving, and the marker keeps the original restorable by eye and by + machine; +* a vhost `proxy_pass` line is commented the same way: the vhost keeps `listen` + + `server_name` and gains no target, a shape the dashboard's warnings already treat + as quiet, so "off" does not read as "broken"; +* a stream `map` entry is **rewritten to the sink address** and marked: commenting + it would fall through to the map's `default` and mis-route the domain instead of + blocking it. The sink file is created on first need (the marker's header, a stream + `server` on `127.0.0.1:44450`), is loaded by the main config's existing + `stream { include /etc/nginx/stream.d/*.conf; }` glob, and the **main config is + never edited**; it is removed only when this helper wrote it (its header present) + and no `map` entry still points at it; +* the `map` `default` entry is **refused** (`"refused": "map-default"`): it is the + fallback for every unmapped SNI, not a service. + +`enable` restores **only** when the current line is byte-identical to the line this +helper wrote (the state sidecar's `written` row, recorded at write time); a hand edit +in the meantime is `{"applied": false, "drift": true}` and no byte changes. The map +arm's original value is not in the file after a disable (it was rewritten), so the +sidecar also carries the `original` line. + +Nothing in this file takes a path, a line number, a pattern, or a command from the +wire. `--serve` accepts exactly the keys `{id, action, dry_run}` and every other key +is a refusal; the checker and unit names come from the flags the operator installed, +never from a request; every subprocess call is a fixed argv list (no `shell=True`), +and a `--file` option does not exist. The config parser is the agent's: +`--agent-module` is imported by file path (the helper is installed beside it at +/usr/local/lib/dashboards-agent/) and a helper that cannot import it refuses to run +at all — two parsers reading one config are two opinions about one file. + +`--serve` (the socket-activated mode, task 02's `StandardInput=socket`): the chosen +idiom is the connected socket on **stdin** — systemd's `StandardInput=socket` dups +the per-connection socket onto fd 0 (and `StandardOutput=socket` onto fd 1), so the +process wraps fd 0 (or fd 3, the classic socket-activation fd, when the platform +hands the connection there instead — the live box put it on fd 0), checks +`SO_PEERCRED` against `--allow-uid` (the **second** look: the socket file's `0600` ++ `SocketUser` is the first) — the uid is read at the field slot a same-process +probe calibrates on this kernel, because the 12-byte ucred buffer's field order is +not portable (the live box's el10 kernel delivers it as `{pid, uid, gid}`, the +classic layout reversed; in the root service the calibration shifts the probe's +own uid with `setuid` while the gid stays 0 — a privilege drop, restored before +any state file is touched — so a matching gid on a foreign-uid caller cannot +open the gate; an unshifted `euid == egid` probe requires both remaining fields +to match) — reads one newline-delimited JSON request, dispatches +one verb, writes one JSON line back to the socket, and exits. One request per +activation is the whole threading policy: the socket's `Accept=yes` spawns the +template instance per connection, so there is no accept loop, no thread, no timeout +to invent. +""" + +import argparse +import contextlib +import datetime +import difflib +import hashlib +import importlib.util +import json +import os +import pwd +import socket +import struct +import subprocess +import sys +import tempfile + +# The managed sink: the nginx `stream` block a disabled `map` entry is pointed at. A +# file this helper creates (the marker arms never delete it), a header that identifies +# it as machine-managed, and an address that is the loopback by construction. +SINK_HEADER = "# managed by dashboards-routectl — do not edit" +SINK_FILE_DEFAULT = "/etc/nginx/stream.d/00-dashboards-sink.conf" +SINK_ADDRESS_DEFAULT = "127.0.0.1:44450" + +# The `id` grammar (`source:in-port->host:out-port`, both ports required — the +# dashboard's own id always carries them) is bounded by construction so a wire +# body stays a parse-and-refuse matter, not a resource one. +ROUTE_ID_MAX = 256 +# `--serve` reads at most this many bytes; more is a malformed refusal. +SERVE_REQUEST_MAX = 65536 +# The checker and the reload are both bounded: a wedged `nginx -t` is not an +# indefinite hold on the write path. +COMMAND_TIMEOUT_SECONDS = 60 + +EXIT_ANSWERED = 0 # the request was answered (it may be a refusal: the JSON says) +EXIT_REFUSED = 1 # a refusal, or the state could not be recorded +EXIT_USAGE = 2 # a flag was used wrongly + + +def _port(value): + """The dashboard's own `_port`: an int or None, never a string.""" + if isinstance(value, int): + return value + if isinstance(value, str) and value.isdigit(): + return int(value) + return None + + +def _directive_part(line): + """The line up to its end-of-line comment: the agent's tokenizer starts a comment + at `#`, and the tokens matched here (a `proxy_pass` target, a `map` entry value) + never contain one.""" + return line.split("#", 1)[0] + + +def _read(path): + try: + with open(path, encoding="utf-8") as handle: + return handle.read() + except OSError: + return None + + +def _replace_file(path, text): + """`mkstemp` in the same directory (one filesystem, so the replace is atomic), + `fsync`, `os.replace`; the temp file is removed on the failure paths.""" + directory = os.path.dirname(path) or "." + os.makedirs(directory, exist_ok=True) + fd, tmp = tempfile.mkstemp(prefix=".dashboards-routectl.", dir=directory) + try: + with os.fdopen(fd, "w", encoding="utf-8") as handle: + handle.write(text) + handle.flush() + os.fsync(handle.fileno()) + os.replace(tmp, path) + except BaseException: + with contextlib.suppress(OSError): + os.remove(tmp) + raise + + +# -- the agent's parser --------------------------------------------------------- # + +def load_agent(path): + """`(module, None)` or `(None, refusal)`: the helper's parser is the agent's, by + file path. A failure names the path (the operator sees which install is wrong) + and the exception's *type* — the module's own error text is not this process's + to print verbatim.""" + if not os.path.isfile(path): + return None, f"cannot import the agent module {path}: not a file" + try: + spec = importlib.util.spec_from_file_location("dashboards_agent_module", path) + if spec is None or spec.loader is None: + return None, f"cannot import the agent module {path}: no loader" + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module, None + except Exception as exc: # noqa: BLE001 — every import failure is a refusal, by type + return None, f"cannot import the agent module {path}: {type(exc).__name__}" + + +def under_root(root, path): + """Resolve a config path the way the agent's `--root` does: absolute under + `root`. A path that is already under `root` (an operator's absolute path for + the sink, say) comes back unchanged — `--root` is a prefix, not a remap.""" + if not path.startswith("/"): + return path + root = str(root) + if root in ("", "/"): + return path + if path == root or path.startswith(root.rstrip("/") + "/"): + return path + return os.path.join(root, path.lstrip("/")) + + +# -- the id and the locate ----------------------------------------------------- # + +def parse_route_id(route_id): + """`(source, in_port, host, out_port)` or None: the id's own grammar, nothing else. + + `out_port` is the target's port — for a `map` entry that is the value's port, the + way the dashboard's claim writes it (a switched-off entry is addressed by the sink + value it carries, which is how `enable` finds the entry again). + """ + head, sep, tail = route_id.partition("->") + if not sep: + return None + source, _, in_port_text = head.partition(":") + if source not in ("haproxy", "nginx") or not in_port_text.isdigit(): + return None + host, sep2, port_text = tail.rpartition(":") + if not sep2 or not port_text.isdigit(): + return None + return source, int(in_port_text), host, int(port_text) + + +def _haproxy_candidates(agent, text, in_port, host, port): + """`(candidates, refusal)`: the `server` lines whose target is `host:port`, in a + backend the frontend on `in_port` reaches (its `default_backend` plus its + `use_backend` names). A candidate is `(index, name)`. Two backends reaching the + same named server are two candidates — the caller's `ambiguous` arm, not a + silent first match.""" + parsed = agent.parse_haproxy_config(text) + if not parsed.get("recognised"): + return [], "not-found" + reachable = set() + for frontend in parsed.get("frontends", ()): + ports = [_port(b.get("port")) for b in frontend.get("binds", ())] + if _port(in_port) not in ports: + continue + default = frontend.get("default_backend") + if default: + reachable.add(default) + reachable.update(frontend.get("backends", ())) + reachable.discard(None) + candidates = [] + for backend in parsed.get("backends", ()): + if backend.get("name") not in reachable: + continue + for server in backend.get("servers", ()): + if server.get("target") != host or _port(server.get("port")) != _port(port): + continue + name = server.get("name") + for index, line in enumerate(text.splitlines()): + words = line.split() + if len(words) < 3 or words[0] != "server" or words[1] != name: + continue + # the line's own target token, parsed by the agent's own helper — the + # grammar is the agent's, and a token that does not parse to the id's + # target is not this line. + if agent._haproxy_addr_port(words[2]) != (host, _port(port)): + continue + candidates.append((index, name)) + return candidates, None + + +def _nginx_documents(agent, collector, main_label): + """`{label: text}`: the main config plus its one level of `include`, the agent's + way (its bounds, its glob expansion, its labels). None when the main file cannot + be read: that is `not-found` for every nginx id, not an error.""" + main_text = _read(under_root(collector.root, main_label)) + if main_text is None: + return None + errors = [] + includes, _dropped, _unread = collector.nginx_include_files( + agent.nginx_include_paths(main_text), errors) + documents = {main_label: main_text} + documents.update(includes) + return documents + + +def _line_matches_token(agent, lines, index, node): + """True when line `index` carries the node's token: the `proxy_pass ` + whose token the agent's own `_nginx_proxy_pass` parses back to the node's three + facts (scheme, target, variable). The grammar is the agent's, not this + file's; the search is text over the lines the parser already parsed.""" + words = _directive_part(lines[index]).strip().split() + if len(words) < 2 or words[0] != "proxy_pass": + return False + # the `;` glues to the token in the file; the agent's tokenizer takes it + # off, so it comes off before the token goes to the agent's helper. + token = words[1].rstrip(";") + pp = agent._nginx_proxy_pass(token, None) + return (pp.get("scheme") == node.get("scheme") + and pp.get("target") == node.get("target") + and pp.get("variable") == node.get("variable")) + + +def _nginx_candidates(agent, documents, in_port, host, port): + """`(candidates, refusal)` over the documents: a candidate is + `(label, index, kind, name)`, kind `vhost` (a `proxy_pass`) or `map` (a `map` + entry). `refusal` is `map-default`: the id addressed the map's `default` + entry, which is refused, not located. The line is the *unique* line whose + token the agent's own helper parses to the id's target — a target written on + two lines (two vhosts, a repeated key) is `ambiguous`, never a silent first + match: with no line number on the wire, a guess between two lines is a write + to the wrong one.""" + candidates = [] + for label, text in documents.items(): + parsed = agent.parse_nginx_config(text, None) + if parsed.get("error"): + continue # the agent drops an unparseable document; it names it in `unparsed` + lines = text.splitlines() + for server in parsed.get("servers", ()): + if (_port(in_port) + not in {_port(b.get("port")) for b in server.get("listen", ())}): + continue + for node in server.get("proxy_passes", ()): + variable = node.get("variable") + if variable: + for block in parsed.get("maps", ()): + if block.get("variable") != variable: + continue + for entry in block.get("entries", ()): + target = entry.get("target") + if target is None: + continue + if (target.get("host") != host + or _port(target.get("port")) != _port(port)): + continue + if entry.get("key") == "default": + return [], "map-default" + index, count = _map_entry_line(lines, entry) + if count != 1: + return [], f"ambiguous: {entry.get('key')}" + candidates.append( + (label, index, "map", entry.get("key"))) + else: + target = node.get("target") + if target is None: + continue + if (target.get("host") != host + or _port(target.get("port")) != _port(port)): + continue + found = [i for i in range(len(lines)) + if _line_matches_token(agent, lines, i, node)] + if len(found) != 1: + return [], f"ambiguous: {_marker_name_for_target(target)}" + candidates.append((label, found[0], "vhost", + _marker_name_for_target(target))) + return candidates, None + + +def _map_entry_line(lines, entry): + """`(index, count)` for the entry's line: ` ;` (or a bare + ` ;` when the value was deleted by hand — the parser then reads a + `None` target, so that arm is the id's `not-found`, and the restore value is + the sidecar's, not the file's). `count` is over the whole document: a key + repeated in one block is the caller's `ambiguous` arm, never a silent first + match. The value word is what the agent's own `_nginx_addr_port` read + (host, port), re-joined as written.""" + target = entry.get("target") + if target is None: + value = "" + elif _port(target.get("port")) is not None: + value = f"{target.get('host')}:{target.get('port')}" + else: + value = target.get("host") or "" + found = [] + for i, line in enumerate(lines): + words = _directive_part(line).strip().rstrip(";").split() + if not words or words[0] != entry.get("key"): + continue + if len(words) > 1: + if words[1] == value: + found.append(i) + elif value == "": + found.append(i) + return (found[0] if found else None), len(found) + + +def _marker_name_for_target(target): + """The vhost arm's marker name, pinned by the captured fixture: the target host's + first label plus the port (`nextcloud.reeselink.com:11000` → `nextcloud-11000`), + the label alone when the target names no port.""" + host = target.get("host") or "" + label = host.split(".", 1)[0] or "unnamed" + port = _port(target.get("port")) + return f"{label}-{port}" if port is not None else label + + +# -- the state sidecar --------------------------------------------------------- # + +def load_state(path): + """`(dict, None)` or `(None, refusal)`. A missing file is the empty state; a + corrupt one is a refusal, not a `{}` — a state this helper cannot read is not a + state it may write over.""" + if not os.path.exists(path): + return {}, None + try: + with open(path, encoding="utf-8") as handle: + state = json.load(handle) + except (OSError, ValueError): + return None, f"state-corrupt: {path}" + if not isinstance(state, dict): + return None, f"state-corrupt: {path}" + return state, None + + +def save_state(path, state): + """The sidecar: written with `os.open(..., O_CREAT, 0o600)` so the mode is its + birth mode (the `dev-certs-export.py` idiom), in a directory created `0700`.""" + directory = os.path.dirname(path) + if directory: + os.makedirs(directory, mode=0o700, exist_ok=True) + fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600) + try: + with os.fdopen(fd, "w", encoding="utf-8") as handle: + json.dump(state, handle, indent=1, sort_keys=True) + handle.write("\n") + except OSError as exc: + return False, type(exc).__name__ + return True, None + + +# -- the one write protocol ---------------------------------------------------- # + +class Write: + """One request's whole life: locate, state, edit, validate, reload, prove, + record. `ctx` carries the flags (paths, units, the actor); the agent module is + the parser (and the config-reader's include machinery).""" + + def __init__(self, ctx, agent, source, in_port, host, port, route_id): + self.ctx = ctx + self.agent = agent + self.source = source + self.in_port = in_port + self.host = host + self.port = port + self.route_id = route_id + + def locate(self, source, in_port, host, port): + """Step 1: `(route, None)` or `(None, refusal)`. `route` is everything the + protocol needs: the file label and its text, the line index, the marker + name, the unit, the checker.""" + ctx = self.ctx + if source == "haproxy": + label = ctx.haproxy_config + text = _read(under_root(ctx.root, label)) + if text is None: + return None, "not-found" + candidates, refusal = _haproxy_candidates( + self.agent, text, in_port, host, port) + if refusal: + return None, refusal + if not candidates: + return None, "not-found" + if len(candidates) > 1: + return None, f"ambiguous: {sorted({c[1] for c in candidates})}" + index, name = candidates[0] + return self._route("haproxy", label, {label: text}, index, name, + ctx.haproxy_unit, "haproxy"), None + collector = self.agent.Collector(ctx.root, smart=False, proxy=False, + resolve=False) + documents = _nginx_documents(self.agent, collector, ctx.nginx_config) + if documents is None: + return None, "not-found" + candidates, refusal = _nginx_candidates(self.agent, documents, in_port, + host, port) + if refusal: + return None, refusal + if not candidates: + return None, "not-found" + distinct = {(c[0], c[1]) for c in candidates} + if len(distinct) > 1: + return None, f"ambiguous: {sorted({c[3] for c in candidates})}" + label, index, kind, name = candidates[0] + return self._route(kind, label, documents, index, name, ctx.nginx_unit, + "nginx"), None + + def _route(self, kind, label, documents, index, name, unit, checker): + return {"kind": kind, "file": label, + "path": under_root(self.ctx.root, label), + "documents": documents, "text": documents[label], + "index": index, "name": name, + "unit": unit, "checker": checker} + + def current_line(self, route): + return route["text"].splitlines()[route["index"]] + + def state_of(self, route): + """Step 2's state: `off` when the line carries the marker, or (the map arm + only) when the value was re-pointed at the sink by hand — the dashboard's + `sink_endpoint` rule, where a bare loopback address alone is not off, the + *address* must be; `on` otherwise. The agent's `split_marker` is the + reader: the grammar is one.""" + line = self.current_line(route) + _clean, name = self.agent.split_marker(line) + if name is not None: + return "off" + if route["kind"] == "map": + words = _directive_part(line).strip().rstrip(";").split() + if len(words) > 1: + host, _, port = words[1].rpartition(":") + sink_host, _, sink_port = self.ctx.sink_address.rpartition(":") + if host == sink_host and _port(port) == _port(sink_port): + return "off" + return "on" + + def edited_line(self, route, wanted, original): + """The edited line: `off` sets the marker (the map arm rewrites the value to + the sink first); `on` strips the marker (the map arm restores the sidecar's + `original`, because the value was rewritten away).""" + line = self.current_line(route) + clean, _name = self.agent.split_marker(line) + if wanted == "off": + if route["kind"] == "map": + indent = line[:len(line) - len(line.lstrip())] + return "{}{} {}; {}".format(indent, route["name"], + self.ctx.sink_address, + self.agent.marker_for(route["name"])) + return clean + " " + self.agent.marker_for(route["name"]) + if route["kind"] == "map": + return original + return clean + + def validate(self, route): + """Step 5: the proxy's own checker. `(ok, first_line)`: on failure the + checker's first non-empty line, bounded — a checker that dumps a stack + trace is not this process's to reprint. A missing binary is a failure naming + the binary: a checker this host does not have is not a pass.""" + argv = ["nginx", "-t"] if route["checker"] == "nginx" else ["haproxy", "-c", "-f", route["path"]] + try: + proc = subprocess.run(argv, capture_output=True, text=True, + timeout=COMMAND_TIMEOUT_SECONDS) + except FileNotFoundError: + return False, f"{argv[0]}: not found" + except (OSError, subprocess.SubprocessError) as exc: + return False, f"{argv[0]}: {type(exc).__name__}" + if proc.returncode == 0: + return True, "" + lines = [line_ for line_ in (proc.stdout + "\n" + proc.stderr).splitlines() + if line_.strip()] + return False, (lines[0] if lines else f"checker failed: {argv[0]}")[:200] + + def reload(self, route): + """Step 6: `systemctl reload ` — the only verb this file executes. + `(ok, status_line)`: on failure the unit's own first status line, so the + answer carries the fact and the operator the next action.""" + unit = route["unit"] + proc = subprocess.run(["systemctl", "reload", unit], + capture_output=True, text=True, + timeout=COMMAND_TIMEOUT_SECONDS) + if proc.returncode == 0: + return True, "" + status = subprocess.run(["systemctl", "status", unit], + capture_output=True, text=True, + timeout=COMMAND_TIMEOUT_SECONDS) + lines = [line_ for line_ in (status.stdout + "\n" + status.stderr).splitlines() + if line_.strip()] + detail = (lines[0] if lines + else f"systemctl reload {unit} failed")[:200] + return False, detail + + def _sink_path(self): + return under_root(self.ctx.root, self.ctx.sink_conf) + + def _sink_block(self): + # the captured fixture's shape: a bare `server` block (the file is loaded by + # the main config's `stream { include …; }` glob, so it must not carry its + # own `stream` wrapper), no `server_name`, no `proxy_pass` — the block, not + # the port, is what `sink_endpoint()` recognises. + return "\n".join([ + SINK_HEADER, + "server {", + f" listen {self.ctx.sink_address};", + "}", + ]) + "\n" + + def ensure_sink(self): + """The sink before the checker runs (a `map` entry rewritten to the sink is a + reference the checker must be able to resolve): create it, or refuse a file + that exists without the header — an operator's file is not this helper's to + overwrite.""" + path = self._sink_path() + existing = _read(path) + if existing is not None: + lines = existing.splitlines() + if not lines or lines[0] != SINK_HEADER: + return False, f"sink-file-not-ours: {self.ctx.sink_conf}" + return True, None + _replace_file(path, self._sink_block()) + return True, None + + def remove_sink_if_empty(self, documents): + """After an `enable`: remove the sink only when this helper wrote it (its + header is present) and no `map` entry still points at it. A file without + the header is left alone, quietly: it is not this helper's.""" + path = self._sink_path() + if not os.path.exists(path): + return + text = _read(path) + lines = (text or "").splitlines() + if not lines or lines[0] != SINK_HEADER: + return + sink_host, _, sink_port = self.ctx.sink_address.rpartition(":") + for _label, doc_text in documents.items(): + parsed = self.agent.parse_nginx_config(doc_text, None) + for block in parsed.get("maps", ()): + for entry in block.get("entries", ()): + target = entry.get("target") + if target is None: + continue + if (target.get("host") == sink_host + and _port(target.get("port")) == _port(sink_port)): + return # a route still points at the sink: it stays + os.remove(path) + + def run(self, body): + """The whole protocol for one validated body: `(answer, code)`. `body` is + `{id, action, dry_run}` — the body has already been shape-checked by the + caller (the wire's gate is the same one).""" + ctx = self.ctx + write = self + source, in_port, host, port = self.source, self.in_port, self.host, self.port + wanted = "off" if body["action"] == "disable" else "on" + route, refusal = write.locate(source, in_port, host, port) + if refusal: + if refusal == "map-default": + return {"ok": False, "applied": False, + "refused": "map-default"}, EXIT_REFUSED + return {"ok": False, "detail": refusal}, EXIT_REFUSED + original = write.current_line(route) + current = write.state_of(route) + # `enable` restores only from the sidecar: the line must be byte-identical + # to what this helper wrote, and the map arm's original value comes from the + # row (it is not in the file after a disable). A missing or drifted row is + # a refusal, and no byte changes. + if wanted == "on": + state, refusal = load_state(ctx.state_file) + if refusal: + return {"ok": False, "detail": refusal}, EXIT_REFUSED + # a `map` route is addressed by its sink value while off, so the + # enable's id is not the write's: the row is found by the line this + # helper wrote in this file, never by the id alone. + row = next((value for value in state.values() + if value.get("file") == route["file"] + and value.get("line") == route["index"]), None) + if row is None: + return {"ok": False, "applied": False, "drift": True, + "detail": "no state row: this helper did not write the line"}, \ + EXIT_REFUSED + if original != row.get("written"): + return {"ok": False, "applied": False, "drift": True, + "detail": "the line was edited after the helper wrote it"}, \ + EXIT_REFUSED + original = row["original"] + if current == wanted: + # step 2: already there. No write, no checker, no reload, no state change. + return {"ok": True, "applied": False, "changed": False, + "id": body["id"], "state": current}, EXIT_ANSWERED + new_line = write.edited_line(route, wanted, original) + lines = route["text"].splitlines() + new_lines = list(lines) + new_lines[route["index"]] = new_line + before = route["text"] + after = "\n".join(new_lines) + if before.endswith("\n"): + after += "\n" + if body["dry_run"]: + # the dry run: the unified diff and nothing else — no file (not even + # the sink's), no checker, no reload, no state. + diff = list(difflib.unified_diff(lines, new_lines, + fromfile=route["file"], + tofile=route["file"] + ".new", + lineterm="")) + return {"ok": True, "applied": False, "changed": True, + "dry_run": True, "id": body["id"], "diff": diff}, EXIT_ANSWERED + # the sink must exist before the checker runs (a `map` entry rewritten to + # it is a reference the checker resolves); an operator's file at that + # path is a refusal. + if wanted == "off" and route["kind"] == "map": + ok, sink_refusal = write.ensure_sink() + if not ok: + return {"ok": False, "detail": sink_refusal}, EXIT_REFUSED + # step 4: the atomic write. + _replace_file(route["path"], after) + # step 5: the checker; a failure restores the previous bytes and reloads + # nothing. + ok, first = write.validate(route) + if not ok: + _replace_file(route["path"], before) + return {"ok": False, "applied": False, "validated": False, + "restored": True, "detail": first}, EXIT_REFUSED + # step 6: the reload (the only verb); a failure leaves the file applied. + reloaded, status = write.reload(route) + # step 7: prove the marker state is live on the re-read file, then record. + text = _read(route["path"]) + proof = "on" + if text is not None: + _clean, name = self.agent.split_marker( + text.splitlines()[route["index"]]) + proof = "off" if name is not None else "on" + if route["kind"] == "map" and wanted == "on": + # the re-pointed entry no longer names the sink: the sink stays only + # while some route does (every document is scanned, not just this one). + new_documents = dict(route["documents"]) + new_documents[route["file"]] = text or "" + write.remove_sink_if_empty(new_documents) + state, refusal = load_state(ctx.state_file) + if refusal: + return {"ok": True, "applied": True, "validated": True, + "reloaded": reloaded, "status": status, + "detail": refusal}, EXIT_REFUSED + if body["action"] == "disable": + state[body["id"]] = { + "actor": ctx.actor, + # `datetime.timezone.utc`, not the `datetime.UTC` alias (3.11+): the + # helper runs on a host's Python 3.10. + "at": datetime.datetime.now(datetime.timezone.utc).isoformat(), # noqa: UP017 + "file": route["file"], + # the file as the helper left it: `verify` compares against it, so a + # hand edit since the write is a fact the operator can see. + "sha256": hashlib.sha256(after.encode("utf-8")).hexdigest(), + "action": "disable", + "line": route["index"], + "written": new_line, + "original": original, + } + else: + # the row is keyed by the write's id, which for a `map` route is not + # the enable's id: drop the row this line came from. + for key in [k for k, value in state.items() + if value.get("file") == route["file"] + and value.get("line") == route["index"]]: + state.pop(key, None) + ok, refusal = save_state(ctx.state_file, state) + if not ok: + return {"ok": True, "applied": True, "validated": True, + "reloaded": reloaded, "status": status, + "detail": f"state-write-failed: {refusal}"}, EXIT_REFUSED + return {"ok": True, "applied": True, "validated": True, + "reloaded": reloaded, "status": status, "state": proof}, \ + EXIT_ANSWERED + + +# -- one request --------------------------------------------------------------- # + +def refused(detail, code=EXIT_REFUSED): + """The one refusal shape: `ok` false and the `detail` (never a traceback, + never a credential — the detail names the arm, the file, the reason, and + nothing else).""" + return {"ok": False, "detail": detail}, code + + +def show(write): + """`show`: the live state of one id — the file, the line as it stands, and + `state` (`on`/`off`) read through the agent's own marker helper. No write, + no checker, no reload. An id the file no longer carries is `not-found`.""" + route, refusal = write.locate(write.source, write.in_port, write.host, write.port) + if refusal: + if refusal == "map-default": + return refused("map-default")[0] + return refused(refusal)[0] + return {"ok": True, "id": write.route_id, "state": write.state_of(route), + "file": route["file"]} + + +def handle_request(ctx, agent, body, actor="manual"): + """The one wire body: exactly `{id, action, dry_run}`. Anything wider or + narrower is a `detail` refusal before any file is read: the wire is the + dashboard's one narrow request, and `dry_run` here means exactly what it + means in the task — the diff, no write, no checker, no reload, no state. + `action` is the request's verb; the read-only verbs (`show`, `verify`) + take no action at all, so `action` is optional for them.""" + if not isinstance(body, dict): + return refused("body: one JSON object") + if set(body) - {"id", "action", "dry_run"}: + return refused("body: exactly {id, action, dry_run}") + route_id = body.get("id") + if not isinstance(route_id, str) or not route_id or len(route_id) > ROUTE_ID_MAX: + return refused("id: one non-empty string of at most 256 characters") + action = body.get("action") + if action is None and "action" not in body: + action = "show" + if action not in ("show", "disable", "enable"): + return refused("action: one of show|disable|enable") + if "dry_run" in body and not isinstance(body["dry_run"], bool): + return refused("dry_run: a boolean") + dry_run = bool(body.get("dry_run", False)) + parsed = parse_route_id(route_id) + if parsed is None: + return refused("id: `source:in-port->host:out-port`") + source, in_port, host, out_port = parsed + write = Write(ctx, agent, source, in_port, host, out_port, route_id) + if action == "show": + return show(write), EXIT_ANSWERED + body = {"id": route_id, "action": action, "dry_run": dry_run} + return write.run(body) + + +def serve(ctx, agent): + """`--serve`: one request per process, from the socket-activated stdin. The + request is one newline-delimited JSON line — exactly the agent write's wire + body, `{id, action, dry_run}` (wider or narrower is a refusal) — and the + answer is one JSON line back on the socket. The + peer check is the second look on top of the unit's `SocketUser`/`SocketMode`: + `SO_PEERCRED` names the caller's uid and it must be `--allow-uid`'s. The + transport is the socket when systemd handed one: fd 3 (the classic + socket-activation contract) or fd 0 (with `Accept=yes` and + `StandardInput=socket`, the accepted connection lands on stdin — that is + what the live box delivered, phase 33 task 07); otherwise a connected pipe, + and the peer check runs on it too — a transport without a readable peer (an + unconnected pipe) is a refusal, not a skipped gate. + """ + sock = None + for fd in (3, 0): + with contextlib.suppress(OSError, ValueError, AttributeError): + sock = socket.fromfd(fd, socket.AF_UNIX, socket.SOCK_STREAM) + if sock is not None: + break + peer_ok, peer_detail = _peer_check(sock, ctx.allow_uid) + if not peer_ok: + # a refusal the dialer cannot see is a timeout in their code: the + # refusal goes on the wire before the process exits. + if sock is not None: + with contextlib.suppress(OSError): + sock.sendall(json.dumps({"ok": False, "detail": peer_detail}, + separators=(",", ":")).encode("utf-8") + b"\n") + return {"ok": False, "detail": peer_detail}, EXIT_REFUSED + # the socket path reads with `recv(2)`: on this fleet's kernel, `read(2)` + # on a socket fd does not wake on data the peer already sent (the + # socketpair test in tests/test_deploy_scripts.py pins it — the helper + # sat in `read` with the request in the buffer, `recv` got it at once). + next_chunk = sock.recv if sock is not None else sys.stdin.buffer.read + buffer = b"" + while b"\n" not in buffer: + chunk = next_chunk(4096) + if not chunk: + break + buffer += chunk + if len(buffer) > SERVE_REQUEST_MAX: + return {"ok": False, + "detail": f"body: longer than {SERVE_REQUEST_MAX} bytes"}, EXIT_REFUSED + line = buffer.split(b"\n", 1)[0] + try: + body = json.loads(line.decode("utf-8")) + except (ValueError, UnicodeDecodeError): + return {"ok": False, "detail": "body: not JSON"}, EXIT_REFUSED + answer, code = handle_request(ctx, agent, body, + actor=f"agent:{ctx.allow_uid}") + # the answer goes back to the socket: with `StandardOutput=socket` that is + # also the process's stdout, but the socket is the contract and stdout is + # not. + if sock is not None: + try: + sock.sendall(json.dumps(answer, separators=(",", ":")).encode("utf-8") + + b"\n") + except OSError as exc: + return {"ok": False, "detail": f"send: {type(exc).__name__}"}, EXIT_REFUSED + return answer, code + + +def _ucred_slots(allow_uid): + """Calibrate which field of `SO_PEERCRED` is the uid, because the 12-byte + ucred buffer's field order is not portable: this fleet's el10 kernel + (6.12.0-211.69.1.el10_2) delivers it as `{pid, uid, gid}` — the classic + layout `{uid, gid, pid}` reversed (probed on the live box, phase 33 task + 07). The probe is a socketpair whose peer is this process: the field equal + to our pid names the pid slot. For a root process (the service mode) the + probe *shifts the uid* to the expected value with `setuid` while the gid + stays 0 (a privilege drop, restored in `finally` before the state file is + touched): the field that becomes `expected` is the uid slot, the one that + stays 0 is the gid slot — a matching gid on a foreign-uid caller cannot + open the gate. A non-root process cannot shift: with `euid == egid` its + two remaining fields are equal, so it gets **both** back and the gate + then requires both to hold the expected value (the service account has + uid == gid, so a true match still passes); `euid != egid` separates the + two by uniqueness. An inconclusive layout refuses: one ambiguous instance + must not open the gate for everyone. + """ + try: + expected = pwd.getpwnam(allow_uid).pw_uid + except KeyError: + return None + if expected is None or expected < 0: + return None + shifted = os.geteuid() == 0 and expected != 0 + if shifted: + try: + os.setuid(expected) # gid stays 0: only the uid field moves + except OSError: + shifted = False + a, b = socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM) + try: + data = a.getsockopt(socket.SOL_SOCKET, socket.SO_PEERCRED, 12) + finally: + if shifted: + with contextlib.suppress(OSError): + os.setuid(0) + a.close() + b.close() + fields = list(struct.unpack_from("<3i", data)) + if len(fields) != 3: + return None + pid_slots = {i for i, v in enumerate(fields) if v == os.getpid()} + if len(pid_slots) != 1: + return None + pid_slot = pid_slots.pop() + non_pid = [i for i in range(3) if i != pid_slot] + if shifted: + uid_slots = tuple(i for i in non_pid if fields[i] == expected) + if len(uid_slots) != 1: + return None + elif expected == 0 or os.geteuid() == os.getegid(): + # unshifted and the two remaining fields are equal (or `root` is + # allowed from an unprivileged probe): require both to hold the value + uid_slots = tuple(non_pid) + else: + uid_slots = tuple(i for i in non_pid if fields[i] == os.geteuid()) + if len(uid_slots) != 1: + return None + return pid_slot, uid_slots + + +def _peer_check(sock, allow_uid): + """`SO_PEERCRED` on the connected socket: `(ok, detail)`. The uid field is + read at the slot `_ucred_slots` calibrated on this kernel (the buffer's + field order is not portable — the live box delivers it reversed); a + transport that cannot report a peer (a plain pipe) or whose layout is + inconclusive fails the check — the gate never opens on a caller the kernel + will not name, and never on an unreadable one.""" + if sock is None: + return False, "peer: no connected socket on this transport" + try: + data = sock.getsockopt(socket.AF_UNIX, socket.SO_PEERCRED, 12) + except OSError: + return False, "peer: the transport reported no peer" + expected = None + try: + expected = pwd.getpwnam(allow_uid).pw_uid + except KeyError: + return False, f"peer: allow-uid {allow_uid} is not a user" + if expected is None or expected < 0: + return False, f"peer: allow-uid {allow_uid} is not a user" + slots = _ucred_slots(allow_uid) + if slots is None: + return False, "peer: the credential layout is inconclusive on this kernel" + pid_slot, uid_slots = slots + fields = struct.unpack_from("<3i", data) + if fields[pid_slot] <= 0: + return False, "peer: the transport reported no peer" + held = [fields[i] for i in uid_slots] + if any(v != expected for v in held): + return False, f"peer: uid {held[0]} is not {allow_uid}" + return True, None + + +# -- the manual arms ----------------------------------------------------------- # + +def verify(ctx, agent): + """`verify`: the sidecar against the files — per id, the file's hash and the + written line. A drift is a fact, not an error: the answer says which, and the + operator decides what it means.""" + state, refusal = load_state(ctx.state_file) + if refusal: + return {"ok": False, "detail": refusal}, EXIT_REFUSED + rows = [] + drifted = False + for key in sorted(state): + row = state[key] + path = under_root(ctx.root, row.get("file", "")) + text = _read(path) + if text is None: + rows.append({"id": key, "file": row.get("file"), + "hash": "missing", "marker": "absent"}) + drifted = True + continue + # the sidecar's `sha256` is the file as the helper left it: a mismatch is + # a hand edit since the write — a fact, not an error. + sha = hashlib.sha256(text.encode("utf-8")).hexdigest() + hash_ok = sha == row.get("sha256") + written = row.get("written") + live = written is not None and written in text.splitlines() + if not hash_ok or not live: + drifted = True + rows.append({"id": key, "file": row.get("file"), + "hash": "ok" if hash_ok else "drift", + "marker": "live" if live else "absent"}) + return {"ok": not drifted, "state": rows}, \ + EXIT_ANSWERED if not drifted else EXIT_REFUSED + + +def main(argv=None): + argv = list(sys.argv[1:] if argv is None else argv) + parser = argparse.ArgumentParser( + prog="dashboards-routectl", + description="the route-switch root helper: one id, one file, one reload") + parser.add_argument("verb", nargs="?", choices=("list", "show", "diff", + "disable", "enable", "verify"), + help="what to do (`--serve` is the socket-activated mode)") + parser.add_argument("--root", default="/", + help="the tree the config paths are read under (rehearsal)") + parser.add_argument("--agent-module", + default="/usr/local/lib/dashboards-agent/host_agent.py", + help="the agent's parser, imported by file path (required)") + parser.add_argument("--haproxy-config", default="/etc/haproxy/haproxy.cfg") + parser.add_argument("--nginx-config", default="/etc/nginx/nginx.conf") + parser.add_argument("--state", + default="/var/lib/dashboards-routectl/state.json") + parser.add_argument("--sink-conf", default=SINK_FILE_DEFAULT) + parser.add_argument("--sink-address", default=SINK_ADDRESS_DEFAULT) + parser.add_argument("--nginx-unit", default="nginx") + parser.add_argument("--haproxy-unit", default="haproxy") + parser.add_argument("--id", default=None, help="the `connections[].id` to act on") + parser.add_argument("--action", choices=("disable", "enable"), default=None, + help="the action to preview (with `diff`)") + parser.add_argument("--apply", action="store_true", + help="write (without it every write verb is a dry run)") + parser.add_argument("--serve", action="store_true", + help="serve one request from the socket-activated stdin") + parser.add_argument("--allow-uid", default="dashboards-agent", + help="the only peer uid --serve accepts (SO_PEERCRED)") + args = parser.parse_args(argv) + + if args.serve and args.verb: + print(json.dumps({"ok": False, "detail": "usage: --serve takes no verb"}, + separators=(",", ":"))) + return EXIT_USAGE + if not args.serve and args.verb is None: + print(json.dumps( + {"ok": False, + "detail": "usage: a verb (list | show | diff | disable | enable | " + "verify) or --serve"}, separators=(",", ":"))) + return EXIT_USAGE + agent, refusal = load_agent(args.agent_module) + if agent is None: + print(json.dumps({"ok": False, "detail": refusal}, + separators=(",", ":"))) + return EXIT_REFUSED + ctx = argparse.Namespace( + root=args.root, haproxy_config=args.haproxy_config, + nginx_config=args.nginx_config, state_file=args.state, + sink_conf=args.sink_conf, sink_address=args.sink_address, + nginx_unit=args.nginx_unit, haproxy_unit=args.haproxy_unit, + actor="manual", allow_uid=args.allow_uid) + if args.serve: + answer, code = serve(ctx, agent) + elif args.verb == "list": + state, refusal = load_state(args.state) + if refusal: + answer, code = {"ok": False, "detail": refusal}, EXIT_REFUSED + else: + answer, code = {"ok": True, "state": [ + {"id": key, **value} for key, value in sorted(state.items())] + }, EXIT_ANSWERED + elif args.verb == "verify": + answer, code = verify(ctx, agent) + else: + if not args.id: + return {"ok": False, "detail": f"usage: {args.verb} needs --id"}, \ + EXIT_USAGE + if args.verb == "diff" and args.action not in ("disable", "enable"): + print(json.dumps( + {"ok": False, "detail": "usage: diff needs --action disable|enable"}, + separators=(",", ":"))) + return EXIT_USAGE + body = {"id": args.id, + "action": args.action if args.verb == "diff" else args.verb, + "dry_run": not args.apply} + answer, code = handle_request(ctx, agent, body, actor="manual") + if not args.serve: + # in `--serve` the answer is already on the socket (the unit's + # StandardOutput=socket is that same socket); a second write here would + # put the line on the wire twice. + sys.stdout.write(json.dumps(answer, separators=(",", ":")) + "\n") + sys.stdout.flush() + return code + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/deploy/dashboards-routectl.socket b/deploy/dashboards-routectl.socket new file mode 100644 index 0000000..9297e1c --- /dev/null +++ b/deploy/dashboards-routectl.socket @@ -0,0 +1,25 @@ +# dashboards-routectl.socket +# +# The socket the agent (and nothing else) may talk to for a route switch. +# +# 0600 + SocketUser=dashboards-agent is the whole access control at the file +# layer: only that uid can connect; everyone else gets EACCES before any code +# of ours runs. The helper's SO_PEERCRED check (its --allow-uid) is the +# *second* look, not the first. +# +# One connection, one instance: `Accept=yes` is what makes it so — the socket +# unit accepts the connection itself and hands the *connection* (not the +# listener) to a fresh `dashboards-routectl@` on stdin (StandardInput=socket); +# the instance answers one request and exits. There is no long-lived privileged +# process. (A `Service=` line naming the template does not load on systemd 257 — +# "missing the instance name" — `Accept=yes` is the documented template path.) +[Socket] +Accept=yes +ListenStream=/run/dashboards-routectl.sock +SocketUser=dashboards-agent +SocketGroup=dashboards-agent +SocketMode=0600 +NoDelay=yes + +[Install] +WantedBy=sockets.target diff --git a/deploy/dashboards-routectl@.service b/deploy/dashboards-routectl@.service new file mode 100644 index 0000000..0133850 --- /dev/null +++ b/deploy/dashboards-routectl@.service @@ -0,0 +1,42 @@ +# dashboards-routectl@.service +# +# The route-switch helper, root, socket-activated, one request per instance. +# +# The privilege model, in one line: the grant is a file mode systemd enforces +# (the socket is 0600 and owned by the agent user), not a sudoers line. This +# unit is how the agent's one narrow write reaches root without the agent's +# own unit being given anything — NoNewPrivileges=true is kept *here* as well, +# because the privilege comes from being started by systemd as root, not from +# escalation. +# +# Write scope, stated out loud: the helper replaces exactly two kinds of +# config files, and os.replace needs write on the *directory*, so +# ReadWritePaths lists every directory whose file it replaces — and nothing +# else: +# /etc/haproxy the haproxy.cfg `server` line (the haproxy arm) +# /etc/nginx/conf.d the vhost `proxy_pass` line (the vhost arm) +# /etc/nginx/stream.d the `map` entry (the stream arm) and the helper's +# own sink file, 00-dashboards-sink.conf, which it +# creates and removes there +# /etc/nginx (the main config) is deliberately not listed: the helper never +# writes it. +# +# One connection, one instance: the socket's `Accept=yes` hands this template a +# fresh connected socket on stdin per connection (StandardInput=socket / +# StandardOutput=socket); the helper answers exactly one newline-delimited JSON +# request and exits, so there is never a long-lived privileged process sitting +# on a socket. +[Service] +Type=simple +ExecStart=/usr/local/sbin/dashboards-routectl.py --serve \ + --allow-uid dashboards-agent --state /var/lib/dashboards-routectl/state.json +User=root +StandardInput=socket +StandardOutput=socket +StateDirectory=dashboards-routectl +ProtectSystem=read-only +ReadWritePaths=/etc/haproxy /etc/nginx/conf.d /etc/nginx/stream.d +ProtectHome=read-only +PrivateTmp=yes +NoNewPrivileges=true +SystemCallFilter=@system-service diff --git a/deploy/gateway-routectl-README.md b/deploy/gateway-routectl-README.md new file mode 100644 index 0000000..18f7bbf --- /dev/null +++ b/deploy/gateway-routectl-README.md @@ -0,0 +1,66 @@ +# The route-switch helper (gateway hosts) + +The dashboard's one narrow write is a route switch: one named connection on the +ingress goes `#off` (a marker the agent's own parser reads) or comes back. It is +not a sudoers line. The grant is a **file mode systemd enforces**: the socket the +agent talks to is `0600` and owned by the agent's own user, and the helper's +peer check (`--allow-uid`) is the second look. No account on the host — `ops` +included — can reach the write; only the agent's uid can connect, and only that +uid is accepted on the other end. + +## The two units + +| unit | what it is | +| ---- | ---------- | +| `dashboards-routectl.socket` | a systemd-activated stream socket at `/run/dashboards-routectl.sock`, `0600`, owned by `dashboards-agent` | +| `dashboards-routectl@.service` | the helper, **root**, one instance per connection (the connected socket on stdin), answers one request and exits | + +The service unit's `ReadWritePaths` lists exactly the directories whose file the +helper replaces — `/etc/haproxy`, `/etc/nginx/conf.d`, `/etc/nginx/stream.d` — +and nothing else. There is never a long-lived privileged process: each request +spawns a fresh instance that exits after its one answer. + +## See them + +``` +systemctl status dashboards-routectl.socket +systemctl status 'dashboards-routectl@*.service' +journalctl -u 'dashboards-routectl@*' +``` + +## Run the helper by hand on the box + +The installed copy is `/usr/local/sbin/dashboards-routectl.py`. It takes its +agent parser by file path and refuses to run without one: + +``` +# what is off (the state sidecar) +sudo /usr/local/sbin/dashboards-routectl.py list + +# preview one route without writing a byte (no checker, no reload, no state) +sudo /usr/local/sbin/dashboards-routectl.py diff --id nginx:443->127.0.0.1:44450 --action enable + +# verify the sidecar against the files (drift is reported, never repaired) +sudo /usr/local/sbin/dashboards-routectl.py verify +``` + +A manual `disable --apply` runs the whole protocol: write, the proxy's own +checker (`nginx -t` / `haproxy -c -f`), restore the previous bytes on a checker +failure, `systemctl reload` (never `restart`) on success, then record the state. + +## Undo it all + +``` +sudo ./deploy/install-agent.sh --without-routectl +``` + +That disables and removes the helper and its two units, **leaves the proxy +configs and the helper's state directory alone**, and prints the list of +still-off routes first — those are yours to switch back on. + +## The one safety sentence + +No sudoers line is involved. The agent's unit keeps every one of its security +directives (`NoNewPrivileges=true`, `ProtectSystem=strict`, …); the read-only +collector stayed read-only in its sandbox and talked to a peer outside it. The +grant is the socket's mode and ownership, enforced by systemd. diff --git a/deploy/install-agent.sh b/deploy/install-agent.sh index e258bd0..7e247b2 100755 --- a/deploy/install-agent.sh +++ b/deploy/install-agent.sh @@ -9,8 +9,11 @@ # # registration fetches its own bundle # sudo ./deploy/install-agent.sh --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN # # a kernel-gated host (see below) +# sudo ./deploy/install-agent.sh --with-routectl # also install the route-switch helper +# # and its two units (the gateway only) # ./deploy/install-agent.sh --status # unit state + /health, no root # sudo ./deploy/install-agent.sh --uninstall +# sudo ./deploy/install-agent.sh --without-routectl # remove helper + units, keep config and state # sudo ./deploy/install-agent.sh --force # re-apply over a unit this would change # # No root shell? `deploy/agent-sudoers` is the single sudoers line that lets the fleet's @@ -102,6 +105,17 @@ # sudo ./deploy/install-agent.sh --bind --port 9120 \ # --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN # +# --with-routectl (gateway hosts only) installs the one narrow write the dashboard can make: +# the root route-switch helper as a socket-activated service. The grant is a file mode systemd +# enforces — the socket is 0600 and owned by the agent's own user, and the helper's peer check +# (--allow-uid) is the second look — not a sudoers line, and the agent's unit is not changed by +# it: an install with the flag renders the agent unit byte-identically to one without it. The +# helper edits exactly one route (a #off marker the agent's parser reads), validates the +# file with the proxy's own checker before it means anything, restores the previous bytes on a +# checker failure, and reloads — never restarts. --without-routectl removes the helper and its +# two units, leaves the proxy configs and the helper's state directory alone, and prints the +# list of still-off routes first, because those are the operator's to switch back on. +# # A re-run is the zero-noise path it has always been as long as the unit this script would # write is the unit already on disk. When the two differ the run stops: the diff between the # installed unit and the generated one is printed and the exit code is 1, before the unit is @@ -113,6 +127,13 @@ # # --force overwrite an existing unit even if the generated unit differs (prints the # diff first) +# +# --allow-writes (gateway hosts) adds the agent's own --allow-writes to the unit's +# ExecStart — the agent-side half of the write path; --with-routectl installs the +# helper it dials, and the two are independent (a host with the flag but without the +# helper answers a write with 503, not with a write). It is the *only* change this +# script makes to the agent's unit for the write path: a re-run diff that shows the +# security directives moving is a stop, not a --force. set -euo pipefail SERVICE="dashboards-agent" @@ -133,6 +154,20 @@ if command -v realpath >/dev/null 2>&1; then SCRIPT_PATH="$(realpath "$0" 2>/dev/null || printf '%s' "$SCRIPT_PATH")" fi SUDOERS_SOURCE="$(dirname "$SCRIPT_PATH")/agent-sudoers" +# The route-switch helper (phase 33) and its two units. The helper is the only writer this +# phase adds: a root, socket-activated service the agent may talk to and nothing else. The +# grant is a file mode systemd enforces (the socket is 0600, owned by the agent's user), not +# a sudoers line, and the agent's own unit is not touched by installing it — the rendered +# unit is byte-identical with and without --with-routectl, and the re-run guard below applies +# to the helper and both units exactly as it does to the agent's. +ROUTECTL_SOURCE="$SCRIPT_DIR/dashboards-routectl.py" +ROUTECTL_SOCKET_UNIT_SRC="$SCRIPT_DIR/dashboards-routectl.socket" +ROUTECTL_SERVICE_UNIT_SRC="$SCRIPT_DIR/dashboards-routectl@.service" +ROUTECTL_TARGET="/usr/local/sbin/dashboards-routectl.py" +ROUTECTL_SOCKET_UNIT="/etc/systemd/system/dashboards-routectl.socket" +ROUTECTL_SERVICE_UNIT="/etc/systemd/system/dashboards-routectl@.service" +ROUTECTL_SOCKET_PATH="/run/dashboards-routectl.sock" +ROUTECTL_STATE_DIR="/var/lib/dashboards-routectl" BIND="127.0.0.1" PORT="9120" @@ -158,6 +193,14 @@ CAPS="" # --force: the re-run guard's opt-in (phase 17 task 03). 0 means a re-run whose generated unit # differs from the installed one refuses after printing the diff. FORCE=0 +# --with-routectl / --without-routectl (phase 33): the route-switch helper and its two units, +# installed beside the agent and never inside its unit. Both off is the default: an install +# without the flag is byte-identical to one before the flag existed. +WITH_ROUTECTL=0 +# --allow-writes (phase 33): appends the agent's own --allow-writes to the unit's ExecStart. +# It is the agent-side gate for POST /route; the unit's security directives do not move +# with it, which is the phase's central safety claim in one flag. +ALLOW_WRITES=0 MODE="install" usage() { @@ -192,6 +235,9 @@ while [ $# -gt 0 ]; do shift 2 ;; --force) FORCE=1; shift ;; + --allow-writes) ALLOW_WRITES=1; shift ;; + --with-routectl) WITH_ROUTECTL=1; shift ;; + --without-routectl) MODE="routectl-off"; shift ;; --status) MODE="status"; shift ;; --uninstall) MODE="uninstall"; shift ;; -h|--help) usage 0 ;; @@ -233,7 +279,23 @@ fi # The fetch belongs to an install. `--status` probes a port and `--uninstall` deletes files; a # run of either would fetch a private key for nothing and leave it in a temp file. if [ "$FETCH_GIVEN" -eq 3 ] && [ "$MODE" != "install" ]; then - echo "--fetch-bundle needs an install (it means nothing to --status or --uninstall)" >&2 + echo "--fetch-bundle needs an install (it means nothing to --status, --uninstall or --without-routectl)" >&2 + exit 1 +fi +# One flag each direction, like --as-root/--caps: a run that installs the helper and removes +# it is a mistake about what to install, and it gets its own sentence before any write. +if [ "$WITH_ROUTECTL" -eq 1 ] && [ "$MODE" = "routectl-off" ]; then + echo "--with-routectl and --without-routectl are one flag each direction; run them separately" >&2 + exit 1 +fi +# --with-routectl installs files; --status probes a port and --uninstall deletes the agent. +if [ "$WITH_ROUTECTL" -eq 1 ] && [ "$MODE" != "install" ]; then + echo "--with-routectl needs an install (it means nothing to --status or --uninstall)" >&2 + exit 1 +fi +# --allow-writes rewrites the agent's unit; none of the non-install modes render a unit. +if [ "$ALLOW_WRITES" -eq 1 ] && [ "$MODE" != "install" ]; then + echo "--allow-writes needs an install (it is an argument to the agent's unit, not to --status, --uninstall or --without-routectl)" >&2 exit 1 fi @@ -335,6 +397,26 @@ uninstall() { echo "removed $SERVICE (the $AGENT_USER user was left in place; remove it with userdel)" } +routectl_off() { + # The inverse of --with-routectl: the helper and its two units go; the proxy configs and + # the helper's state directory stay — the state is the memory of what was switched, and + # a route the helper left off is the operator's to switch back on, so the still-off list + # prints first and the removal ends with the reminder, not the file list. + need_root --without-routectl + if [ -f "$ROUTECTL_TARGET" ]; then + echo "routes the helper has switched off (these are still off after this removal — switch them back on):" + "$ROUTECTL_TARGET" --state "$ROUTECTL_STATE_DIR/state.json" list || true + else + echo "the helper is not installed; nothing to list" + fi + systemctl disable --now dashboards-routectl.socket 2>/dev/null || true + systemctl disable --now 'dashboards-routectl@.service' 2>/dev/null || true + rm -f "$ROUTECTL_SOCKET_UNIT" "$ROUTECTL_SERVICE_UNIT" "$ROUTECTL_TARGET" + systemctl daemon-reload + echo "removed dashboards-routectl (the proxy configs and $ROUTECTL_STATE_DIR are left in place)" + echo "reminder: a route still off in the list above is still off — switch it back on" +} + new_token() { # urlsafe and no padding: the token goes into HOST_TARGETS as URL userinfo # (`name=http://TOKEN@host:9120`), where '+' and '=' are annoying to quote. @@ -590,6 +672,14 @@ RuntimeDirectory=dashboards-agent RuntimeDirectoryMode=0700" fi + # The write path's one unit change: the flag on ExecStart, and nothing else. The four + # security directives below are rendered exactly as before; a diff that shows them moving + # is a stop, not a --force (the phase's central safety claim, in one flag). + WRITES_EXEC_ARGS="" + if [ "$ALLOW_WRITES" -eq 1 ]; then + WRITES_EXEC_ARGS=" --allow-writes" + fi + # The kernel-gate variant (--as-root / --caps). Default: the dedicated user and an empty # CapabilityBoundingSet, i.e. the capability-less unit this script has always written. UNIT_USER="$AGENT_USER" @@ -630,7 +720,7 @@ Wants=network-online.target [Service] Type=simple User=$UNIT_USER -ExecStart=/usr/bin/env python3 $AGENT_TARGET --host $BIND --port $PORT --token-file ${TOKEN_FILE}${TLS_EXEC_ARGS} +ExecStart=/usr/bin/env python3 $AGENT_TARGET --host $BIND --port $PORT --token-file ${TOKEN_FILE}${TLS_EXEC_ARGS}${WRITES_EXEC_ARGS} Restart=on-failure RestartSec=5 Environment=PYTHONUNBUFFERED=1 @@ -677,6 +767,33 @@ SystemCallFilter=@system-service WantedBy=multi-user.target UNIT_FILE + # The helper and its two units land beside the agent's unit and never inside it: the + # re-run guard is the same one, per file — a differing installed helper or unit is a diff + # plus exit 1 unless --force, so a routine redeploy cannot reset a helper somebody fixed by + # hand. The *guard* for the three runs here, before the agent unit guard, and the *writes* + # run after the agent unit write: a refusal does nothing after it — no routectl file, no + # agent unit, no daemon-reload, no enable — and an apply writes both before the one reload. + ROUTECTL_REFUSED=0 + if [ "$WITH_ROUTECTL" -eq 1 ]; then + for _pair in "$ROUTECTL_SOURCE $ROUTECTL_TARGET" \ + "$ROUTECTL_SOCKET_UNIT_SRC $ROUTECTL_SOCKET_UNIT" \ + "$ROUTECTL_SERVICE_UNIT_SRC $ROUTECTL_SERVICE_UNIT"; do + _src="${_pair%% *}" + _dst="${_pair##* }" + [ -f "$_src" ] || { echo "no $_src beside this script" >&2; exit 1; } + if [ -f "$_dst" ] && ! cmp -s "$_dst" "$_src"; then + if [ "$FORCE" -eq 1 ]; then + diff -u "$_dst" "$_src" || true + else + echo "refusing to overwrite $_dst (installed copy differs); run with --force to apply:" + diff -u "$_dst" "$_src" || true + ROUTECTL_REFUSED=1 + break + fi + fi + done + fi + # No silent clobber (phase 17 task 03). $UNIT is the operator's file as much as this # template's output: it gets edited on the host to fix something the template did not know # (driveripper: User=root plus the two capabilities the kernel-7.2 passthrough gate needs), @@ -699,6 +816,22 @@ UNIT_FILE exit 1 fi fi + # Both guards have passed now: the writes are one step. The routectl refusal flag exists + # only so this write can be skipped when a routectl file refused above — but the agent + # unit guard below exits the run, so a routectl refusal alone must stop the run too: + # the refusal is the answer, and a half-apply (agent unit written, routectl not, and no + # reload) is the failure it exists to prevent. + if [ "$ROUTECTL_REFUSED" -eq 1 ]; then + rm -f "$TMP_UNIT" + exit 1 + fi + if [ "$WITH_ROUTECTL" -eq 1 ]; then + install -d -m 0755 "$(dirname "$ROUTECTL_TARGET")" + install -m 0755 "$ROUTECTL_SOURCE" "$ROUTECTL_TARGET" + install -m 0644 "$ROUTECTL_SOCKET_UNIT_SRC" "$ROUTECTL_SOCKET_UNIT" + install -m 0644 "$ROUTECTL_SERVICE_UNIT_SRC" "$ROUTECTL_SERVICE_UNIT" + echo " installed the route-switch helper in $ROUTECTL_TARGET and its two units" + fi cat "$TMP_UNIT" > "$UNIT" rm -f "$TMP_UNIT" @@ -712,6 +845,24 @@ UNIT_FILE journalctl -u "$SERVICE" -n 20 --no-pager >&2 exit 1 fi + if [ "$WITH_ROUTECTL" -eq 1 ]; then + systemctl enable --now dashboards-routectl.socket + if ! systemctl is-active --quiet dashboards-routectl.socket; then + echo " dashboards-routectl.socket failed to start — last lines:" >&2 + journalctl -u dashboards-routectl.socket -n 20 --no-pager >&2 + exit 1 + fi + # The reachability check the install owes: a socket the agent's own uid cannot connect + # to is a feature that fails at 03:00 during an incident, so the check is part of the + # install and its output is printed. $UNIT_USER is the service user (the agent runs as + # root under --as-root, and root may always connect — the check then passes trivially). + if sudo -u "$UNIT_USER" python3 -c 'import socket, sys; s = socket.socket(socket.AF_UNIX); s.connect(sys.argv[1])' "$ROUTECTL_SOCKET_PATH"; then + echo " $UNIT_USER can connect to $ROUTECTL_SOCKET_PATH (the whole grant at the file layer)" + else + echo " $UNIT_USER cannot connect to $ROUTECTL_SOCKET_PATH — the helper is installed but the agent cannot reach it" >&2 + exit 1 + fi + fi if [ -n "$TLS_EXEC_ARGS" ]; then cat < AppContext: @@ -255,8 +264,9 @@ def _endpoint_values(connections: Any) -> list[Any]: def _gateway_summary(payload: Mapping[str, Any] | None, settings: Settings) -> dict[str, Any]: """`/api/summary`'s gateway block: the ingress's totals, or a stub of unknowns. - Additive in phase 29, reshaped by phase 30 task 02 into eleven fields, with `warnings` joined - by task 03 as the twelfth. The page charts + Additive in phase 29, reshaped by phase 30 task 02 into eleven fields, `warnings` joined by + task 03 as the twelfth, and `disabled` — the switch-off tally — by phase 33 task 04 as the + thirteenth. The page charts from `/api/gateway/snapshot`, which has the rows themselves; this block is the **tally** a programmatic consumer (and the overview's one line) reads without walking them. Three arms, and they are three different sentences: @@ -304,6 +314,11 @@ def _gateway_summary(payload: Mapping[str, Any] | None, settings: Settings) -> d "map_targets": None, # the config-integrity tally; `null` until the agent answers, exactly like the row counts "warnings": None, + # the switch-off tally (phase 33 task 04): the number of rows reading `enabled: false` + # — routes the operator's own marker (or the helper's sink) switched off. `null` in the + # unsupported arm, exactly like the other tallies: an agent that never read the configs + # has no route in an off state. + "disabled": None, } if payload is None: return block @@ -322,6 +337,12 @@ def _gateway_summary(payload: Mapping[str, Any] | None, settings: Settings) -> d # malformed counts as zero rather than raising — this module does not own the projection, and a # drift inside one row must not 500 the overview (see `_mappings`). block["warnings"] = len(_mappings(payload.get("warnings"))) + # The tally over the rows' own `enabled` state, not a second reading of the configs: a row + # says `enabled: false` when the agent read the helper's marker (or the sink address) off + # that route's line. `0` is the truth ("nothing switched off"), `null` is the unsupported arm + # above, and a row from an agent older than the marker (no `enabled` key) is counted as on — + # absence of the flag is "not switched off", never "broken". + block["disabled"] = sum(1 for row in connections if row.get("enabled") is False) # Row counts come from the config and are therefore known whenever the agent answered at all — # `0` here is the truth ("this host declares no listener"), not a missing measurement, so it is # the *live* pair below that carries the nulls. @@ -498,9 +519,9 @@ def auth_router() -> APIRouter: every cookie write — a plain `` link that signs the operator out is a CSRF, and one planted on a page they read during an outage is the version that actually hurts. - These are the app's only writes besides `POST /api/ups/command`, and they are writes to + These are the app's only writes besides `POST /api/ups/command` and `POST /api/gateway/route`, and they are writes to **this process** and its SQLite `meta` table, never to an upstream: invariant 1 is about the - fleet, and `tests/test_api.py` pins the two lists separately rather than widening the first. + fleet, and `tests/test_api.py` pins the lists separately rather than widening the first. """ router = APIRouter(prefix="/api/auth", tags=["auth"]) @@ -712,6 +733,11 @@ def common_router() -> APIRouter: "url": _gateway_url(settings), "interval_seconds": settings.gateway_interval, "timeout_seconds": settings.gateway_timeout, + # `GATEWAY_ALLOW_WRITES`, the write's own flag (invariant 1's second + # exception): the page renders the switch's honest arm from it instead of + # assuming (rule 2) — `read_only` stays in the `app` block, where both + # write paths read it. + "writes_enabled": settings.gateway_allow_writes, }, "ups": { "enabled": settings.nut_enabled, @@ -994,9 +1020,10 @@ def _disk_visible(host: Mapping[str, Any]) -> bool: def disks_router() -> APIRouter: """`/api/disks/*` — GETs only, like every other dashboard's surface. - There is deliberately no POST here: the app's one write stays `POST /api/ups/command` - (invariant 1). Nothing about reading a drive's SMART attributes justifies a mutation, and - `smartctl -t` (a self-test) would put real load on a disk something is plugged into. + There is deliberately no POST here: the app's two upstream writes stay `POST /api/ups/command` + and `POST /api/gateway/route` (invariant 1). Nothing about reading a drive's SMART attributes + justifies a mutation, and `smartctl -t` (a self-test) would put real load on a disk something + is plugged into. """ router = APIRouter(prefix="/api/disks", tags=["disks"]) @@ -1071,11 +1098,15 @@ def disks_router() -> APIRouter: # Gateway dashboard (the ingress's pipes) # --------------------------------------------------------------------------- # def gateway_router() -> APIRouter: - """`/api/gateway/*` — GETs only, like every other dashboard's surface. + """`/api/gateway/*` — the two read routes, and the app's second upstream write. - A fifth data page grows no write (invariant 1's shape): reading two config files and a kernel - socket table justifies no mutation, and the app's only route that dials an upstream stays - `POST /api/ups/command`. `tests/test_api.py` enumerates the served verbs to keep that true. + `GET /snapshot` and `GET /history` are like every other dashboard's surface: read-only, and + an unconfigured `GATEWAY_HOST` answers `404 {"configured": false}` before anything that + could open a socket (invariant 3). `POST /route` is the second exception to invariant 1 + (the first is `POST /api/ups/command`), and its ten gates are the design of the page it + serves: nine refusals before a socket opens, then one dial through + `collectors.post_route`. `tests/test_api.py` enumerates the served verbs to keep both + write lists true, and `tests/test_auth_api.py` pins the gate's public surface. """ router = APIRouter(prefix="/api/gateway", tags=[GATEWAY_DASHBOARD]) @@ -1152,9 +1183,275 @@ def gateway_router() -> APIRouter: "series": _history_batch(ctx.store, gateway_source.SOURCE, wanted_hosts, wanted_metrics, span, points)} + @router.post("/route") + async def switch_route(request: Request, + ctx: AppContext = Depends(get_context)) -> Any: + """The app's second and last upstream write: turn one ingress route off or back on. + + The gate order is the design, and it is written as numbered comments because a + reordering is the bug. Gates 1-3 run in the session-gate middleware (`app.py`) before + this body sees anything — the session (a `303` to `/login` for a page, the one JSON + refusal for the rest), the `Origin`/`Sec-Fetch-Site` check on a write, and + `X-CSRF-Token` for this session on a cookie write — and `tests/test_auth_api.py` + pins all three on the wire for this route, the way it does for the UPS one. Gates + 4-10 run here; every one refuses **before the single dial** (`collectors.post_route` + is what the first nine keep off), and each refusal names its gate in its 4xx/429 + sentence and in the one event this call writes, and says nothing about which check + the caller came closest to passing. + + The response is the helper's own answer object, verbatim (a `applied: false` stays + `false` — no success is invented, rule 2), plus `host`, `id`, `action`, `dry_run` and + the `hint` sentences this module owns. + """ + settings = ctx.settings + address = request.client.host if request.client else "unknown" + throttle = ctx.login_throttle + + def event(kind: str, severity: str, message: str, data: dict[str, Any]) -> None: + # One event per call (invariant: the feed is a window, not a printout), under the + # dashboard name, for the host the route lives on. No diff, no file path, no config + # text — the message is a fixed shape and the data carries ids and flags only. + ctx.store.log_event(Event(time.time(), GATEWAY_DASHBOARD, settings.gateway_host, + kind, severity, message, data)) + + def row_text(row: Mapping[str, Any]) -> tuple[str, str]: + out = row.get("out") or {} + selectors = out.get("selectors") or [] + subject = (out.get("name") or (selectors[0] if selectors else None) + or out.get("host") or "?") + port = out.get("port") + target = str(out.get("host") or "?") + (f":{port}" if port else "") + return str(subject), target + + def denied_row(row: Mapping[str, Any], gate: int, name: str, action: str) -> None: + subject, target = row_text(row) + event("route-denied", "error", + f"{action} {subject} → {target}: denied at gate {gate} — {name} (admin)", + {"gate": gate, "id": row.get("id")}) + + # Gate 4 — `DASH_READ_ONLY` beats everything, including the flag below (invariant 1): + # the read-only refusal is read before `GATEWAY_ALLOW_WRITES` is, and the sentence + # names the variable so `grep DASH_READ_ONLY` finds both write paths. + if settings.read_only: + event("route-denied", "error", + "route write denied at gate 4 — read-only (admin)", {"gate": 4}) + raise HTTPException(403, "this dashboard is read-only (DASH_READ_ONLY)") + + # Gate 5 — `GATEWAY_ALLOW_WRITES`, its own flag: a deployment that may test a UPS + # battery is not one that may cut the fleet's only ingress, so the two writes are + # two independent decisions (invariant 1's two exceptions, two sentences). + if not settings.gateway_allow_writes: + event("route-denied", "error", + "route write denied at gate 5 — writes disabled (admin)", {"gate": 5}) + raise HTTPException(403, "route switching is disabled — set GATEWAY_ALLOW_WRITES=1 " + "to enable it") + + # Gate 6 — the write's own throttle key (`route:
`, the third shape on the + # per-process `login_throttle`; the per-process caveat stays a documented gap). The + # answer is `refusal()`'s, like the password change's, because it is the one 4xx + # that carries `Retry-After`, and `retry_after` is read-only: a refused request does + # not extend its own block by asking. + key = f"route:{address}" + blocked = throttle.retry_after(key) + if blocked > 0: + seconds = math.ceil(blocked) + event("route-denied", "error", + "route write denied at gate 6 — throttled (admin)", {"gate": 6}) + return refusal(429, f"too many route writes — try again in {seconds}s", + headers={"Retry-After": str(seconds)}) + + # Gate 7 — the body, parsed by hand for the same reason the login body is: the 422 + # must be *ours* — one fixed sentence naming the contract — not a framework's + # per-field schema errors a caller has to diff to fix the request. + try: + body = await request.json() + except Exception as exc: # not JSON at all — the same refusal as a missing key + event("route-denied", "error", + "route write denied at gate 7 — bad body (admin)", {"gate": 7}) + raise HTTPException(422, "send a JSON body with `id`, `action` and `danger`") from exc + if not isinstance(body, dict): + event("route-denied", "error", + "route write denied at gate 7 — bad body (admin)", {"gate": 7}) + raise HTTPException(422, "send a JSON body with `id`, `action` and `danger`") + route_id = body.get("id") + action = body.get("action") + dry_run = body.get("dry_run", False) + extra = sorted(set(body) - {"id", "action", "dry_run", "danger"}) + if extra: + event("route-denied", "error", + "route write denied at gate 7 — bad body (admin)", {"gate": 7}) + raise HTTPException(422, + f"unknown key {extra[0]!r} — send only `id`, `action`, " + f"`dry_run` and `danger`") + if not isinstance(route_id, str) or not route_id: + event("route-denied", "error", + "route write denied at gate 7 — bad body (admin)", {"gate": 7}) + raise HTTPException(422, "`id` must be a non-empty string") + if len(route_id) > 256: + event("route-denied", "error", + "route write denied at gate 7 — bad body (admin)", {"gate": 7}) + raise HTTPException(422, "`id` is longer than 256 characters") + if action not in ("disable", "enable"): + event("route-denied", "error", + "route write denied at gate 7 — bad body (admin)", {"gate": 7}) + raise HTTPException(422, "`action` must be `disable` or `enable`") + if not isinstance(dry_run, bool): + event("route-denied", "error", + "route write denied at gate 7 — bad body (admin)", {"gate": 7}) + raise HTTPException(422, "`dry_run` must be a boolean") + + # Gate 8 — the id's state in the *current* payload. A route the payload has not + # published is not a route this app may flip: an unknown id is a 409, an id already + # in the requested state is a 409 `{changed: false}`, and a gateway that has never + # answered is a 409 too — never a guess about a route it cannot see. Both spellings + # are accepted: with or without the `#off` suffix; the helper's wire id is the base. + job = ctx.hub.jobs.get(gateway_source.SOURCE) + if job is None or job.state.payload is None: + event("route-denied", "error", + "route write denied at gate 8 — never answered (admin)", + {"gate": 8, "id": route_id}) + raise HTTPException(409, "the gateway has not answered yet — no route is visible") + payload = job.state.payload + base = route_id[:-4] if route_id.endswith("#off") else route_id + rows = _mappings(payload.get("connections")) + row_on = next((row for row in rows + if row.get("id") == base and row.get("enabled") is True), None) + row_off = next((row for row in rows + if row.get("id") == base + "#off" and row.get("enabled") is False), + None) + row, already = (row_on, row_off) if action == "disable" else (row_off, row_on) + if row is None: + if already is not None: + denied_row(already, 8, + f"already {'off' if action == 'disable' else 'on'}", action) + raise HTTPException(409, detail={"changed": False}) + event("route-denied", "error", + "route write denied at gate 8 — unknown id (admin)", + {"gate": 8, "id": route_id}) + raise HTTPException(409, f"unknown id {route_id!r} — not in the current payload") + + # Gate 9 — freshness: "do not rewrite a config you cannot currently read end to + # end". All six facts are in the job state and the payload; the sentence and the + # event name the one that refused. + now = time.time() + host_job = ctx.hub.jobs.get(f"host:{settings.gateway_host}") + fresh_name: str | None = None + if not job.state.ok: + fresh_name = "the gateway job is not ok" + elif job.state.last_ok is None: + fresh_name = "the gateway has not answered" + elif now - job.state.last_ok > 3 * job.interval: + fresh_name = "the gateway is stale" + elif host_job is None or not host_job.state.ok: + fresh_name = "the host job is not ok" + else: + raw_sections = payload.get("sections") + sections = raw_sections if isinstance(raw_sections, Mapping) else {} + if sections.get("haproxy") is not True: + fresh_name = "the haproxy section is not read" + elif sections.get("nginx") is not True: + fresh_name = "the nginx section is not read" + elif not _version_at_least(payload.get("agent_version"), (1, 4, 0)): + fresh_name = "the agent predates the marker read" + if fresh_name is not None: + denied_row(row, 9, fresh_name, action) + raise HTTPException(409, f"{fresh_name} — do not rewrite a config you cannot " + f"currently read end to end") + + # Gate 10 — the typed `danger: "DANGER"`, for **both** verbs: switching a route off + # and switching one back on are both writes to the fleet's only ingress, and the + # word is the confirmation (the UPS `confirm="DANGER"` sentence's sibling). A dry + # run is the one body that does not carry it: the helper validates and answers with + # a diff without writing a byte, so there is nothing for the word to confirm — the + # preview the page asks for before the operator types it (phase 33 task 06). + if not dry_run and body.get("danger") != "DANGER": + denied_row(row, 10, "danger confirmation missing", action) + raise HTTPException(422, "switching an ingress route can cut the fleet's only " + 'ingress: send danger="DANGER" to prove you read the ' + "warning") + + # The one dial — what gates 1-10 exist to keep off until they pass. `post_route` + # dials the same web server the reads do: the per-dial `AgentTls` client, the + # revocation check before any socket opens, no `Authorization` on the certificate + # path, the write's own 35 s budget (`GATEWAY_WRITE_TIMEOUT_SECONDS`, longer than + # the agent's 30 s peer timeout on purpose), and the redacted URL in every error. + client = ctx.client + if client is None: + # A context assembled outside the lifespan has no dialer (the `auth` `None` + # branch's sibling): it answers the 503, not a 500 with a traceback. + raise HTTPException(503, "the server's dialer is not ready") + target = next((item for item in settings.host_targets + if item.name == settings.gateway_host), None) + if target is None: + # A context assembled without the source it claims to write for: answer the 503, + # not a 500 with a traceback. (The job's existence makes this unreachable in a + # live app — the source is configured exactly when its host is in `host_targets` + # — but a hand-built context is what the 503 is for.) + raise HTTPException(503, "the server's gateway is not configured") + tls = (agent_tls_map(ctx.certs, ctx.store, (target,)).get(target.name) + if ctx.certs is not None else None) + subject, target_text = row_text(row) + try: + answer = await post_route(client, target, {"id": base, "action": action, + "dry_run": dry_run}, + timeout=settings.gateway_write_timeout, tls=tls) + except UpstreamError as exc: + event("route-denied", "error", + f"{action} {subject} → {target_text}: refused at the dial — {exc} (admin)", + {"id": base, "action": action, "dry_run": dry_run}) + throttle.failure(key) + raise + applied = answer.get("applied") is True + + # One event per call: a change is `warn` under its own kind, a refusal (the helper's + # `applied: false`, or the dial's own error above) is `error` under `route-denied`. + # A dry run is neither — it changed nothing and refused nothing, and a feed line + # reading `applied=false` for a preview would say the helper refused a write it + # was never asked to make. + triple = (f"applied={str(applied).lower()} " + f"validated={str(answer.get('validated') is True).lower()} " + f"reloaded={str(answer.get('reloaded') is True).lower()}") + if not dry_run: + event("route-disabled" if (applied and action == "disable") + else ("route-enabled" if applied else "route-denied"), + "warn" if applied else "error", + f"{action} {subject} → {target_text}: {triple} (admin)", + {"id": base, "action": action, "dry_run": dry_run, + "applied": applied, + "validated": answer.get("validated"), + "reloaded": answer.get("reloaded")}) + if dry_run: + hint = "a dry run — nothing was written; the diff above is what the change would do" + elif applied: + # The attempt completed: the backoff resets. A helper-side `applied: false` + # records neither a success nor a failure — the caller did nothing wrong and + # the helper said so; a failed dial charged one above. + throttle.success(key) + hint = (f"the helper reloaded the proxy — the switch shows on the next gateway " + f"poll (≤ {settings.gateway_interval:.0f}s)") + else: + hint = ("no byte of config was written by this call — the helper's word above " + "says why") + return {**answer, "host": settings.gateway_host, "id": route_id, + "action": action, "dry_run": dry_run, "hint": hint} + return router +def _version_at_least(version: Any, minimum: tuple[int, ...]) -> bool: + """`"1.4.1" >= (1, 4, 0)`. A version this cannot read is *below* the floor, never above: + a payload without an `agent_version` cannot prove it reads the marker, and a write aimed + at a config the dashboard cannot read is gate 9's whole point.""" + if not isinstance(version, str): + return False + chunks = version.strip().split(".") + if not chunks or any(not chunk.isdigit() for chunk in chunks): + return False + got = tuple(int(chunk) for chunk in chunks) + got = got + (0,) * (len(minimum) - len(got)) + return got[:len(minimum)] >= minimum + + # --------------------------------------------------------------------------- # # LLM dashboard # --------------------------------------------------------------------------- # @@ -1245,7 +1542,7 @@ def ups_router() -> APIRouter: @router.post("/command") async def send_command(ctx: AppContext = Depends(get_context), payload: dict[str, Any] = Body(...)) -> dict[str, Any]: - """The one write endpoint in the application. Deliberately loud about it. + """One of the application's two upstream write endpoints. Deliberately loud about it. Gated on: NUT_ALLOW_ACTIONS, not DASH_READ_ONLY, credentials that carry `actions` in upsd.users, the command being advertised by the driver for that UPS, and — for diff --git a/src/dashboards/app.py b/src/dashboards/app.py index a629cf1..355861c 100644 --- a/src/dashboards/app.py +++ b/src/dashboards/app.py @@ -196,7 +196,7 @@ def build_context(settings: Settings, client: httpx.AsyncClient) -> tuple[ hub, nut_monitor = build_hub(settings, store, client, tls=agent_tls_map(material, store, settings.host_targets)) ctx = AppContext(settings=settings, store=store, hub=hub, nut_monitor=nut_monitor, - started_at=time.time(), certs=material) + started_at=time.time(), certs=material, client=client) return ctx, store, hub.start @@ -340,7 +340,7 @@ def create_app(settings: Settings | None = None, store: SampleStore | None = Non settings.host_targets)) ctx = AppContext(settings=settings, store=store, hub=hub, nut_monitor=nut_monitor, started_at=time.time(), - certs=material) + certs=material, client=client) start = hub.start else: ctx, owned_store, start = build_context(settings, client) diff --git a/src/dashboards/collectors.py b/src/dashboards/collectors.py index 671bbc9..0529e5f 100644 --- a/src/dashboards/collectors.py +++ b/src/dashboards/collectors.py @@ -1,8 +1,9 @@ """Collectors: the fetch half of each poll. Parse + store live in the Job wrapper. Every collector returns a `Result`; every failure is an exception the Job turns into a -red dot plus an event. Nothing here writes to any upstream — with one deliberate exception, -`run_ups_command`, which is the app's only mutating path and is gated in three places. +red dot plus an event. Nothing here writes to any upstream — with **two** deliberate +exceptions, and both are gated: `run_ups_command` (the UPS battery tests) and +`post_route` (turning one ingress route off or back on, phase 33 task 05). """ from __future__ import annotations @@ -235,6 +236,63 @@ async def fetch_snapshot(client: httpx.AsyncClient, target: Target, return body +async def post_route(client: httpx.AsyncClient, target: Target, body: Mapping[str, Any], + *, timeout: float | None = None, + tls: AgentTls | None = None) -> dict[str, Any]: + """One `POST /route` to the agent — the app's second and last upstream write. + + The dial is the snapshot dial, because it is the same web server: the per-dial + :class:`AgentTls` client, the revocation check **before any socket opens** (a host the + operator struck out is not a host a write may touch, and the refusal costs no network + round trip), and the **redacted** URL in every message, because an `UpstreamError` string + lands in the event table and the browser reads it (invariant 6). There is one deliberate + difference from `fetch_snapshot`, and it is the point: **`Authorization` is never sent on + the certificate path** — the bundle is the credential there, and a token alongside it is a + second opinion the agent's write handler refuses; the token is sent only on the plain + token path, where a 401 from it is about the token. The response is the helper's own + object, **verbatim** — `applied: false` comes back `false`, no success is invented (rule + 2), and a non-2xx answer is an :class:`UpstreamError` carrying the agent's own word. + """ + if tls is not None and tls.revoked: + raise UpstreamError(f"{target.name}: agent certificate revoked") + request_timeout: Any = httpx.USE_CLIENT_DEFAULT if timeout is None else timeout + url = f"{target.url}/route" + shown = f"{target.display_url}/route" + secure = tls if (tls is not None and target.url.startswith(HTTPS_SCHEME)) else None + # The certificate path sends no `Authorization`: a token next to a certificate is the + # second opinion the write refuses. The token path sends the header exactly as the read + # does — and the agent's answer to a token-mode write is its own, carried by `refusal_detail`. + headers: dict[str, str] = {} if secure is not None else dict(target.headers) + try: + if secure is not None: + async with secure.client(shared=client) as scoped: + response = await scoped.post(url, json=dict(body), headers=headers, + timeout=request_timeout) + else: + response = await client.post(url, json=dict(body), headers=headers, + timeout=request_timeout) + except httpx.TimeoutException as exc: + raise UpstreamError(f"{target.name}: {shown} timed out", status=504) from exc + except (httpx.HTTPError, OSError) as exc: + raise UpstreamError(f"{target.name}: {shown} unreachable ({type(exc).__name__})") from exc + if response.status_code == 401: + raise UpstreamError(f"{target.name}: agent rejected the token (401) — check the " + f"userinfo token in HOST_TARGETS") + if response.status_code >= 400: + # a plain 403 here is the agent's own write refusal ("writes require a certificate", + # "writes are not enabled on this agent"): its word is the answer, carried verbatim + raise UpstreamError(f"{target.name}: {shown} -> HTTP {response.status_code} " + f"({refusal_detail(response)})") + try: + answer = response.json() + except ValueError as exc: + raise UpstreamError(f"{target.name}: route helper answered something unreadable " + f"({response.text[:80]!r})") from exc + if not isinstance(answer, dict): + raise UpstreamError(f"{target.name}: route helper answered a non-object") + return answer + + async def fetch_host_snapshot(client: httpx.AsyncClient, target: Target, host_collector: HostCollector, *, tls: AgentTls | None = None) -> Result: diff --git a/src/dashboards/config.py b/src/dashboards/config.py index ec1818e..667c407 100644 --- a/src/dashboards/config.py +++ b/src/dashboards/config.py @@ -203,8 +203,21 @@ class Settings: # plain agent fetch's (the snapshot handler does no work for these sections beyond reading # two configs and two /proc tables), with headroom because the same handler may be mid- # smartctl pass for a different page. - gateway_interval: float = _num("GATEWAY_INTERVAL_SECONDS", 30) - gateway_timeout: float = _num("GATEWAY_TIMEOUT_SECONDS", 10) + gateway_interval: float = _num("GATEWAY_INTERVAL_SECONDS", 30.0) + gateway_timeout: float = _num("GATEWAY_TIMEOUT_SECONDS", 10.0) + # the second and last upstream write (phase 33 task 05): turning one ingress route + # off or back on. Off by default, and its **own** flag rather than reusing + # `NUT_ALLOW_ACTIONS`: a deployment that may test a UPS battery is not one that may + # cut the fleet's only ingress, so the two writes are two independent decisions, each + # gated by its own sentence at its own endpoint (invariant 1). Same truthy rule as + # `NUT_ALLOW_ACTIONS`. + gateway_allow_writes: bool = _flag("GATEWAY_ALLOW_WRITES") + # the write's budget, on purpose a different number from `gateway_timeout`: the dial + # has to fit the agent's own 30 s peer timeout **inside** a helper that re-reads the + # file, runs the proxy's own checker and reloads one unit, so 35 s makes the + # dashboard wait for the helper's own answer instead of timing out first and leaving + # the operator wondering whether the reload happened + gateway_write_timeout: float = _num("GATEWAY_WRITE_TIMEOUT_SECONDS", 35.0) # --- llm dashboard ------------------------------------------------------------ gufo_url: str = _base_url("GUFO_BASE_URL") @@ -228,8 +241,9 @@ class Settings: nut_password: str = _env("NUT_PASSWORD") nut_interval: float = _num("NUT_INTERVAL_SECONDS", 5) nut_timeout: float = _num("NUT_TIMEOUT_SECONDS", 4) - # Instant commands (battery tests, beeper, calibration) are the only write path in - # the app. Off unless BOTH the env flag is set and DASH_READ_ONLY is not. + # Instant commands (battery tests, beeper, calibration) are the only write path to a + # **NUT** — the app's second upstream write is the gateway route switch, a different + # flag, a different dial. Off unless BOTH the env flag is set and DASH_READ_ONLY is not. nut_allow_actions: bool = _flag("NUT_ALLOW_ACTIONS") # --- history ------------------------------------------------------------------ diff --git a/src/dashboards/sources/gateway.py b/src/dashboards/sources/gateway.py index cd07c98..f458c38 100644 --- a/src/dashboards/sources/gateway.py +++ b/src/dashboards/sources/gateway.py @@ -77,6 +77,20 @@ dialled). That is the same traffic seen from both sides, which is exactly what t for — and it is why the loop `continue`s the moment a socket is counted: one row may never be drawn down two pipes. +A line's `enabled` field (phase 33 task 04) is the route's **config state**: `true` when the +config declares the route and it is on, `false` when the operator's own helper switched it off — +the `#off ` marker on the route line (haproxy `server`, nginx `proxy_pass`, an `upstream` +member, a `map` entry), read by the agent and published as `disabled`, or the helper's sink +address in a `map` entry's value, which is the off state the helper left before the agent knew +markers. Three facts, never conflated: a route **never declared** is absent from the payload, a +route **declared and on** is an `enabled: true` line, a route **declared and off** is an +`enabled: false` line that is still drawn and still counted — the kernel does not know markers, so +a switched-off route with traffic in it is drawn as an off line carrying a number. An arm that +never read the config (the `stub`, the `predates` arm) publishes **no lines at all**, which is what +keeps `false` out of a payload nobody answered: a `false` is always a claim the config made. No +count is ever *derived* from `enabled` here: the join and the gauges read sockets, and the field +is the state the page prints and the write path targets. + The connection graph -------------------- @@ -100,10 +114,16 @@ files express a route: one connection per entry with a target, selector the entry's own key (the SNI domain, or the literal `default`, which is the config's word for "everything nobody mapped", not a guess). -**Merge rule.** Identical `(source, in_port, host, port)` is **one** connection whose `selectors` -are the union in first-seen order — one vhost's two names arriving at one pool is one line, not -two. Never merge across `in_port`: `:4443 → keycloak:8080` and `:8443 → keycloak:8080` are two -pipes on two socket spaces, which is the `minecraft` 25565/25566 lesson told from the other side. +**Merge rule.** Identical `(source, in_port, host, port, enabled)` is **one** connection whose +`selectors` are the union in first-seen order — one vhost's two names arriving at one pool is one +line, not two. `enabled` is in the key on purpose: a claim to one endpoint that is on in one +config and off in another is **two lines** — one of these selectors is off is two facts, not one +row, and a silent union would print the switched-off route as live and the live one as off, at +once. The off line's id carries a `#off` suffix (on the line's `id`, where the page and the write +path both key on it), which is what makes the two halves of such a split distinct rows; the suffix +is the state, and the state is the only reason an id ever carries one. Never merge across +`in_port`: `:4443 → keycloak:8080` and `:8443 → keycloak:8080` are two pipes on two socket +spaces, which is the `minecraft` 25565/25566 lesson told from the other side. **Two refusals**, both rule 2, both silent (a warning needs a payload that can tell broken from merely unreadable, and neither of these can): @@ -131,17 +151,21 @@ What this module warns about With everything outside the two config files ignored, the only warning left worth an operator's attention is a config that **does not connect to itself**: a route naming a backend nobody declares, a backend nothing routes to, a `proxy_pass $var` with no `map` behind it. `warnings()` -answers the seven such kinds, each derived from config text alone and each actionable: +answers the seven such kinds, each derived from config text alone and each actionable — and since +phase 33 task 04 two of them are **marker-aware**: a shape the operator's own switch caused (a +line carrying the helper's marker) is a state, not a fault, while the same shape with the line +hand-edited is still a fault, and the checks are written so that stays true without a second code +path: | kind | condition | |------|-----------| | `haproxy_route_dangling` | a frontend's `default_backend`/`use_backend` names no `backend` block | | `haproxy_backend_orphan` | a backend with `server` lines that no frontend routes to | -| `haproxy_backend_empty` | a `backend` block with no `server` line at all | +| `haproxy_backend_empty` | a `backend` block with no `server` line **in the document** at all — a line the helper switched off stays in the config, `disabled: true`, so it is not empty and gets no warning (its state is the line's `enabled: false`); this kind is the hand-deleted line | | `nginx_server_unlistened` | a `server` block with names and/or `proxy_pass` and no `listen` | | `nginx_pool_undeclared` | `proxy_pass http://` with no port and no `upstream` of that name | | `nginx_variable_unmapped` | `proxy_pass $var` with no `map` block defining `$var` | -| `nginx_map_route_missing` | an entry of a **referenced** `map` whose target is null | +| `nginx_map_route_missing` | a `map` entry's value is neither an address nor a variable — **except** a line the helper switched off (marked, or pointing at the helper's sink block), which is a state, not this fault; the hand-edited shape still reaches it | **Three things it refuses to warn about**, each because the payload cannot tell broken from merely unreadable (and a warning is a claim, not a hunch): @@ -444,12 +468,17 @@ def haproxy_frontends(section: Mapping[str, Any] | None) -> list[dict[str, Any]] def haproxy_targets(section: Mapping[str, Any] | None) -> list[dict[str, Any]]: - """Every `server` line's `{name, host, port, backend}` — the outbound half of a haproxy config. + """Every `server` line's `{name, host, port, backend, disabled}` — the outbound half of a + haproxy config. `backend` is the name of the `backend` block the `server` was declared in, and it is the only reason the join in `config_connections()` is possible at all: the frontend side speaks backend names (`default_backend` / `use_backend`) and the endpoint side speaks hosts, so without the - declaring block's name a `server` line is an endpoint nobody routes to. + declaring block's name a `server` line is an endpoint nobody routes to. `disabled` is the + agent's read of the helper's marker on the line (phase 33): `true` when the operator + switched the route off and the line stayed in the document, `false` otherwise, `True` + **never** for a line the agent did not report the flag on (an older agent's rows stay on — + absence of the flag is "not switched off", not "broken"). """ rows: list[dict[str, Any]] = [] for backend in _mappings((section or {}).get("backends")): @@ -459,7 +488,8 @@ def haproxy_targets(section: Mapping[str, Any] | None) -> list[dict[str, Any]]: if not host: continue rows.append({"name": _text(server.get("name")), "host": host, - "port": _port(server.get("port")), "backend": declared_in}) + "port": _port(server.get("port")), "backend": declared_in, + "disabled": server.get("disabled") is True}) return rows @@ -560,6 +590,59 @@ def _maps_by_variable(section: Mapping[str, Any] | None) -> dict[str, list[Mappi return found +def _loopback(value: str) -> bool: + """Is this listen address the machine's own loopback (`127.0.0.0/8` or `::1`)? + + `ipaddress` parses text and never dials, the `_is_ip_literal` rule; a name is not a + loopback, and a non-loopback literal is not one either. + """ + try: + return ipaddress.ip_address(value).is_loopback + except ValueError: + return False + + +def sink_endpoint(nginx: Mapping[str, Any] | None) -> tuple[str, int] | None: + """The helper's catch-all stream server, recognised from this payload's own text. + + The root helper's `off` state for a route it switched is "the entry's value is the sink + address" — a `server` block that listens on loopback, carries **no** `server_names` and + proxies **nowhere** (its whole job is to answer the switched-off route with a 502). The + block, not a port constant, is what this looks for: the port is the helper's to choose, and + a vhost that legitimately `proxy_pass`es to `127.0.0.1:` is not a switch. `None` + when the config has no such block, which is every config the helper has not run on. + """ + for server in _mappings((nginx or {}).get("servers")): + if _names(server.get("server_names")): + continue + if _mappings(server.get("proxy_passes")): + continue + for listen in _mappings(server.get("listen")): + addr = _text(listen.get("addr")) + if addr is not None and _loopback(addr): + port = _port(listen.get("port")) + if port is not None: + return addr, port + return None + + +def _map_entry_off(entry: Mapping[str, Any], sink: tuple[str, int] | None) -> bool: + """Did this map entry's own line say the route is off? + + The agent's `disabled` flag (a line carrying the helper's marker, read by the agent's own + `split_marker`), or a target that **is** the helper's sink block — the state the helper left + before the agent knew markers. An address alone, without the block, is not off (the + `sink_endpoint` refusal, in one call): a map that legitimately lands on loopback is routing, + not a switch. + """ + if entry.get("disabled") is True: + return True + if sink is None: + return False + target = _target_of(entry.get("target")) + return target is not None and target == sink + + def _pools(section: Mapping[str, Any] | None) -> dict[str, list[Mapping[str, Any]]]: """The `upstream` blocks by name, each mapped to its `server` rows, in config order.""" found: dict[str, list[Mapping[str, Any]]] = {} @@ -572,7 +655,7 @@ def _pools(section: Mapping[str, Any] | None) -> dict[str, list[Mapping[str, Any def _connection(source: str, in_port: int, host: str, out_port: int | None, name: str | None, - kind: str, selectors: Sequence[str]) -> dict[str, Any]: + kind: str, selectors: Sequence[str], *, enabled: bool) -> dict[str, Any]: """One line of the graph: an inbound port, an outbound endpoint, and the config's word for it. Three of these fields are set later, and each says so by being `None` here: @@ -582,6 +665,16 @@ def _connection(source: str, in_port: int, host: str, out_port: int | None, name number, so a reader never mistakes two lines for twice the traffic. * `metric` — the gauge this line publishes. + `enabled` is the line's **config state** (phase 33 task 04): `true` = the config declares the + route and it is on, `false` = the config declares it and the operator switched it off (the + helper's marker on the line, or the helper's sink address in its value). The line is drawn + either way — "declared, off" is a different fact from "never declared" (rule 2) — and an arm + that never read the config publishes **no lines at all** (`stub` / the predates arm carry + `connections: []`), which is what keeps `false` out of an arm nobody answered: a `false` is + always a claim the config made. A merge that would mix states splits on this flag instead + (the merge rule in the module docstring), and the `#off` suffix on the id is what keeps the + two halves of such a split distinct rows when the page and the write path both key on it. + `apply_resolution` then widens all three on one question: can the kernel name this line's far end? An address the config wrote can, and so can a name the gateway resolved on the box that dials it. A name nobody resolved never can, and keeps `connections: null`, `metric: null` and @@ -593,8 +686,9 @@ def _connection(source: str, in_port: int, host: str, out_port: int | None, name address = _is_ip_literal(host) joinable = address and isinstance(out_port, int) and not isinstance(out_port, bool) return { - "id": f"{source}:{in_port}->{suffix}", + "id": f"{source}:{in_port}->{suffix}" + ("" if enabled else "#off"), "source": source, + "enabled": enabled, "in": {"port": in_port}, "out": {"host": host, "port": out_port, "name": name, "kind": kind, "selectors": list(selectors), "address": address}, @@ -609,19 +703,27 @@ def config_connections(haproxy: Mapping[str, Any] | None, """Every connection the two configs declare, in config order. Pure, clock-free, resolver-free. The four derivations, the merge rule and the two refusals are stated in the module docstring - ("The connection graph"); this is that text made executable. What it deliberately does **not** - do is join a socket: a connection says the config routes here, which is a fact readable from - a file, and never that a connection is open now, which is not. + ("The connection graph"); this is that text made executable, with the phase-33 `enabled` + state on every line (the flag read of the helper's marker, per arm: the `server` line's own + `disabled` for haproxy and the pool members, the `proxy_pass` line's own for a vhost, and + `_map_entry_off` — the flag **or** the sink block — for a map entry). What it deliberately + does **not** do is join a socket: a connection says the config routes here, which is a fact + readable from a file, and never that a connection is open now, which is not. """ - claims: dict[tuple[str, int, str, int | None], dict[str, Any]] = {} + claims: dict[tuple[str, int, str, int | None, bool], dict[str, Any]] = {} def claim(source: str, in_port: int, host: str, out_port: int | None, *, name: str | None, - kind: str, selectors: Sequence[str]) -> None: - """One line per `(source, in_port, host, out_port)`; selectors union, first-seen order.""" - key = (source, in_port, host, out_port) + kind: str, selectors: Sequence[str], enabled: bool) -> None: + """One line per `(source, in_port, host, out_port, enabled)`; selectors union, first-seen + order. `enabled` is in the key on purpose: two claims to one endpoint, one on and one off, + are two facts (the module docstring's merge rule), so the merge splits into one line per + state rather than unioning a `false` into an `on` line — a silent union would print the + switched-off route as live and the live one as off, at once.""" + key = (source, in_port, host, out_port, enabled) seen = claims.get(key) if seen is None: - claims[key] = _connection(source, in_port, host, out_port, name, kind, selectors) + claims[key] = _connection(source, in_port, host, out_port, name, kind, selectors, + enabled=enabled) return merged = seen["out"]["selectors"] for selector in selectors: @@ -641,8 +743,10 @@ def config_connections(haproxy: Mapping[str, Any] | None, for route in frontend["routes"]: for target in servers_by_backend.get(route, ()): claim("haproxy", int(frontend["port"]), target["host"], target["port"], - name=target["name"], kind=KIND_BACKEND, selectors=(route,)) + name=target["name"], kind=KIND_BACKEND, selectors=(route,), + enabled=not target["disabled"]) + sink = sink_endpoint(nginx) maps = _maps_by_variable(nginx) pools = _pools(nginx) for block in nginx_server_blocks(nginx): @@ -658,16 +762,19 @@ def config_connections(haproxy: Mapping[str, Any] | None, continue for port in block["ports"]: claim("nginx", port, target[0], target[1], name=None, kind=KIND_MAP, - selectors=(key,)) + selectors=(key,), enabled=not _map_entry_off(entry, sink)) continue target = _target_of(proxy.get("target")) if target is None: continue # refusal: neither a target nor a variable names an endpoint + # the `proxy_pass` line's own flag: a marked line stays in the document (the agent + # reports it `disabled: true`), so this is the arm's state, not the block's + proxy_enabled = proxy.get("disabled") is not True if target[1] is not None: # rule 2: a concrete `host:port`, one line per listen port of this block for port in block["ports"]: claim("nginx", port, target[0], target[1], name=None, - kind=KIND_SERVER_NAME, selectors=block["names"]) + kind=KIND_SERVER_NAME, selectors=block["names"], enabled=proxy_enabled) continue # rule 3: a name with no port is an `upstream` block, and no block of that name is # no endpoint — the payload cannot say where the traffic lands, so it claims nothing @@ -678,7 +785,8 @@ def config_connections(haproxy: Mapping[str, Any] | None, continue for port in block["ports"]: claim("nginx", port, host, _port(member.get("port")), name=None, - kind=KIND_POOL, selectors=(target[0],)) + kind=KIND_POOL, selectors=(target[0],), + enabled=proxy_enabled and member.get("disabled") is not True) return list(claims.values()) @@ -757,8 +865,12 @@ def haproxy_warnings(haproxy: Mapping[str, Any] | None) -> list[dict[str, str]]: continue servers = _mappings(backend.get("servers")) if not servers: - # an empty backend routes nowhere whichever frontend names it, so this is the only - # thing said about it: `haproxy_backend_orphan` needs a server to have anywhere to go + # `servers` is the lines **in the document**, and that is the whole of the marker + # decision (phase 33 task 04): the helper's marker leaves a switched-off line in the + # config and the agent reports it `disabled: true`, so a backend whose only server + # the operator turned off is *not* empty here and gets no warning — its state is the + # line's `enabled: false`. This arm is the **hand-deleted** line: the config no + # longer says anything, and that is the fault. found.append(_warning( WARN_BACKEND_EMPTY, name, f'backend "{name}" declares no server — it routes nowhere')) @@ -844,6 +956,12 @@ def nginx_warnings(nginx: Mapping[str, Any] | None) -> list[dict[str, str]]: continue # refusal: an unreferenced map is not a routing table bare = variable.lstrip("$") for entry in _mappings(block.get("entries")): + # a line carrying the helper's marker is the state the operator wrote, not a fault + # (phase 33 task 04): its value is the sink address or nothing at all, and both are + # the switch. The same shape with the line hand-edited (value deleted, no marker) + # falls through to the check below and still warns. + if entry.get("disabled") is True: + continue if isinstance(entry.get("target"), Mapping): continue key = entry.get("key") diff --git a/src/dashboards/sources/nut.py b/src/dashboards/sources/nut.py index b84fff7..41c5865 100644 --- a/src/dashboards/sources/nut.py +++ b/src/dashboards/sources/nut.py @@ -737,7 +737,8 @@ def _duration(seconds: float) -> str: # --------------------------------------------------------------------------- # -# Instant commands (the app's only write path) +# Instant commands (the app's only write to a NUT; the gateway route switch is the other +# upstream write, on its own flag) # --------------------------------------------------------------------------- # RISK_SAFE = "safe" RISK_CAUTION = "caution" diff --git a/src/dashboards/web/pages/gateway.html b/src/dashboards/web/pages/gateway.html index 4f5e55e..7582023 100644 --- a/src/dashboards/web/pages/gateway.html +++ b/src/dashboards/web/pages/gateway.html @@ -198,8 +198,9 @@ here is coloured by a verdict: a busy ingress is the job.

a dash means the gateway could not resolve that name in this reading — hover a counted line for the address it matched and when that answer was taken.

-

the switch shows what the config says; toggling a route needs a config write - path and is not wired up yet.

+

the switch shows what the config says; with `GATEWAY_ALLOW_WRITES=1` set on + the dashboard, activating it switches that route in its config file — otherwise it is a + reading that does nothing, and its title says so.

the filter box hides the routes whose own labels do not contain what you typed — domain, server_name, map key, endpoint or port. It asks for nothing: the tallies, the warnings and the charts above it are still the whole ingress, not the filtered @@ -207,6 +208,32 @@ + +

+
+ danger +

Switch this route?

+
+
+

+

+

+

+    

Type DANGER to apply:

+

+ +
+
+ + +
+
+ diff --git a/src/dashboards/web/static/dash.css b/src/dashboards/web/static/dash.css index e35af96..d5e3647 100644 --- a/src/dashboards/web/static/dash.css +++ b/src/dashboards/web/static/dash.css @@ -1403,19 +1403,16 @@ table.data td.san { white-space: normal; } .gate-link.on .gate-elbow { border-top-color: var(--gate-on); } .gate-link.off .gate-elbow { border-top-color: var(--gate-off); } .gate-switch-slot { display: inline-flex; align-items: center; flex: none; } -/* The switch is a **reading** shaped like a control, and it is the shape the operator sketched: a - * pill with a knob, green when the route is in the config. It is keyed on `[aria-checked]` and - * **never** on `:enabled`, because a switch this app cannot wire (invariant 1: nothing here writes - * to an upstream) must still look green while it is on — styling `:enabled` would paint every one - * of them red by accident. `cursor: not-allowed` says what a hover would otherwise promise, and - * there is no `:hover` rule and no listener anywhere in `page-gateway.js`. `aria-disabled="true"` - * rather than `disabled`, which would drop a real `