phase: 31_gateway_route_readability
**Phase 31 verification complete — all 5 tasks already in `complete/`; this was the final verification pass.**
Verified / changed
- Re-ran every phase-31 artifact live and green: `verify-31-page-only.py` 6 PASS/0 FAIL; browser pass `31-route-readability.cjs` **64 PASS / 0 FAIL** (exit 0); `27-task02-mobile-audit.cjs` **371 PASS / 0 FAIL / 0 FINDING** (exit 0); rebuilt `data/logs/31-route-readability.out` as the 4-part transcript.
- Added the 4 degradation-arm screenshots the criterion asked for but the pass lacked (`31-gateway-{unconfigured,unreachable,no-nginx,sockets-quiet}.png`) — 12 frames now, one per arm.
- Grep criteria: no gateway connector rule uses `border-strong`; no verdict token in the gateway CSS section or `page-gateway.js` beyond the `alerts.py`-owned `.warn`; exactly 4 `addEventListener` calls (none for the switch); `routeInConfig()` returns `true`; **no `.py` under `src/dashboards/` changed**.
Tests / lint (all 100% green)
- `uv run pytest` → **1796 passed, 16 skipped**; `--cov` → **98.94%** (floor 94); `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; `bash .agents/validate.sh` → **exit 0**.
- `uv run pytest -m live --live` → **12 passed / 0 failed**; `./scripts/dev-server.sh status` → all **10 sources ok**.
Completion criteria: colour (33/33 rows every segment `--gate-on`, dark `rgb(63,185,80)` + light `rgb(26,127,55)`, zero grey dashes) ✔ · inbound-first rows + filter (33/33 line 1 = `out.selectors`, line 2 = `→ host:port · kind`; `stackexpected` → **6** rows, not the overview's "five" — the capture declares 6 such routes; tallies byte-identical `10/33/16/0`, zero requests) ✔ · trunk motion truth table + `null` arm prints `n/a` never `0` + reduced-motion stills 43 paths and every knob ✔ · screenshots/transcripts ✔ · payload page-only ✔ (literal byte-diff of a live socket table can never pass; answered by the no-`.py`-diff + shape-equality proof, as recorded in the transcript) · no behaviour change elsewhere ✔.
Docs: fixed 6 inconsistencies — two identifiers split across line breaks (`test_the_pipe_animation_stills_in_both_states_under_reduced_motion`, `32_gateway_resolved_target_liveness`), stale "sixteen routes" count, stale "four machines/hosts" in README **and** AGENTS (fleet is three), "three fields"→"three field types", gap-14 headline wording; updated the pass's screenshot list.
Deviations: `uv pip list --outdated` shows one advisory (`starlette 1.9.0 → 1.10.0`) — not upgraded, this phase moved no dependency (rule 8 / DoD apply "if deps moved"). `TODO.md` carries two new operator disk requests, left untouched.
Next pending phase: `32_gateway_resolved_target_liveness`.
This commit is contained in:
@@ -0,0 +1,71 @@
|
||||
# Phase 32 — Names become addresses where they are actually dialled: every route gets a real live count
|
||||
|
||||
**Source:** operator request, 2026-10-23 — "The only animated line is `:443 -> 127.0.0.1:4443`." + the decision taken in that conversation: **the agent resolves the names it reads**, because the resolution nginx's `resolver 10.1.0.1` and haproxy's `default-server init-addr last,libc,none` perform happens *on the gateway*, and a laptop- or dashboard-side lookup is a different vantage answering a different question.
|
||||
**Story:** n/a
|
||||
**Context:** `agent/host_agent.py` (1.2.0 → **1.3.0**: the `haproxy`/`nginx` parsers that already produce every target name, the `sockets` reader, the `AGENT_VERSION` + `/schema` contract, the `_smart_lock`/`_smart_refreshing`/`_refresh_smart` cache-and-background-pass precedent, `--no-proxy`/`--no-smart`/`--root`), `src/dashboards/sources/gateway.py` (`_connection()`'s `address`/`joinable`/`metric` rule, `join_sockets()`, `endpoint_of()`, `samples()`), `src/dashboards/collectors.py` (`ADDON_ERROR_PREFIXES` — the cross-module contract AGENTS.md says a new add-on section must satisfy **in the same commit**), `scripts/capture-gateway-config.sh` (the six fixtures + its "no privilege, no copy tool, a bad read is never a fixture" rules + the RFC 5737/3849 address scrub), `tests/test_gateway_source.py`, `tests/test_agent.py`, `tests/fixtures/gateway_*`, `data/logs/verify-28-final.py` (the side-by-side additivity control this phase copies).
|
||||
|
||||
## Objective
|
||||
|
||||
`agent/host_agent.py` **1.3.0** resolves every target name the two proxy configs declare — with `socket.getaddrinfo`, i.e. the same libc resolution haproxy itself uses, on the machine that opens the connections — and publishes the address set, the vantage and the timestamp per name. `sources/gateway.py` then joins the kernel's ESTABLISHED sockets through those addresses, so `brain.reeseapps.com → reeseapps-brain-of-reese.reeselink.com:443` carries a **number** and its line **moves**, instead of a permanent em dash on 31 of 33 routes.
|
||||
|
||||
This is a measurement with provenance, not a guess with better marketing: the payload says *which* addresses, *how* they were obtained (`via: "libc"`), and *when*; a name that does not resolve right now publishes no addresses, gets no gauge, and prints the dash with the reason. `sources/gateway.py` still never calls a resolver — that spy pin survives this phase on purpose, because a dashboard resolving through its own network would be measuring a different box's DNS than the one routing the traffic.
|
||||
|
||||
## Scope decisions taken with the operator (binding for this phase)
|
||||
|
||||
| # | Decision | Consequence |
|
||||
|---|----------|-------------|
|
||||
| 1 | **Resolution happens on the gateway, in the agent.** | `host_agent.py` gains a `resolver` section. `sources/gateway.py` keeps its `socket.getaddrinfo` spy pin (fails if the module ever resolves), and `tests/test_gateway_source.py` gains the matching assertion that the only resolver in the system is the agent's. The dashboard's own network vantage is not the routing vantage; a lookup from here would be a fact about *this* box's `resolv.conf`. |
|
||||
| 2 | **libc, and the vantage is printed.** | `via: "libc"` on every entry (what `getaddrinfo` is), plus the nginx config's own `resolver` directive captured verbatim as **`resolver.config_directive`** (`"10.1.0.1"` on this fleet) so the page can name both and an operator can see a divergence. It lives inside the new section rather than inside `nginx` so that `nginx` stays byte-stable and the phase's additivity control remains a strict top-level key comparison with no exception list. The agent does **not** hand-roll UDP DNS queries to that resolver: stdlib only, no new dependency, and a divergence is a config finding for the operator rather than a second resolver implementation to maintain. |
|
||||
| 3 | **Only names the configs wrote are ever resolved.** | The name set is derived from the parsed `haproxy` `server` targets and the parsed `nginx` `proxy_pass`/`upstream`/`map` targets — nothing else, ever. Not a name from `/proc`, not a client address, not anything a request supplied. Pinned by asserting the spied `getaddrinfo` calls are exactly the config's target set. |
|
||||
| 4 | **Bounded, cached, never on the request path after the first pass.** | `RESOLVE_MAX_NAMES = 64` (the excess is **one** line naming the count — the include/smart budget shape), `RESOLVE_CACHE_SECONDS = 300`, one single-flight background refresher (the `_smart_refreshing` idiom), a per-pass budget so a black-holing resolver cannot hold `/snapshot`. A wedged lookup must never blow `HOST_TIMEOUT_SECONDS` and turn `/system` red over an add-on it never draws — the exact lesson SMART learned. |
|
||||
| 5 | **Three-way `available`, and `--no-resolve` removes the key in silence.** | `true` resolved what it was asked, `false` no proxy sections to read from, `null` the pass could not run — each non-`true` arm exactly one `resolver:` line (the pinned-wording contract shared with `smartctl:`/`haproxy:`/`nginx:`/`sockets:`). `--no-resolve` is `--no-smart`'s precedent: the key is **absent** ("not collected"), which stays distinguishable from "collected and empty". |
|
||||
| 6 | **A resolved name is joinable; an unresolved one is not.** | `_connection()`'s `metric`/`joinable` test becomes `address is True or resolved-address-set is non-empty`. Name-keyed gauges (`gateway.out_conns@<host>:<port>`) come back **because they now have measurements** — the AGENTS.md passage that retired them said a name-keyed series "would be a chart of a number that is always `null`", and this phase is the reason that stopped being true. The port stays in the name (`:25565`/`:25566` on one host are two services). |
|
||||
| 7 | **The shared-endpoint rule now has a real case and stays exactly as written.** | Two names resolving to the same `(address, port)` are one count **repeated** on both lines with `shared: true` — never halved, never attributed to one of them. `samples()` keeps de-duplicating by metric, and `live_out` in `/api/summary` keeps summing **distinct endpoints** (`api._endpoint_values()`). |
|
||||
| 8 | **Nothing moves that the operator did not ask about.** | `warnings()` is unchanged (resolution is not a config-integrity question); `/api/summary`'s `gateway` block gains **no** thirteenth key; `listeners[]` and the inbound join are untouched; the switch stays unwired and unwire-able (phase 33), and `routeInConfig()` keeps returning `true`. |
|
||||
|
||||
**Out of scope:** any config write, any new sample *family* (only the meaning of `gateway.out_conns@…` widens), any change to `/api/health` or `/api/config`, any `severity` (a count is not graded), the enable/disable path (phase 33), and re-bundling or re-issuing certificates (the agent moves in place on its existing bundle, exactly as phase 28 moved the gateway 1.1.0 → 1.2.0).
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `31_gateway_route_readability` (todo, **must complete first**) — the row that reads inbound → counterpart and the green end-to-end route are what a live count paints. Its payload-purity guarantee (`sources/gateway.py` untouched) is what makes this phase's rewrite land cleanly.
|
||||
- `30_gateway_connection_map` (complete) — `connections[]`, the join, `endpoint_of()`, `shared`, the gauge names, `warnings()`.
|
||||
- `28_gateway_agent_proxy` (complete) — the parsers whose target set this phase resolves, the fixture capture script and its scrub rule, and `tests/test_agent.py`'s fake-tree `--root` machinery the resolver tests reuse.
|
||||
- `24_dev_mtls_rollout` / the fleet record — the rollout is phase 28 task 05's procedure on one host (`gateway`), on its existing bundle, with `systemctl restart` and a read-only probe first.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_the_agent_resolves_the_names_it_reads.md` — agent 1.3.0: the `resolver` section (`available`, `via`, `config_directive`, entries), `RESOLVE_MAX_NAMES`/`RESOLVE_CACHE_SECONDS`/pass budget, the single-flight background pass, three-way `available`, `--no-resolve`, `/schema`, `AGENT_VERSION`.
|
||||
2. `02_resolution_ground_truth.md` — `capture-gateway-config.sh` records the resolver document and the two config files' `resolver` line; the RFC-5737/3849 scrub extended to it; the 1.2.0-vs-1.3.0 **additivity control** (`data/logs/verify-32-additive.py`, copied from `verify-28-final.py`).
|
||||
3. `03_join_through_the_address_the_name_became.md` — `sources/gateway.py`: `out.resolved`/`via`/`resolved_ts`, the widened `joinable`/`metric` rule, the address-set match, `shared` unchanged, `null` refusals, and `collectors.ADDON_ERROR_PREFIXES` gaining `resolver:` **in this commit**.
|
||||
4. `04_the_page_says_which_unknown_it_is.md` — `page-gateway.js` + `gateway.html`: the dash splits into "the name did not resolve in this reading" vs "resolved and nothing is open", the `libc`/timestamp provenance on `title`, footer paragraph 2 rewritten, tallies unchanged.
|
||||
5. `05_roll_the_gateway_to_1_3_0.md` — read-only probe first, then the phase-28 procedure on `gateway` (re-stage, install, restart, verify both ways), the live join numbers recorded, and the fleet record updated with the new version spread.
|
||||
6. `06_docs_and_live_verify.md` — `AGENTS.md` (the agent bullet, `sources/gateway.py`, `collectors.py`, the metric-naming passage, the `/gateway` UI paragraph, the fleet record, gap 13) + `README.md`, the browser pass with screenshots, the mobile-audit re-run, the full gate.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- **Hermetic by construction.** No test dials real DNS: `tests/test_agent.py` monkeypatches `socket.getaddrinfo` and asserts (a) it is called with **exactly** the parsed target names and ports and nothing else, (b) the cache means a second `/snapshot` costs zero lookups, (c) a pass over `RESOLVE_MAX_NAMES` reports one line and keeps `available: true`, (d) a raising/timeout-simulating resolver yields `status`-bearing entries and never fails `/snapshot` or `/health`.
|
||||
- **Real captures for shape.** `tests/fixtures/gateway_resolver.json` (documentation addresses, verbatim names) plus the existing six fixtures drive `tests/test_gateway_source.py`: every captured name route gets a count when the recorded `/proc/net/tcp{,6}` holds a socket to its recorded address; `minecraft.reeselink.com:25565` and `:25566` stay two distinct gauges; two names → one endpoint is one count twice with `shared: true`; an entry with no addresses publishes `connections: null`, **no** metric, and no sample.
|
||||
- **The spy pins are restated, not relaxed.** `socket.getaddrinfo` stays forbidden in `sources/gateway.py` (spy) and `0` becomes "no sample", never "an empty chart" — a pipe whose name resolved to nothing publishes no point.
|
||||
- **Live (`-m live --live`)**: one new test reads the real gateway and asserts every `map` entry whose `resolver` status is `ok` carries a finite count **or** `0`, and that none of them is `null` — the operator's complaint, pinned against the box. Skips when `GATEWAY_HOST` is not a target.
|
||||
- Coverage **>90%** on `agent/host_agent.py`, `sources/gateway.py`, `collectors.py`; the whole suite passes 100% and `bash .agents/validate.sh` is green, coverage floor included.
|
||||
|
||||
## Documentation & Consistency
|
||||
|
||||
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase, fixing every inconsistency **including pre-existing ones**. Specifically: AGENTS.md's agent bullet gains the `resolver` section (and its `--no-resolve`, its bounds, its three-way `available`), its `sources/gateway.py` bullet's join paragraph is rewritten (the join now has two address sources: written-as-address and resolved), the **"no name-keyed `gateway.out_conns@…` name is ever inserted"** sentence in the metric-naming passage is replaced with the widened rule and *why it changed*, the `collectors.ADDON_ERROR_PREFIXES` list gains `resolver` (the "prefix in the same commit" rule), the `/gateway` UI paragraph's dash sentence splits in two, and the **fleet record** moves `gateway` to 1.3.0 with the deskwork/driveripper spread and what the spread costs.
|
||||
- `README.md`: the `/gateway` table row and the gateway prose now describe counted name routes, the resolution vantage, and the capture script's new outputs.
|
||||
- Tasks update both files immediately when their own changes are user-visible.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `GET /schema` on the gateway agent answers `agent_version` **1.3.0** with a `resolver` block (its `available` flag named) and `--no-resolve` listed; `tests/test_agent.py` pins the constant and the schema text.
|
||||
- [ ] `GET /api/gateway/snapshot` on the live ingress: **0** `connections[]` rows with `connections: null` among those whose target resolved; `brain.reeseapps.com → reeseapps-brain-of-reese.reeselink.com:443` carries a finite count; `socket.getaddrinfo` is still never called anywhere under `src/dashboards/` (spied) and every address in `out.resolved` names `via: "libc"` with a `resolved_ts`.
|
||||
- [ ] Two names resolving to one `(address, port)` draw one count repeated with `shared: true` and **one** stored sample; `/api/summary`'s `gateway.live_out` still equals the sum over **distinct** endpoints (pinned before and after).
|
||||
- [ ] A stubbed `nxdomain` for one name publishes `out.resolved: null`, no `metric`, no sample, and the page prints the "did not resolve in this reading" sentence — never `0`, never a silent blank.
|
||||
- [ ] `--no-resolve` on a loopback agent removes the `resolver` key entirely and `/api/gateway/snapshot` behaves exactly as phase 30/31 (names uncounted, `supported: true`); a 1.2.0 agent on any host behaves identically (the spread is harmless — pinned by a fixture without a `resolver` key).
|
||||
- [ ] `/system` is not `degraded` on a host whose agent reports `resolver: nxdomain …` — the `ADDON_ERROR_PREFIXES` pin (invariant 3), asserted both ways.
|
||||
- [ ] `data/logs/verify-32-additive.py` proves the 1.3.0 document equals the 1.2.0 document key-for-key **plus** `resolver`, and 1.3.0 with `--no-resolve` equals it exactly.
|
||||
- [ ] `uv run pytest` (incl. `--cov` ≥ 94), `uv run pytest -m live --live`, `uv run ruff check src tests agent`, `uv run pyright`, `node --check`, `bash .agents/validate.sh` — all green; `uv pip list --outdated` empty (no dependency moved, and this phase must not add one).
|
||||
- [ ] `data/logs/32-*.png` at 1440 px and 360 px (counted routes, an unresolved arm, both themes) + `data/logs/32-*.out`; `data/logs/27-task02-mobile-audit.cjs` exits 0 with 0 FAIL / 0 FINDING.
|
||||
- [ ] `./scripts/dev-server.sh status` lists all ten sources `ok`, `gateway` included, with `host:gateway`/`disks:gateway` unaffected by their host's agent upgrade.
|
||||
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass).
|
||||
- [ ] No behaviour change in completed work: the other five dashboards, `/settings`, `/certs`, `/login`, the warnings panel, the unwired switch and every non-gateway source render and behave as phase 31 left them.
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
# Task 01 — The agent resolves the names its own configs declare, off the request path
|
||||
|
||||
**Phase:** `32_gateway_resolved_target_liveness` · **Source:** operator decision 2026-10-23 — "the agent resolves the names it reads"
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
`agent/host_agent.py` goes **1.2.0 → 1.3.0** and gains one additive top-level section, `resolver`: for every `(name, port)` the two parsed proxy configs write as a target, the libc address set, its status, the vantage and a timestamp — obtained on the machine that dials, bounded, cached, and refreshed on a background thread so a slow or black-holing resolver can never hold `/snapshot` open. Everything the agent published before is byte-identical; `--no-resolve` removes the key in silence.
|
||||
|
||||
## Work
|
||||
|
||||
1. `agent/host_agent.py`
|
||||
- Constants beside the proxy section's bounds, each with a comment naming what it protects:
|
||||
```python
|
||||
AGENT_VERSION = "1.3.0"
|
||||
RESOLVE_MAX_NAMES = 64 # a gateway's config-declared targets, with room to grow
|
||||
RESOLVE_CACHE_SECONDS = 300.0 # a LAN target moves when the operator moves it, not every poll
|
||||
RESOLVE_PASS_BUDGET_SECONDS = 3.0 # a black-holing resolver must not hold /snapshot open
|
||||
RESOLVE_VIA = "libc" # getaddrinfo, i.e. the resolution haproxy's libc path performs
|
||||
```
|
||||
- **New pure helper** `target_names(haproxy: Mapping | None, nginx: Mapping | None) -> list[tuple[str, int | None]]`
|
||||
Derived **only** from the parsed sections: haproxy `backends[].servers[].target` (in config order), then nginx `upstreams` members, `server`-block `proxy_pass` targets, and `maps[].entries[].target`, deduplicated keeping first-seen order. An IP literal is kept as a target with `addresses` set without any lookup (it is already an address, and the lookup would be a round trip to say so) and marked `status: "literal"`. A name from `/proc`, from a request, or from anywhere else must never enter this list — that is decision 3 and it is pinned by asserting the spied call set equals this function's output.
|
||||
- **The pass, following the SMART idiom exactly** (`_smart_lock` / `_smart_cache` / `_smart_refreshing` / `_refresh_smart`):
|
||||
`_resolver_lock`, `_resolver_cache: dict[str, dict[str, Any]]` keyed on `f"{name}:{port}"` (a name on two ports is two targets — `minecraft.reeselink.com` on `25565` **and** `25566` is on this fleet today), `_resolver_refreshing: bool`, `_resolver_errors: list[str]`.
|
||||
```python
|
||||
def resolver_pass(self, targets) -> tuple[dict[str, dict[str, Any]], list[str]]:
|
||||
"""One bounded libc lookup per declared target. Never raises out of this function."""
|
||||
```
|
||||
For each target: `socket.getaddrinfo(name, port, proto=socket.SOCK_STREAM)`, collect the unique `sockaddr[0]` values, keep both families, sort address-family-stable (v4 ascending then v6 ascending) so the payload is deterministic across polls. `socket.gaierror` with `EAI_NONAME`/`EAI_NODATA` → `status: "nxdomain"`, any other `gaierror`/`OSError` → `status: "error"`, both with `addresses: []`. After each lookup compare elapsed against `RESOLVE_PASS_BUDGET_SECONDS`; on exhaustion stop and append **one** line — `resolver: pass budget 3.0s exhausted, 41 of 58 targets looked up` — mirroring the smart pass-budget wording the agent already prints (phase 16 pins that family of tests; match its shape so the next reader has one idiom, not two).
|
||||
- Entry shape (and the whole section):
|
||||
```python
|
||||
"resolver": {
|
||||
"available": True | False | None,
|
||||
"via": "libc",
|
||||
"config_directive": "10.1.0.1" | None, # nginx's own `resolver` line, verbatim arguments
|
||||
"ts": <wall clock of the pass>,
|
||||
"entries": [{"name": …, "port": …, "status": "ok|nxdomain|error|literal",
|
||||
"addresses": […], "ts": <per-entry, when it was answered>}],
|
||||
"errors": [ …at most one line per arm… ],
|
||||
}
|
||||
```
|
||||
`config_directive` lives **here** rather than inside the `nginx` section on purpose: it keeps the `nginx` document byte-stable so phase 32's additivity control stays a strict top-level key comparison instead of an exception list. Read it from the nginx config text the agent already has (`resolver` directive, arguments joined by a space, `None` when the file has none) — no second file read.
|
||||
- `available` is the three-way answer the other add-ons use: `true` the pass ran, `false` there is nothing to resolve because the proxy sections were not read, `null` a pass that could not run — and **exactly one `resolver:` line per non-`true` arm**, the missing/no-proxy case saying **not found** (the pinned cross-module wording the dashboard reads).
|
||||
- Cache policy: the **first** pass runs inline (the SMART precedent, bounded by the budget); an expired cache afterwards refreshes on a single-flight daemon `threading.Thread` while the request is served the previous result; a failed refresh keeps the previous entries and retries on the next expiry instead of hammering DNS. `resolver.entries[].ts` is always the time that entry was actually answered, so a served-from-cache entry cannot pretend to be current.
|
||||
- `--no-resolve` (argparse, no env default — `--no-smart`/`--no-proxy` precedent): the key is **absent** from `snapshot()` ("not collected", which stays distinguishable from "collected and empty"), silently, and `--once` prints the same absence. `--no-proxy` also removes it (nothing to resolve on a box that collects no proxy) — say so in one comment.
|
||||
- `AGENT_VERSION` bump, the module docstring's `/schema` block gains the `resolver` section with its `available` flag and `--no-resolve`, and the status page's path list is unchanged (this task adds no path — task 02 of phase 33 is what adds a POST).
|
||||
2. `tests/test_agent.py`
|
||||
- `test_the_agent_version_is_1_3_0_and_the_schema_names_the_resolver_section` (the constant + the `/schema` text names `resolver`, its `available`, and `--no-resolve`).
|
||||
- `test_only_the_targets_the_configs_declare_are_resolved` — monkeypatch `socket.getaddrinfo`, run a snapshot against the fake-tree configs, assert the `(name, port)` pairs it was called with are **exactly** `target_names(...)`' output for the same tree, in config order, with no duplicates, and that no call carries an address from `/proc/net/tcp`.
|
||||
- `test_the_resolver_cache_means_one_lookup_per_target` — two snapshots, one call set; expiry triggers a background refresh and the first of the two responses is served from cache with its original per-entry `ts`.
|
||||
- `test_a_slow_resolver_cannot_hold_the_snapshot_open` — a `getaddrinfo` stub that sleeps: the pass stops at the budget, prints the one budget line naming the counted and total figure, `available` stays `true`, `/snapshot` and `/health` still answer.
|
||||
- `test_nxdomain_is_a_status_and_never_a_failure` — one raising name yields `status: "nxdomain"`, `addresses: []`, other entries intact, snapshot `ok`.
|
||||
- `test_no_resolve_removes_the_key_in_silence` and `test_a_host_with_no_proxy_sections_says_so_once`.
|
||||
- `test_the_resolver_lines_never_fail_the_snapshot` — every arm, plus the existing "no section can fail `/snapshot`" family extended to cover the new one.
|
||||
- An IP-literal target is reported `status: "literal"` with its address and **no** lookup.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the pins above; hermetic by monkeypatching `socket.getaddrinfo` — **no test resolves a real name**.
|
||||
- Coverage **>90%** on the new agent code; `uv run ruff check agent` + `uv run pyright` clean.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] Run against a fake tree built the way `tests/test_agent.py::nginx_tree()` builds one (the captured `gateway_nginx_main.conf` + `conf.d` + `stream.d` under `<tmp>/etc/nginx/`, with `--haproxy-config tests/fixtures/gateway_haproxy.cfg`): `python3 agent/host_agent.py --once --no-smart --root <tmp> --haproxy-config tests/fixtures/gateway_haproxy.cfg --nginx-config <tmp>/etc/nginx/nginx.conf` prints a `resolver` block with `available`, `via: "libc"`, `config_directive: "10.1.0.1"` and one entry per declared target — and prints **no** `resolver` key with `--no-resolve` added.
|
||||
- [ ] `grep -n 'AGENT_VERSION = ' agent/host_agent.py` reads `1.3.0` and the `/schema` docstring names `resolver`.
|
||||
- [ ] Two consecutive `--once` runs with the stubbed resolver perform one lookup set, not two.
|
||||
- [ ] `uv run pytest -k agent` **100% passing**; the whole suite green; coverage not down.
|
||||
- [ ] README.md/AGENTS.md updated for the new agent section and flag (this task's own user-visible surface) — the phase's final pass re-checks both project-wide.
|
||||
- [ ] No behaviour change in completed work: every pre-1.3.0 section is byte-identical for the same `--root` (formally proven in task 02's control).
|
||||
@@ -0,0 +1,52 @@
|
||||
# Task 02 — Ground truth for the resolution, and a control that proves 1.3.0 is only additive
|
||||
|
||||
**Phase:** `32_gateway_resolved_target_liveness` · **Source:** AGENTS.md invariant 7 ("verify against the deployed upstream; never guess a metric name") + the phase-28 additivity precedent
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Two things this phase must be able to point at: a **captured** resolution document from the real gateway, recorded through the same no-privilege, never-a-fixture-on-a-bad-read discipline `scripts/capture-gateway-config.sh` already keeps; and a **control** that proves agent 1.3.0's snapshot equals 1.2.0's key-for-key plus `resolver` — so the upgrade of a live ingress is provably additive rather than "the page still looks fine".
|
||||
|
||||
## Work
|
||||
|
||||
1. `scripts/capture-gateway-config.sh`
|
||||
- After the existing config captures, derive the target set **from the repo's own parser**, never by re-implementing one:
|
||||
```python
|
||||
# The name list is what the agent will ask for, from the configs we just captured — one
|
||||
# source of truth. If `target_names()` changes, this capture follows it, and a fixture
|
||||
# that lists a name the agent no longer resolves is a fixture that fails a test.
|
||||
from agent.host_agent import target_names
|
||||
```
|
||||
(`bash` calls `uv run python - <<'PY'`, same interpreter the tests use.)
|
||||
- One **world-readable, unprivileged** read per name, over the same `$GATEWAY_SSH` alias, which is libc resolution *on the box that dials* — the exact vantage `via: "libc"` claims:
|
||||
`ssh "$GATEWAY_SSH" getent ahostsv4 <name>` then `getent ahostsv6 <name>`. No `sudo`, no `dig`/`host` assumption, no copy tool.
|
||||
- `config_directive` is read from the **already captured** nginx text (the `resolver` directive's arguments) — no extra round trip, and the fixture then matches what the agent will read out of the same bytes.
|
||||
- Write `tests/fixtures/gateway_resolver.json` in the agent's own entry shape, sorted in the agent's order, with `ts` set to a fixed capture constant so the fixture is byte-stable, and:
|
||||
- **names verbatim** — the domains are the ingress map, exactly as the config fixtures keep them;
|
||||
- **addresses scrubbed by the socket table's rule**: keep RFC 1918 / ULA / loopback, rewrite any address `ipaddress` calls globally routable into documentation space (RFC 5737 / RFC 3849), first-seen order, same spelling, **idempotent** — invariant 6, the same reason `/proc/net/tcp` is scrubbed (a global address is not this repo to publish);
|
||||
- a `nxdomain` is recorded as a **status**, not dropped.
|
||||
- Failure discipline, unchanged in spirit and pinned: a failed `ssh`, a missing `getent`, or **every** name failing → exit 1 having written nothing (a broken vantage is not a fixture); a half-written file is removed. One name failing among successes is a fact and is kept.
|
||||
- `--help`/usage comment updated in the same edit (the script's header is the documentation `README.md` points at).
|
||||
2. `tests/fixtures/gateway_resolver.json` — the capture itself, committed.
|
||||
3. `data/logs/verify-32-additive.py` (copy the shape of `data/logs/verify-28-final.py`)
|
||||
- `git stash`-safe pattern: load `agent/host_agent.py` from `git show HEAD:agent/host_agent.py` into a temp file and import it side by side with the working tree's, via `importlib.util.spec_from_file_location` under distinct module names.
|
||||
- Build one fake tree (`nginx_tree`-shaped) plus the haproxy capture, monkeypatch `socket.getaddrinfo` in **both** modules to the same deterministic stub, call `snapshot()` on both with identical arguments, and demand:
|
||||
- `new == old | {"resolver": …}` — every pre-1.3.0 key equal, `resolver` the only addition;
|
||||
- `new --no-resolve == old` **exactly**;
|
||||
- no key removed, renamed, or reordered-meaning-changed anywhere, printed as a diff on failure.
|
||||
- `config_directive` lives inside `resolver` (not inside `nginx`) precisely so this comparison stays a strict top-level key test with no exception list — record that reason in the script's header so nobody later "tidies" the field into the nginx section and weakens the control.
|
||||
- Transcript: `data/logs/32-additive.out`, exit non-zero on any difference.
|
||||
4. `tests/test_deploy_scripts.py` (or the capture-script test module that already covers this script)
|
||||
- Extend the existing pins: the script names `getent` and no `sudo`; it derives the name list by importing `target_names` (so the capture cannot out-ask the agent); a forced-`ssh`-failure run exits 1 and leaves the existing fixture byte-identical; a re-run over the same inputs is idempotent; the new fixture carries no globally routable address (`test_the_gateway_fixtures_carry_no_routable_endpoint`-style walk, extended to JSON).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the pins above; `tests/test_agent.py` and `tests/test_gateway_source.py` both consume `gateway_resolver.json` (task 03 wires the second).
|
||||
- Coverage: shell + fixtures, so the gate is the existing script pins plus no Python regression; `bash -n scripts/capture-gateway-config.sh` clean.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `./scripts/capture-gateway-config.sh` run on a box with the `gateway-ops` key writes `tests/fixtures/gateway_resolver.json` and exits 0; the file has one entry per name `target_names()` derives from the captured configs, `via: "libc"`, `config_directive: "10.1.0.1"`, and no globally routable address anywhere in it.
|
||||
- [ ] Killing the alias (or `GATEWAY_SSH=nope`) exits non-zero with every existing fixture byte-identical (`sha256sum` before/after, printed).
|
||||
- [ ] `uv run python data/logs/verify-32-additive.py` exits 0 and its transcript shows the two documents equal plus `resolver`, and the `--no-resolve` arm equal exactly.
|
||||
- [ ] `uv run pytest` **100% passing**, coverage not down; `bash .agents/validate.sh` green.
|
||||
- [ ] README.md's capture section and AGENTS.md's invariant-7/fixture prose name the new capture output (this task's user-visible change) — the phase's final pass re-checks both project-wide.
|
||||
- [ ] No behaviour change in completed work: the six existing fixtures are unchanged unless the live router's config genuinely moved, in which case that drift is recorded in the transcript and `tests/test_gateway_source.py`'s tallies are re-counted in the same commit (never edited to match a live read).
|
||||
+56
@@ -0,0 +1,56 @@
|
||||
# Task 03 — The join counts a name route through the address the gateway itself dials
|
||||
|
||||
**Phase:** `32_gateway_resolved_target_liveness` · **Source:** operator complaint — "the only animated line is `:443 -> 127.0.0.1:4443`"
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
`sources/gateway.py` reads the agent's `resolver` section and widens the one rule that decided a route could never be counted: an endpoint is joinable when the config wrote an **address** *or* when the gateway's own libc resolution published one for the name. `connections[]` grows three provenance fields, name-keyed gauges start carrying measurements, and the two rules that made phase 30 honest stay exactly where they were — the dashboard still never resolves, and a pipe nobody measured still publishes **no** sample.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/sources/gateway.py`
|
||||
- New pure reader `resolution_index(snapshot) -> dict[tuple[str, int | None], Mapping[str, Any]]`, keyed on `(name, port)` from `snapshot["resolver"]["entries"]`, ignoring entries with no `addresses` (a `nxdomain` contributes an empty set, never a `None` address). Absent `resolver` section (a 1.2.0 agent, or `--no-resolve`) → `{}`, which is exactly today's behaviour and must be pinned with a fixture that has no `resolver` key at all.
|
||||
- New pure pass `apply_resolution(connections, index, via, ts)` — called between `config_connections()` and the socket join — writing on each connection's `out`:
|
||||
- `resolved`: the address list, or `null` when nothing published one (never `[]`, so the page's "did not resolve" arm is one `is null` test rather than a length check that confuses empty with unknown);
|
||||
- `via`: `"libc"` (the agent's string, copied, never invented here) and `resolved_ts`: that entry's own timestamp;
|
||||
- `joinable`-driven `metric`, widened:
|
||||
```python
|
||||
# A line is countable when the kernel can name its far end: either the config wrote an
|
||||
# address, or the gateway's own libc resolution — the one that opens this connection —
|
||||
# published one. Phase 30 retired name-keyed gauges because they could only ever be
|
||||
# empty; this is the fact that made that untrue. The NAME keeps the config's spelling so
|
||||
# the stored series is the same one phase 29/30 would have written.
|
||||
```
|
||||
`metric` stays `gateway.out_conns@<host>:<port>` with the **config's** host spelling (history continuity, and `:25565`/`:25566` stay two gauges), and stays `null` for a line whose address is unknown.
|
||||
- `endpoint_of(connection)` → returns a **list** of `(address, port)` keys: the written address, or every resolved address. `join_sockets()` builds its `endpoints` map by pointing **every** key at the **same** counter row, so a multi-homed name is one endpoint with several spellings in the kernel's table, and two lines reaching one endpoint still share one counter (`shared: true`, unchanged).
|
||||
- **Address comparison is normalised, spelling is not.** Match with `ipaddress.ip_address(...)` on both sides so `/proc`'s `inet_ntop` output, an IPv4-mapped `::ffff:10.1.2.64`, and a resolver's `10.1.2.64` are the same address; the **payload** keeps publishing whatever the config and the agent wrote, verbatim. No string equality anywhere in the match path — that is the bug class where a route resolves and still never counts.
|
||||
- `_skeleton()` gains `"resolver": {"available": None, "via": None, "entries": []}` so `parse` and `stub` keep one key set (the parity pin), and `parse()` fills `available`/`via` from the section while `entries` carries the entries the index was built from — the page reads provenance from the document it was drawn from, not from a second call.
|
||||
- `samples()`: unchanged logic, now wider in effect (a resolved name route emits its gauge). A `null` count still publishes nothing. De-duplication by metric stays, so a shared endpoint costs one insert.
|
||||
- `supported()` — a 1.3.0 snapshot must answer `supported: true` (its three proxy sections are what the check reads); add the assertion rather than assuming it.
|
||||
- `warnings()` untouched. The `/api/summary` block grows **no** field.
|
||||
2. `src/dashboards/collectors.py` — **the same commit**, per AGENTS.md's cross-module rule for a new add-on section:
|
||||
- `ADDON_ERROR_PREFIXES` gains `resolver`, so a `resolver: … nxdomain …` or `resolver: pass budget … exhausted` line is an *unconfigured* report on `/system`, not `degraded`, and never an overview WARN. Invariant 3, both directions pinned: a host with that line stays out of `alerts`, and a `nginx:`/`smart:` line still behaves as before.
|
||||
3. `tests/test_gateway_source.py`
|
||||
- `test_a_name_route_counts_through_the_address_the_gateway_published` — captured configs + `gateway_resolver.json` + the recorded `/proc/net/tcp{,6}`: a socket whose remote is `10.1.2.64:443` lands on `brain.reeseapps.com → reeseapps-brain-of-reese.reeselink.com:443`'s line, with `out.via == "libc"` and a `resolved_ts`.
|
||||
- `test_the_match_is_an_address_not_a_string` — the same address in `::ffff:` / leading-zero / uppercase spellings on either side still matches; the payload keeps verbatim spelling.
|
||||
- `test_two_names_one_endpoint_is_one_count_shared` and `test_two_ports_on_one_host_are_two_gauges` (`minecraft` `25565`/`25566`).
|
||||
- `test_an_unresolved_name_publishes_no_metric_and_no_sample` — `connections: null`, `metric: null`, zero samples, never `0`.
|
||||
- `test_the_dashboard_still_never_resolves` — `socket.getaddrinfo` spied and asserted uncalled through the whole `parse()` path, **and** a new comment naming where the resolver lives and why (vantage). This pin is restated, not relaxed.
|
||||
- `test_a_snapshot_without_a_resolver_section_is_phase_30_verbatim` — the 1.2.0 fixture: identical payload to today's except `resolver`/`out.resolved`/`via`/`resolved_ts` being absent/`null`, and `supported` still `true`.
|
||||
- The counted-once identity and the `sum(listeners) + sum(endpoints) <= established` inequality are restated with the widened match set, including the `127.0.0.1:4443` loopback hop counting once inbound **and** once outbound on two different pipes.
|
||||
- `tests/test_api.py`: the "no name-keyed `gateway.out_conns@…` series is ever inserted" pin is **rewritten** to the new rule (an endpoint whose address is known — written or resolved — gets its gauge; an unresolved name never does), and `test_the_summary_gateway_block_is_the_payloads_sums_not_the_pages_arithmetic` gains the resolved case so `live_out` still sums **distinct endpoints** (`api._endpoint_values()`), never lines.
|
||||
- `tests/test_live_fleet.py` (marked `live`): against the real ingress, every `connections[]` row whose resolver status is `ok` has a finite `connections` (a number, `0` allowed) — never `null`. Skips when `GATEWAY_HOST` is not a configured target.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit + fixture as above; `tests/test_gateway_source.py` keeps running the **real** `host_agent` parsers over the captures so the join is compared with the agent's document, not with a copy of its field names.
|
||||
- Coverage **>90%** on `sources/gateway.py` and `collectors.py`; whole suite 100%; `uv run ruff check src tests agent` and `uv run pyright` clean.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] Against the captures: the number of `connections[]` rows with a non-null `connections` rises from 2 (the two loopback targets) to every row whose name resolved in `gateway_resolver.json`, and `tests/test_gateway_source.py` prints that count so the change is visible rather than implied.
|
||||
- [ ] `grep -rn "getaddrinfo" src/dashboards/` finds nothing; `grep -n "resolver" src/dashboards/collectors.py` shows `ADDON_ERROR_PREFIXES` carrying it.
|
||||
- [ ] `GET /api/gateway/snapshot` on a fixture-driven store shows `out: {…, address: false, resolved: ["10.1.2.64"], via: "libc", resolved_ts: …, …}` with a finite `connections` on that row, and `null`/`null` on an injected `nxdomain` row.
|
||||
- [ ] The stored series for one shared endpoint is **one** insert per poll (asserted by counting rows), and `/api/summary`'s `gateway.live_out` equals the sum over distinct endpoints.
|
||||
- [ ] `uv run pytest` **100% passing**, coverage not down; `bash .agents/validate.sh` green.
|
||||
- [ ] README.md/AGENTS.md updated for the widened join rule and the metric-naming passage in the same commit (the "no name-keyed series" sentence is replaced, with the reason) — the phase's final pass re-checks both project-wide.
|
||||
- [ ] No behaviour change in completed work: `warnings()`, `/api/summary`'s key set, the inbound join, `/system`'s `degraded` for the other add-on prefixes, and every non-gateway source are untouched.
|
||||
+50
@@ -0,0 +1,50 @@
|
||||
# Task 04 — The dash now says *which* unknown it is, and the resolution rides on the tooltip
|
||||
|
||||
**Phase:** `32_gateway_resolved_target_liveness` · **Source:** phase-31 decision 8 ("the `—` keeps its meaning and loses its mystery") meeting this phase's new fact
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Two facts that used to share one glyph separate: **"the gateway could not resolve this name in this reading"** and **"resolved, and nothing is open on it right now"**. The first keeps the em dash with a sentence that names the cause; the second prints its `0`. Every counted line carries where its address came from (`libc`) and when, on `title` and in the `aria-label` — provenance the operator needs to explain a number, kept off the ink because the page's copy rule is *would you act differently without it?*
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/web/static/page-gateway.js`
|
||||
- Replace the single `NO_COUNT_WHY` with the two sentences the payload can now distinguish, chosen from data and never guessed:
|
||||
```js
|
||||
const UNRESOLVED_WHY = 'no number: the gateway could not resolve this name in this reading';
|
||||
const IDLE_TITLE = (n) => `${n} open connection(s) on this endpoint`
|
||||
+ (resolved ? ` · address from ${via}, resolved ${age}` : '');
|
||||
```
|
||||
`age` comes from `Dash.duration(Dash.ageSeconds(conn.out.resolved_ts))`; `null`/absent prints nothing rather than `NaN`, and `via` is the payload's word (`libc`) copied, not paraphrased.
|
||||
- `leafAria(conn, on)`: the unknown arm becomes `'connections not attributed: the gateway could not resolve this name in this reading'`; the counted arm keeps its existing facts and adds nothing else. A measured `0` keeps its own sentence (`0 open connections`) — never the unresolved wording, because it is a measurement.
|
||||
- `countText()` and the `—` glyph are unchanged; a `0` continues to print `0`.
|
||||
- **Filter amendment** (phase 31 task 04's `haystack`): include `conn.out.resolved` addresses, so `10.1.2.64` finds `brain.reeseapps.com`. They are payload strings, the filter is still an eye, and this is the one case where matching an address helps the operator — pinned with the rest of the haystack contract.
|
||||
- `cardStreams()` is untouched: a line with no `metric` still draws no series, and the chart legend still names only pipes that have points.
|
||||
2. `src/dashboards/web/pages/gateway.html`
|
||||
- Footer paragraph 2 is rewritten to the new truth and stays **one** paragraph (the file's three-paragraph shape is pinned in `tests/test_api.py`):
|
||||
> `a dash means the gateway could not resolve that name in this reading — hover a counted line for the address it matched and when that answer was taken.`
|
||||
- The card comment above `.gate-map` gains one line: `out.resolved` is the agent's libc answer and is why a name line can carry a number.
|
||||
- The `#gate-empty` setup block keeps naming agent **1.2.0** as the minimum (a 1.3.0 agent is not required for the page to work, only for name routes to count) — do not bump it here; the version floor is `sources/gateway.supported()`.
|
||||
3. `tests/test_api.py`
|
||||
- `GATEWAY_PAGE_FIELDS` gains `connections[].out.resolved`, `connections[].out.via`, `connections[].out.resolved_ts`, resolved against a live document in both directions (the existing device), so a rename on the Python side fails by name here.
|
||||
- **New** `test_an_unresolved_name_and_an_idle_route_say_two_different_sentences`: served JS picks the sentence from `out.resolved === null`, the two strings differ, neither is `n/a`, and a `0` is described as a measurement.
|
||||
- `test_an_unattributed_endpoint_prints_a_dash_and_never_a_zero` stays green with its claim intact and its docstring amended to name the new cause (restated, not deleted).
|
||||
- **New** `test_a_counted_line_names_where_its_address_came_from`: the `title` builder reads `out.via` and `out.resolved_ts` through the shared age helpers, and a missing `resolved_ts` yields no age fragment rather than `NaN`/`undefined`.
|
||||
- The filter pin from phase 31 (`test_the_filter_matches_only_payload_text_and_costs_no_request`) is extended: the haystack includes `out.resolved` and still contains no `fetchJSON`, no DOM text, and no `"null"` matchable value.
|
||||
- `test_no_page_narrates_its_own_pipeline` must still pass on the new footer sentence (it names an operator-facing cause and a place to look, not the app's design); if it objects, shorten the sentence rather than touching the audit's allow-list.
|
||||
- The degradation-arm pin gains a case: `resolver: {available: null}` — no red, no extra ink, `why` still the page's one sentence.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: served-bytes pins above + a fixture render for each of the three arms (resolved+live, resolved+idle, unresolved) asserting the glyph, the `title`, and the `aria-label` differ exactly as specified.
|
||||
- Integration: with the resolver section stubbed to `available: null`, every name line keeps the unresolved sentence and the page stays uncoloured; no line anywhere prints `NaN`, `undefined` or `null`.
|
||||
- Coverage: no new Python in this task; `node --check` clean; floor does not fall.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] On live data: a counted name line's `title` reads `… · address from libc, resolved <age>`; an unresolved line (stub one) prints the em dash with the resolution sentence; a resolved line with no socket prints `0`.
|
||||
- [ ] Reading five lines' `aria-label`s aloud (or dumping them) yields three distinct claims and no invented number.
|
||||
- [ ] Typing `10.1.2.64` (or any address from the live `resolver` section) into `#gate-filter` reveals exactly the route(s) that resolve to it, with zero network requests.
|
||||
- [ ] `grep -c "<p" ` on the served footer is still 3 and `test_no_page_narrates_its_own_pipeline` is green.
|
||||
- [ ] `uv run pytest` **100% passing**, coverage not down; `node --check` clean; `bash .agents/validate.sh` green.
|
||||
- [ ] README.md/AGENTS.md's `/gateway` dash sentence updated (this task's user-visible change) — the phase's final pass re-checks both project-wide.
|
||||
- [ ] No behaviour change in completed work: tallies, warnings panel, chart disclosure, switch, phase 31's colour and row order untouched.
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
# Task 05 — Roll the gateway to 1.3.0, read-only first, exactly as phase 28 moved it to 1.2.0
|
||||
|
||||
**Phase:** `32_gateway_resolved_target_liveness` · **Source:** AGENTS.md's fleet record (the rollout discipline that kept `host:gateway`/`disks:gateway` `ok` through phase 28) + the standing rule that a path grant is only as current as the file it points at
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
One host moves: `gateway` runs agent **1.3.0** on its existing bundle and its existing unit, and the live page gets real counts on its name routes. Nothing else moves — no `HOST_TARGETS` edit, no certificate, no URL flip, no other host (a 1.1.0 `deskwork`/`driveripper` is unaffected, and a 1.2.0 gateway left running after a failed attempt is a working gateway). The transcript is the artifact.
|
||||
|
||||
## Work
|
||||
|
||||
1. **Read-only probe first** — `data/logs/verify-32-task05-probe.py` (copy `data/logs/verify-28-task04.py`), printing and asserting:
|
||||
- the CA-only `--status`-style probe against `https://10.1.1.27:9120` answers `version` **1.2.0** on `/health`, and still refuses `/snapshot` with `client certificate required`;
|
||||
- the dashboard's own mTLS dial (`collectors.fetch_snapshot` with `certs.agent_tls_map` over the stored authority — the `verify-28-task05.py` idiom, never a hand-rolled `client.get`) reads every pre-1.3.0 section with live numbers and **no** `resolver` key: the control that makes the install provably additive before anything is touched;
|
||||
- `/api/health` reads `host:gateway`, `disks:gateway` `ok` before the change, and the `gateway` job's payload counts how many rows carry a number (the before figure).
|
||||
2. **Re-stage, then install** over the `gateway-ops` key, as `ops`:
|
||||
- `scp agent/host_agent.py deploy/install-agent.sh → /home/ops/agent-install/…` (the directory is `ops`-owned; no root), leaving the previous files beside the grant as `host_agent.py.20261023-backup` / `install-agent.sh.20261023-backup`, which sudo will not run;
|
||||
- read the live unit back (`systemctl cat dashboards-agent`) and run the **same** arguments it carries:
|
||||
`sudo /home/ops/agent-install/deploy/install-agent.sh --bind 10.1.1.27 --port 9120`
|
||||
— no `--cert-bundle` (the bundle on disk, not the flag, is what puts the TLS lines into the render — the phase-28 finding, quoted in the transcript), no `--as-root`/`--caps` (that is `driveripper`'s shape, not this host's);
|
||||
- expect the re-run guard to stay silent (identical render). `hashsum.sha256` the unit before and after and record both; a differing unit means the run did something this phase did not plan, so stop and report rather than `--force`ing;
|
||||
- `sudo systemctl restart dashboards-agent` (the file changed; `enable --now` would not restart it), then `systemctl is-active`.
|
||||
3. **Verify both ways, read-only**:
|
||||
- the CA-only probe answers `version 1.3.0` and still refuses `/snapshot`;
|
||||
- the dashboard's mTLS read shows `agent_version: "1.3.0"`, `resolver.available: true` with an entry per declared target, `haproxy`/`nginx`/`sockets` and every pre-1.3.0 section intact with live numbers, and `/api/gateway/snapshot` answering a **finite** `connections` on every row whose resolver status is `ok` (the after figure, printed beside the before figure);
|
||||
- `./scripts/dev-server.sh status` → all ten sources `ok`; `/system` shows no new `degraded` on any host (the `resolver:` prefix, invariant 3);
|
||||
- a failure at any step is a rollback, not a fix-forward: copy `host_agent.py.20261023-backup` back to `/usr/local/lib/dashboards-agent/host_agent.py` and restart — the unit, the bundle and the URL never changed, so that *is* the whole rollback. Record which arm happened.
|
||||
4. `data/logs/NOTES-32-rollout.md` — sha256 of what was staged where, the unit hash before/after, the before/after "rows with a number" counts, the live `resolver` entry table, the `nxdomain` list if any, the screenshots' names, and the rollback command. The AGENTS.md fleet record's version spread is task 06's edit; this note is the evidence it cites.
|
||||
5. **Do not** touch `deskwork` or `driveripper`. Their 1.1.0 agents have no proxy sections and nothing to resolve; the spread costs nothing because 1.3.0 is additive, and task 02's control is the proof of that claim.
|
||||
|
||||
## Testing & Quality
|
||||
- This task changes no repo code; its quality gate is that the suite is still green afterwards and the two probe scripts exit 0 with their transcripts saved (`data/logs/32-task05-probe.out`, `data/logs/32-task05-verify.out`).
|
||||
- `uv run pytest -m live --live` afterwards: the new live pin (every resolved name route carries a number, never `null`) passes rather than skips.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `systemctl is-active dashboards-agent` is `active` on `gateway`, and the CA-only probe answers **1.3.0**.
|
||||
- [ ] The unit's sha256 is identical before and after the install run, and the guard printed no diff.
|
||||
- [ ] `/api/gateway/snapshot` on the live ingress: `resolver.available: true`, and **0** `connections[]` rows with `connections: null` among the rows whose resolver status is `ok`; the before/after counts are in the transcript.
|
||||
- [ ] `host:gateway` and `disks:gateway` read `ok` across the whole exercise, and no other host changed `degraded` state.
|
||||
- [ ] `data/logs/NOTES-32-rollout.md` exists with the hashes, the two probe transcripts, and the rollback command named.
|
||||
- [ ] `uv run pytest` **100% passing**; `uv run pytest -m live --live` green (the gateway pin passing, not skipping).
|
||||
- [ ] No repo file changed except `data/logs/` (gitignored).
|
||||
@@ -0,0 +1,47 @@
|
||||
# Task 06 — Docs that match the counted page, the browser measurement, and the full gate
|
||||
|
||||
**Phase:** `32_gateway_resolved_target_liveness` · **Source:** AGENTS.md definition of done (browser verification against live data, plus a full `README.md`/`AGENTS.md` consistency pass)
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Prove the counted page in a browser on the upgraded ingress, carry the resolution facts into the two documents that outlive this conversation, and close the phase with every gate green — including the ones that only bite if something drifted (dependency freshness, the mobile audit, the live suite).
|
||||
|
||||
## Work
|
||||
|
||||
1. `data/logs/32-counted-routes.cjs` — a Playwright pass over the live page (house shape: `NODE_PATH=<playwright node_modules>`, `DASH_BASE`/`DASH_PW`, sign in with `DASH_ADMIN_PASSWORD`, exit non-zero on FAIL, print every measured number):
|
||||
- **motion where the complaint said there was none**: count the animating elements on load (`getComputedStyle(path).animationName === 'pipe-flow'`) and assert the set is non-empty **and** that at least one of them is a **name** endpoint (a `connections[]` row whose `out.address` is `false`); record the counts per panel;
|
||||
- **provenance**: for 5 sampled rows, `title` contains `address from libc` and an age fragment, and `aria-label` states a number where the payload has one and the resolution sentence where it does not;
|
||||
- **no invented zero**: with a stubbed `resolver: {available: null}` and with `sockets.available: null`, no line prints `0` that the payload did not measure, and the trunk prints `n/a`;
|
||||
- **the filter still costs nothing**: query by an IP from the `resolver` section and by a domain, asserting zero network requests and unchanged tallies;
|
||||
- **colour is still existence**: every drawn route — spine, tick, run, switch — computes to `--gate-on` in both themes, and a row given `off` in devtools turns all four together (phase 31's contract, re-asserted after this phase's DOM/provenance changes);
|
||||
- **reduced motion**: nothing animates under emulated `prefers-reduced-motion: reduce`;
|
||||
- **degradation arms**: `unreachable`, `supported: false`, `sections {nginx: false}`, `resolver {available: null}`, `404 {configured:false}` — each in the payload's own words, none red;
|
||||
- **no sideways scroll** at 1440 px and 360 px with the widest fan-out expanded and both charts open;
|
||||
- screenshots `data/logs/32-gateway-{1440,360,counted,light,unresolved,filtered,reduced,phone}.png`, transcript `data/logs/32-counted-routes.out`.
|
||||
2. Re-run `data/logs/27-task02-mobile-audit.cjs` (nine pages, both widths) — exit 0, **0 FAIL / 0 FINDING**, tally recorded in the transcript.
|
||||
3. `AGENTS.md` — update every passage this phase made stale, in full:
|
||||
- the agent bullet: the `resolver` section (its `available` three-way and one-line-per-arm rule, `via: "libc"`, `config_directive`, `RESOLVE_MAX_NAMES`/`RESOLVE_CACHE_SECONDS`/pass budget, the single-flight background pass and *why* it is off the request path, `--no-resolve` removing the key in silence, and the never-resolve-anything-but-config-targets rule), `AGENT_VERSION` **1.3.0**, and the `/schema` addition;
|
||||
- `sources/gateway.py`'s join paragraph: two address sources (written-as-address and resolved), `out.resolved`/`via`/`resolved_ts`, normalised-not-string matching, the shared rule's now-real case, and the sentence that the dashboard **still** never resolves with the vantage reason;
|
||||
- the metric-naming passage: replace "no name-keyed `gateway.out_conns@…` name is ever inserted" with the widened rule and the reason it changed, keeping the note that an old name-keyed series stops growing rather than being rewritten;
|
||||
- `collectors.py`'s `ADDON_ERROR_PREFIXES` list gains `resolver`, beside the "a new add-on section adds its prefix here in the same commit" rule;
|
||||
- the `/gateway` UI-conventions paragraph: the dash now has two meanings and one glyph, provenance lives on `title`, the filter matches resolved addresses;
|
||||
- the **fleet record**: `gateway` runs **1.3.0**, `deskwork`/`driveripper` remain **1.1.0**, what the spread costs (nothing, with the additivity control named), the ten sources `ok` re-verified, and a pointer to `data/logs/NOTES-32-rollout.md`;
|
||||
- **gap list**: add the two follow-on lines (name-route switching = phase 33, amends invariant 1 and needs `PLAN.md` sign-off) and re-check gap 13's clip counts against this phase's measurements.
|
||||
4. `README.md` — the `/gateway` table row (name routes are counted through the gateway's own libc resolution), the gateway prose (the resolver section, its bounds, `--no-resolve`, what a dash means now), the capture-script section (its new output and its scrub rule), and the agent-version references wherever the page's minimum and the fleet's actual are named — kept as two different facts, not one number.
|
||||
5. Freshness and the whole gate:
|
||||
`uv sync --python 3.14 && uv pip list --outdated` → empty (this phase adds **no** dependency; if `cryptography` or anything else moved, that is a separate decision);
|
||||
`uv run pytest`, `uv run pytest --cov`, `uv run pytest -m live --live`, `uv run ruff check src tests agent`, `uv run pyright`, `bash .agents/validate.sh`.
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: this task **is** the live proof of the phase's headline claim; every number in the Completion Criteria is printed by `32-counted-routes.out` or a probe transcript, never asserted in prose.
|
||||
- Coverage: the floor is met on the phase's modified Python (`sources/gateway.py`, `collectors.py`, `agent/host_agent.py`); the run is recorded.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `data/logs/32-counted-routes.out` exits 0 and prints: animating-element counts per panel with at least one **name** endpoint among them, the five provenance `title`s, the zero-invention arms, the filter's zero-request proof, the both-themes colour equality, the reduced-motion pass, and 0 px sideways scroll at both widths.
|
||||
- [ ] The eight `data/logs/32-gateway-*.png` screenshots exist, including `counted` (name routes with numbers) and `phone`.
|
||||
- [ ] `data/logs/27-task02-mobile-audit.cjs` exits 0 with 0 FAIL / 0 FINDING, tally recorded.
|
||||
- [ ] `uv pip list --outdated` is empty; `uv run pytest` and `uv run pytest --cov` green above the floor; `uv run pytest -m live --live` green; `ruff` + `pyright` clean; `bash .agents/validate.sh` green.
|
||||
- [ ] `grep -n "1.3.0" AGENTS.md` shows the agent bullet **and** the fleet record; `grep -n "no name-keyed" AGENTS.md README.md` finds nothing (the retired sentence is gone, not contradicted); `resolver` appears in `ADDON_ERROR_PREFIXES`' documented list.
|
||||
- [ ] A full read of `README.md` and `AGENTS.md` finds no inconsistency anywhere in either file, including pre-existing ones — fixed in this phase's commits.
|
||||
- [ ] No behaviour change in completed work: `/`, `/system`, `/disks`, `/llm`, `/ups`, `/settings`, `/certs`, `/login` render as before and the unwired switch stays unwired (phase 33 wires it).
|
||||
@@ -0,0 +1,79 @@
|
||||
# Phase 33 — The switch is wired: turning one route off, and back on, without ever weakening the box
|
||||
|
||||
**Source:** operator request, 2026-10-23 — "Also write the phase to disable and enable services. I want to be able to turn off and on connections as needed in the event of security concerns or upgrades." + the two decisions taken with them in that conversation: **enforcement is a config edit, validated and reloaded, by a narrow root helper on the gateway** (so a disabled route stays disabled across a reload *and* a reboot), and **one switch per connection line, as already drawn**.
|
||||
**Story:** n/a
|
||||
**Context:** `deploy/install-agent.sh` (the unit template, whose `NoNewPrivileges=true` / `ProtectSystem=strict` / `SystemCallFilter=@system-service` / empty `CapabilityBoundingSet` this phase **does not touch**, the `--force` re-run guard, `deploy/agent-sudoers` as the precedent for a path-pinned grant), `agent/host_agent.py` (1.3.0 → **1.4.0**: `do_GET`-only `Handler`, `_certificate_allows`, the `haproxy`/`nginx` parsers that must learn the disabled marker), `src/dashboards/sources/gateway.py` (`connections[]`, `config_connections`, `warnings()`' seven kinds), `src/dashboards/api.py` (`/api/ups/command`'s four-gate shape and the `certs_router()`'s audit-event rule), `src/dashboards/collectors.py` (`AgentTls`, `fetch_gateway_snapshot`, the revoke-before-dial rule), `src/dashboards/config.py` (`GATEWAY_*` block), `src/dashboards/web/static/page-gateway.js` (`buildSwitch`, `routeInConfig`, `renderLeaf`), `tests/test_api.py` (the verb-enumeration pins), `tests/test_auth_api.py` (the served-route walk), `tests/test_deploy_scripts.py` (the fake-root sandbox the helper's tests extend).
|
||||
|
||||
## Objective
|
||||
|
||||
`/gateway`'s switch becomes a control: **disable** `brain.reeseapps.com` and the nginx `stream` map entry stops routing it, persistently, with the config validated and the proxy reloaded and the page showing that exact route **red** until it is re-enabled. One new write route, `POST /api/gateway/route`, gated seven times, with the write itself performed off-box by a **root helper on the gateway that the agent reaches over a systemd-activated unix socket** — because the agent's own unit forbids privilege (`NoNewPrivileges=true`) and this phase is not going to weaken the sandbox of a read-only collector to give it a root shell.
|
||||
|
||||
This amends AGENTS.md invariant 1 from *one* exception to **two**, and that is the phase's real risk: a bug here is an outage on the fleet's only ingress. Everything below exists to make the blast radius one line of one file, and to make the app refuse rather than improvise whenever it cannot see what it is doing.
|
||||
|
||||
## Scope decisions taken with the operator (binding for this phase)
|
||||
|
||||
| # | Decision | Consequence |
|
||||
|---|----------|-------------|
|
||||
| 1 | **Invariant 1 becomes "two exceptions".** | `POST /api/ups/command` and `POST /api/gateway/route`. Both: session, CSRF/origin on a cookie write, an env gate (`NUT_ALLOW_ACTIONS` / `GATEWAY_ALLOW_WRITES`, default **off**), a typed `DANGER`, and every gate re-checked server-side on every request. `PLAN.md`'s anchor table records a read-only app, so this phase's overview carries the sign-off request; the phase never edits `PLAN.md`. |
|
||||
| 2 | **`DANGER` is required for both verbs.** | Enabling is not the safe direction: applying either change re-validates the file and **reloads the proxy**, and a reload touches every route on the box. A word that means "I know this affects the ingress" has to mean it both ways. |
|
||||
| 3 | **The agent keeps its sandbox; the privilege lives outside it.** | No sudo from the agent, no setuid, no change to `NoNewPrivileges`/`ProtectSystem`/`SystemCallFilter`. The write peer is `dashboards-routectl.socket` + `dashboards-routectl@.service` (systemd socket activation, `Accept=yes`, `User=root`, `SocketUser=dashboards-agent`, `SocketGroup=dashboards-agent`, `SocketMode=0600`): the agent connects to a socket it is *filemode-permitted* to connect to, and no other uid on the box can even open it, and the helper proves the peer uid with `SO_PEERCRED`. Access control is a file mode plus the mTLS client certificate that already authorises the agent's other paths — not a capability the agent process can be tricked into. |
|
||||
| 4 | **Only an id crosses a wire. Never a path, never a line, never text.** | The body is `{id, action, danger?, dry_run?}` where `id` is the payload's own `connections[].id`. The helper re-parses the configs itself, maps that id to the one line it declared, and refuses anything it cannot map. There is no `file`, no `line`, no `pattern`, no `command` anywhere in the contract, and the argv to the helper is fixed with no shell. |
|
||||
| 5 | **A route is off in the way that actually stops traffic for its proxy.** | haproxy: the `server` line is commented with a `# dashboards:off … was: <verbatim original>` marker. nginx vhost: the `proxy_pass` line is commented the same way. nginx **stream `map` entry**: commenting it would fall through to the map's `default` (`127.0.0.1:4443`) and **mis-route** the domain instead of blocking it, so the helper rewrites that entry's value to a sink (`127.0.0.1:44450`) and keeps the original in the marker; the sink is a `server` block the helper maintains in `/etc/nginx/stream.d/00-dashboards-sink.conf`, which the existing `stream { include /etc/nginx/stream.d/*.conf; }` glob already loads — so **no hand edit of the main config is ever required**. The map's `default` entry is **refused**: it is the fallback for every unmapped name, not a service. |
|
||||
| 6 | **Validate, restore on failure, reload — never restart.** | Write the new bytes, run `nginx -t` / `haproxy -c -f …`; on failure put the previous bytes back, do **not** reload, and answer `applied: false, validated: false, restored: true`. On success `systemctl reload <unit>` — reload, because a restart cuts every Minecraft session and every in-flight upload on an ingress that is doing its job. A `reload` failure is reported with the unit's own status and the file is left in the applied state (the config is correct; the reload is the operator's next action, said out loud). |
|
||||
| 7 | **Self-describing marker + drift refusal, not silent trust.** | The disabled line carries the original verbatim after `was:`, so a human can restore it by eye and the helper can restore it mechanically. A sidecar `/var/lib/dashboards-routectl/state.json` (`0600`, root) records actor, timestamp and the sha256 of what was written. Enabling restores only if the current line still matches what the helper wrote; otherwise it **refuses** and reports `drift` — a file an operator hand-edited is theirs to fix, not the app's to guess at. |
|
||||
| 8 | **Refuse rather than act blind.** | The write is refused when: writes are not enabled; the id is not in the **current** payload; the route is already in that state (`changed: false`, no reload); either config section is not `available: true`; the `gateway` job is `ok: false` or stale beyond 3× its interval; the `host:<GATEWAY_HOST>` system job is not `ok`; the agent is revoked (refused before any socket, the existing rule); the throttle is tripped. That is the UPS "refused while on battery" rule, translated: do not rewrite a config you cannot currently read end to end. |
|
||||
| 9 | **`off` finally means something the payload said.** | The agent's parsers learn the marker and report the marked `server`/`proxy_pass`/`map` entry with `disabled: true`, so the route stays in the graph as a declaration and `connections[].enabled` becomes **`false`**. `routeInConfig()` stops being a literal, the `off` styling phase 30 built becomes reachable, and `warnings()` learns the marker: a backend emptied by a marker is not `haproxy_backend_empty` (the operator chose that; the page shows it red), while a backend with no `server` line at all still is. |
|
||||
| 10 | **A live route's count is never stamped.** | Disabling does not write `0` into the payload. The sockets drain on the reload and the next poll reports what `/proc` says. Invariant: no code path in this phase writes a count. |
|
||||
| 11 | **The audit is one event, fixed in shape, with no config text in it.** | `gateway`-source events `route-disabled` / `route-enabled` / `route-denied`: verb, id, the session's actor name, `applied`/`validated`/`reloaded`/`drift`, and the proxy's own one-line error when there was one. The unified diff is returned to **the caller's response** (it is what the confirm dialog shows) and never stored — events are served to a browser and `/api/events` is a feed, not a config archive. |
|
||||
| 12 | **The overview is told, because "something is switched off" is not a page-only fact.** | `/api/summary`'s `gateway` block gains **one** field, `disabled` (an int count of `enabled is false` rows, `null` in the arms that did not measure) — the thirteenth key, restated in the pinned key set with its reason. A notifier has to learn about a deliberate outage without scraping a page, and it prints nothing at `0` (the `warnings` rule: a permanent `0 switched off` is ink that trains the eye). |
|
||||
| 13 | **One route, one file, one reload per request.** | A request touches exactly one config file and reloads exactly one proxy. There is no batch verb, no "disable all to this host", no `dry_run` that writes. If a security incident needs five routes off, that is five reviewed requests, each with its own event. |
|
||||
|
||||
**Out of scope:** editing anything a marker cannot express (adding/removing vhosts, certificates, `bind` lines, the map's `default`), haproxy runtime `disable server` via a stats socket (a runtime-only state that a reload silently undoes — decision 5 is persistent instead), disabling an entire **host** or **port**, per-user authorisation (one admin, gap 6), a scheduler ("off at 02:00"), and any change to the agent's unit security directives or to another host's agent (`gateway` alone runs 1.4.0; `deskwork`/`driveripper`'s 1.1.0 agents have no routes and are unaffected).
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `32_gateway_resolved_target_liveness` (todo, **must complete first**) — agent 1.3.0's resolver section and the counted name routes are what a switched-off line reads against: `off` + a draining count to `0` is the proof the switch worked, and without resolution a disabled route and a never-counted route look identical.
|
||||
- `31_gateway_route_readability` (todo) — the row that names the inbound domain, the green end-to-end route, and the `off` styling this phase finally reaches.
|
||||
- `30_gateway_connection_map` (complete) — `connections[].id` as the vocabulary, the switch markup and its a11y contract, `routeInConfig()` as the named seam.
|
||||
- `15_agent_mtls` / `24_dev_mtls_rollout` (complete) — the verified-client-certificate gate the new POST path inherits, and the revoke-before-dial rule this phase's collector reuses.
|
||||
- `11_admin_login` (complete) — session, CSRF, origin checks, `LoginThrottle` (the `route:<address>` bucket is its third key shape).
|
||||
- **Operator actions, not repo work:** the `dashboards-agent`-to-helper socket grant lands through the units (no sudoers line needed), but installing units on a host and running the first live drill are the operator's — and the drill **stops** unless they are present. `PLAN.md`'s anchor sign-off for invariant 1 is theirs too.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_the_root_helper_edits_exactly_one_route.md` — `deploy/dashboards-routectl.py`: `show`/`list`/`diff`/`disable`/`enable`/`verify`, the marker with `was:`, the sink file it manages, validate-restore-reload, drift refusal, `--serve` over a systemd-passed socket, `--root` for tests.
|
||||
2. `02_the_privileged_peer_is_a_socket_not_a_sudo_line.md` — the two systemd units + `install-agent.sh --with-routectl`; the proof that the agent's `NoNewPrivileges=true`/`ProtectSystem=strict` unit is unchanged and can still reach the socket.
|
||||
3. `03_the_agent_accepts_one_write.md` — agent **1.4.0**: `POST /route` behind the verified client certificate **and** `--allow-writes`, body validation, one fixed peer call, plus the parsers reporting `disabled` for a marked line; `/schema` and the additivity control.
|
||||
4. `04_the_source_publishes_enabled_and_the_warnings_learn_the_marker.md` — `connections[].enabled`, `off` reachable in the payload, `warnings()` marker-aware, stub/skeleton parity, the one new `/api/summary` field (`gateway.disabled`), and no count ever written.
|
||||
5. `05_the_api_gates_the_write_ten_times.md` — `GATEWAY_ALLOW_WRITES`, `DASH_READ_ONLY`, `collectors.post_route`, `POST /api/gateway/route`, the ten-gate order, the throttle, the events, `/api/config.gateway.writes_enabled`, and the verb-enumeration + served-route pins restated (this is where invariant 1 is amended in `AGENTS.md`).
|
||||
6. `06_the_switch_obeys.md` — `page-gateway.js` + `gateway.html` + `dash.css`: `DANGER` confirm with the diff from a `dry_run` first, patch from the response, `routeInConfig()` reading `enabled`, the writes-disabled arm that keeps the switch honest, footer paragraph 3.
|
||||
7. `07_rehearse_then_drill_on_the_live_ingress.md` — the fake-tree rehearsal, then the live install on `gateway` and a drill on the one named-safe route, restore, verify, `data/logs/NOTES-33-switching.md`. Stops, correctly, without the operator present.
|
||||
8. `08_docs_and_live_verify.md` — `AGENTS.md` (invariant 1, the agent bullet, `sources/gateway.py`, `api.py`, `collectors.py`, `config.py`'s block, the `/gateway` UI paragraph, the fleet record, gap list) + `README.md` (including the operator's sudo/units steps), the browser pass, the mobile audit, the full gate.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- **The helper is tested unprivileged, against a fake tree, with the proxy commands stubbed.** `tests/test_routectl.py` runs the real helper in a `tmp_path` tree (`--root`) with `nginx`/`haproxy`/`systemctl` faked on `PATH`, and pins: one line changed and nothing else in the file (byte-diff assertion); `was:` restores verbatim; a failing validator restores the exact previous bytes and reloads nothing; `default` refused; an unknown id refused; a second disable `changed: false`; the sink file created on first use, left alone if an operator's file is there, removed only when it wrote it and nothing is disabled; drift refused rather than restored; `SO_PEERCRED` refused for a non-`dashboards-agent` uid in the fake-peer test; no `shell=True` anywhere (`grep` pin, the installer's precedent).
|
||||
- **The agent's write path is proven against a fake peer, not a real one.** `tests/test_agent.py` runs the real handler over a TLS loopback port (the phase-15 harness) plus a unix-socket stub: `POST /route` without a client certificate is 403 `client certificate required`; with `--allow-writes` absent it is 403 `writes are not enabled on this agent`; a body with an unexpected key, a bad id, or a missing `danger` is refused **before** the peer is touched (asserted by the stub logging nothing); `GET` behaviour is byte-identical to 1.3.0 (the additivity control re-run).
|
||||
- **The API gates are proven twice, as the UPS ones are.** `tests/test_api.py` walks every gate with spies: writes-disabled refuses without dialling; a stale `gateway` job refuses; a revoked agent refuses without a socket; `DANGER` is required for `disable` **and** `enable`; a right request produces one peer call, one event, and the response's `applied`/`validated`/`reloaded` triple. `tests/test_auth_api.py`'s served-route walk demands a session for `/api/gateway/route`, and the CSRF pin is extended to it on the **wire** (a scripted helper records nothing without the header, once with it) — a 403 after the write is an outage with an error message attached.
|
||||
- **Live, opt-in, and reversible:** `-m live --live` gains `test_the_switch_round_trip_on_one_named_route` which is **skipped unless `--allow-live-writes` is passed as well**, names the route in the skip reason, and always restores in a `finally`. The default live suite stays read-only (AGENTS.md's rule survives this phase; the drill belongs to task 07 with the operator watching).
|
||||
- Coverage **>90%** on `deploy/dashboards-routectl.py`, `agent/host_agent.py`, `sources/gateway.py`, `collectors.py`, `api.py`; the whole suite 100%; `bash .agents/validate.sh` green.
|
||||
|
||||
## Documentation & Consistency
|
||||
|
||||
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it will stand after this phase and fixes every inconsistency found, **including pre-existing ones**. This phase's specific list: **invariant 1** rewritten to two exceptions with the seven gates named (and the sentence "hiding a button is not a safety control" kept, because it is the reason the switches are gated server-side rather than greyed in JS); the agent bullet's `POST /route` and the `disabled` marker; the `sources/gateway.py` bullet's `enabled`/`off` rules and the marker-aware `warnings()`; the `api.py` bullet's write lists (fleet writes are now two); the `collectors.py` bullet (`post_route`, the revocation check it shares); `config.py`'s `GATEWAY_ALLOW_WRITES` row and README's configuration table; the `/gateway` UI-conventions paragraph (the switch is wired, what `off` looks like, what the writes-disabled arm says); the fleet record (gateway **1.4.0**, its unit's security directives unchanged, the two new systemd units, `NOTES-33-switching.md`); and the gap list (name-route switching **closed**, plus the honest limits: one admin, no scheduler, no per-host kill).
|
||||
- `README.md` gains the operator's install steps for the helper units and the exact `--with-routectl` invocation, plus the recovery command for the state file (`systemctl status dashboards-routectl.socket`, `routectl list`, `routectl verify`).
|
||||
- Tasks update both files immediately when their own changes are user-visible.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] With `GATEWAY_ALLOW_WRITES=1`, `/gateway` disables `brain.reeseapps.com`: the confirm dialog shows the **diff from a `dry_run`**, asks for `DANGER`, and the row turns **red** (`off`) on the poll after the reload while its live count drains to `0` on its own — no code wrote that zero.
|
||||
- [ ] `ssh gateway-ops 'sudo /usr/local/sbin/dashboards-routectl.py list'` names exactly that route; `verify` reports no drift; the marked line contains the original verbatim after `was:`; the map entry points at the sink and `stream.d/00-dashboards-sink.conf` exists and is owned by the helper's header.
|
||||
- [ ] Enabling restores the original bytes exactly (byte-compare against a pre-drill copy), the row turns green, and `list` is empty afterwards. **The live ingress ends the phase in the state it started in.**
|
||||
- [ ] Every gate refuses with its own sentence and **without** touching a peer: writes disabled, no session, no CSRF, wrong `Origin`, missing `DANGER`, unknown id, already-in-state, section unreadable, `gateway` job stale, `host:gateway` not `ok`, agent revoked. Each is a test, and each test asserts the stub peer logged nothing.
|
||||
- [ ] The agent's unit is unchanged: `systemctl cat dashboards-agent` still shows `NoNewPrivileges=true`, `ProtectSystem=strict`, `SystemCallFilter=@system-service`, and the same sha256 the phase recorded before installing the units — the sandbox was not traded for the feature.
|
||||
- [ ] A failing validator (`nginx -t` stubbed to exit 1) restores the previous bytes, reloads nothing, answers `applied: false, validated: false, restored: true`, and the page's line never turned `off`.
|
||||
- [ ] `POST` inventory: the app now has exactly two upstream writes (`/api/ups/command`, `/api/gateway/route`) and the enumeration pin lists them as such; `AGENTS.md` invariant 1 says **two** and names both gates; `/api/health` gained nothing.
|
||||
- [ ] `uv run pytest` (with `--cov` ≥ 94), `uv run ruff check src tests agent deploy`, `uv run pyright`, `node --check`, `bash .agents/validate.sh` all green; `uv run pytest -m live --live` green **without** `--allow-live-writes` (the round-trip test skipped by default, its reason naming the route it would touch).
|
||||
- [ ] `data/logs/33-*.png` (switch off in both themes, the `DANGER` dialog with its diff, the writes-disabled arm, 360 px) and `data/logs/33-*.out`; `data/logs/27-task02-mobile-audit.cjs` exits 0 with 0 FAIL / 0 FINDING; `data/logs/NOTES-33-switching.md` records the drill, the hashes and the restore.
|
||||
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass), including the operator's steps for undoing all of this (`--without-routectl`, remove the units, re-enable every route).
|
||||
- [ ] No behaviour change in completed work: the other five dashboards, `/settings`, `/certs`, `/login`, the UPS command path and its four gates, and every non-gateway source behave exactly as phase 32 left them.
|
||||
+71
@@ -0,0 +1,71 @@
|
||||
# Task 01 — The root helper edits exactly one line, proves it validates, and reloads or gives up
|
||||
|
||||
**Phase:** `33_gateway_route_switching` · **Source:** operator decision 2026-10-23 — "Edit the config + validate + reload, via a narrow root helper on the gateway"
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
`deploy/dashboards-routectl.py` — a stdlib-only root program that turns one `connections[].id` off or back on in one of the two proxy configs, validates the file with the proxy's own checker, reloads that one proxy, and refuses everything else. It is the phase's only writer, it has no network listener of its own (task 02 hands it a systemd-activated socket), and it can be run by hand on the gateway by the operator to see exactly what the dashboard would do.
|
||||
|
||||
The helper **imports the agent's config parser by file path** and never re-implements a rule: two parsers reading one config is two opinions about the same file, which is the failure mode this repo has pinned against since phase 28. `agent/host_agent.py` is world-readable at `/usr/local/lib/dashboards-agent/host_agent.py`, so the helper imports it with `importlib` and refuses to run at all if it cannot.
|
||||
|
||||
## Work
|
||||
|
||||
1. `agent/host_agent.py` — the shared grammar only (no version bump here; `AGENT_VERSION` moves in task 03):
|
||||
```python
|
||||
ROUTE_MARKER = "# dashboards:off" # the only marker this app writes into a proxy config
|
||||
ROUTE_WAS = "was:" # `<marker> <id> was: <original line verbatim>`
|
||||
def marker_for(route_id: str, original: str) -> str: ...
|
||||
def split_marker(line: str) -> tuple[str, str] | None: # (route_id, original) or None
|
||||
def config_line_index(haproxy_text, haproxy_path, nginx_docs) -> dict[str, dict[str, Any]]:
|
||||
"""route id -> {"file", "line", "original", "kind", "disabled"} for every declared target."""
|
||||
```
|
||||
`config_line_index` walks the parsed declarations **with their source lines**, so the id → (file, line, verbatim text) map is derived by the same parser that produced the graph, and a marked line is reported as the same id with `disabled: true`. It is pure over text; the file reading stays in the callers (`Collector` for the agent, the helper for the writes), both resolved under `--root`.
|
||||
2. `deploy/dashboards-routectl.py` (new, `0755`, stdlib, no third-party import, no `shell=True`)
|
||||
- **Arguments** (all default to the deployed paths, all overridable for the fake tree):
|
||||
`--root /` · `--haproxy-config /etc/haproxy/haproxy.cfg` · `--nginx-config /etc/nginx/nginx.conf` · `--state /var/lib/dashboards-routectl/state.json` · `--sink-conf /etc/nginx/stream.d/00-dashboards-sink.conf` · `--sink-address 127.0.0.1:44450` · `--agent-module /usr/local/lib/dashboards-agent/host_agent.py` · `--nginx-unit nginx` · `--haproxy-unit haproxy` · `--apply` (without it every write verb is a dry run).
|
||||
- **Verbs**, one JSON object to stdout, non-zero exit on refusal:
|
||||
`list` · `show --id ID` · `diff --id ID --action disable|enable` · `disable|enable --id ID [--apply]` · `verify` (marker/hash vs file → drift report) · `--serve` (read one JSON request from the systemd-passed socket, answer one JSON line).
|
||||
- **Per-kind edit, exactly as decision 5** (the three arms and the `default` refusal are this file's core; a fourth arm is a new decision, not a fall-through):
|
||||
| id kind | action | why |
|
||||
|---|---|---|
|
||||
| haproxy `server` | comment the line with `ROUTE_MARKER` + `was:` | the backend stops receiving; the marker keeps the original restorable by eye and by machine |
|
||||
| nginx vhost `proxy_pass` | comment the line the same way | the vhost keeps `listen` + `server_name` and gains no target — a shape `warnings()` already treats as quiet (the `:80 return 301` arm), so "off" does not read as "broken" |
|
||||
| nginx stream `map` entry | **rewrite the value** to the sink address, marker + `was:` on the line, ensure the sink file exists | commenting would fall through to the map's `default` (`127.0.0.1:4443`) and **mis-route the domain instead of blocking it** |
|
||||
| nginx `map` `default` | **refuse** — `{"applied": false, "refused": "map-default"}` | it is the fallback for every unmapped SNI, not a service |
|
||||
- **The sink file**: created on first need with a `# managed by dashboards-routectl — do not edit` header and a stream `server` block that listens on `127.0.0.1:44450` and returns a short refusal string; loaded by the existing `stream { include /etc/nginx/stream.d/*.conf; }` glob, so **the main config is never edited**. If a file already exists at that path without the header, the helper refuses (an operator's file is not its own to overwrite). It deletes the file only when it wrote it **and** nothing is disabled, and never removes it while any map entry still points at the sink.
|
||||
- **Write protocol** (order is the safety property, so it is written as numbered steps in the file):
|
||||
1. re-read the file; index it; map `id` → line (unknown id → refuse, nothing written);
|
||||
2. already in the requested state → `{"applied": false, "changed": false}` and **no reload**;
|
||||
3. `--apply` absent → answer with the unified `diff` only;
|
||||
4. write to `mkstemp` in the same directory, `fsync`, `os.replace`;
|
||||
5. run `nginx -t` or `haproxy -c -f <that file>`; on failure `os.replace` the previous bytes back and answer `{"applied": false, "validated": false, "restored": true}` with the checker's own first line — **and reload nothing**;
|
||||
6. `systemctl reload <unit>`; a failure is reported as `reloaded: false` with the unit's status line, leaving the applied file in place (the config is right; the reload is the operator's next action and the answer says so);
|
||||
7. re-read and re-index to prove the marker is live; record state.
|
||||
- **Never `restart`** — the docstring says why in one sentence: a restart cuts every live connection through the fleet's only ingress, and a reload does not.
|
||||
- **State sidecar**: `0600`, root, `{id: {"actor", "at", "file", "sha256", "action"}}`, written with `os.open(..., O_CREAT, 0o600)` so the mode is its birth mode (the `dev-certs-export.py` idiom). `enable` restores **only** when the current line hashes to what the helper wrote; otherwise `{"applied": false, "drift": true, "hint": "…"}` and no byte changes.
|
||||
- **`--serve`**: `socket.fromfd(3, AF_UNIX, SOCK_STREAM)` (or the `LISTEN_FDS`/stdin idioms, chosen once and documented), read one newline-delimited JSON request, `getsockopt(SO_PEERCRED)` and **refuse any uid that is not the agent's** (`--allow-uid`, default `dashboards-agent`, resolved with `pwd.getpwnam`), dispatch one verb, write one JSON line, exit. One request per activation: systemd spawns the instance per connection (`Accept=yes`), so there is no accept loop, no thread, no timeout policy to invent.
|
||||
- **Nothing in this file takes a path, a line number, a pattern or a command from the wire.** `--serve`'s accepted keys are exactly `{id, action, dry_run}` and every other key is a refusal. Assert it structurally in the tests.
|
||||
- Every subprocess call is a fixed argv list (`subprocess.run([...])`, no `shell=True`), and the checker/unit names come from **flags the operator installed**, never from the request.
|
||||
3. `tests/test_routectl.py` (new) — real helper, fake tree (`--root`), fake `nginx`/`haproxy`/`systemctl` scripts on `PATH` that record their argv and answer to order:
|
||||
- `test_disabling_one_route_changes_exactly_one_line` — unified diff of the whole file is one hunk, one line; every other byte identical (assert on the bytes, not on a count).
|
||||
- `test_a_disabled_map_entry_points_at_the_sink_and_keeps_its_original` — the sink file exists with its header, the marked line carries `was: <verbatim>`, and re-enabling restores the file byte-identical to the pre-drill capture.
|
||||
- `test_the_map_default_is_refused_and_nothing_is_written`, `test_an_unknown_id_is_refused`, `test_disabling_a_disabled_route_changes_nothing_and_reloads_nothing` (the fake `systemctl` log stays empty).
|
||||
- `test_a_failing_validator_restores_the_previous_bytes` — byte-identical restore, `validated: false`, `restored: true`, zero reload calls, non-zero exit.
|
||||
- `test_a_failing_reload_says_so_and_leaves_the_file_applied`, and `test_reload_is_never_a_restart` (grep the source for `restart` in a call position, the installer's `--status` style pin).
|
||||
- `test_the_helper_imports_the_agents_parser_and_refuses_without_it` — point `--agent-module` at a non-file: exit non-zero, no write, and the message names the path (a helper that silently skipped the parser would be a second opinion about the config).
|
||||
- `test_state_is_0600_and_drift_is_refused` — mode at birth; hand-edit the marked line and `enable` refuses with `drift: true` and no byte change.
|
||||
- `test_serve_refuses_a_peer_that_is_not_the_agent_uid`, `test_serve_accepts_exactly_id_action_dry_run`, `test_no_shell_and_no_path_from_the_wire` (structural greps over the file, the `certs.py`/`auth.py` precedent).
|
||||
- `test_the_sink_file_is_left_alone_when_someone_elses_wrote_it`, `test_the_sink_is_removed_only_when_it_is_empty_of_our_routes`.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the whole matrix above, hermetic and unprivileged, over the **captured** `gateway_haproxy.cfg` / nginx tree so the edits are pinned against real config bytes rather than a toy file.
|
||||
- Coverage **>90%** on `deploy/dashboards-routectl.py` and the new `agent/host_agent.py` functions; `uv run ruff check agent deploy` + `uv run pyright` clean; `python3 -m py_compile` on the helper (it must run on a host's Python 3.10, so no 3.14-only syntax — pin that with a `# requires: python >=3.10` header and a test that greps for `match `/`typing.Self`-style 3.11+ syntax if the repo has such a pin; otherwise assert `ast` parses with `feature_version=(3,10)`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run python deploy/dashboards-routectl.py --root <tmp> … diff --id nginx:443->reeseapps-brain-of-reese.reeselink.com:443` prints a one-hunk diff and writes nothing; with `--apply` it edits, validates, reloads (against the fakes) and answers `{"applied": true, "validated": true, "reloaded": true}`.
|
||||
- [ ] Re-enabling leaves the tree byte-identical to its pre-drill `sha256sum` for **both** config files.
|
||||
- [ ] `grep -n "shell=True\|restart" deploy/dashboards-routectl.py` finds neither in a call position; `grep -n "def \|argparse" ` shows no `--file`/`--line`/`--pattern` option anywhere.
|
||||
- [ ] `uv run pytest tests/test_routectl.py -q` 100% passing; whole suite green; coverage not down.
|
||||
- [ ] README.md/AGENTS.md note the helper's existence and its verbs (this task's user-visible surface — the operator can run it by hand) — the phase's final pass re-checks both project-wide.
|
||||
- [ ] No behaviour change in completed work: `/snapshot`'s document is unchanged by this task (the `AGENT_VERSION` bump and the `disabled` field are task 03's, with the additivity control).
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
# Task 02 — The privileged peer is a socket systemd activates, not a privilege the agent is given
|
||||
|
||||
**Phase:** `33_gateway_route_switching` · **Source:** the agent unit's `NoNewPrivileges=true` + `ProtectSystem=strict` (read from `deploy/install-agent.sh`), which this phase must not weaken to get a write
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Install the helper on the gateway as a **socket-activated root service** the agent may connect to and nothing else: `dashboards-routectl.socket` owned by the agent user at `0660`, `dashboards-routectl@.service` running as root with `Accept=yes`. The agent's own unit keeps every one of its security directives — the phase's central safety claim is that a read-only collector stayed read-only in its sandbox and talked to a peer outside it.
|
||||
|
||||
## Work
|
||||
|
||||
1. `deploy/dashboards-routectl.socket` (new)
|
||||
```ini
|
||||
[Socket]
|
||||
ListenStream=/run/dashboards-routectl.sock
|
||||
SocketUser=dashboards-agent
|
||||
SocketGroup=dashboards-agent
|
||||
SocketMode=0600
|
||||
NoDelay=yes
|
||||
[Install]
|
||||
WantedBy=sockets.target
|
||||
```
|
||||
`0600` + `SocketUser=dashboards-agent` is the whole access control at the file layer: only that uid can connect, everyone else gets `EACCES` before any code of ours runs. The peer check in task 01's `--serve` is the **second** look, not the first.
|
||||
2. `deploy/dashboards-routectl@.service` (new)
|
||||
```ini
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/local/sbin/dashboards-routectl.py --serve \
|
||||
--allow-uid dashboards-agent --state /var/lib/dashboards-routectl/state.json
|
||||
User=root
|
||||
StandardInput=socket
|
||||
StandardOutput=socket
|
||||
StateDirectory=dashboards-routectl
|
||||
ProtectSystem=read-only # it writes two config files by absolute path; see ReadWritePaths
|
||||
ReadWritePaths=/etc/haproxy /etc/nginx/stream.d
|
||||
ProtectHome=read-only
|
||||
PrivateTmp=yes
|
||||
NoNewPrivileges=true
|
||||
SystemCallFilter=@system-service
|
||||
```
|
||||
Three notes the file must carry as comments: the helper needs `/etc/haproxy` and `/etc/nginx/stream.d` writable and **nothing else** (`/etc/nginx/conf.d` is deliberately absent — vhost edits land there, so the *vhost* arm is listed in `ReadWritePaths` too and that addition is a decision recorded in the file, not smuggled in); `NoNewPrivileges=true` here as well, because it gets its privilege from being started by systemd as root, not from escalation; and `Accept=yes` on the template means systemd spawns one instance per connection with the socket on stdin, which is why the helper handles exactly one request and exits.
|
||||
- **`ReadWritePaths` must list every directory whose file the helper replaces** (`os.replace` needs write on the directory): `/etc/haproxy`, `/etc/nginx/conf.d`, `/etc/nginx/stream.d`. Getting this wrong is a `EROFS` in production, so the live drill in task 07 is the test that matters and the units file names it.
|
||||
3. `deploy/install-agent.sh` — one new flag, `--with-routectl`, off by default, and its inverse `--without-routectl`:
|
||||
- installs `dashboards-routectl.py` to `/usr/local/sbin/` (`0755`, root) and the two units to `/etc/systemd/system/`, `daemon-reload`, `systemctl enable --now dashboards-routectl.socket`;
|
||||
- **changes nothing about the agent's own unit** — assert that structurally: the rendered `dashboards-agent.service` bytes are identical with and without the flag (`tests/test_deploy_scripts.py`), because the temptation to "help" the agent reach the socket by relaxing it is exactly what this task exists to close;
|
||||
- verifies reachability from the agent's own uid before reporting success: `sudo -u dashboards-agent python3 -c 'import socket; s=socket.socket(socket.AF_UNIX); s.connect("/run/dashboards-routectl.sock")'` — a socket the agent cannot connect to is a feature that fails at 03:00 during an incident, so this check is part of the install, and its output is printed;
|
||||
- `--without-routectl` disables and removes the units and the helper, leaves `/etc` and the state directory alone, and reminds the operator to re-enable any route still off (`routectl list` output printed first).
|
||||
- The re-run guard applies as it does to the unit: a differing installed helper is a diff + exit 1 unless `--force`.
|
||||
4. `deploy/gateway-routectl-README.md` (new, short) — the operator's page: what the two units are, how to see them (`systemctl status dashboards-routectl.socket`, `journalctl -u 'dashboards-routectl@*'`), how to run the helper by hand on the box, how to undo it all, and the sentence that **no sudoers line is involved**: the grant is a file mode systemd enforces.
|
||||
5. `tests/test_deploy_scripts.py` — extend the fake-root sandbox:
|
||||
- `test_with_routectl_installs_helper_and_units_and_leaves_the_agent_unit_byte_identical`;
|
||||
- `test_the_socket_is_0600_and_owned_by_the_agent_user` (the fake `install`/`chown`/`chmod` log, the precedent that proved the bundle's `0640`);
|
||||
- `test_the_helper_is_installed_0755_root_and_its_state_dir_is_created`;
|
||||
- `test_without_routectl_removes_the_units_and_not_the_config_or_the_state`;
|
||||
- `test_no_new_sudoers_line_exists` (the `deploy/agent-sudoers` file is unchanged, and nothing in the new units grants sudo);
|
||||
- `test_the_install_checks_the_agent_can_reach_the_socket` — a fake where that connect fails makes the install exit non-zero rather than reporting a success nobody can use.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the pins above; the units are also parsed (`systemd-analyze verify` is not available in the sandbox, so the assertions are on the directive lines, and task 07 runs the real `systemctl` on the gateway).
|
||||
- Coverage: the installer is shell; the gate is the sandbox tests plus `bash -n`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `bash -n deploy/install-agent.sh` clean; `grep -n "NoNewPrivileges\|ProtectSystem\|SystemCallFilter" deploy/install-agent.sh` shows the agent unit's values unchanged from before this task (the guard against "fixing" reachability by weakening the sandbox), and `git diff` for this task shows no edit inside the agent unit template block.
|
||||
- [ ] `uv run pytest tests/test_deploy_scripts.py -q` 100% passing, including the byte-identical agent-unit assertion; whole suite green; coverage not down.
|
||||
- [ ] The two unit files list exactly the paths the helper writes and no others, with the `os.replace`-needs-directory reason stated in a comment.
|
||||
- [ ] README.md/AGENTS.md gain the two unit names and the "no sudoers line" sentence (this task's user-visible surface) — the phase's final pass re-checks both project-wide.
|
||||
- [ ] No behaviour change in completed work: an install **without** `--with-routectl` is byte-identical to an install before this phase, on every host.
|
||||
@@ -0,0 +1,50 @@
|
||||
# Task 03 — Agent 1.4.0: one POST path, behind the certificate and a flag, plus the marker read back as `disabled`
|
||||
|
||||
**Phase:** `33_gateway_route_switching` · **Source:** the agent's `do_GET`-only `Handler` and `_certificate_allows` (phase 15), and phase 30's `routeInConfig()` seam
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
`host_agent.py` goes **1.3.0 → 1.4.0** with two additions and no other change: the two config parsers report a marker-authored line as a declared target with **`disabled: true`**, and one new path, `POST /route`, which is reachable only with a verified client certificate **and** an agent started with `--allow-writes`, forwards exactly `{id, action, dry_run}` to the task-02 socket, and answers with the helper's JSON. Every `GET` behaviour stays byte-identical, proven by the additivity control.
|
||||
|
||||
## Work
|
||||
|
||||
1. `agent/host_agent.py`
|
||||
- Parsers: each haproxy `server` node and each nginx `proxy_pass` / `upstream` member / `map` entry gains `"disabled": <bool>` — `false` for an ordinary line, `true` for one carrying `ROUTE_MARKER` (read with `split_marker`). Additive: no existing key changes meaning, and a route that is off **stays in the document** because "the config declares this and the operator switched it off" is a different fact from "the config never said it" (rule 2).
|
||||
- **New path** in `Handler`:
|
||||
```python
|
||||
def do_POST(self) -> None: # the BaseHTTPRequestHandler API name — keep it
|
||||
if self.path.split("?", 1)[0].rstrip("/") != "/route":
|
||||
self._send(405, {"ok": False, "detail": "no such write path"})
|
||||
return
|
||||
# the certificate check first, exactly as do_GET does it, revoked serial included;
|
||||
# then the flag; then the body; then the peer. Nothing else in this method writes.
|
||||
```
|
||||
Order, and each refusal is its own sentence: no bundle / no verified certificate → `403 client certificate required` (and `403 client certificate revoked` for a withdrawn serial, on this path too); agent started **without** `--allow-writes` → `403 writes are not enabled on this agent`; body not a JSON object, longer than `_ROUTE_MAX_BODY = 4096`, carrying a key outside `{id, action, dry_run}`, an `action` outside `{disable, enable}`, or an `id` longer than 256 chars → `422` with a fixed message and **no peer call**; no `--routectl-socket` configured → `503 no route helper on this host`.
|
||||
- Peer call: one newline-delimited JSON request over `AF_UNIX`, `settimeout(ROUTE_TIMEOUT_SECONDS = 30)` (the helper may run `nginx -t` and a reload), one JSON line back, oversized or unparsable replies are `502 route helper answered something unreadable` and **never** retried — a retry of a write is a second reload. Any `OSError` → `502` with the errno word, no traceback text, no path echoed beyond the flag's own configured value.
|
||||
- **No token path ever gets this**: on the plain-`http` bearer-token arm, `POST /route` is refused with `403 writes require a certificate` — the same reasoning phase 15 applied to reads (the handshake is the identity), and this app's fleet is mTLS anyway.
|
||||
- `--allow-writes` (default off) and `--routectl-socket /run/dashboards-routectl.sock`; both appear in the `/schema` doc block. `--status` and `--uninstall` are unaffected.
|
||||
- `log_message`: the POST line prints the path and the status only — never the body, never the id (an id is a hostname, and the agent's log line convention is that query strings stay out).
|
||||
- `AGENT_VERSION = "1.4.0"`; `/schema` names `POST /route`, `--allow-writes`, `--routectl-socket`, and the `disabled` field on the proxy sections; the status page lists the write path with its refusal sentence so a human with a browser is not surprised by a 405.
|
||||
2. `tests/test_agent.py` — the TLS loopback harness (phase 15) plus a unix-socket stub peer that logs what it received:
|
||||
- `test_the_agent_version_is_1_4_0_and_the_schema_names_the_write_path`;
|
||||
- `test_a_post_without_a_client_certificate_is_refused` and `test_a_revoked_serial_is_refused_on_the_write_path_too` (the denylist is not a read-side rule);
|
||||
- `test_writes_off_refuses_before_the_peer_is_touched` (stub logs nothing);
|
||||
- `test_the_body_vocabulary_is_id_action_dry_run_exactly` — a key added, a wrong `action`, a 5 KB body, a non-object: all refused, all with the peer idle;
|
||||
- `test_a_good_request_reaches_the_peer_once_and_the_answer_is_passed_through` (including `applied:false` staying `false` — the agent invents no success), `test_the_peer_is_called_once_even_when_it_times_out`, `test_an_unreadable_peer_answer_is_502_and_is_not_retried`;
|
||||
- `test_token_mode_cannot_write`; `test_every_other_path_is_405`;
|
||||
- `test_a_marked_server_is_reported_disabled_and_stays_in_the_document` (haproxy + all three nginx arms), `test_an_unmarked_config_has_disabled_false_everywhere`.
|
||||
- **Additivity control re-run**: `data/logs/verify-33-additive.py` (from task 02 of phase 32) extended to demand 1.4.0's `GET /snapshot` equals 1.3.0's with `disabled` added at the target level and nothing else, and equal *exactly* on an unmarked config.
|
||||
3. `src/dashboards/collectors.py` — a `post_route(tls, target, body)` beside `fetch_gateway_snapshot`, reusing the same `AgentTls` material and the **same revoke-before-socket check**, and redacting the URL in every error string it produces (invariant 6: those strings become events).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the matrix above; hermetic (a stub unix socket, no helper, no root, no proxy binaries).
|
||||
- Coverage **>90%** on the new agent code and `collectors.post_route`; `ruff` + `pyright` clean.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `python3 agent/host_agent.py --help` lists `--allow-writes` and `--routectl-socket`; `/schema` names `POST /route` and `disabled`.
|
||||
- [ ] Against the fake peer: the full refusal matrix answers without a peer call, and the success path calls it **exactly once** and passes the JSON through unaltered.
|
||||
- [ ] `uv run python data/logs/verify-33-additive.py` exits 0: 1.4.0 equals 1.3.0 plus `disabled`, and equals it exactly on unmarked configs.
|
||||
- [ ] `uv run pytest` **100% passing**, coverage not down; `bash .agents/validate.sh` green.
|
||||
- [ ] README.md/AGENTS.md note the write path and its two agent-side gates (this task's user-visible surface) — the phase's final pass re-checks both project-wide.
|
||||
- [ ] No behaviour change in completed work: `GET /snapshot`, `/health`, `/info`, `/schema` and the TLS matrix behave exactly as 1.3.0; `deskwork`/`driveripper` are untouched.
|
||||
+46
@@ -0,0 +1,46 @@
|
||||
# Task 04 — `enabled` joins the payload, `off` becomes reachable, and the warnings stop calling a chosen outage a fault
|
||||
|
||||
**Phase:** `33_gateway_route_switching` · **Source:** phase 30 decision 1 ("red is the **off** state, styled now, unreachable until the write path exists")
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
`connections[].enabled` carries the config's own state (`true` declared and active, `false` declared and marker-off), so `routeInConfig()` stops being a literal and the red styling phase 30 built has a referent. And `warnings()` learns the marker: an empty backend, a targetless vhost or a sink-pointed map entry **caused by the operator's own switch** is not a config-integrity fault, while the same shape caused by a hand-deleted line still is. No count is ever written by this phase.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/sources/gateway.py`
|
||||
- `_connection()` gains `"enabled": True`; `config_connections()` sets it from the agent's per-target `disabled` flag (`enabled = not disabled`), and the merge rule keeps its meaning: **a line is enabled only if every claim that merged into it is enabled** — a union of selectors with one disabled line is two lines, so if a merge would mix states the source must not silently union them; state which arm wins and test it (recommended: split into two rows keyed by `enabled`, because "one of these selectors is off" is two facts).
|
||||
- `_skeleton()`/`stub()` gain the key so the parity pin keeps holding; `enabled` is `None` in the stub (nobody read the config), `true`/`false` in an answer — never `false` in an arm that did not read it.
|
||||
- `warnings()` marker-awareness, per kind, and each is a decision to write down rather than a suppression to hope nobody notices:
|
||||
| kind | new rule |
|
||||
|---|---|
|
||||
| `haproxy_backend_empty` | not raised when the backend's only servers are `disabled: true` (the operator emptied it from `/gateway`); still raised when it has no `server` line at all |
|
||||
| `haproxy_route_dangling` / `haproxy_backend_orphan` | unchanged — a switch never renames or removes a `backend`/`use_backend`, so a dangling name is still a broken config |
|
||||
| `nginx_server_unlistened` | unchanged (a disabled `proxy_pass` does not remove a `listen`) |
|
||||
| `nginx_pool_undeclared` / `nginx_variable_unmapped` | unchanged |
|
||||
| `nginx_map_route_missing` | a map entry pointing at the **sink address** is not a missing route; it is a switched-off route, and the sink address is recognised from the payload's own `resolver`/config text rather than hard-coded in the source (a literal in two places is how this drifts) |
|
||||
- The **socket join** is unchanged in code and in ethics: a disabled haproxy `server` has no live sockets after the reload because they drain, and a disabled map entry's sockets to the sink are the sink's own loopback pair. Nothing stamps `0`, nothing zeroes a count, and the `shared` rule is untouched.
|
||||
- `samples()` publishes the gauge for an `enabled: false` line only if the join measured a real count for it (usually `0`, and `0` measured is a legitimate measurement — the distinction is *measured*, never *assumed*).
|
||||
- No `severity` anywhere, still: `off` is a state, not a verdict. The recursive no-verdict walk in `tests/test_api.py` must stay green without an exception for `enabled`.
|
||||
2. `src/dashboards/api.py` — `/api/gateway/snapshot` is otherwise unchanged, but the `gateway` block in `/api/summary` is the one place this phase **does** add a field, because the overview's one line is where "something is switched off" belongs and a page-only fact would be invisible to a notifier: `disabled` (int, count of `enabled is false` rows, `null` in the arms that did not measure). That makes thirteen keys — restate the pinned key set in the same commit, with the reason, and keep `warnings` as a count beside it.
|
||||
3. `tests/test_gateway_source.py`
|
||||
- `test_a_marker_off_route_is_declared_and_disabled` — a config tree with one marked haproxy server + one marked vhost `proxy_pass` + one sink-pointed map entry yields three rows, each `enabled: false`, all their selectors intact, every other row `enabled: true`.
|
||||
- `test_the_source_never_calls_a_route_broken_because_the_operator_disabled_it` and `test_the_same_shape_without_a_marker_is_still_a_warning` — both directions for each of the two affected kinds, run over the **captured** configs plus a marker-injected copy of them (never a toy file).
|
||||
- `test_the_sink_address_is_recognised_not_hard_coded` — a config that writes a different sink port still reads as switched-off, because the sink is identified from the marker+config, not from `44450`.
|
||||
- `test_a_disabled_line_still_publishes_only_a_measured_count` — with no sockets, `connections: 0` only when the table answered; `null` when it did not, and no sample in the second case.
|
||||
- `test_the_stub_and_the_answer_share_one_key_set` gains `enabled`; `test_the_gateway_snapshot_carries_no_verdict_and_no_window` unchanged (extended walk).
|
||||
4. `tests/test_api.py` — the `/api/summary` `gateway` block key set grows `disabled` with its reason in the test docstring; `test_the_overview_line_reads_the_gateway_block_it_claims` learns the new field (and that `0` prints nothing — a permanent `0 switched off` is the ink the empty-state rules exist to avoid, the `warnings` rule restated).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: every rule above from real captures plus injected markers; both directions for each suppression.
|
||||
- Coverage **>90%** on `sources/gateway.py` and `api.py`; whole suite 100%.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `GET /api/gateway/snapshot` on a fixture with three markers shows three rows with `enabled: false`, their selectors and endpoints intact, and `warnings == []` for the marker-caused shapes.
|
||||
- [ ] Removing the markers from the same tree brings the corresponding warnings back (the suppression is marker-caused, not shape-caused) — asserted as one test pair.
|
||||
- [ ] `/api/summary`'s `gateway.disabled` is `3` on that fixture, `0` on the clean capture, `null` on the stub, and the overview line prints nothing for `0`.
|
||||
- [ ] No `severity` key at any depth, still (the recursive walk passes unmodified).
|
||||
- [ ] `uv run pytest` **100% passing**, coverage not down; `bash .agents/validate.sh` green.
|
||||
- [ ] README.md/AGENTS.md's `/gateway` paragraph names `enabled`/`off` and the marker-aware warnings rule — the phase's final pass re-checks both project-wide.
|
||||
- [ ] No behaviour change in completed work: the socket join, inbound counts, chart gauges and the phase-32 provenance fields are untouched.
|
||||
+52
@@ -0,0 +1,52 @@
|
||||
# Task 05 — One route, ten gates: `POST /api/gateway/route` and the amendment of invariant 1
|
||||
|
||||
**Phase:** `33_gateway_route_switching` · **Source:** operator request — "turn off and on connections as needed in the event of security concerns or upgrades" · **Shape borrowed from:** `/api/ups/command`'s four gates and `/api/certs/*`'s fixed-text audit events
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The app's **second and last** upstream write. `POST /api/gateway/route {id, action, danger?, dry_run?}` refuses in nine places before it dials anything, and in the tenth it forwards one reviewed request through `collectors.post_route` and answers with the helper's own triple. `GATEWAY_ALLOW_WRITES` defaults to **off**, `DASH_READ_ONLY` still refuses everything, and `AGENTS.md` invariant 1 is rewritten in this commit to name two exceptions — because a change to a written invariant that leaves the document describing the old one is the bug, not the omission.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/config.py` — two settings beside the existing `GATEWAY_*` block, each with the reason in its comment:
|
||||
- `GATEWAY_ALLOW_WRITES: bool` (default `False`, the same truthy rule as `NUT_ALLOW_ACTIONS`);
|
||||
- `GATEWAY_TIMEOUT_SECONDS` **unchanged** (reads), plus `GATEWAY_WRITE_TIMEOUT_SECONDS: float = 35.0` — deliberately longer than the agent's own 30 s peer timeout so the dashboard never times out first and leaves the operator guessing whether the reload happened.
|
||||
2. `src/dashboards/collectors.py` — `post_route(tls, target, body, timeout)` beside `fetch_snapshot`: the same `AgentTls` client (built per dial, the phase-15 rule), the same **revocation check before any socket opens**, `Authorization` never used on the certificate path, and every error message run through `strip_userinfo` (those strings reach events and `/api/health`-adjacent payloads — invariant 6).
|
||||
3. `src/dashboards/api.py` — the route, and the gate order is the design, so it is written in the function as numbered comments and mirrored in the tests:
|
||||
| # | Gate | Refusal |
|
||||
|---|---|---|
|
||||
| 1 | session (middleware) | `303`/`401`, unchanged |
|
||||
| 2 | `Origin` / `Sec-Fetch-Site` | `403`, unchanged |
|
||||
| 3 | this session's `X-CSRF-Token` on a cookie write | `403`, unchanged |
|
||||
| 4 | `DASH_READ_ONLY` | `403 "this dashboard is read-only (DASH_READ_ONLY)"` |
|
||||
| 5 | `GATEWAY_ALLOW_WRITES` | `403 "route switching is disabled — set GATEWAY_ALLOW_WRITES=1 to enable it"` |
|
||||
| 6 | throttle `route:<address>` on `ctx.login_throttle` | `429` + `Retry-After` (its third key shape; the per-process caveat stays gap 7) |
|
||||
| 7 | body: `id` non-empty string ≤ 256, `action` in `{disable, enable}`, `dry_run` optional bool, **no other key** | `422`, and it is a `422` rather than a `400` for the same reason the login body is |
|
||||
| 8 | `id` exists in the **current** payload (`409` if the job has never answered) and is not already in the requested state (`409 {changed: false}`) | never a guess about a route it cannot see |
|
||||
| 9 | freshness: `gateway` job `ok` and not stale beyond 3× its interval, `host:<GATEWAY_HOST>` `ok`, `sections.haproxy`/`sections.nginx` both `true`, the agent's `version >= 1.4.0` | `409` naming which one — "do not rewrite a config you cannot currently read end to end" |
|
||||
| 10 | `danger == "DANGER"` for **both** verbs | `422` naming the word, the UPS wording's sibling |
|
||||
Then one `collectors.post_route`, and the response is the helper's object plus `host`, `id`, `action`, `dry_run`, and `hint` sentences owned by `api.py` (the `restart_required` precedent: the prose lives in the route, so the page prints what the route said).
|
||||
- **Events**, one per call, `source="gateway"`, `subject=<the route's host>`, `kind` in `{route-disabled, route-enabled, route-denied}`, severity `warn` for a change and `error` for a refusal-that-matters, message fixed-shape: `disable brain.reeseapps.com → reeseapps-brain-of-reese.reeselink.com:443: applied=true validated=true reloaded=true (admin)`. **No diff, no file path, no config text** — `/api/events` is a feed a browser reads, and a config's bytes are not a log line. A `route-denied` from gates 4–10 writes the event with the gate named and **nothing** about which check the caller came closest to passing.
|
||||
- `/api/config`'s `gateway` row gains `writes_enabled` (bool) so the page can render the honest arm instead of assuming (rule 2), and nothing else in that body changes.
|
||||
4. `tests/test_api.py`
|
||||
- `test_the_gateway_surface_is_gets_only_and_the_write_stays_the_ups_one` is **rewritten**, not deleted: the verb enumeration now lists exactly two fleet writes — `/api/ups/command` and `/api/gateway/route` — beside the session writes and the certificate-authority writes, and the test still fails if a third appears anywhere.
|
||||
- `test_each_gate_refuses_without_touching_the_peer`, parameterised over all ten with spies: `collectors.post_route` swapped for a recorder, and each refusal asserts the recorder is empty **and** the event is what the gate named. The CSRF half is proven on the wire in `tests/test_auth_api.py` the way the UPS one is: a scripted helper records nothing without the header and once with it.
|
||||
- `test_danger_is_required_to_enable_as_well_as_disable`, `test_read_only_refuses_and_names_the_variable`, `test_a_route_that_is_already_off_says_so_and_dials_nothing`, `test_a_stale_gateway_or_dead_host_job_refuses_the_write`, `test_a_revoked_agent_refuses_the_write_without_a_socket`, `test_the_write_timeout_is_longer_than_the_agents_peer_timeout` (the ordering, not the numbers).
|
||||
- `test_no_diff_or_config_text_reaches_an_event` — a successful write whose helper answer carries a `diff` field: the event message contains no `diff`, no `--- `/`+++ ` line, no path, and no `was:` marker.
|
||||
- `test_the_writes_enabled_flag_reaches_the_page_config`.
|
||||
5. `tests/test_auth_api.py` — the served-route walk (`walk_routes`) now sees `POST /api/gateway/route` and demands a session; the public-surface list stays exactly four entries; the CSRF-on-the-wire test gains this endpoint.
|
||||
6. `AGENTS.md` — **in the same commit**: invariant 1 rewritten from "exactly one exception" to "exactly two, and both are gated on purpose", naming the two endpoints and the gate list, keeping the sentences that make it work ("Hiding a button is not a safety control", "the UI confirmation … `DANGER` for anything that can cut power" now extended: *or that can cut a route through the fleet's only ingress*). Also: `api.py`'s bullet (its three write lists become four facts and now include the gateway route), `collectors.py`'s bullet (`post_route`), `config.py`'s paragraph (the two new settings), and the `/gateway` UI-conventions sentence about the unwired switch is **removed** — it becomes the wired description in task 06, and the two must not both exist.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: every gate in both directions; the wire-level CSRF proof; the event-shape proof; the verb enumeration restated.
|
||||
- Coverage **>90%** on `api.py` and `collectors.py`; the whole suite 100%; `ruff` + `pyright` clean.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `curl` against the dev server with writes off answers the `GATEWAY_ALLOW_WRITES` sentence; with it on and no `DANGER` answers the `DANGER` sentence; with both and a live fake agent it answers `applied: true` and the event line is in `/api/events?source=gateway`.
|
||||
- [ ] `grep -n "DASH_READ_ONLY" src/dashboards/api.py` shows it refusing on **both** write paths.
|
||||
- [ ] The verb-enumeration test lists exactly two fleet writes and fails on a third by name; `walk_routes` covers the new POST; `PUBLIC_PATHS` is still four entries.
|
||||
- [ ] `/api/config` serves `gateway.writes_enabled` and the page's own arm is task 06's.
|
||||
- [ ] `AGENTS.md` invariant 1 names two exceptions and their gates in the same commit as the code that made it true.
|
||||
- [ ] `uv run pytest` **100% passing**, coverage not down; `bash .agents/validate.sh` green.
|
||||
- [ ] No behaviour change in completed work: `/api/ups/command`'s four gates behave exactly as before (the suite's existing UPS pins are untouched), and no read route changed shape.
|
||||
@@ -0,0 +1,58 @@
|
||||
# Task 06 — The switch works, asks for `DANGER` with the diff on screen, and never claims a change it didn't get
|
||||
|
||||
**Phase:** `33_gateway_route_switching` · **Source:** the operator's sketch (phase 30) + this phase's decision that a route off is a config edit with a reload behind it · **Shape borrowed from:** `page-ups.js`'s typed-`DANGER` confirm and `dash.css`'s `dialog` + `input[type="text"].confirm`
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Clicking a switch on `/gateway` shows the **exact one-line diff** the gateway would apply, asks for `DANGER`, applies it, and paints the row from the **answer** — not from hope. Where writes are disabled the switch stays visibly unwired and says which variable would change that, because `/gateway` must look the same on a deployment that chose not to have this feature as it did before the feature existed.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/web/static/page-gateway.js`
|
||||
- `routeInConfig(conn)` becomes a reading, not a literal: `return conn.enabled !== false;` — with the comment rewritten: `off` is now reachable, it means "declared in the config, switched off by the operator", and `enabled === undefined` (a pre-1.4.0 agent, or the stub arm) still means "on", so an old agent's page never goes red.
|
||||
- `buildSwitch(conn)` splits into two arms, driven by `/api/config`'s `gateway.writes_enabled` (never inferred):
|
||||
- **writes off** → today's markup exactly: `aria-disabled="true"`, `cursor: not-allowed`, `title` = `Route switching is off — set GATEWAY_ALLOW_WRITES=1 on the dashboard to enable it.` (the sentence from `api.py`, copied into a constant that `tests/test_api.py` compares against the route's refusal so the two cannot drift);
|
||||
- **writes on** → a real control: no `aria-disabled`, `aria-checked` from `enabled`, a full accessible name (`endpoint, reached by <selectors>, in the <source> config, on|off, activate to switch it off`), `title` naming what activating does (edits the file, validates, reloads the proxy).
|
||||
- **The flow**, one async handler, and the order is the safety:
|
||||
1. `POST /api/gateway/route` with `{id, action, dry_run: true}` → `diff` + `hint`;
|
||||
2. open a real `<dialog>` (`showModal()`) whose body is: the row's inbound word and endpoint, the config's own `selectors`, the diff in a labelled `<pre>`, the route's `hint`, and a `input[type="text"].confirm` asking for `DANGER`;
|
||||
3. submit only when the typed word is exactly `DANGER` (client-side check of the *client's* convenience; the server re-checks, always);
|
||||
4. switch `aria-busy="true"`, `POST` with `danger: "DANGER"`, `dry_run: false`;
|
||||
5. paint from the **response**: `applied: true` → the row's `enabled` flips on the next poll (and immediately, optimistically **only** for the switch's own `aria-checked`, with `aria-busy` cleared and a note that the config is applied and the page re-reads); `applied: false` → nothing flips, the refusal prints verbatim through `textContent`;
|
||||
6. `await refresh()` so the numbers come from the agent's next reading rather than from the click.
|
||||
- No optimistic colour, ever: the page does not turn a line green or red because a button was pressed. `pending` is the only state a click may create, and `aria-busy` is how it says so.
|
||||
- A `429`/`409`/`403`/`422` prints the server's `detail` + `hint` in the dialog (`role="alert"`), leaves the dialog open so the operator can read it, and returns focus to the confirm field — the `/login` lesson about a form that eats its own error.
|
||||
2. `src/dashboards/web/pages/gateway.html`
|
||||
- One `<dialog id="gate-switch-dialog" aria-labelledby="gate-switch-title" aria-modal="true">` with the four slots the script fills, shipped `hidden`/closed; no per-row dialogs (the same rows are patched, so one dialog is the whole surface, and a dialog per row is 33 focus targets in the tab order).
|
||||
- Footer paragraph 3 becomes the truth and stays **one** paragraph:
|
||||
> `the switch edits that route in its own config file, validates it with the proxy's own checker and reloads that proxy — nothing is restarted, and no route goes off unless GATEWAY_ALLOW_WRITES is set on the dashboard.`
|
||||
- The `#gate-empty` setup block is untouched.
|
||||
3. `src/dashboards/web/static/dash.css`
|
||||
- `#gate-switch-dialog pre` — mono, `overflow-x: auto`, `max-height` bounded, added/removed lines tinted from the **existing** tokens only (`.diff-add`/`.diff-del` deriving from `--gate-on`/`--gate-off`, the phase-30 existence pair, so no new colour enters the map and the diff reads as the same red/green the rows do);
|
||||
- `button.gate-switch:not([aria-disabled="true"]):hover` and `cursor: pointer` — added **only** under that selector, so a writes-off switch never grows a hover affordance it cannot honour;
|
||||
- any new transition (dialog, knob while `aria-busy`) is named in the file's one `@media (prefers-reduced-motion: reduce)` block — naming one and missing the other is the phase-18 lesson, and there is a test for it;
|
||||
- at ≤640 px the dialog stays inside the viewport (`width: calc(100% - 2rem)` is already the shared rule) and the diff scrolls in its own box, never the document (phase 27's rule).
|
||||
4. `tests/test_api.py`
|
||||
- `test_the_switch_is_built_and_explicitly_unwired` is **replaced** by `test_the_switch_is_wired_only_when_the_deployment_enables_writes`: with `writes_enabled: false` the served script keeps `aria-disabled="true"`, the `GATEWAY_ALLOW_WRITES` sentence, and no listener; with it true the listener exists, `aria-disabled` is gone, and the accessible name states the state and the consequence. The old test's *intent* survives: a dead-looking-live control still fails.
|
||||
- `test_a_toggle_asks_for_a_preview_then_the_word_then_sends_once` — the served script issues `dry_run: true` before any `DANGER` send, requires the exact word, and sends exactly once per activation (no retry loop in the handler).
|
||||
- `test_the_row_is_painted_from_the_answer_not_the_click` — no code path sets `off`/`on` other than the payload/`applied` pass; `applied: false` leaves `aria-checked` untouched.
|
||||
- `test_the_switch_sentence_matches_the_routes_refusal` — the writes-off `title` and `api.py`'s `GATEWAY_ALLOW_WRITES` refusal are the same string, read from both files.
|
||||
- `test_no_gateway_count_is_coloured_by_a_verdict` gains the reachable-`off` fact and keeps every assertion: the diff tints are `--gate-*`, no `.ok/.warn/.fail` or `--ok/--warn/--fail` enters the gateway section, and the warnings panel's `.warn` stays the one exception.
|
||||
- `test_the_pipe_animation_stills_in_both_states_under_reduced_motion` extended to the new transitions (knob busy state, dialog) — and it still asserts there is **one** such block in the file.
|
||||
- `test_the_gateway_page_is_exactly_two_connection_panels_and_a_warnings_list` and the footer-paragraph pin stay green (three paragraphs, one dialog, `<section>` count unchanged).
|
||||
- The orphan-`id` audit must find `#gate-switch-dialog` referenced by the script; every slot inside it is `[data-k=…]`, not an `id`.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the pins above; a fixture render for each arm (writes on/off, `enabled: false` from the payload, `applied: false` from the response) asserting class, `aria-checked`, `aria-busy`, `title` and the accessible name.
|
||||
- Integration: the dev server against a stub agent (the `tests/conftest.py` fake) answering `applied: true` then a payload with the row `enabled: false` — the row turns red **because of the payload**, and the second read of the switch's name says `off`.
|
||||
- Coverage: `node --check` clean; floor does not fall.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] With `GATEWAY_ALLOW_WRITES=0` the page is indistinguishable from phase 32's: `aria-disabled`, the `GATEWAY_ALLOW_WRITES` sentence, no click effect, no hover affordance.
|
||||
- [ ] With it `=1` (and the fake agent): clicking shows the one-line diff and the `DANGER` field; the wrong word sends nothing; the right word sends once; the row's colour changes on the payload that follows and never before.
|
||||
- [ ] A stubbed `applied: false, validated: false, restored: true` leaves every row exactly as it was and prints the route's own sentence.
|
||||
- [ ] Tab order: dialog opens with focus in the confirm field, Escape closes it without a request, focus returns to the switch. At 360 px the dialog and its diff stay inside the viewport and the document scrolls no sideways pixels.
|
||||
- [ ] `uv run pytest` **100% passing**, coverage not down; `node --check` and `bash .agents/validate.sh` green.
|
||||
- [ ] README.md/AGENTS.md's `/gateway` paragraph describes the wired switch and both arms (replacing the "unwired" sentence task 05 deleted — the two must not coexist) — the phase's final pass re-checks both project-wide.
|
||||
- [ ] No behaviour change in completed work: tallies, warnings panel, charts, filters, colour semantics, and the other seven pages.
|
||||
+52
@@ -0,0 +1,52 @@
|
||||
# Task 07 — Install the helper, prove it from both sides, and drill one named route with the operator watching
|
||||
|
||||
**Phase:** `33_gateway_route_switching` · **Source:** AGENTS.md's rollout discipline (phases 24 and 28) applied to the one change in this repo that can take the fleet's ingress down
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
`gateway` runs agent **1.4.0** with the socket-activated helper, and one route is switched off and back on **for real**, with the live ingress ending the task in the state it started in. The drill is not a formality: it is the only proof that `ReadWritePaths`, `SocketMode`, `SO_PEERCRED`, `nginx -t` and the reload all line up on a machine, which no sandbox can tell us. **If the operator is not present in the session, this task stops after step 3 and reports — that is the correct outcome, not a failure**, and ntfy is not a way to ask (it is outbound-only).
|
||||
|
||||
## Work
|
||||
|
||||
1. **Backups first, hashed.** On `gateway-ops` as root:
|
||||
`cp -a /etc/haproxy /root/dashboards-33-backup-<utc>/haproxy && cp -a /etc/nginx /root/dashboards-33-backup-<utc>/nginx`, then `sha256sum` every file that the helper can edit (`haproxy.cfg`, `nginx.conf`, `conf.d/*.conf`, `stream.d/*.conf`) into the transcript. This is the restore point for every later step.
|
||||
2. **Sandbox rehearsal on the box** (writes nothing real): run the helper with `--apply` against a copy tree (`--root /tmp/dashboards-33-rehearsal`) with the *real* `nginx -t` and `haproxy -c` binaries on `PATH`, for each of the three marker arms and the `default` refusal. Record the three diffs. A checker that fails on a copied tree for path reasons is expected and is *not* the rehearsal's point — the arms and the restore byte-equality are.
|
||||
3. **Install, read-only-verified first**:
|
||||
- re-stage `/home/ops/agent-install/` (repo `agent/host_agent.py` **1.4.0**, `deploy/install-agent.sh`, `deploy/dashboards-routectl.py`, the two unit files), leaving `*.20261023-backup` copies sudo will not execute;
|
||||
- `systemctl cat dashboards-agent` → record its sha256 and the four security directives verbatim;
|
||||
- `sudo /home/ops/agent-install/deploy/install-agent.sh --bind 10.1.1.27 --port 9120 --allow-writes --with-routectl` (arguments read back off the unit; **no** `--cert-bundle`, the bundle on disk is what drives the TLS lines — the phase-28 finding);
|
||||
- the guard's diff is inspected before `--force`: the only legitimate change to the agent unit is the added `--allow-writes` in `ExecStart` — if `NoNewPrivileges`, `ProtectSystem`, `PrivateDevices`, `SystemCallFilter` or `CapabilityBoundingSet` moved, **stop** and report: the sandbox was not traded for this feature;
|
||||
- `sudo systemctl restart dashboards-agent`; `systemctl is-active` for it and for `dashboards-routectl.socket`;
|
||||
- reachability both ways: as `dashboards-agent`, connect to `/run/dashboards-routectl.sock` and call `list` (must answer `[]`); as any other non-root uid, connect and get `EACCES` (the mode is the first gate and the peer check is the second);
|
||||
- read-only probes: CA-only `--status` answers **1.4.0**; the dashboard's mTLS read shows `agent_version 1.4.0`, every section intact with live numbers, `disabled: false` everywhere, and `/api/gateway/snapshot` rendering identically to phase 32's.
|
||||
4. **Preconditions for the drill — all checked, all recorded, any failure stops it:**
|
||||
- the operator has said go, in this session, in words;
|
||||
- `nut` is not on battery and no UPS event is in the last hour (the same "don't do the risky thing during a power event" rule the UPS commands keep);
|
||||
- `gateway`, `host:gateway`, `disks:gateway` all `ok` and fresh; `warnings == []`;
|
||||
- `nginx -t` and `haproxy -c -f /etc/haproxy/haproxy.cfg` pass on the box before anything is touched;
|
||||
- `routectl list` is empty (no half-finished state from an earlier attempt);
|
||||
- the drill target is **`iperf3-5201`** (`haproxy:5201->web.reeselink.com:5201`) — a benchmark service with no human dependency. **Never** the auth, git, alerts or file routes: not `keycloak*`, `gitea*`, `forgejo*`, `ntfy*`, `nextcloud*`, `immich*`, `brain*`, `keycloak`, or any `map` entry an operator would notice within a minute. If even `iperf3` is in doubt, the drill does not run and the task reports that instead.
|
||||
5. **The drill, in order, with the page open:**
|
||||
1. `dry_run` through `POST /api/gateway/route` → the diff, and the page's dialog shown;
|
||||
2. apply with `DANGER`; the helper answers `applied/validated/reloaded`;
|
||||
3. prove it four independent ways: `routectl list` names the route; the marked line is in `haproxy.cfg` with `was:`; `haproxy -c` still passes; `curl -sv --resolve <host>:25565:127.0.0.1` (or the drill port) from the box connects and is refused/closed rather than proxied;
|
||||
4. the page: the row is **red** within one poll, its live count drains to `0` on its own, no other row changed, and `/api/summary`'s `gateway.disabled` is `1`;
|
||||
5. re-enable through the page; `routectl list` empty; `sha256sum` of every file equal to step 1's capture, byte-for-byte;
|
||||
6. the page green again, `disabled` back to `0`, and `journalctl -u 'dashboards-routectl@*'` showing exactly the two activations.
|
||||
6. **Abort rules, written where they can be followed:** if any step fails, restore from the step-1 backup (`cp -a` back, `haproxy -c`, reload), re-read the page, and write down what happened. Never leave a marked line behind without saying so in the transcript's first line. Never run two drills at once, never run one while the `gateway` job is stale, and never re-run an apply to "make sure" — the helper is the place idempotence lives, and the answer tells you which state you are in.
|
||||
7. `data/logs/NOTES-33-switching.md`: the hashes before and after, the unit diff that was guarded, the socket's owner/mode, both probe transcripts, the four proofs, the `journalctl` window, the screenshots' names, and — if the drill did not run — **exactly** which precondition stopped it, so the next operator can finish it deliberately.
|
||||
8. `tests/test_live_fleet.py`: `test_the_switch_round_trip_on_one_named_route`, marked `live` **and** gated on a separate `--allow-live-writes` flag, skipped by default with a reason naming the route it would touch, restoring in a `finally`. The default live suite stays read-only.
|
||||
|
||||
## Testing & Quality
|
||||
- This task's code change is the one opt-in live test; everything else is procedure plus transcript.
|
||||
- After it: `uv run pytest`, `uv run pytest -m live --live` (green, round-trip **skipped**), `./scripts/dev-server.sh status` (ten sources `ok`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `systemctl cat dashboards-agent` shows the four security directives **identical** to step 3's capture, with `--allow-writes` as the only added argument, and its pre/post sha256 pair is in the note.
|
||||
- [ ] `dashboards-routectl.socket` is `active`, owned `dashboards-agent:dashboards-agent` mode `0600`; a non-agent uid gets `EACCES`; the agent's own uid gets a JSON answer.
|
||||
- [ ] The drill's four proofs are in the transcript, plus the byte-identical `sha256sum` restore of every editable config file, and `routectl list` is empty at the end.
|
||||
- [ ] `/gateway` showed the row red and its count draining, and returned to green — screenshotted at both states.
|
||||
- [ ] `uv run pytest -m live --live` passes with the round-trip **skipped** by default; `uv run pytest -m live --live --allow-live-writes` passes too, if and only if the drill ran.
|
||||
- [ ] `data/logs/NOTES-33-switching.md` exists and its first line states the final state of the ingress (`all routes enabled`, or exactly what is off and why).
|
||||
- [ ] No repo file changed except the opt-in live test and `data/logs/`.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Task 08 — Docs that describe a dashboard which can change the fleet, the browser pass, the full gate
|
||||
|
||||
**Phase:** `33_gateway_route_switching` · **Source:** AGENTS.md definition of done + the fact that this phase changed an invariant, so the documents are part of the change rather than a summary of it
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
`README.md` and `AGENTS.md` describe the write path honestly — what it touches, what refuses, how to turn it off entirely, and how to recover a route by hand at 03:00 when the dashboard is the thing that is broken. Then the browser pass measures the wired page in both arms, the mobile audit re-runs, and every gate is green.
|
||||
|
||||
## Work
|
||||
|
||||
1. `AGENTS.md` — the full consistency pass, with these named as the passages this phase makes stale if left:
|
||||
- **invariant 1** — two exceptions, both gated, the gate list, and the reason a typed word survives in the design (`Hiding a button is not a safety control`);
|
||||
- `agent/host_agent.py` bullet — **1.4.0**, the marker grammar (`ROUTE_MARKER`, `split_marker`, `config_line_index`), `disabled` on every target, `POST /route` behind certificate + `--allow-writes`, the single peer call and its no-retry rule;
|
||||
- the new deploy surface — `deploy/dashboards-routectl.py` (verbs, marker + `was:`, validate-restore-reload, never restart, the sink file, drift refusal), the two units, `--with-routectl`/`--without-routectl`, and the sentence that the agent's unit lost no security directive and gained no sudoers line;
|
||||
- `sources/gateway.py` bullet — `enabled`, the merge rule with mixed states, the marker-aware warnings table, the `/api/summary` `disabled` field;
|
||||
- `api.py` bullet — the two fleet writes, `/api/gateway/route`'s gate order, the fixed-shape audit events with no config text, `/api/config.gateway.writes_enabled`;
|
||||
- `collectors.py` bullet — `post_route` and the shared revoke-before-socket rule;
|
||||
- `config.py` paragraph + README's table — `GATEWAY_ALLOW_WRITES` (default off) and `GATEWAY_WRITE_TIMEOUT_SECONDS` (and why it exceeds the agent's own timeout);
|
||||
- the `/gateway` UI-conventions paragraph — the switch is wired, the preview-then-`DANGER`-then-paint-from-the-answer flow, `off`'s appearance, the writes-disabled arm's sentence, the diff box scrolling in its own container;
|
||||
- the **fleet record** — `gateway` at **1.4.0** with the helper units installed, the unit's directives re-read, the drill result and `NOTES-33-switching.md`;
|
||||
- **gap list** — name-route switching **closed**, and the honest limits written as gaps: one admin (gap 6 unchanged, so anyone with the session can switch a route), no scheduler, no per-host kill switch, no runtime haproxy state (a hand reload cannot undo a marker), and `GATEWAY_ALLOW_WRITES` off by default means the feature is invisible on a fresh deployment until it is deliberately enabled.
|
||||
2. `README.md` + both tracked env templates — `.env.example` and `.env.dev.example` gain `GATEWAY_ALLOW_WRITES` with its default and a comment naming exactly what enabling it authorises, and `tests/test_auth_docs.py` gains the pin (the same device that keeps the auth settings declared in the templates), because a write gate no template names is a write gate nobody finds — and never a value, only the declaration. Then the operator-facing half: the `/gateway` table row (routes can be switched off **and on**), a short *Switching a route* subsection (what the dialog does, what `DANGER` means here, that the edit is persistent across reload and reboot, and the three recovery commands: `routectl list`, `routectl verify`, `routectl enable --id …` on the box), the *Enabling it* subsection (`GATEWAY_ALLOW_WRITES=1`, `--with-routectl`, the two units, how to undo everything), and the configuration table's two new rows. Also: the sentence that a route off is a **deliberate outage** and appears in the event feed, so an operator does not mistake it for an incident.
|
||||
3. `data/logs/33-switching.cjs` — Playwright pass (house shape, exit non-zero on FAIL, prints what it measured):
|
||||
- writes-**off** arm: `aria-disabled`, the `GATEWAY_ALLOW_WRITES` sentence, a click does nothing (no request on the wire), no hover affordance;
|
||||
- writes-**on** arm against the stubbed agent: the dialog opens with the diff and focus in the confirm field; the wrong word sends nothing; the right word sends exactly one request; the row's colour changes only after the payload says `enabled: false`; `applied: false` leaves the row untouched and prints the route's sentence; Escape closes with no request and returns focus;
|
||||
- `off` colour contract: spine halves, tick, run, switch and diff tint all compute to the same `--gate-off` value in dark **and** light, and an `on` row is fully `--gate-on` — no fifth colour, no verdict token anywhere in the section;
|
||||
- reduced motion: nothing animates, including the dialog and the busy knob;
|
||||
- 360 px: the dialog and its `<pre>` stay inside the viewport, the diff scrolls in its own box, and the document scrolls zero sideways pixels;
|
||||
- screenshots `data/logs/33-gateway-{writes-off,dialog,danger,off,off-light,refused,phone,phone-dialog}.png`, transcript `data/logs/33-switching.out`.
|
||||
4. Re-run `data/logs/27-task02-mobile-audit.cjs` (nine pages, both widths): exit 0, **0 FAIL / 0 FINDING**, tally recorded — the dialog is new markup and the audit is what catches a fact painted past a card edge.
|
||||
5. The whole gate, recorded in the transcript:
|
||||
`uv run pytest`, `uv run pytest --cov`, `uv run pytest -m live --live`, `uv run ruff check src tests agent deploy`, `uv run pyright`, `bash .agents/validate.sh`, `uv pip list --outdated` (empty — no dependency is added by this phase, and `deploy/dashboards-routectl.py` is stdlib on a host's Python 3.10).
|
||||
6. `./scripts/dev-server.sh status` with writes **off** → ten sources `ok` and no change; with `GATEWAY_ALLOW_WRITES=1` and the fake agent → still ten `ok`, and a write through `curl` round-trips.
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: this task is the phase's proof that the documents and the shipped page agree; every completion-criteria number comes from `33-switching.out`, `NOTES-33-switching.md` or a gate's own output.
|
||||
- Coverage: floor green with the phase's new modules included; no `# pragma: no cover` added to reach it.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `grep -n "one exception\|exactly one write" AGENTS.md` finds nothing; invariant 1 names **two** endpoints and their gates, and the `/gateway` paragraph no longer says the switch is unwired (the two statements must not coexist).
|
||||
- [ ] `grep -n "GATEWAY_ALLOW_WRITES" README.md AGENTS.md .env.example .env.dev.example` — declared in both templates with its default and a comment saying what enabling it authorises, and documented in both documents.
|
||||
- [ ] `data/logs/33-switching.out` exits 0 with the both-arms results, the `--gate-off`/`--gate-on` equalities in both themes, the reduced-motion pass and 0 px sideways scroll at 360 px; the eight screenshots exist.
|
||||
- [ ] `data/logs/27-task02-mobile-audit.cjs` exits 0, 0 FAIL / 0 FINDING, tally recorded.
|
||||
- [ ] All five gates green, `uv pip list --outdated` empty, and the live suite green with the round-trip skipped by default.
|
||||
- [ ] A full read of `README.md` and `AGENTS.md` finds no inconsistency anywhere, including pre-existing ones, fixed in this phase's commits.
|
||||
- [ ] No behaviour change in completed work: the UPS write path and its four gates, the other five dashboards, `/settings`, `/certs`, `/login`, and every read route's payload are exactly as phases 31–32 left them.
|
||||
Reference in New Issue
Block a user