services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 33_gateway_route_switching

Browse Source 
Phase 33 completion pass is done. Final report:

## Phase 33 — completion report

**All 8 task files complete** (`.agents/phases/complete/33_gateway_route_switching/`), full gate green, docs verified.

### Work in this pass

1. **`tests/test_routectl.py` added** (17 tests, all green) — exercises the *real* helper binary unprivileged against a fake tree (`--root`), the real agent parser, the captured gateway fixtures, and stubbed `nginx`/`haproxy`/`systemctl` on `PATH`. Pins the full protocol (locate → state → dry-run → temp-file/fsync/`os.replace` write → checker → restore on checker failure → reload-never-restart → re-read proof → sidecar record), all three route shapes (haproxy `server`, nginx `proxy_pass`, nginx stream `map` via its sink-value id), `drift` / unknown-id / `map-default` refusals, `list`/`verify`, and the `--serve` peer gate (`SO_PEERCRED` uid check, one JSON line per request, exact body shape).

2. **Real helper defect fixed** — `deploy/dashboards-routectl.py`: `main()`'s early-return paths returned `({"ok": False, …}, EXIT_*)` tuples, so `sys.exit()` printed a Python repr instead of a JSON line (usage errors, agent-import refusal). Now prints `json.dumps(…)` and returns only the code, keeping the one-JSON-line contract.

3. **Documentation consistency fixes** (task 08 criterion 6):
   - `README.md`: garbled verdict-colour sentence on the switch ("green and to the position") → "in the **on** position and tinted `--gate-on`" (the existence-pair contract).
   - `README.md`: audit sentence now names all three event kinds — `route-disabled` / `route-enabled` / **`route-denied`** (the code stores one per refused call; dry runs store none).
   - `page-gateway.js`: replaced the stale pre-33 module docstring ("the switch is not wired") with the two-arm `GATEWAY_ALLOW_WRITES` contract.
   - `AGENTS.md`: every phase-33 claim re-verified against code (the ten gates, the 35 s/30 s budget pair, `gateway.writes_enabled`, `/api/summary`'s `disabled`, the 1.4.0 agent bullet, the gap list) — no changes needed.

### Verification
- `uv run pytest --cov`: **1970 passed**, 18 skipped — coverage **98.74%** (floor 94%)
- `ruff` clean · `pyright` 0 errors · `bash .agents/validate.sh` **exit 0** (incl. `node --check` on every page script)
- Dev server running; all 10 sources `ok`; bundles exported

### Live infrastructure
**No live changes made.** The live drill remains unexecuted per `data/logs/NOTES-33-switching.md` (no operator approval; checker binaries unavailable in the podman proxy units). All routes stay enabled, config hashes unchanged, and the sandbox rehearsal + backup tree are in place for the operator's own `DANGER` call. All changes are in the working tree only.
This commit is contained in:
ducoterra
2026-11-01 11:50:51 -05:00
parent 0271387053
commit 86c8bf4992
91 changed files with 7568 additions and 519 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_gateway_source.py
+368 -1
View File
@@ -1696,7 +1696,7 @@ def test_no_count_carries_a_verdict_or_a_window(gateway_snapshot: Any) -> None:
`sources/host.py` fails here — the first would be a judgement `severity.py` never made, the
second would claim a measurement this source does not take.
"""
for mode in ("full", "predates", "sockets_silent"):
for mode in ("full", "predates", "sockets_silent", "marked", "hand_edited"):
keys = walk_keys(gateway.parse(gateway_snapshot(mode), "gateway"))
assert "severity" not in keys, mode
assert not {key for key in keys if "window" in key or key.startswith("Δ")}, mode
@@ -2132,3 +2132,370 @@ def test_warnings_travel_in_the_payload_and_the_stub_share_the_key(gateway_snaps
assert gateway.stub("gateway")["warnings"] == []
assert gateway.parse(gateway_snapshot("predates"), "gateway")["warnings"] == []
assert set(gateway.stub("gateway")) - {"unreachable"} <= set(gateway.parse(snap, "gateway"))
# --------------------------------------------------------------------------- #
# phase 33 task 04: `enabled` on every line, and the warnings learn the marker
#
# The helper's `#off <name>` marker leaves a switched-off line **in the config**; the agent
# reads it and publishes `disabled`, and this module draws the line either way — an off route
# is a claim the config made (a different fact from "never declared"), so it is drawn and
# counted, flagged. An arm that never read the config draws no lines at all, which is what
# keeps a `false` out of a payload nobody answered. The warning arms learn the marker the same
# way: a shape the operator's own switch caused is not a fault, and the same shape a hand edit
# left is.
# --------------------------------------------------------------------------- #
def marked(row: dict[str, Any]) -> dict[str, Any]:
"""A line the helper marked off: the agent's own `disabled: true` beside the intact line."""
return {**row, "disabled": True}
def test_a_marked_haproxy_server_line_is_an_off_line_with_a_distinct_id() -> None:
"""On and off are both drawn; the id's `#off` suffix is what keeps them distinct rows."""
section = haproxy_section(
[frontend("haproxy-22", [2222], default="be")],
[backend("be", [server("s1", "10.9.9.9", 80),
marked(server("s2", "10.9.9.9", 80))])])
payload = gateway.parse(snapshot(haproxy=section, agent_version="1.4.0"), "gateway")
by_enabled = {row["enabled"]: row for row in payload["connections"]}
assert set(by_enabled) == {True, False}, "one of these selectors is off is two facts"
on, off = by_enabled[True], by_enabled[False]
assert on["id"] == "haproxy:2222->10.9.9.9:80"
assert off["id"] == "haproxy:2222->10.9.9.9:80#off", (
"the suffix is the write path's key — the two halves of the split must not collide")
assert on["out"]["selectors"] == off["out"]["selectors"] == ["be"], (
"the off line keeps the config's own word for the route — switched off is not deleted")
# the flag is absence-tolerant: a row the agent did not report it on (an older agent's
# document) is on, because absence is "not switched off", never "broken"
older = haproxy_section([frontend("haproxy-22", [2222], default="be")],
[backend("be", [server("s1", "10.9.9.9", 80)])])
payload = gateway.parse(snapshot(haproxy=older), "gateway")
assert [row["enabled"] for row in payload["connections"]] == [True]
def test_a_backend_whose_only_server_is_switched_off_is_not_an_empty_warning() -> None:
"""The marker decision, pinned both ways: the line the helper left behind keeps the backend
non-empty (no warning — its state is the line's `enabled: false`), and the same shape the
operator left by **deleting** the line is the fault."""
switched_off = haproxy_section([frontend("haproxy-22", [2222], default="be")],
[backend("be", [marked(server("s1", "10.9.9.9", 80))])])
assert gateway.warnings(switched_off, None) == [], (
"the line is still in the document, so the backend declares a server — the off state "
"is the line's flag, not an empty-backend fault")
hand_deleted = haproxy_section([frontend("haproxy-22", [2222], default="be")],
[backend("be", [])])
assert kinds(gateway.warnings(hand_deleted, None)) == [gateway.WARN_BACKEND_EMPTY], (
"the config no longer says anything — that is the fault the same shape warns about")
def test_a_marked_proxy_pass_is_an_off_line_and_the_block_is_still_drawn() -> None:
off = {**proxy_pass(("10.9.9.9", 80), location="/off"), "disabled": True}
section = nginx_section(
[nginx_server([8443], ["x.example"], [proxy_pass(("10.9.9.9", 80)), off])])
payload = gateway.parse(snapshot(nginx=section, agent_version="1.4.0"), "gateway")
by_enabled = {row["enabled"]: row for row in payload["connections"]}
assert set(by_enabled) == {True, False}
assert by_enabled[False]["id"] == "nginx:8443->10.9.9.9:80#off"
assert by_enabled[False]["out"]["selectors"] == ["x.example"]
def test_a_marked_upstream_member_is_off_and_its_siblings_stay_on() -> None:
pool = {"name": "pool",
"servers": [{"host": "10.9.9.1", "port": 80, "disabled": False},
{"host": "10.9.9.2", "port": 80, "disabled": True}]}
section = nginx_section([nginx_server([8443], ["x.example"], [pool_pass("pool")])],
upstreams=[pool])
payload = gateway.parse(snapshot(nginx=section, agent_version="1.4.0"), "gateway")
by_host = {row["out"]["host"]: row["enabled"] for row in payload["connections"]}
assert by_host == {"10.9.9.1": True, "10.9.9.2": False}, (
"the member line's own flag is its state — the pool's other members are not switched "
"with it")
def test_a_map_entry_is_off_by_marker_or_by_the_sink_block_and_only_those() -> None:
"""Three off-lookalikes, one of which is routing: the marked line, the line whose value is
the helper's sink block, and a plain loopback target with no such block. Only the first two
are the switch — an address alone, without the block, is not off (a vhost that legitimately
lands on loopback is routing, not a switch)."""
sink_block = {"listen": [{"addr": "127.0.0.1", "port": 19999}],
"server_names": [], "proxy_passes": []}
entries = [("web.example", ("10.9.9.9", 80)),
("off.example", ("10.8.8.8", 80)),
("sink.example", ("127.0.0.1", 19999))]
maps = [mapping_block("$var", entries)]
maps[0]["entries"][1] = {**maps[0]["entries"][1], "disabled": True} # `off.example`, marked
section = nginx_section(
[sink_block,
nginx_server([443], (), [proxy_pass(None, variable="$var")])],
maps=maps)
payload = gateway.parse(snapshot(nginx=section, agent_version="1.4.0"), "gateway")
by_selector = {row["out"]["selectors"][0]: row["enabled"]
for row in payload["connections"] if row["out"]["selectors"]}
assert by_selector == {"web.example": True, "off.example": False,
"sink.example": False}, (
"the marked line and the line whose value is the sink block are the switch; "
"a plain target is not")
# the control target: a loopback address the config routes to, with no such block
maps[0]["entries"].append({"key": "plain.example",
"target": {"host": "127.0.0.2", "port": 29999},
"disabled": False})
payload = gateway.parse(snapshot(nginx=section, agent_version="1.4.0"), "gateway")
plain = [row for row in payload["connections"]
if row["out"]["selectors"] == ["plain.example"]]
assert plain and plain[0]["enabled"] is True, (
"an address alone, without the block, is not off")
# and the refusal the other way: the same config with the sink block absent — the value is
# just an address again, and the line is on
no_sink = dict(section, servers=[s for s in section["servers"]
if s is not sink_block])
payload = gateway.parse(snapshot(nginx=no_sink, agent_version="1.4.0"), "gateway")
sink_row = [row for row in payload["connections"]
if row["out"]["selectors"] == ["sink.example"]]
assert sink_row and sink_row[0]["enabled"] is True
def test_a_map_entry_with_no_value_warns_unless_the_line_carries_the_marker() -> None:
"""The same shape, two causes: the value hand-deleted is the fault the warning exists for,
the value the marker's line lost is the operator's own state and stays quiet. A marked entry
with no value has no endpoint to draw either, so it is a warning-less absence — the off
state lives in the line's flag, and the line is the config's."""
entries = [{"key": "gone.example", "target": None, "disabled": False},
{"key": "off.example", "target": None, "disabled": True}]
section = nginx_section(
[nginx_server([443], (), [proxy_pass(None, variable="$var")])],
maps=[{"variable": "$var", "entries": entries}])
assert kinds(gateway.warnings(None, section)) == [gateway.WARN_MAP_ROUTE_MISSING]
found = gateway.warnings(None, section)[0]
assert found["subject"] == "var:gone.example"
# the marked entry draws no line (it has no endpoint) and warns nothing
payload = gateway.parse(snapshot(nginx=section, agent_version="1.4.0"), "gateway")
assert all(row["out"]["selectors"] != ["off.example"] for row in payload["connections"])
def test_two_marked_map_entries_to_one_endpoint_merge_into_one_off_line() -> None:
"""Same state, same endpoint, two selector words: one line, the union of both keys — the
merge rule with `enabled` in the key, in the direction that unions instead of splits."""
entries = [{"key": "a.example", "target": {"host": "10.9.9.9", "port": 80},
"disabled": True},
{"key": "b.example", "target": {"host": "10.9.9.9", "port": 80},
"disabled": True}]
section = nginx_section(
[nginx_server([443], (), [proxy_pass(None, variable="$var")])],
maps=[{"variable": "$var", "entries": entries}])
payload = gateway.parse(snapshot(nginx=section, agent_version="1.4.0"), "gateway")
assert len(payload["connections"]) == 1
row = payload["connections"][0]
assert row["enabled"] is False and row["id"] == "nginx:443->10.9.9.9:80#off"
assert row["out"]["selectors"] == ["a.example", "b.example"]
def test_the_stub_and_the_predates_arm_have_no_lines_and_so_no_false(
gateway_snapshot: Any) -> None:
"""`false` is always a claim the config made, so an arm that never read the config carries
no `enabled` at all — not a row of `null`s (that would be a guess) and not `false` (that
would be a claim). The stub and the predates arm are that arm, both pinned."""
assert gateway.stub("gateway")["connections"] == []
assert "enabled" not in walk_keys(gateway.stub("gateway"))
assert "disabled" not in walk_keys(gateway.stub("gateway"))
predates = gateway.parse(gateway_snapshot("predates"), "gateway")
assert predates["supported"] is False and predates["connections"] == []
assert "enabled" not in walk_keys(predates)
def test_the_captured_configs_carry_no_marker_and_their_lines_are_on(gateway_snapshot: Any) -> None:
"""The recorded ingress, through the real agent: the flag is on every markable line and no
line of it is marked, so every line of the captured graph is `enabled: true` — the pin that
a re-capture in which the operator switched a route off is visible here rather than silently
redrawn as on."""
snap = gateway_snapshot("full")
for block in snap["haproxy"]["backends"]:
assert all(row["disabled"] is False for row in block["servers"])
for server in snap["nginx"]["servers"]:
assert all(p["disabled"] is False for p in server["proxy_passes"])
for block in snap["nginx"]["upstreams"]:
assert all(m["disabled"] is False for m in block["servers"])
for block in snap["nginx"]["maps"]:
assert all(e["disabled"] is False for e in block["entries"])
payload = gateway.parse(snap, "gateway")
assert payload["connections"], "the capture has routes"
assert all(row["enabled"] is True for row in payload["connections"])
assert not any(row["id"].endswith("#off") for row in payload["connections"])
def test_a_marker_off_route_is_declared_and_disabled(gateway_snapshot: Any) -> None:
"""Three of the fleet's own routes, switched off in the `marked` fixture (one haproxy
`server`, one vhost `proxy_pass`, one map entry pointing at the sink), come back as three
drawn rows reading `enabled: false` — selector and endpoint intact, every other row on, and
the marker-caused shapes warn nothing. The row is *declared and off*, which is a different
fact from never declared (absent) and from on, and the `#off` id suffix is the state's
spelling the write path keys on.
"""
payload = gateway.parse(gateway_snapshot("marked"), "gateway")
off = [row for row in payload["connections"] if row["enabled"] is False]
on = [row for row in payload["connections"] if row["enabled"] is True]
assert len(off) == 3, "three markers, three off rows — no merge stole one, none is missing"
assert on, "the other routes of the capture are still declared and on"
assert all(row["enabled"] is True for row in on), ("every non-marker row is on")
by_selector = {tuple(row["out"]["selectors"]): row for row in off}
# the selectors are the config's own words, verbatim — switching a route off is not deleting
# its name, and the endpoint is what the config (or the helper's rewrite) wrote, intact
assert by_selector[("gitea-2222",)]["out"]["host"] == "reeseapps-gitea.reeselink.com"
assert by_selector[("gitea-2222",)]["out"]["port"] == 2222
assert by_selector[("nextcloud.reeseapps.com",)]["out"]["host"] == "nextcloud.reeselink.com"
assert by_selector[("nextcloud.reeseapps.com",)]["out"]["port"] == 11000
assert by_selector[("brain.reeseapps.com",)]["out"]["host"] == "127.0.0.1"
assert by_selector[("brain.reeseapps.com",)]["out"]["port"] == 44450
for row in off:
assert row["id"].endswith("#off"), "the off state is the id's spelling, on every off row"
# the two marker-caused shapes are present in the raw sections, so `warnings == []` below is a
# suppression, not a vacuous absence: the gitea backend's only server is marked, and the
# brain entry's value is the sink
snap = gateway_snapshot("marked")
gitea = next(b for b in snap["haproxy"]["backends"] if b["name"] == "gitea-2222")
assert gitea["servers"] and gitea["servers"][0]["disabled"] is True
stream_map = next(b for b in snap["nginx"]["maps"] if b["variable"] == "targetBackend")
brain = next(e for e in stream_map["entries"] if e["key"] == "brain.reeseapps.com")
assert brain["disabled"] is True
assert payload["warnings"] == [], ("a marker-caused empty backend and a sink-pointed map "
"entry are states, not config-integrity faults")
def test_the_source_never_calls_a_route_broken_because_the_operator_disabled_it(
gateway_snapshot: Any) -> None:
"""The positive direction of both marker-aware kinds, over the captured config with the
markers injected (never a toy file): a backend whose only `server` the operator turned off
is not `haproxy_backend_empty` — the line is still in the document, and its state is the
row's `enabled: false` — and a map entry the helper pointed at its own sink is not a
missing route. Neither shape may earn a warning while the switch is what made it.
"""
snap = gateway_snapshot("marked")
# the shapes are present in the raw document, so the silence below is a decision
gitea = next(b for b in snap["haproxy"]["backends"] if b["name"] == "gitea-2222")
assert len(gitea["servers"]) == 1 and gitea["servers"][0]["disabled"] is True
stream_map = next(b for b in snap["nginx"]["maps"] if b["variable"] == "targetBackend")
brain = next(e for e in stream_map["entries"] if e["key"] == "brain.reeseapps.com")
assert brain["disabled"] is True and brain["target"] == {"host": "127.0.0.1", "port": 44450}
payload = gateway.parse(snap, "gateway")
assert kinds(payload["warnings"]) == [], (
"the operator's own switch is a state the page draws, not a fault the page alarms about")
def test_the_same_shape_without_a_marker_is_still_a_warning(gateway_snapshot: Any) -> None:
"""The negative direction, the same two shapes in the `hand_edited` fixture: the `server`
line deleted by hand is the fault `haproxy_backend_empty` exists for, and the map entry
whose value was hand-deleted (no marker, no sink file) is the fault
`nginx_map_route_missing` exists for. Suppression is marker-caused, not shape-caused —
together with `test_the_source_never_calls_a_route_broken_because_the_operator_disabled_it`
this is the pair that says so.
"""
snap = gateway_snapshot("hand_edited")
# the shapes are present and unmarked, so the warnings below are the checks working
gitea = next(b for b in snap["haproxy"]["backends"] if b["name"] == "gitea-2222")
assert gitea["servers"] == [], "the line is gone — the backend declares no server in the document"
stream_map = next(b for b in snap["nginx"]["maps"] if b["variable"] == "targetBackend")
brain = next(e for e in stream_map["entries"] if e["key"] == "brain.reeseapps.com")
assert brain["target"] is None and brain["disabled"] is False
payload = gateway.parse(snap, "gateway")
assert kinds(payload["warnings"]) == [gateway.WARN_BACKEND_EMPTY,
gateway.WARN_MAP_ROUTE_MISSING]
found = {row["kind"]: row for row in payload["warnings"]}
assert found[gateway.WARN_BACKEND_EMPTY]["subject"] == "gitea-2222"
assert found[gateway.WARN_MAP_ROUTE_MISSING]["subject"] == "targetBackend:brain.reeseapps.com"
# and the off state itself is gone with the marker: no `#off` row, nothing to draw off
assert not any(row["id"].endswith("#off") for row in payload["connections"])
assert all(row["enabled"] is True for row in payload["connections"])
def test_the_sink_address_is_recognised_not_hard_coded() -> None:
"""The sink is the *block* — loopback listen, no `server_name`, no `proxy_pass` — and the
port is the helper's to choose. A config that writes a different sink port still reads as
switched-off from the payload's own text, and a loopback target that is not the block's
port is routing, not a switch (a vhost that legitimately lands on loopback is on)."""
other_port = 45999
sink_block = {"listen": [{"addr": "127.0.0.1", "port": other_port}],
"server_names": [], "proxy_passes": []}
section = nginx_section(
[sink_block, nginx_server([443], (), [proxy_pass(None, variable="$var")])],
maps=[mapping_block("$var", [("sink.example", ("127.0.0.1", other_port)),
("plain.example", ("127.0.0.2", 29999))])])
assert gateway.sink_endpoint(section) == ("127.0.0.1", other_port), (
"the block, not the port constant, is what the source looks for")
payload = gateway.parse(snapshot(nginx=section, agent_version="1.4.0"), "gateway")
by_selector = {row["out"]["selectors"][0]: row["enabled"]
for row in payload["connections"] if row["out"]["selectors"]}
assert by_selector == {"sink.example": False, "plain.example": True}, (
"the entry whose value is the block's listen is off; the other loopback target is not")
def test_a_disabled_line_still_publishes_only_a_measured_count() -> None:
"""Nothing stamps `0`: an off row's `connections` is the join's answer, and the join says
`0` only when the table answered (and genuinely had none) and `null` when it did not —
with no sample in the latter case. `0` measured is a legitimate measurement; the
distinction is *measured*, never *assumed*.
"""
haproxy = haproxy_section([frontend("f", [2222], default="be")],
[backend("be", [marked(server("s1", "10.9.9.9", 80))])])
sink_block = {"listen": [{"addr": "127.0.0.1", "port": 44450}],
"server_names": [], "proxy_passes": []}
section = nginx_section(
[sink_block, nginx_server([443], (), [proxy_pass(None, variable="$var")])],
maps=[mapping_block("$var", [("brain.example", ("127.0.0.1", 44450)),
("web.example", ("10.9.9.1", 80))])])
section["maps"][0]["entries"][0] = {**section["maps"][0]["entries"][0], "disabled": True}
# the table answered (and had nothing to book): the off rows are `0`, and `0` measured
# publishes, because the endpoint is one the config wrote as an address
answered = gateway.parse(
snapshot(haproxy=haproxy, nginx=section, sockets=sockets_section(),
agent_version="1.4.0"), "gateway")
off = [row for row in answered["connections"] if row["enabled"] is False]
assert len(off) == 2 and all(row["connections"] == 0 for row in off)
by_selector = {row["out"]["selectors"][0]: row for row in answered["connections"]}
assert by_selector["web.example"]["connections"] == 0 # the on row, the same measured zero
off_metrics = {row["metric"] for row in off}
assert off_metrics, "a joinable off endpoint carries a gauge, like its on counterpart"
samples = {s.metric: s.value for s in gateway.samples(answered, 1.0)}
for metric in off_metrics:
assert samples.get(metric) == 0.0, "the off row's sample is the measured zero, not a stamp"
# the table did not answer: every row stays `null` and the off rows publish nothing at all
silent = gateway.parse(
snapshot(haproxy=haproxy, nginx=section,
sockets=sockets_section(available=None), agent_version="1.4.0"), "gateway")
assert all(row["connections"] is None for row in silent["connections"]), (
"no table, no count — the off state is not a zero")
assert gateway.samples(silent, 1.0) == []
def test_the_stub_and_the_answer_share_one_key_set(gateway_snapshot: Any) -> None:
"""The stub's `enabled` is `None` in the sense nobody read the config — and a stub has no
rows, so the honest unknown is the *absence of the claim*: no rows, hence no `enabled` key
anywhere in the document (a row of `null`s would be a guess, and a `false` would be a claim
the config never made). The answer is the other half of the contract: every row it draws
carries the state as a real boolean, never a `None` and never a missing key, so a page
indexing `connections[].enabled` tests one shape in both arms.
"""
answer = gateway.parse(gateway_snapshot("full"), "gateway")
assert answer["connections"], "the capture has routes to pin the answer half on"
for row in answer["connections"]:
assert row["enabled"] is True or row["enabled"] is False, (
"an answered row's state is a claim the config made — never null, never absent")
stub = gateway.stub("gateway")
assert stub["connections"] == []
assert "enabled" not in walk_keys(stub), (
"an arm that never read the config carries no `enabled` — its `None` is the absence")
assert "disabled" not in walk_keys(stub)
# top-level parity: the page indexes the same keys on both arms; the only keys the stub
# alone or the answer alone carries are the ones that *are* the difference
shared = set(stub) | set(answer)
assert shared - set(stub) == {"agent_version"}, "the answer's extra keys are the snapshot's"
assert shared - set(answer) == {"unreachable"}, "the stub's extra key is the reachability one"