phase: 32_gateway_resolved_target_liveness
Fixture and live stream.conf are byte-identical (modulo header comment). Let me test the parser directly:
This commit is contained in:
@@ -29,7 +29,7 @@ file — plus an overview page that says which of the five you need:
|
||||
| `/disks` | Are any of my physical disks failing, and which one — health, temp, hours, wear | the same agents, `smartctl -i -A -H` per whole drive (never a partition) |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo `/metrics` + `/slots`, vLLM `/metrics` |
|
||||
| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) |
|
||||
| `/gateway` | Which domains come in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`), a trunk per bound port with its configured routes branching off it, each line reading the config's own **inbound** word (`server_name`, `map` key, `backend` name) over the counterpart it routes to, plus its live socket count, with one filter box above the map that hides the routes whose own labels do not contain what was typed | the same agent (1.2.0) on the one host `GATEWAY_HOST` names: its `haproxy` + `nginx` config sections joined to its `/proc/net/tcp{,6}` sockets |
|
||||
| `/gateway` | Which domains come in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`), a trunk per bound port with its configured routes branching off it, each line reading the config's own **inbound** word (`server_name`, `map` key, `backend` name) over the counterpart it routes to, plus its live socket count, with one filter box above the map that hides the routes whose own labels do not contain what was typed | the same agent (1.2.0's three proxy sections, 1.3.0's `resolver`) on the one host `GATEWAY_HOST` names: its `haproxy` + `nginx` config sections joined to its `/proc/net/tcp{,6}` sockets through the addresses the gateway itself resolved |
|
||||
|
||||
History is a first-class feature, not a sparkline afterthought: every number worth graphing
|
||||
is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`.
|
||||
@@ -124,13 +124,17 @@ page still renders.
|
||||
global IPv6 is EUI-64, so it encodes this box's NIC, and an inbound visitor's address was never
|
||||
ours to publish. Documentation space (RFC 5737 / RFC 3849), first-seen order, same width, same
|
||||
`::ffff:` spelling, idempotent; ports, states and every other column stay verbatim.
|
||||
`tests/test_agent.py` fails the day a fixture grows a routable endpoint back.
|
||||
**A captured resolver answer is the same case too** (phase 32): what `getent` returns for
|
||||
`reeseapps-immich.reeselink.com` on this router is as much an endpoint as a socket row is, so the
|
||||
same substitution runs over `tests/fixtures/gateway_resolver.json`, and the name itself is kept
|
||||
verbatim because the name is the thing the page draws. `tests/test_agent.py` fails the day either
|
||||
fixture grows a routable endpoint back.
|
||||
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
|
||||
`sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
|
||||
2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
|
||||
`scripts/capture-fixtures.sh` (and `scripts/capture-node-exporter.sh` when comparing
|
||||
exporters, `scripts/capture-gateway-config.sh` for the agent's `haproxy`/`nginx`/`sockets`
|
||||
ground truth) rather than trusting a README or a memory of an upstream.
|
||||
exporters, `scripts/capture-gateway-config.sh` for the agent's `haproxy`/`nginx`/`sockets`/
|
||||
`resolver` ground truth) rather than trusting a README or a memory of an upstream.
|
||||
8. **Keep dependencies current on purpose.** LLM-written version floors are usually stale.
|
||||
Check after touching dependencies: `uv pip list --outdated` must come back empty.
|
||||
Python is 3.14 (`requires-python >=3.14`, ruff `py314`, pyright `3.14`, Containerfile
|
||||
@@ -319,17 +323,57 @@ Responsibilities, one line each:
|
||||
uninstalled, and `false` beside an empty list would read as "no listeners", which is rule 2 — and
|
||||
**no join with the config**: a socket does not know its domain, so phase 29's page joins these
|
||||
ports to the `haproxy`/`nginx` sections and an unclaimed socket stays visible as one.
|
||||
**The resolver section (phase 32 task 01) is the fourth and the only one that leaves the host.**
|
||||
`target_names(haproxy, nginx)` is the whole allow-list — haproxy `backends[].servers[].target`,
|
||||
then nginx `upstream` members, `server`-block `proxy_pass` targets and `maps[].entries[].target`,
|
||||
deduplicated keeping first-seen order — and nothing else may ever enter it: not a name from
|
||||
`/proc/net/tcp`, not a client address, not anything a request supplied. That is decision 3 and
|
||||
`tests/test_agent.py` pins it by asserting the spied `getaddrinfo` call set **equals** this
|
||||
function's output minus the address literals, in config order, with no duplicates, and that no
|
||||
address from the recorded socket table ever appears in it. The lookup is
|
||||
`socket.getaddrinfo(name, port, proto=SOCK_STREAM)` — `via: "libc"`, which is the resolution
|
||||
haproxy's `init-addr last,libc,none` performs **on the box that dials**; the dashboard resolving
|
||||
the same name through its own `resolv.conf` would be a different vantage answering a different
|
||||
question, so `sources/gateway.py`'s resolver spy pin survives this phase on purpose and the
|
||||
agent does **not** hand-roll UDP queries to the config's own `resolver` server either. That line
|
||||
is published instead, verbatim, as `config_directive` (`"10.1.0.1"` on this fleet) — and it lives
|
||||
inside this section rather than inside `nginx` so the `nginx` document stays **byte-stable**, which
|
||||
keeps phase 32's additivity control a strict top-level key comparison with no exception list
|
||||
(`Collector.nginx_document()` is the seam that returns the pair; `Collector.nginx()` is its
|
||||
one-line wrapper). Entry shape is `{name, port, status: ok|nxdomain|error|literal, addresses,
|
||||
ts}`: `addresses` is the unique answer sorted v4-ascending then v6-ascending (libc's order is not
|
||||
stable and `entries` is a list a page renders), an address literal is `literal` with **no** lookup
|
||||
(it is already an address), and `nxdomain`/`error` carry `addresses: []` — never a stale guess,
|
||||
never a `0`, and `EAI_AGAIN` is `error` and not `nxdomain`, because "the server did not answer"
|
||||
and "the name has no record" are two fixes. The pass is SMART's idiom for SMART's reason
|
||||
(`_resolver_lock`/`_resolver_cache`/`_resolver_refreshing`, `RESOLVE_MAX_NAMES` 64 /
|
||||
`RESOLVE_CACHE_SECONDS` 300 / `RESOLVE_PASS_BUDGET_SECONDS` 3.0): the **first** pass inline, every
|
||||
later expiry refreshed on a single-flight daemon thread while the request is served the previous
|
||||
result with its **original per-entry `ts`**, a failed refresh keeping the old entries and
|
||||
re-stamping so the retry lands on the next expiry rather than on every poll, and the budget/name-
|
||||
bound excess each reported in **one** line (`resolver: pass budget 3.0s exhausted, 41 of 58
|
||||
targets looked up`) with `available` still `true` — a black-holing resolver must never blow
|
||||
`HOST_TIMEOUT_SECONDS` and blink `/system` over an add-on it never draws. `available` is the
|
||||
family's three-way answer (`false` = nothing to resolve because no proxy config was read, one
|
||||
line saying **not found** or **unreadable** per section; `null` = a pass that could not run,
|
||||
which `snapshot()`'s `_safe` turns into the empty section plus its line), and **`--no-resolve`
|
||||
removes the key in silence** the way `--no-smart` does — while `--no-proxy` removes it too, since
|
||||
a box that collects no proxy has no declared names.
|
||||
**Task 04 made the three sections a versioned contract with captured ground truth:**
|
||||
`AGENT_VERSION` is **1.2.0** (`tests/test_agent.py` pins the constant and asserts the
|
||||
`GET /schema` block names `haproxy`/`nginx`/`sockets` — each with its `available` flag — plus
|
||||
`--no-proxy`, `--haproxy-config` and `--nginx-config`), and `scripts/capture-gateway-config.sh`
|
||||
re-records the six fixtures the parsers are pinned to over `GATEWAY_SSH` (`gateway-ops`): the
|
||||
haproxy config, the nginx main config **plus every document its own glob `include` names** (so
|
||||
the fixture tree cannot drift from the config's tree while the parser still follows one level),
|
||||
and `/proc/net/tcp{,6}`. Three rules it exists to keep. (a) **No privilege and no copy tool**: one
|
||||
`ssh <alias> cat <path>` per document, because `/proc/net/tcp` reports size 0 and `scp` would copy
|
||||
an empty file, and every path it reads is world-readable — the same read the agent performs as its
|
||||
own user, so a capture that needed sudo would be recording something the agent cannot see.
|
||||
`AGENT_VERSION` is **1.3.0** (`tests/test_agent.py` pins the constant and asserts the
|
||||
`GET /schema` block names `haproxy`/`nginx`/`sockets` — each with its `available` flag — plus the
|
||||
`resolver` section, its own `available`, and `--no-resolve`, beside `--no-proxy`,
|
||||
`--haproxy-config` and `--nginx-config`), and `scripts/capture-gateway-config.sh`
|
||||
re-records the seven fixtures the parsers and the join are pinned to over `GATEWAY_SSH`
|
||||
(`gateway-ops`): the haproxy config, the nginx main config **plus every document its own glob
|
||||
`include` names** (so the fixture tree cannot drift from the config's tree while the parser still
|
||||
follows one level), `/proc/net/tcp{,6}`, and `gateway_resolver.json` — the resolver section
|
||||
itself, captured the way the agent will produce it (task 02, below). Three rules it exists to
|
||||
keep. (a) **No privilege and no copy tool**: one `ssh <alias> cat <path>` per document and one
|
||||
`getent` per name per family, because `/proc/net/tcp` reports size 0 and `scp` would copy an
|
||||
empty file, and every path and binary it uses is world-readable and world-runnable — the same
|
||||
reads the agent performs as its own user, so a capture that needed sudo would be recording
|
||||
something the agent cannot see.
|
||||
(b) **A bad read is never a fixture**: a nonzero `cat`/`ls`, an empty body or a missing glob include
|
||||
exits non-zero having written nothing, and a half-written file is removed — the pin is that the
|
||||
recorded bytes survive a dead alias and an empty read both. (c) **Configs verbatim, socket tables
|
||||
@@ -343,7 +387,41 @@ Responsibilities, one line each:
|
||||
publish: the same rule that scrubs a drive's serial (invariant 6). Those two tables are volatile by
|
||||
nature, so `tests/test_agent.py` asserts their **shape**, the ports the recorded configs claim, and
|
||||
the absence of a globally routable endpoint — never a particular row, which a re-run may lawfully
|
||||
change. What phase 28 stopped at was the wire: `sources/host.py` still passes **none** of the
|
||||
change.
|
||||
**The seventh fixture is the resolver section, and it is derived rather than listed (phase 32
|
||||
task 02).** `tests/fixtures/gateway_resolver.json` is the agent's own `resolver` document — same
|
||||
keys, `via: "libc"`, entries in `target_names()` order — so a source test can exercise the
|
||||
name→address join hermetically instead of wishing for DNS. Three rules keep it honest. (a) **The
|
||||
name list is the parser's, never the script's**: a `plan` step rebuilds the agent's `--root` tree
|
||||
from the config fixtures in this repo, imports `agent.host_agent`, and asks `target_names()` +
|
||||
`address_family()` what to look up — so a route added to the gateway moves the fixture with it, an
|
||||
address literal costs no remote read, and `tests/test_deploy_scripts.py` fails the day a domain is
|
||||
hardcoded into the script. (b) **`getent ahostsv4` and `ahostsv6`, one name at a time**, because
|
||||
that is libc on the box that dials; `rc 2` is "no record" and becomes `status: "nxdomain"` — a
|
||||
fact, kept beside its successful neighbours — while `rc 255` (the ssh died), `rc 127` (no `getent`)
|
||||
or **not one name answering** is a broken vantage, so the run exits 1 having left the recorded
|
||||
document byte-identical: a fixture of nothing but `nxdomain` would make every live route on
|
||||
`/gateway` a permanent dash and every join test a test of nothing. The `ts` is a constant and the
|
||||
answer set is deduplicated and sorted with the agent's own `sort_resolver_addresses`, so
|
||||
re-capturing a gateway nobody changed is a no-op. (c) **A resolver answer is an identifier**, so the
|
||||
socket scrub runs over it too (`8.8.8.8` → `203.0.113.1`), and the `::ffff:` spelling `getent
|
||||
ahostsv6` synthesises for a name with no AAAA record is dropped when the A record answered — the
|
||||
agent's one `getaddrinfo` never publishes it. This fleet's targets all answer private LAN addresses,
|
||||
which is why the recorded document is scrub-free; `tests/test_agent.py` pins that absence as a
|
||||
warning sign, because a documentation-space address there would not necessarily match the one the
|
||||
socket fixtures gave the same endpoint, and `tests/test_gateway_source.py`'s join depends on their
|
||||
agreeing. The version's additivity is pinned two ways: permanently, by
|
||||
`SNAPSHOT_KEYS_BEFORE_1_3_0` (1.2.0's top-level keys) in `tests/test_agent.py`, whose only legal
|
||||
successor is that set plus `resolver` — and value-for-value, by `data/logs/verify-32-additive.py`,
|
||||
which imports `HEAD`'s agent file beside the working tree's and calls both against one `--root`.
|
||||
One defect only the second kind of control can find: the lookup is
|
||||
`getaddrinfo(name, port, type=SOCK_STREAM)`, and `proto=SOCK_STREAM` — the spelling that reads as
|
||||
"the TCP addresses" to anyone who has not read the man page — is `EAI_SERVICE` on real libc for
|
||||
**every** name, because `proto` takes a protocol number (`IPPROTO_TCP` = 6) and `SOCK_STREAM` is 1.
|
||||
A stubbed `getaddrinfo` accepts any keyword, so the whole resolver suite stayed green while the
|
||||
agent reported `error` on a healthy gateway; `test_the_lookup_the_agent_performs_is_one_real_libc_accepts`
|
||||
removes the module-wide stub and asks real libc about `localhost`, the one name it answers from
|
||||
`/etc/hosts` with no network. What phase 28 stopped at was the wire: `sources/host.py` still passes **none** of the
|
||||
three into `/api/system/snapshot`, and `store.py` still records no series for a config — the join
|
||||
and the two gauges it publishes are phase 29's `sources/gateway.py`, which reads the same
|
||||
`/snapshot` through its own `gateway` job. What the *system* page does read today is their
|
||||
@@ -434,14 +512,16 @@ Responsibilities, one line each:
|
||||
the agent's smartctl pass, and `None` is translated into httpx's sentinel because a
|
||||
literal `None` means *no timeout at all*). Agent `errors`
|
||||
become the payload's `degraded` **except** the add-on lines named by
|
||||
`ADDON_ERROR_PREFIXES` (`smart`, `haproxy`, `nginx`, `sockets`) — a host without smartmontools,
|
||||
without device read access, without a proxy config, or whose kernel socket table did not answer
|
||||
is unconfigured, not broken (invariant 3), and `degraded` is `alerts.py`'s input, so a fleet-wide
|
||||
`ADDON_ERROR_PREFIXES` (`smart`, `haproxy`, `nginx`, `sockets`, `resolver`) — a host without
|
||||
smartmontools,
|
||||
without device read access, without a proxy config, whose kernel socket table did not answer, or
|
||||
whose resolver had nothing to resolve / stopped mid-pass is unconfigured, not broken (invariant
|
||||
3), and `degraded` is `alerts.py`'s input, so a fleet-wide
|
||||
agent upgrade would otherwise put an overview WARN on every box that never ran haproxy or nginx.
|
||||
Those lines stay verbatim in `agent.errors` for the page that owns the add-on — `/disks` explains
|
||||
the `smart:` ones (and since phase 22 a host whose agent said `smartctl` is *missing* is not
|
||||
listed on that page at all, `api._disk_visible`), phase 29's gateway page reads the
|
||||
`haproxy:`/`nginx:`/`sockets:` ones the same way.
|
||||
`haproxy:`/`nginx:`/`sockets:`/`resolver:` ones the same way.
|
||||
**A new agent add-on section adds its prefix here in the same commit**, or its absence-report
|
||||
becomes a red banner on a page that has nothing to say about it.
|
||||
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card
|
||||
@@ -490,14 +570,18 @@ Responsibilities, one line each:
|
||||
not warned about. What that retires is the guessing and the bookkeeping it existed to explain —
|
||||
the port-only guess for a name target, `unmatched` / `unmatched.groups` / `partial` /
|
||||
`unmatched_listeners`, and `by_port` with its `MAX_GROUP_HOSTS` chip. A **name** target
|
||||
(`reeseapps-immich.reeselink.com:443`) is therefore never credited: crediting it needs DNS (which
|
||||
the module still never calls, spy-pinned) and every name here resolves back through this same
|
||||
router, so the count would be attributed to something the config never said. Two consequences the
|
||||
(`reeseapps-immich.reeselink.com:443`) is credited **only through an address the gateway itself
|
||||
published** (phase 32, below): crediting it any other way needs DNS, which the module still never
|
||||
calls (spy-pinned), and every name here resolves back through this same router, so a vantage of
|
||||
ours would attribute a socket to a route on the strength of a query the router never made. Two
|
||||
consequences the
|
||||
page and the tests both depend on: **`live_out` is not the bandwidth** and the socket table never
|
||||
reconciles to it — a pipe count means "this route has N live connections" and nothing more — and
|
||||
several `(listen, endpoint)` lines reaching one endpoint carry one count **repeated** with
|
||||
`shared: true` (never a split nobody measured), which is why `/2`'s totals and the tests sum
|
||||
**distinct endpoints**. A pipe is `null` (never `0`) whenever the socket table did not answer or no
|
||||
`shared: true` (never a split nobody measured) — a shape that is real now that a name route is
|
||||
counted: two vhosts writing one `(address, port)`, two **names** the resolver placed on one
|
||||
address, or a multi-homed name sharing one address with a single-homed one — which is why the
|
||||
API's totals and the tests sum **distinct endpoints**. A pipe is `null` (never `0`) whenever the socket table did not answer or no
|
||||
line reaches its endpoint, and publishes **no sample** then, so a chart cannot draw a zero for an
|
||||
unknown. `supported: false`
|
||||
(a 1.1.0 agent, or `--no-proxy`) keeps every key with empty lists and null counts plus
|
||||
@@ -517,15 +601,44 @@ Responsibilities, one line each:
|
||||
line whose selectors union in first-seen order; **never** merged across `in_port`) and two silent
|
||||
refusals (a `proxy_pass` with neither target nor variable, and a haproxy frontend that routes by
|
||||
a fetch expression, which the agent reports as no name — that trunk is drawn alone, because "no
|
||||
name" and "no routing directive" are one value in this payload). **Name-keyed outbound gauges are
|
||||
retired** (phase 30 task 02): `gateway.out_conns@<host>:<port>` is written **only** for an endpoint
|
||||
the config wrote as an address (`out.address` true), because the sole reason a name-keyed gauge
|
||||
existed was the port-only guess the join no longer makes — such a series would be a chart of a
|
||||
number that is always `null`. `gateway.in_conns@<port>` is unchanged, and the port **stays** in the
|
||||
outbound name, because this fleet's two haproxy Minecraft backends share `minecraft.reeselink.com`
|
||||
on 25565 and 25566 and two gauges on one name would overwrite each other in silence; lines sharing
|
||||
an endpoint write the same value to the same name (they are one count, not two) and `samples()`
|
||||
de-duplicates by metric. `parse_gateway_snapshot()` is the
|
||||
name" and "no routing directive" are one value in this payload). **A name-written endpoint is
|
||||
joinable through the address the gateway published for it** (phase 32 task 03): `resolution_index()`
|
||||
keys agent 1.3.0's `resolver.entries` by `(name, port)` — `{}` when the section is absent or its
|
||||
`available` is not `true`, which is phase 30 byte-for-byte, and never an entry with no addresses —
|
||||
and `apply_resolution()` stamps every `out` with `resolved` (the address list, **never `[]`**: an
|
||||
entry with no addresses and no entry at all are one fact, so the page's dash arm is one `is null`
|
||||
test), `via` (the agent's own `"libc"`, copied, `null` where nothing was resolved) and
|
||||
`resolved_ts` (**that entry's** stamp, falling back to the section's, because a name refreshed five
|
||||
minutes ago and one answered on this poll are two claims). All three stay `null` on an
|
||||
`address: true` row: the config wrote that address, and copying the agent's `literal` echo of it
|
||||
into `resolved` would report a `getaddrinfo` that never ran. With an address and a port the line
|
||||
becomes countable, so **name-keyed gauges are back** — `gateway.out_conns@<domain>:<port>`, under
|
||||
the config's own spelling so history continues under one name. That is the reverse of phase 30
|
||||
task 02's retirement for exactly one reason: the retirement existed because the only figure such a
|
||||
series could carry was the *port's*, and the figure here is the sockets to the address this domain's
|
||||
own libc resolution produced **on the box that dials**. An unresolved name keeps `metric: null`,
|
||||
`connections: null`, no sample and the dash the footer explains. `gateway.in_conns@<port>` is
|
||||
unchanged, and the port **stays** in the outbound name, because this fleet's two haproxy Minecraft
|
||||
backends share `minecraft.reeselink.com` on 25565 and 25566 and two gauges on one name would
|
||||
overwrite each other in silence; lines sharing an endpoint write the same value to the same name
|
||||
(they are one count, not two) and `samples()` de-duplicates by metric.
|
||||
**The join key is a normalised address, never a spelling** (`_address_key`, `endpoint_of`,
|
||||
`_endpoint_groups`): `ipaddress` canonicalises and folds `/proc`'s `::ffff:10.1.2.64` to the
|
||||
`10.1.2.64` `getaddrinfo` returned — one kernel endpoint, one count, however many spellings or names
|
||||
reach it — while a zero-padded `010.1.2.64` stays **no one's address** (decimal versus octal is a
|
||||
guess, and a guess on the far end of a pipe is rule 2), so `out.resolved` keeps the agent's bytes and
|
||||
only the key is canonical. `_endpoint_groups()` is a union-find over those keys (a name route and an
|
||||
address route reaching one endpoint are one group; overlapping multi-homed sets merge; a route with
|
||||
no endpoint is no group) and `distinct_endpoint_counts()` is what `api._endpoint_values` sums for
|
||||
`live_out`, which is the only reason the head tally and the pipes cannot disagree. Matching stays
|
||||
exact equality on `(address, port)` — no prefix, no range — and this module still resolves nothing:
|
||||
`socket.getaddrinfo` is spy-pinned absent over a name-only synthetic, a resolver-full synthetic and
|
||||
the captured `resolved` ingress, plus `samples()` (a gauge name is built from a host string and is
|
||||
the other place a resolver could creep in). The payload's own `resolver` block is
|
||||
`{available, via, entries}` with each entry **projected** to `name`/`port`/`status`/`addresses`/`ts`
|
||||
— every entry including `nxdomain` and `error`, because the index filters *within* the published
|
||||
list and a page that says *this name has no record* needs the entry that says so — and a section
|
||||
that did not answer publishes none. `parse_gateway_snapshot()` is the
|
||||
`Result` the poller consumes; there are **no events** (a connection count has no transition
|
||||
worth a line — a proxy that stops routing shows up as its host's own poll failing). The
|
||||
never-answered arm is `stub(host, ts)` — the **same skeleton** `parse` fills in, plus
|
||||
@@ -939,7 +1052,11 @@ Responsibilities, one line each:
|
||||
or a `window_seconds`, at any depth (pinned by a recursive walk: a busy proxy is the job and no
|
||||
Δ was ever measured). Phase 30 task 02 took `outbound[]`, `unmatched` and `by_port` out of it: the
|
||||
flat list duplicated the graph with a worse count, and the residual existed only to explain a
|
||||
guessing rule that no longer runs. `/history` serves the `gateway.in_conns@<port>` /
|
||||
guessing rule that no longer runs. Phase 32 task 03 added one top-level key, `resolver` — the
|
||||
agent's own block, `{available, via, entries}`, each entry projected to the five fields the agent
|
||||
publishes — so a dash on one line is answerable from the document the page already holds; `stub()`
|
||||
grows it with nulls, which is what the stub/answer key-parity pin compares. `/history`
|
||||
serves the `gateway.in_conns@<port>` /
|
||||
`gateway.out_conns@<host>:<port>` gauges with `hosts=` defaulting to the ingress **alone** (the
|
||||
name the samples are stored under — defaulting to every target would draw empty plots for
|
||||
machines nobody asked to route), and it carries no configured gate because it dials nothing:
|
||||
@@ -989,9 +1106,12 @@ Responsibilities, one line each:
|
||||
`live_out` the **distinct outbound endpoints** — never the connection lines — because several
|
||||
`(listen, endpoint)` lines carry one socket count repeated with `shared: true`, and summing the
|
||||
lines would book the same sockets once per line and print a router busier than it is (the same
|
||||
rule that keys the outbound gauges by `(address, port)`); `api._endpoint_values()` is the one
|
||||
reader of that, and a line with no `metric` (its target is a name the join can never count) is
|
||||
absent rather than an unmeasured endpoint holding the total hostage. `map_targets` counts the
|
||||
rule that keys the outbound gauges by `(address, port)`, with the address normalised so two names
|
||||
— or two spellings — resolving to one endpoint book it **once**); `api._endpoint_values()` is the one
|
||||
reader of that (it delegates to `gateway_source.distinct_endpoint_counts()`, the same union-find the
|
||||
join builds its counters with, so the block's total and the pipes cannot disagree), and a line with
|
||||
no `metric` (a name nothing resolved) is absent rather than an unmeasured endpoint holding the total
|
||||
hostage. `map_targets` counts the
|
||||
stream map's routing entries. Every figure is `null` — never `0` — in the arms that did not
|
||||
measure (unconfigured, configured-but-never-answered, a 1.1.0/`--no-proxy agent, a silent socket
|
||||
table), and a total is `null` rather than a part of itself when any countable endpoint is unknown
|
||||
@@ -1317,10 +1437,12 @@ dashboard family — `host`, `llm`, `ups`, `disks` for the SMART series
|
||||
`disk.uncorrectable@…`, `disk.percentage_used@…`), and `gateway` for the phase-29 connection
|
||||
gauges (`gateway.in_conns@<listen-port>`, `gateway.out_conns@<host>:<port>` — the port belongs
|
||||
in the outbound name because two services on one host are two facts and one name would make the
|
||||
second insert overwrite the first; since phase 30 task 02 the outbound gauge is written **only** for
|
||||
an endpoint the config wrote as an **address**, so no name-keyed `gateway.out_conns@…` name is ever
|
||||
inserted again — a host that had one before this change keeps the old series until `maintain()` ages
|
||||
it out, and its chart simply stops growing points). Those keep their own family instead of
|
||||
second insert overwrite the first; the outbound gauge is written for an endpoint whose address the
|
||||
join knows — one the config wrote as an **address**, or (phase 32 task 03) one the gateway's own
|
||||
`resolver` section published for the name it wrote, under the config's own spelling, so the
|
||||
name-keyed `gateway.out_conns@<domain>:<port>` series phase 30 task 02 stopped inserting continue
|
||||
instead of restarting. A name nothing resolved is still never inserted, and its chart simply stops
|
||||
growing points). Those keep their own family instead of
|
||||
sitting beside `host`'s `disk.used_pct@/mount` because the cadences differ thirty-fold and
|
||||
`GET /api/metrics?source=disks` has to answer "what does *this page* have history for".
|
||||
That scheme is the history API's key — treat it as a public interface.
|
||||
@@ -1378,14 +1500,22 @@ That scheme is the history API's key — treat it as a public interface.
|
||||
`https://` agent as `host:gateway` with the same certificate entry, and re-verified live on the
|
||||
dev server with no change to any pre-existing source.
|
||||
**The fleet is mTLS since 2026-10-21 (phase 24 task 03), so the bearer-token transport is now
|
||||
history rather than the designed shape.** `gateway` runs `agent/host_agent.py` **1.2.0** (phase 28
|
||||
task 05, 2026-10-22) and `deskwork`/`driveripper` still run **1.1.0**; the spread costs nothing,
|
||||
because 1.2.0 is purely additive — the three proxy-config sections are new keys and a 1.1.0
|
||||
snapshot simply has none — so those two move at their next re-stage rather than on a flag day.
|
||||
The control that proves "purely additive" is `data/logs/verify-28-final.py`: it stashes the whole
|
||||
history rather than the designed shape.** `gateway` runs `agent/host_agent.py` **1.3.0** (phase 32
|
||||
task 05, 2026-10-25, on its existing bundle and an unchanged unit — `data/logs/NOTES-32-rollout.md`)
|
||||
and `deskwork`/`driveripper` still run **1.1.0**; the spread costs nothing,
|
||||
because both upgrades were purely additive — the three proxy-config sections and the `resolver`
|
||||
section are new keys, and a 1.1.0 snapshot simply has **none of them**: no `available` flag, no
|
||||
error line, nothing for `collectors.ADDON_ERROR_PREFIXES` to keep out of `/system`'s `degraded`, and
|
||||
`sources/gateway.py` answers that absence as `supported: false` + `WHY_PREDATES` rather than an empty
|
||||
gateway — so those two move at their next re-stage rather than on a flag day. (The three-way
|
||||
`available: false` arm belongs to a **1.2.0/1.3.0** host that has no proxy config to read, which is
|
||||
a different fact from a host whose agent predates the section.)
|
||||
The controls that prove "purely additive" are `data/logs/verify-28-final.py` and
|
||||
`data/logs/verify-32-additive.py`: each stashes the whole
|
||||
working tree, imports both `host_agent.py` files side by side, calls both `collect()`s against
|
||||
the same `--root`, and demands the 1.2.0 document equal the 1.1.0 document key-for-key with
|
||||
`haproxy`/`nginx`/`sockets` added — and 1.2.0 with `--no-proxy` equal it exactly.
|
||||
the same `--root`, and demands the newer document equal the older one key-for-key with
|
||||
`haproxy`/`nginx`/`sockets` (phase 28) and `resolver` (phase 32) added — and the newer one with
|
||||
`--no-proxy` / `--no-resolve` equal it exactly.
|
||||
All three are installed from a bundle, every unit carries `--cert-bundle /etc/dashboards-agent.certs` (plus
|
||||
`RuntimeDirectory=dashboards-agent`, and still its `--token-file` line), and `HOST_TARGETS` is
|
||||
`"driveripper=https://10.1.0.126:9120 deskwork=https://10.1.0.51:9120
|
||||
@@ -1495,7 +1625,8 @@ That scheme is the history API's key — treat it as a public interface.
|
||||
+ `CapabilityBoundingSet` unit with the snapshot's capability-less one, **silently**. That trap
|
||||
is now closed on all three hosts: `/home/ops/agent-install/` is `ops`-owned, so the repo's
|
||||
`deploy/install-agent.sh` (sha256 `ffbcd91f…`) and `agent/host_agent.py` (**1.1.0**, re-staged as
|
||||
**1.2.0** on `gateway` in phase 28) were staged over it without root, and each snapshot was left
|
||||
**1.2.0** on `gateway` in phase 28 and **1.3.0** in phase 32) were staged over it without root, and
|
||||
each snapshot was left
|
||||
beside the grant as
|
||||
`install-agent.sh.oct18-backup`, which sudo will not run. **Re-staging is therefore the first
|
||||
step of any install, not an optional one** — the grant is only as current as the file it points
|
||||
@@ -1653,11 +1784,14 @@ uv run ruff check src tests agent && uv run pyright
|
||||
SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh
|
||||
# ... plus smartctl per physical disk (needs device read
|
||||
# access: plain smartctl, else `sudo -n`; scrubs serials)
|
||||
./scripts/capture-gateway-config.sh # ... plus the gateway's proxy tree and socket tables:
|
||||
# haproxy.cfg, nginx.conf + its glob includes, and
|
||||
# /proc/net/tcp{,6} over $GATEWAY_SSH (gateway-ops);
|
||||
# world-readable paths only, and a failed or empty read
|
||||
# never replaces a good fixture
|
||||
./scripts/capture-gateway-config.sh # ... plus the gateway's proxy tree, socket tables and
|
||||
# resolved targets: haproxy.cfg, nginx.conf + its glob
|
||||
# includes, /proc/net/tcp{,6}, and gateway_resolver.json
|
||||
# (the names `target_names()` derives from those configs,
|
||||
# read back with `getent ahostsv4/6` — no DNS client, no
|
||||
# sudo) over $GATEWAY_SSH (gateway-ops); a failed or empty
|
||||
# read never replaces a good fixture, and a vantage where
|
||||
# not one name answers is refused rather than recorded
|
||||
|
||||
bash .agents/validate.sh # the phase gate: the four checks above (pytest
|
||||
# --cov --cov-fail-under=94, ruff, pyright, plus a
|
||||
@@ -1743,20 +1877,25 @@ branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only
|
||||
is compared with the agent's document and not with a copy of its field names, and then demands the
|
||||
two attributions and nothing beyond them: a socket whose **local** port names a listener is
|
||||
inbound, a socket whose remote `(address, port)` equals a configured target is outbound, and
|
||||
**every other socket is ignored** — the capture's `established` (18 that day) and the two totals
|
||||
(9 in, 2 out) are therefore both printed and deliberately never reconciled, because the identity
|
||||
this suite used to demand (in + out + unmatched = the table) was the arithmetic of a guessing rule.
|
||||
The rules that must never come back are pinned as absences: no `by_port`/`partial`/`unmatched`
|
||||
**every other socket is ignored** — the capture's `established` and the two totals are therefore
|
||||
both printed and deliberately never reconciled, because the identity this suite used to demand
|
||||
(in + out + unmatched = the table) was the arithmetic of a guessing rule, and because most of a
|
||||
busy router's sockets are nobody's pipe, which is why **`live_out` is not the bandwidth**. The
|
||||
rules that must never come back are pinned as absences: no `by_port`/`partial`/`unmatched`
|
||||
vocabulary in the payload or on the page, no `0` for a pipe the join could not count, and
|
||||
`socket.getaddrinfo` spied so the day the join resolves a name the run fails — a name is never
|
||||
credited with a connection (rule 2), which is why 14 of this fleet's 18 live sockets are nobody's
|
||||
pipe and why **`live_out` is not the bandwidth**. One number in that suite is a **date**, not a fact:
|
||||
the captured `stream.conf` has 17 `map` entries (so 32 connections and 17 leaves on `:443`), while the
|
||||
live router answered **18** entries / 33 connections on 2026-10-23 (`data/logs/30-live-verify.out`)
|
||||
because a domain was added after the capture. That is the designed difference between a fixture and a
|
||||
reading — folding the drift in means re-running `./scripts/capture-gateway-config.sh` **and**
|
||||
re-tallying `tests/test_gateway_source.py` in the same commit, never editing a test to match a live
|
||||
read.
|
||||
credited with a connection (rule 2).
|
||||
**Config-derived counts are literals in these tests; socket-derived ones are computed from the
|
||||
same capture.** `tests/test_api.py` asserts 10 listeners, 33 connections and 18 `map_targets`
|
||||
as numbers because only the operator moves those, and derives `live_in`, `live_out`, each pipe's
|
||||
count and `established` from `gateway_snapshot("full")` because a second capture of a running
|
||||
router lawfully changes them — pinning the live totals as literals is how a re-capture turns a
|
||||
green suite red over nothing. The rule is the assertion; the reading is not.
|
||||
Which is the same discipline that closed the drift phase 32 task 02 folded in: that capture's
|
||||
`stream.conf` gained a `map` entry (a domain was added to the router after phase 30 recorded it),
|
||||
so 17 became 18 and 32 connections became 33 **in the commit that re-ran
|
||||
`./scripts/capture-gateway-config.sh`** — re-tallying the config-derived counts, never editing a
|
||||
test to match a live reading while the fixture stays where it was.
|
||||
- Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host
|
||||
rebooting mid-poll, a counter reset — is a fake in `tests/conftest.py`.
|
||||
- **A verdict the operator sees is pinned on the surface that shows it, in both directions.**
|
||||
@@ -2215,17 +2354,37 @@ for a difference of one connection (the `/llm` slot-grid rule). `pruneCards()` w
|
||||
map is the only bulk clear, for the case where the family itself left the payload.
|
||||
|
||||
A count is printed by `countText()`, which answers `null` for anything that is not a finite number —
|
||||
**never `0`** (rule 2) — and the two unknowns stay two sentences: a **trunk** whose socket table went
|
||||
quiet prints `n/a` (`setTxt`'s own word), while a **leaf** whose endpoint the config wrote as a
|
||||
**name** prints an em dash (`NO_COUNT`) whose `title` says `connections not attributed — the config
|
||||
writes a name and nothing here resolves names`. That is the normal look of 31 of this ingress's 33
|
||||
lines, and a column of `n/a` would be twelve identical words competing with the page's one footnote.
|
||||
The footer's second paragraph is the same sentence in the operator's terms rather than the join's —
|
||||
*a dash means the endpoint is written in the config as a name; the page matches sockets by address and
|
||||
resolves nothing, so this route is drawn but not counted* — because a column of dashes needs one
|
||||
explanation on the page and not thirty-two on the rows, and because it is a **name** that is the reason
|
||||
and not a fault. `32_gateway_resolved_target_liveness` is what removes the reason; until then the dash
|
||||
stays unexplained-looking by design rather than borrowing the trunk's `n/a` word.
|
||||
**never `0`** (rule 2). Phase 32 task 04 made the missing count say **which** missing count it is:
|
||||
`countCause()` reads the same two fields `sources/gateway.py` joins on (`out.address === true` or a
|
||||
non-empty `out.resolved`) and returns one of two fragments, so `no number: the gateway could not
|
||||
resolve this name in this reading` is printed only when nothing is known for the endpoint, and
|
||||
`no number: the socket table did not answer this reading` only when an address **is** known and
|
||||
nobody measured — because those are two operator actions, and either one drawn as the other sends
|
||||
someone to the wrong box. Both go out through `countWhy()` on the `title` and through the same
|
||||
`countCause()` in `leafAria()` (`connections not attributed: …`), so a row's tooltip and its
|
||||
accessible name cannot disagree, and a dash is never the trunk's `n/a`, which is the trunk's own
|
||||
fact about its own port. **A `0` is not an unknown at all**: a resolved endpoint with no socket is a
|
||||
measurement, keeps `0 open connection(s) on this endpoint`, and never borrows either sentence
|
||||
(`test_an_unresolved_name_and_an_idle_route_say_two_different_sentences`, and
|
||||
`data/logs/32-task04-sentences.cjs`, which serves all four arms through route interception because
|
||||
the live 1.2.0 ingress can only produce one of them). Every **counted** line gets its provenance on
|
||||
the same `title` rather than on a column of ink: `countTitle()` appends
|
||||
`· address from <via>, resolved <age>` — `via` copied from the payload (`libc`) and the age through
|
||||
`duration(ageSeconds(out.resolved_ts))` — and prints neither fragment on a row the config wrote as an
|
||||
address, because `out.address` is already the whole story and echoing the agent's `literal` back as a
|
||||
lookup would report a `getaddrinfo` that never ran; a missing `resolved_ts` costs the age and nothing
|
||||
else, so no row prints `n/a` or `NaN` where an age belongs. Phase 32 task 03 left the rendering alone
|
||||
and changed what lands in the arms: a name-written route is now counted through the address the
|
||||
**gateway** resolved, so on the **captured** ingress all 33 lines carry a number (26 of them a
|
||||
measured `0`) and a dash has become the exception that means *this name has no record*
|
||||
(`nxdomain`/`error` in `resolver.entries`, or no entry beside it — including no section at all, which
|
||||
is what a 1.2.0 agent or `--no-resolve` answers (the live ingress has run 1.3.0 since phase 32 task 05
|
||||
on 2026-10-25, where **33 of 33** rows carried a number and none dashed; the same pass recorded
|
||||
`resolver.available: true`, 32 entries, `via: "libc"`, `config_directive: "10.1.0.1"`, 30 `ok` + 2
|
||||
`literal` targets). The footer's second paragraph says the same thing once for the whole page — a dash means the
|
||||
gateway could not resolve that name in this reading, hover a counted line for the address it matched
|
||||
and when that answer was taken — and the footer is still four `<p>`s (phase 31 task 04's filter
|
||||
footnote is the fourth, which is why the count that phase's pin demands is four and not three).
|
||||
The line itself is hand-rolled SVG (`role="img"` + an `aria-label` naming the inbound port, the
|
||||
endpoint, the selector and the count) and the row carries **two independent class pairs** (phase 30
|
||||
task 05, moved onto the `<li>` by phase 31 task 01), because the operator's green and the socket count
|
||||
@@ -2316,7 +2475,10 @@ the field is the one control that could be read as a query, and it is not one.
|
||||
**`#gate-filter` is an eye, not a query** (phase 31 task 04). 33 routes in config order is not a
|
||||
lookup, and the operator's question is always about one domain, so one field above the panels hides the
|
||||
rows whose **own payload text** does not contain what was typed: `in.port`, `out.host`, `out.name`,
|
||||
`out.kind` and every `out.selectors` entry for a route (`haystack()`), `:${port}` plus
|
||||
`out.kind`, every `out.selectors` entry and every `out.resolved` address for a route (`haystack()`;
|
||||
the sixth field is phase 32 task 04's, and it is what makes typing `10.1.2.64` find the route whose
|
||||
name target resolves there — a haystack of names could not answer a question asked about a socket),
|
||||
`:${port}` plus
|
||||
`listeners[].domains` and `.clients` for a trunk (`trunkHaystack()`). Nothing else is readable and
|
||||
nothing else is matched — no DNS, no re-ranking, no rule about `.reeselink.com`, and no page-level
|
||||
split by domain suffix, because "`reeseapps.com` vs `stackexpected.com`" is a fact neither config file
|
||||
@@ -3011,7 +3173,7 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
|
||||
The rollout is done and verified: gateway, deskwork and driveripper each run `agent/host_agent.py`
|
||||
**1.1.0** with `/etc/dashboards-agent.certs` installed `0640 root:dashboards-agent`, their units
|
||||
carry `--cert-bundle` + `RuntimeDirectory=dashboards-agent` (and keep `--token-file`) — the
|
||||
gateway has since moved to **1.2.0** for phase 28's proxy-config sections on the same unit and
|
||||
gateway has since moved to **1.2.0** (phase 28) and then **1.3.0** (phase 32) on the same unit and
|
||||
the same bundle, see the fleet record above —
|
||||
`HOST_TARGETS` is `https://` for all three with no userinfo, `agent.bundle_rev` reads `2` on
|
||||
every host and equals what `/api/certs` issued, and the nine sources are `ok`. The transcript is
|
||||
@@ -3275,10 +3437,16 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
|
||||
|
||||
13. **`/gateway`'s long endpoint labels are ellipsised on a phone — a column-width question, not a
|
||||
lost fact, and the filter is what a clipped label used to block** (found by phase 29 task 06's
|
||||
live pass, **closed at the desktop width by phase 30 task 04's trunk→leaf row and re-measured
|
||||
after phase 31 re-ranked the row on 2026-10-26**; `data/logs/31-route-readability.out` is the
|
||||
current measurement, `data/logs/30-connection-map.out`, `data/logs/30-task04-panels.out`,
|
||||
`data/logs/29-task06-live.out` and `data/logs/29-task06-card-haproxy.png` the history).
|
||||
live pass, **closed at the desktop width by phase 30 task 04's trunk→leaf row, re-measured
|
||||
after phase 31 re-ranked the row on 2026-10-26 and again after phase 32 put the count
|
||||
provenance on the `title` on 2026-10-26**; `data/logs/32-counted-routes.out` is the current
|
||||
measurement, `data/logs/31-route-readability.out`, `data/logs/30-connection-map.out`,
|
||||
`data/logs/30-task04-panels.out`, `data/logs/29-task06-live.out` and
|
||||
`data/logs/29-task06-card-haproxy.png` the history). Phase 32's re-check found the clip
|
||||
counts **unchanged** by the DOM/provenance move — **19 of 33** leaf inbound lines and **33
|
||||
of 33** counterpart lines at 360 px (the 10 trunk names, none clipped), 0 of 33 at 1440 px,
|
||||
and the whole pair on **33 of 33** rows' tooltips — so the trade is the same and only the
|
||||
tooltip's second sentence (the provenance) is new.
|
||||
|
||||
**Phase 30 task 04 closed the desktop half by changing the row, not the font.** The two
|
||||
equal-width pipe columns of phase 29 became a group of `minmax(0, 9.5rem) minmax(0, 1fr)` — a
|
||||
@@ -3330,16 +3498,40 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
|
||||
declarations and removed the reason they were needed: with the trunk in its own track, the leaf's
|
||||
name is the widest thing in a column that is now wide enough for it.
|
||||
|
||||
14. **A route whose endpoint the config writes as a *name* is drawn and never counted —
|
||||
`32_gateway_resolved_target_liveness` is the phase that removes the reason** (recorded by phase
|
||||
31 task 05, which wrote no design for it). Today 31 of this ingress's 33 routes end in a
|
||||
`server_name`/`map` target and so carry an em dash where a live count would go, because
|
||||
`sources/gateway.py` matches sockets by
|
||||
address and resolves nothing (rule 2 by choice, and pinned as an absence). The follow-on phase is
|
||||
the one that makes such a target answer with its liveness and its live connections anyway — the
|
||||
agent reporting resolved targets on the box that owns the config, which is an `AGENT_VERSION` bump
|
||||
to **1.3.0** and therefore an agent + source + pin change, not a page change. Its acceptance line
|
||||
is that a dash disappears from `/gateway` for a reason and not by hiding the column.
|
||||
14. **A name-written route is counted through the address the gateway published — closed, the
|
||||
re-stage included** (recorded by phase 31 task 05, implemented by
|
||||
phase 32 tasks 01-03). Until phase 32, 31 of this ingress's 33 routes ended in a
|
||||
`server_name`/`map` target and carried an em dash where a live count would go, because
|
||||
`sources/gateway.py` matches sockets by address and resolves nothing. That is still true — the
|
||||
module calls no resolver, spy-pinned over the parse, the gauge names and the capture — but it no
|
||||
longer has to, because agent **1.3.0** publishes `resolver`: every `(name, port)` its own two
|
||||
config files write as a target, answered by libc **on the box that opens the connection**, with
|
||||
the nginx `resolver` directive printed beside it as `config_directive` so a divergence between the
|
||||
two vantages is visible instead of guessed. `sources/gateway.py` reads that document, stamps
|
||||
`out.resolved` / `via` / `resolved_ts`, joins sockets through the normalised address, and writes
|
||||
the `gateway.out_conns@<domain>:<port>` gauge under the config's own spelling — so the dash is gone
|
||||
on the live page **because the gateway said what the name became**, which was the acceptance line,
|
||||
and not because a column was hidden.
|
||||
**The deployment half is done.** `gateway` ran **1.2.0** until phase 32 task 05 re-staged it on
|
||||
2026-10-25 (`data/logs/NOTES-32-rollout.md`: unit sha256 unchanged, bundle untouched, the install
|
||||
run guard silent, one file and one restart), and that pass read `/api/gateway/snapshot` going from
|
||||
**2 of 33** rows carrying a number to **33 of 33**, with all 31 resolver-placed rows finite and
|
||||
`test_a_resolved_target_on_the_live_ingress_is_never_a_dash` passing instead of skipping. What is
|
||||
left of the gap is rotation hygiene, not a step: `deskwork` and `driveripper` stay on 1.1.0 (no
|
||||
proxy configs, therefore nothing to resolve), and a host that later rolls **back** to a pre-1.3.0
|
||||
file simply returns its name routes to dashes — `sources/gateway.py` treats a missing `resolver`
|
||||
section as phase-30 behaviour. A name that stops
|
||||
resolving is `nxdomain`/`error` and goes back to being a dash on the next poll (the agent caches
|
||||
300 s and never publishes a stale address as current), which is the designed degradation rather
|
||||
than a gap — and since task 04 the page says **which** degradation it is: `countCause()` separates
|
||||
that dash (`the gateway could not resolve this name in this reading`) from the one an unreadable
|
||||
socket table causes (`the socket table did not answer this reading`), while a resolved endpoint with
|
||||
nothing open prints its `0` beside `· address from libc, resolved <age>`, so the sentence a line
|
||||
carries names the box to go and look at. `data/logs/32-task04-sentences.cjs` measures all four arms,
|
||||
serving the three a pre-1.3.0 host could not produce through route interception (43 checks, and
|
||||
each stub leg
|
||||
first proves its DOM equals the document that leg was served, so a leg that rendered the live page
|
||||
instead fails by name rather than three checks later).
|
||||
|
||||
15. **`/gateway`'s switches are knobs with no write behind them — `33_gateway_route_switching` is the
|
||||
phase that decides whether they ever get one** (recorded by phase 31 task 05, no design written).
|
||||
|
||||
Reference in New Issue
Block a user