services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
51772ffef2361d8f2fe2a87d07ae58475d45538d
dashboards/AGENTS.md
T
ducoterra 51772ffef2 phase: 32_gateway_resolved_target_liveness 
Fixture and live stream.conf are byte-identical (modulo header comment). Let me test the parser directly:
2026-10-26 14:03:42 -04:00

3576 lines
313 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# AGENTS.md — homelab dashboards
The plan, the rules, and the environment facts an agent needs to work on this repo without
re-deriving them. `README.md` is the operator-facing how-to; this file is the why, the
invariants, and the state of the fleet.
## Working protocol
This project runs the `.agents/phases/` phased-execution pipeline.
1. Always read `.agents/PLAN.md` first to understand the project context and goals.
2. Follow the phased execution protocol in `.agents/phases/`.
3. Never modify `.agents/PLAN.md` or any files in `.agents/phases/complete/`.
4. If you need to update any file in `.agents/phases/todo/`, you must ask the user for permission first.
5. Strictly adhere to the **LOCKED DECISIONS** listed in `.agents/PLAN.md`.
Everything below this section is the project's own content: what the app is, the rules that
keep its numbers honest, and the state of the fleet it watches.
## What this is
Five dashboards for one homelab, behind one domain, in one process, backed by one SQLite
file — plus an overview page that says which of the five you need:
| Route | Question it answers | Source |
| ---------- | ------------------------------------------------------------------------------ | ------------------------------------------------- |
| `/` | Is anything wrong, and where do I look first? | all five, in one request |
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent per host |
| `/disks` | Are any of my physical disks failing, and which one — health, temp, hours, wear | the same agents, `smartctl -i -A -H` per whole drive (never a partition) |
| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo `/metrics` + `/slots`, vLLM `/metrics` |
| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) |
| `/gateway` | Which domains come in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`), a trunk per bound port with its configured routes branching off it, each line reading the config's own **inbound** word (`server_name`, `map` key, `backend` name) over the counterpart it routes to, plus its live socket count, with one filter box above the map that hides the routes whose own labels do not contain what was typed | the same agent (1.2.0's three proxy sections, 1.3.0's `resolver`) on the one host `GATEWAY_HOST` names: its `haproxy` + `nginx` config sections joined to its `/proc/net/tcp{,6}` sockets through the addresses the gateway itself resolved |
History is a first-class feature, not a sparkline afterthought: every number worth graphing
is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`.
None of it is public. One admin signs in at `/login` (the only page an anonymous visitor
reaches), and the two admin pages live behind it: `/settings` — `⚙ Account` in the masthead —
where that session is read, ended and re-credentialled, and `/certs` — `⚷ Certs`, next to it —
where the fleet's certificate authority is listed, rotated, revoked and downloaded. Every page
above, `/settings`, `/certs`, and every `/api` route but the four named in invariant 12 require
the session.
## Non-negotiable rules
These are the things that make the numbers trustworthy. Breaking one is a bug even if the
page still renders.
1. **Read-only by construction, with exactly one exception.** Nothing in this app writes to
an upstream except `POST /api/ups/command`. That endpoint is gated four times
(`NUT_ALLOW_ACTIONS`, not `DASH_READ_ONLY`, the upsd.users account actually having the
command, and a UI confirmation — `DANGER` for anything that can cut power) and the server
re-checks every gate on every request. Hiding a button is not a safety control.
Commands that would cut output are refused outright while the UPS is already on battery. Since
phase 11 there are two more in front of it — a session, and this session's CSRF token for a
cookie write (invariant 12) — and they are additive: `tests/test_auth_api.py` proves the
`DANGER` requirement and the wire send still behave exactly as they did without them.
2. **Never invent a number.** A value the upstream does not publish renders as `n/a`, never
as `0`. Every derived value carries the `source` string it came from and `derived: true`
so the UI can print *how* it was obtained (`computed:Δtokens/Δt` labelled `Δ2.0s`, the LLM's
own poll interval since phase 23). Aggregating
two different series into one rate is the failure mode this rule exists to catch — the
rate estimator is keyed by the *resolved* upstream series name **plus its label set**
for exactly that reason (phase 13: a label set joining a summed counter mid-window
contributes nothing until it has its own baseline, never its carried cumulative value),
and the estimator refuses a window too short or too stale to carry the claim.
3. **Unconfigured is not broken.** A source with no env config answers
`404 {"configured": false}` **without dialling**, and the page says "not configured".
A red panel for a server the operator never installed destroys the signal from the
panels that matter.
4. **One dead source must not blink another — and nothing may stop polling in silence.** Each
source is its own asyncio task with its own timeout and health record (`poller.py`). A
failed poll keeps the last good payload and marks it stale in the UI instead of zeroing it —
on `/` too, where a host that has never answered still gets a fleet row
(`unreachable: true`, dimmed, every number `n/a`) because dropping the row would turn a dead
agent into "never configured". Transitions write events; repeated failure writes one line,
not 500. **The task is the source's only poller, so no exception may end it**: a store that
refuses an otherwise good poll is a *reported* failure (`ok: false`, the real reason in
`error`, `last_ok` frozen so the page ages the numbers, one `store-error` event then one
`recording` event on recovery), and anything that escapes the poll body is logged and the
loop continues. Four dev-server sources once froze behind an unretrieved SQLite exception
while `/api/health` went on calling them `ok` — a frozen source that looks healthy is the
one failure mode worse than an outage.
5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app;
charts are hand-rolled SVG (`src/dashboards/web/static/charts.js`). The engines live on a network with
no outbound access — a CDN reference is a request that never comes back.
6. **Never leak a credential.** Target URLs may carry a userinfo token (`parse_targets`
moves it into an `Authorization` header, so `Target.url` ends up clean). Nothing that leaves
the process may carry one: `/api/config` serves `Target.display_url` and
`config.strip_userinfo`-redacted engine URLs, and every URL inside an `UpstreamError` message
is redacted too — those strings go into the health record *and* the `event` table, which
`/api/events` serves to the browser and `page-system.js` prints. No token, password, or
personal hostname goes into tracked files, fixtures, or the browser. **The admin's own
credential is the same case** (phase 11): `auth.py` records nothing and answers
`False`/`None` to every rejection, because the *reason* a login was refused is a verifier, a
salt, or a session token — and this app turns reasons into health records and `event` rows
that a browser reads. Its one `logging` call and its two `raise` sites (the fail-closed
startup and the empty-new-password refusal) carry **fixed** operator text with nothing
interpolated, not even a prefix of a hash; `tests/test_auth.py` pins that structurally, so
the day someone adds `logger.warning(f"bad verifier {v}")` a test fails instead of a log line.
**A private key is the same case** (phase 15): the root CA key and the dashboard's client key
live only in `meta`, `certs.py` has no `logging` import and one fixed-text raise naming the rows
to delete (pinned structurally by `tests/test_certs.py`, which also asserts the refusal quotes
no line of the stored key), the root key is never written to the temp dir the client material
lives in, and a bundle carries the root *certificate* — never the key — because an agent only
has to verify its clients and a stolen agent install must not be a stolen CA.
**A session token used to fetch a bundle is the same case** (phase 24): the installer's
`--fetch-bundle <url> --cert-name <name> --session <token>` sends that token on exactly one
request, hands it to its `python3` child through the **environment** rather than argv, prints
only the URL in its log line, `unset`s it before anything is written, and never repeats it in
the sudoers hint's re-quoted command (which names `--bind`/`--port` and nothing else).
`tests/test_deploy_scripts.py` signs in for a real token and then greps every file the run
could have touched plus both output streams.
**A bundle exported for the dev loop is the same case** (phase 24):
`scripts/dev-certs-export.py` writes only into the gitignored `data/certs/` tree, creates each
file with `os.open` so `0600` is its *birth* mode (and re-chmods a file that already existed
looser), dials `GET /api/certs/<name>/bundle` over HTTP instead of importing `ensure_certs` — so
it cannot mint even by accident — and prints one line per host with no PEM, no token and no
password in it (`tests/test_deploy_scripts.py` greps the output for all three).
**A captured socket table is the same case** (phase 28): `scripts/capture-gateway-config.sh`
keeps every config byte (a `server_name` and its `proxy_pass` are the ingress map the page draws,
and a key *path* is a filename, not a key) and rewrites **only** the two address columns of
`/proc/net/tcp{,6}`, and only when `ipaddress` calls the address globally routable — a Linux
global IPv6 is EUI-64, so it encodes this box's NIC, and an inbound visitor's address was never
ours to publish. Documentation space (RFC 5737 / RFC 3849), first-seen order, same width, same
`::ffff:` spelling, idempotent; ports, states and every other column stay verbatim.
**A captured resolver answer is the same case too** (phase 32): what `getent` returns for
`reeseapps-immich.reeselink.com` on this router is as much an endpoint as a socket row is, so the
same substitution runs over `tests/fixtures/gateway_resolver.json`, and the name itself is kept
verbatim because the name is the thing the page draws. `tests/test_agent.py` fails the day either
fixture grows a routable endpoint back.
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
`sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
`scripts/capture-fixtures.sh` (and `scripts/capture-node-exporter.sh` when comparing
exporters, `scripts/capture-gateway-config.sh` for the agent's `haproxy`/`nginx`/`sockets`/
`resolver` ground truth) rather than trusting a README or a memory of an upstream.
8. **Keep dependencies current on purpose.** LLM-written version floors are usually stale.
Check after touching dependencies: `uv pip list --outdated` must come back empty.
Python is 3.14 (`requires-python >=3.14`, ruff `py314`, pyright `3.14`, Containerfile
`python3.14`) — do not lower these to make something else work.
9. **SQLite owns the history and must stay bounded.** Writes go through `store.py`; new
metrics need a rollup path and a retention story, or a year of charts becomes a disk
emergency. Put the DB on a volume in deployment.
10. **The dev server stays up while work happens** (`./scripts/dev-server.sh restart`,
http://127.0.0.1:8090), and UI changes are verified in a real browser with a screenshot
before they are called done. `data/logs/*.png` is where those land.
11. **The dashboard never starts open, and the stored password beats the environment**
(phase 11). The admin verifier and the session-signing key are durable rows in the SQLite
`meta` table (`auth.verifier`, `auth.session_secret`) and are the only place either secret
lives; `DASH_ADMIN_PASSWORD` is a **seed** that hashes into the row on the first boot of a
database and is ignored thereafter (one warning line, value never quoted), so a password is
changeable without a redeploy and no container env can roll it back. A server with neither a
stored verifier nor the seed **refuses to start** — a `RuntimeError` out of the lifespan, a
non-zero exit, and a message naming the variable, the meta key, and the
`sqlite3 <db> "DELETE FROM meta WHERE key IN ('auth.verifier','auth.session_secret')"`
reset command. "Start open with a warning" is not an option: the domain is public and the
fleet's hostnames and UPS controls are behind that login. `testpass` is the local-testing
password (dev config and the hermetic suite) and never a production fallback. Invariant 12 is
the request-time half; this one is why there is always a credential to check against.
12. **The gate is default-deny and its public surface is four entries** (phase 11). `app.py`'s
`PUBLIC_PATHS` (`/login`, `/api/auth/login`, `/api/health`) plus `PUBLIC_PREFIXES`
(`/static/`) are the only things an anonymous caller can reach — every page, every `/api`
data route, and `/api/auth/status`/`logout` require a session read from the
`dash_session` cookie or an `Authorization: Bearer` token. `/api/health` is the one data
route that stays open on purpose (the Containerfile `HEALTHCHECK` and
`./scripts/dev-server.sh status` poll it unsigned, and it leaks nothing); its shape is now a
contract with deployment, which means **nothing may be added to it without re-deciding that**
— a new field there is a field the whole internet reads. Adding a path to that list is a
security decision, and
`tests/test_auth_api.py::test_every_route_that_is_not_public_requires_a_session` walks the
**served** route table so a route that forgets the gate fails a test instead of shipping —
the middleware guards by default, which is why no new route has to remember to. Rules the
gate keeps and each has a test: a page `GET` gets `303` to `/login?next=…` (query string
included) and everything else the one JSON refusal, because a redirect a script cannot render
is not an answer; every answer carries `no-store`; `?next=` passes `auth.safe_next` and is
never followed raw (an open redirect on a login page is a phishing kit with a homelab theme);
a write additionally needs a matching `Origin`/`Sec-Fetch-Site` and — when it came in on the
**cookie** — this session's `X-CSRF-Token`, because a bearer caller is something no other site
can forge. A process whose lifespan never produced `ctx.auth` answers `503` for **everything,
public paths included**: "healthy" from a server that cannot say who its admin is is the
invariant-4 failure mode, and a login form that cannot verify a password is a worse promise
than no form. `POST /api/ups/command` keeps all four of its own gates behind this one (the
`DANGER` word is still what cuts power), and `POST /api/auth/logout` is gated like any other
write because a sign-you-out link is a CSRF. `POST /api/auth/password` (task 04) is the third
gated write and the app's only revocation: guesses at the current password spend their own
`password:<address>` bucket, and a right one replaces `auth.verifier` **and**
`auth.session_secret` so every browser that is not in front of the form dies on its next
request — which is why the response is minted from the new state and never the one being
replaced.
## Architecture
```
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD
│ │ + /slots (gufo) │
▼ ▼ ▼
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command ──┐
│ parse + Δcounter/Δt │
▼ ▼
sources/host.py · disks.py · llm.py · nut.py poller.py (task per source)
│ Result(payload, samples, events) │
▼ ▼
store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ──► api.py ──► web/
```
Responsibilities, one line each:
- `agent/host_agent.py` — stdlib-only (3.10+), reads `/proc`, `/sys/block`, `/sys/class/drm`,
`statvfs` — plus, for the proxy add-on, the haproxy and nginx configs and `/proc/net/tcp{,6}`
on the host that has them; serves `GET /snapshot` + `GET /info` + `GET /schema` + `GET /health`, plus an
HTML status page at `GET /`. Refuses to bind off-loopback without a token **or** a certificate
bundle (exit 2). Counters stay counters: the dashboard computes
rates so the agent never remembers who polled it last.
**TLS mode (phase 15 task 02, `AGENT_VERSION` 1.1.0):** `--cert-bundle`/`AGENT_CERT_BUNDLE`
loads the one PEM the dashboard signs (`resolve_bundle`, the `resolve_token` idiom: server cert
+ server key + root CA + `# BUNDLE-REV <n>` + `# REVOKED <serial>` lines, grepped by text
because the agent has no `cryptography`), unpacks it into the unit's `RuntimeDirectory` when
there is one else a temp dir, builds a `PROTOCOL_TLS_SERVER` context with
`verify_mode = CERT_OPTIONAL`, and **deletes the PEMs once loaded** (the key then lives only in
this process, so a `kill -9` has no cleanup to forget). `CERT_OPTIONAL` is the assumption that
keeps `/health` probeable: the *handler* enforces, refusing a peer with no verified certificate
403 `client certificate required` on every path but `/health`, and refusing a serial on the
bundle's denylist 403 `client certificate revoked` on **every** path including `/health` (the
exemption is for a probe holding the CA and no certificate, not for an identity the operator
withdrew). The denylist is re-read when the bundle's `(mtime_ns, size)` stamp changes — that is
the whole reason the `# REVOKED` lines live in the bundle — while a new certificate needs a
restart, because `ssl` cannot swap a chain under a live socket. Serial matching accepts the hex
`getpeercert()` renders *and* the decimal `certs.py` writes, because comparing one spelling to
the other accepts a revoked client forever. On the TLS path the bearer token is **not** read (the
handshake is the auth, and a second credential that outlives a revocation defeats the list),
`/snapshot` gains additive `tls: true` + `bundle_rev` only in TLS mode, and token mode is
byte-identical to before. Task 03 has since flipped the collector (see the `collectors.py`
bullet), so those two fields are how the dashboard answers "is this host running the bundle
`/certs` issued?" — `sources/host.py` and `sources/disks.py` carry them into both payloads as
`agent.tls` / `agent.bundle_rev`, present only when the agent published them.
SMART (phase 09): `smart()` runs `smartctl -i -A -H` per whole `/sys/block` disk
(`parse_smart_output` covers ATA + NVMe; NVMe media errors map to
`offline_uncorrectable`, hex-packed vendor raws are never decoded, and an attribute row is
read from its **RAW_VALUE column** rather than the end of the line because this binary
annotates that column — `35 (Min/Max 34/44)`, `43894h+43m+32.519s` — so its last token is
not the value), 5 s per device / a pass budget whose 20 s floor scales to 5 s × the disk count
/ 240 s cache, attached as `disk_io.devices[].smart` (gap 10: a 16-drive shelf takes longer
than the floor, and truncating the pass would null the tail drives). Missing binary →
**one** `errors` line for the pass (the word `not found` in it is what
`sources/disks.py` reads to answer "is smartctl installed", and
`tests/test_agent.py` pins that the line still says it); one denied/wedged device → that
disk's `smart` is `null` with its own line; `--no-smart` opts out silently. Never fails
the snapshot. **An expired cache refreshes on a background thread while the request is
served the previous result** — only the first pass after startup runs inline — because a wedged
pass on the request path blows `HOST_TIMEOUT_SECONDS` and would turn the
*system* source red every four minutes over an add-on that page never draws. The refresher
is single-flight and keeps the old cache on failure, so a bad pass retries on the next
expiry instead of hammering the disk.
**The proxy-config add-on (phase 28 task 01)** is the `haproxy` section: `Collector.haproxy()`
reads `--haproxy-config` (default `/etc/haproxy/haproxy.cfg`, resolved under `--root` so the
fake-tree tests read the fake tree, one size-bounded read per snapshot) and
`parse_haproxy_config()` turns its text into `{frontends: [{name, binds: [{addr, port}],
default_backend, backends}], backends: [{name, servers: [{name, target, port}]}]}`. Three rules
keep it honest. (a) `available` is `smartctl_available`'s three-way answer — `true` read and
parsed, `false` no file at that path, `null` a file that could not be read or holds no haproxy
section — and each non-`true` arm costs exactly **one** `haproxy:` line for the pass, the missing
one saying **not found** (pinned wording, the same cross-module contract as smartctl's, because
the dashboard reads that word to answer "is haproxy installed?"). (b) The section reports the
config's own text and infers nothing about a request: a bind keeps the address as written
(`:::22` splits at the **last** colon into `"::"` and `22`, so an IPv6 bind survives, and
`bind :8080` keeps `addr: null` because `*` is a spelling the config did not use), a `server`
with no port keeps `port: null` and never `0`, a `use_backend` whose target is a fetch
expression (`%[req.hdr(host),map_dom(...)]`) contributes **no name** rather than an expression
where a backend belongs, and everything outside `bind`/`default_backend`/`use_backend`/`server`
is ignored without a word. The frontend→backend routing is therefore the names, which the
dashboard joins. (c) A `listen` section is reported as the frontend+backend pair haproxy itself
defines it as, so a config written in that shorthand cannot read as "this gateway routes
nothing". The section is absent from the document with `--no-proxy` (silence, the `--no-smart`
precedent — "not collected" stays distinguishable from "collected and empty"), it is additive in
the payload, and nothing in it can fail `/snapshot` or `/health`.
`tests/fixtures/gateway_haproxy.cfg` is the verbatim capture the parser is pinned to.
**The same add-on's `nginx` section (phase 28 task 02)** reads `--nginx-config` (default
`/etc/nginx/nginx.conf`) **plus the files its own `include` lines name, one level deep** — the
gateway keeps its vhosts in `conf.d/*.conf` and its SNI routing in `stream.d/*.conf`, so the
main config alone is not the picture. `parse_nginx_config(main_text, included)` is the pure
half over text, and it needs a real scanner rather than haproxy's indent loop because nginx
nests by `{}`: `_nginx_tokens` keeps a `;` inside a quoted value as data (the real config writes
`add_header … "max-age=31536000; includeSubDomains" always;`), `_nginx_statements` builds the
block tree and raises only for a file nginx itself would refuse to start on, and the walk fills
`server` blocks (`listen`, `server_name`, `proxy_pass`), `upstream` blocks and `map` blocks.
Same three rules as `haproxy`: `available` is three-way with exactly one `nginx:` line per arm
(the missing one saying **not found**, the pinned wording the dashboard reads), `--no-proxy`
removes the key in silence, and nothing in the section can fail `/snapshot` or `/health` —
including a config that will not tokenize, which is `available: null` plus its one line with
every other section intact. What it refuses to do is interpret: `listen` keeps the address as
written (`[::]:8443` → `"::"`, a bare `80` → `addr: null`) and ignores the `ssl`/`proxy_protocol`
flags, `server_name _` stays the literal `_`, a map's value becomes a `target` only when it is
an address (the `$connection_upgrade` map's `upgrade`/`close` are header words, and calling one
a host is inventing a fact to fill a field), and a `proxy_pass $var` travels with `variable`
(the `$` dropped once here, so both sides of the join spell it identically) and a **null
`target`** — the map decides per connection, so a merged target would be rule 2 with a page's
clothes on. Include-following is one level and bounded (8 files / 1 MB of included text, the
smart pass-budget shape): the excess is dropped with **one** line naming the count and the
section stays `available: true`; `files` is the main config then the documents actually merged,
spelled the way the config spells them (`/etc/nginx/conf.d/http.conf`, never the path under
`--root`, which is also what lets the parser match its own `include` lines); a literal include
naming a file this host does not have costs nothing and no line; and an `include` inside an
included file is not followed at all — `root=False` makes that structural, not a convention.
**The same add-on's `sockets` section (phase 28 task 03)** is the live half: `read_proc_table`
reads `/proc/net/tcp` and `/proc/net/tcp6` (resolved under `--root`, the config rule) through at
most `PROC_NET_MAX_LINES` = **4096 lines per table** — `/proc`'s own header among them, so 4095
sockets — because a gateway's file is one line per socket and the read must not cost the poll
somebody else's connection count; the excess is reported in **one** line naming the bound and the
section stays `available: true`. `parse_proc_net_tcp(text, family)` keeps two kernel states only:
`0A` LISTEN → `{family, addr, port}` and `01` ESTABLISHED →
`{family, local: {addr, port}, remote: {addr, port}}`; every other state is a socket in transit
(`TIME_WAIT` above all) and is skipped in silence, since drawing it would put two pipes on one
connection and bury the live pairs. `_proc_net_addr` unrolls `/proc`'s per-32-bit-word
little-endian spelling through `socket.inet_ntop`, so a wildcard keeps its literal `0.0.0.0` /
`::` — the value the inbound join compares against — and a word that is not an address is `addr:
null` with the port kept, never a guess and never `0`. The two tables are read **independently**:
`available` is `true` when either answered (a kernel built without IPv6 has no `tcp6`, which is
absence, so no line — the nginx include rule), `null` with one combined `sockets:` line when
neither did, and a table that is there and denied costs one line even beside a good one. This
section has **no `false` arm on purpose** — `/proc/net/tcp` is not something a host leaves
uninstalled, and `false` beside an empty list would read as "no listeners", which is rule 2 — and
**no join with the config**: a socket does not know its domain, so phase 29's page joins these
ports to the `haproxy`/`nginx` sections and an unclaimed socket stays visible as one.
**The resolver section (phase 32 task 01) is the fourth and the only one that leaves the host.**
`target_names(haproxy, nginx)` is the whole allow-list — haproxy `backends[].servers[].target`,
then nginx `upstream` members, `server`-block `proxy_pass` targets and `maps[].entries[].target`,
deduplicated keeping first-seen order — and nothing else may ever enter it: not a name from
`/proc/net/tcp`, not a client address, not anything a request supplied. That is decision 3 and
`tests/test_agent.py` pins it by asserting the spied `getaddrinfo` call set **equals** this
function's output minus the address literals, in config order, with no duplicates, and that no
address from the recorded socket table ever appears in it. The lookup is
`socket.getaddrinfo(name, port, proto=SOCK_STREAM)` — `via: "libc"`, which is the resolution
haproxy's `init-addr last,libc,none` performs **on the box that dials**; the dashboard resolving
the same name through its own `resolv.conf` would be a different vantage answering a different
question, so `sources/gateway.py`'s resolver spy pin survives this phase on purpose and the
agent does **not** hand-roll UDP queries to the config's own `resolver` server either. That line
is published instead, verbatim, as `config_directive` (`"10.1.0.1"` on this fleet) — and it lives
inside this section rather than inside `nginx` so the `nginx` document stays **byte-stable**, which
keeps phase 32's additivity control a strict top-level key comparison with no exception list
(`Collector.nginx_document()` is the seam that returns the pair; `Collector.nginx()` is its
one-line wrapper). Entry shape is `{name, port, status: ok|nxdomain|error|literal, addresses,
ts}`: `addresses` is the unique answer sorted v4-ascending then v6-ascending (libc's order is not
stable and `entries` is a list a page renders), an address literal is `literal` with **no** lookup
(it is already an address), and `nxdomain`/`error` carry `addresses: []` — never a stale guess,
never a `0`, and `EAI_AGAIN` is `error` and not `nxdomain`, because "the server did not answer"
and "the name has no record" are two fixes. The pass is SMART's idiom for SMART's reason
(`_resolver_lock`/`_resolver_cache`/`_resolver_refreshing`, `RESOLVE_MAX_NAMES` 64 /
`RESOLVE_CACHE_SECONDS` 300 / `RESOLVE_PASS_BUDGET_SECONDS` 3.0): the **first** pass inline, every
later expiry refreshed on a single-flight daemon thread while the request is served the previous
result with its **original per-entry `ts`**, a failed refresh keeping the old entries and
re-stamping so the retry lands on the next expiry rather than on every poll, and the budget/name-
bound excess each reported in **one** line (`resolver: pass budget 3.0s exhausted, 41 of 58
targets looked up`) with `available` still `true` — a black-holing resolver must never blow
`HOST_TIMEOUT_SECONDS` and blink `/system` over an add-on it never draws. `available` is the
family's three-way answer (`false` = nothing to resolve because no proxy config was read, one
line saying **not found** or **unreadable** per section; `null` = a pass that could not run,
which `snapshot()`'s `_safe` turns into the empty section plus its line), and **`--no-resolve`
removes the key in silence** the way `--no-smart` does — while `--no-proxy` removes it too, since
a box that collects no proxy has no declared names.
**Task 04 made the three sections a versioned contract with captured ground truth:**
`AGENT_VERSION` is **1.3.0** (`tests/test_agent.py` pins the constant and asserts the
`GET /schema` block names `haproxy`/`nginx`/`sockets` — each with its `available` flag — plus the
`resolver` section, its own `available`, and `--no-resolve`, beside `--no-proxy`,
`--haproxy-config` and `--nginx-config`), and `scripts/capture-gateway-config.sh`
re-records the seven fixtures the parsers and the join are pinned to over `GATEWAY_SSH`
(`gateway-ops`): the haproxy config, the nginx main config **plus every document its own glob
`include` names** (so the fixture tree cannot drift from the config's tree while the parser still
follows one level), `/proc/net/tcp{,6}`, and `gateway_resolver.json` — the resolver section
itself, captured the way the agent will produce it (task 02, below). Three rules it exists to
keep. (a) **No privilege and no copy tool**: one `ssh <alias> cat <path>` per document and one
`getent` per name per family, because `/proc/net/tcp` reports size 0 and `scp` would copy an
empty file, and every path and binary it uses is world-readable and world-runnable — the same
reads the agent performs as its own user, so a capture that needed sudo would be recording
something the agent cannot see.
(b) **A bad read is never a fixture**: a nonzero `cat`/`ls`, an empty body or a missing glob include
exits non-zero having written nothing, and a half-written file is removed — the pin is that the
recorded bytes survive a dead alias and an empty read both. (c) **Configs verbatim, socket tables
identifier-free**: the configs keep every `server_name`, `proxy_pass` and key path (a domain is the
map the page draws and a key path is a filename, not a key, while an agent-only bundle carries no
key at all), but a globally routable address in a socket table becomes a documentation one (RFC
5737 `203.0.113.0/24`, RFC 3849 `2001:db8::/32`) through an idempotent column substitution — ports,
states, every other column and every space are verbatim, loopback/private/link-local/wildcard
addresses are kept, and the IPv4-mapped `::ffff:` spelling survives — because a global IPv6 on
Linux is EUI-64 (derived from the NIC MAC) and an inbound visitor's address is not this repo to
publish: the same rule that scrubs a drive's serial (invariant 6). Those two tables are volatile by
nature, so `tests/test_agent.py` asserts their **shape**, the ports the recorded configs claim, and
the absence of a globally routable endpoint — never a particular row, which a re-run may lawfully
change.
**The seventh fixture is the resolver section, and it is derived rather than listed (phase 32
task 02).** `tests/fixtures/gateway_resolver.json` is the agent's own `resolver` document — same
keys, `via: "libc"`, entries in `target_names()` order — so a source test can exercise the
name→address join hermetically instead of wishing for DNS. Three rules keep it honest. (a) **The
name list is the parser's, never the script's**: a `plan` step rebuilds the agent's `--root` tree
from the config fixtures in this repo, imports `agent.host_agent`, and asks `target_names()` +
`address_family()` what to look up — so a route added to the gateway moves the fixture with it, an
address literal costs no remote read, and `tests/test_deploy_scripts.py` fails the day a domain is
hardcoded into the script. (b) **`getent ahostsv4` and `ahostsv6`, one name at a time**, because
that is libc on the box that dials; `rc 2` is "no record" and becomes `status: "nxdomain"` — a
fact, kept beside its successful neighbours — while `rc 255` (the ssh died), `rc 127` (no `getent`)
or **not one name answering** is a broken vantage, so the run exits 1 having left the recorded
document byte-identical: a fixture of nothing but `nxdomain` would make every live route on
`/gateway` a permanent dash and every join test a test of nothing. The `ts` is a constant and the
answer set is deduplicated and sorted with the agent's own `sort_resolver_addresses`, so
re-capturing a gateway nobody changed is a no-op. (c) **A resolver answer is an identifier**, so the
socket scrub runs over it too (`8.8.8.8` → `203.0.113.1`), and the `::ffff:` spelling `getent
ahostsv6` synthesises for a name with no AAAA record is dropped when the A record answered — the
agent's one `getaddrinfo` never publishes it. This fleet's targets all answer private LAN addresses,
which is why the recorded document is scrub-free; `tests/test_agent.py` pins that absence as a
warning sign, because a documentation-space address there would not necessarily match the one the
socket fixtures gave the same endpoint, and `tests/test_gateway_source.py`'s join depends on their
agreeing. The version's additivity is pinned two ways: permanently, by
`SNAPSHOT_KEYS_BEFORE_1_3_0` (1.2.0's top-level keys) in `tests/test_agent.py`, whose only legal
successor is that set plus `resolver` — and value-for-value, by `data/logs/verify-32-additive.py`,
which imports `HEAD`'s agent file beside the working tree's and calls both against one `--root`.
One defect only the second kind of control can find: the lookup is
`getaddrinfo(name, port, type=SOCK_STREAM)`, and `proto=SOCK_STREAM` — the spelling that reads as
"the TCP addresses" to anyone who has not read the man page — is `EAI_SERVICE` on real libc for
**every** name, because `proto` takes a protocol number (`IPPROTO_TCP` = 6) and `SOCK_STREAM` is 1.
A stubbed `getaddrinfo` accepts any keyword, so the whole resolver suite stayed green while the
agent reported `error` on a healthy gateway; `test_the_lookup_the_agent_performs_is_one_real_libc_accepts`
removes the module-wide stub and asks real libc about `localhost`, the one name it answers from
`/etc/hosts` with no network. What phase 28 stopped at was the wire: `sources/host.py` still passes **none** of the
three into `/api/system/snapshot`, and `store.py` still records no series for a config — the join
and the two gauges it publishes are phase 29's `sources/gateway.py`, which reads the same
`/snapshot` through its own `gateway` job. What the *system* page does read today is their
**error lines**, through `collectors.ADDON_ERROR_PREFIXES`, so a host with no proxy config stays
out of `/system`'s `degraded` (invariant 3).
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
env var always wins. Defaults are safe except the sources, which default to unset. The auth
block (phase 11) is `DASH_ADMIN_PASSWORD` (the **seed**, and `_env()` strips, so `VAR=` means
*unset* rather than "the password is empty"), `DASH_SESSION_HOURS` (720 = 30 days: a dashboard
that logs the operator out mid-outage gets bypassed), `DASH_COOKIE_SECURE` (off, because dev is
`http://127.0.0.1`; task 03 also treats `X-Forwarded-Proto: https` as secure), and
`DASH_PBKDF2_ITERATIONS` (default `auth.PBKDF2_ITERATIONS`, imported so the primitive and the
setting cannot drift) — that knob exists so the hermetic suite hashes in microseconds instead
of 0.3 s per client and is not a production tuning parameter. The gateway block (phase 29) is
`GATEWAY_HOST` (default `gateway`, and **a name, never a URL**: `gateway_enabled` is true only
when a `HOST_TARGETS` entry has it, so the page is unconfigured rather than red for a
deployment with no ingress — the `nut_enabled` shape, and the reason `any_source_enabled`
includes it), `GATEWAY_INTERVAL_SECONDS` (30 — a connection count is the shape of half a
minute, not of ten seconds) and `GATEWAY_TIMEOUT_SECONDS` (10, the plain agent fetch with
headroom for a handler that may be mid-smartctl pass for `/disks`).
- `poller.py` — job loop, health records, stale/last-good semantics, transition events.
`/disks` is a **source of its own** (`disks:<host>`, `DISKS_INTERVAL_SECONDS` 300 s,
`DISKS_TIMEOUT_SECONDS` 30 s, events under the dashboard name `disks`) reading the same
agents the `system` jobs read: the separate task is what keeps a SMART poll out of the
10-second utilisation loop and lets `/disks` age on its own instead of blinking `/system`.
`gateway` is a **job of its own** for the same reason (phase 29 task 02) — `GATEWAY_INTERVAL_SECONDS`
30 s, `GATEWAY_TIMEOUT_SECONDS` 10 s, health and events under the dashboard name `gateway` — but
**one** job rather than one per host (the `nut` shape), dialling the `HOST_TARGETS` entry
`GATEWAY_HOST` names with its `label` set to that host's name, because the page is about the single
ingress and a record per machine would ask every host a question only one of them can answer. It exists only
when that name is a target: an unconfigured source has no task that could look red
(invariant 3), and it is handed no `RateEstimator` because a count is not a rate (rule 2).
The job list is built once (`app.build_hub()`) because it is wired twice — production and
the injected-store test path — and two hand-written copies drifting is how a source ends up
polling in tests and not in the container. The
success path records (payload → samples → events) and only *then* moves `last_ok`, so a
refused SQLite write ages the source and alerts instead of leaving it current with charts
that have silently stopped drawing.
Events are stored under the **dashboard** name (`system`/`disks`/`gateway`/`llm`/`ups`), not the sample
source (`host`) — a page filtering `/api/events` by `host` gets an empty feed forever
(found on `/system` during phase 03 browser verification).
- `collectors.py` — the fetch half. `AgentTls` (phase 15 task 03) is what makes an `https://`
`HOST_TARGETS` entry a **certificate-authenticated** dial: `verify` and `cert` are the paths
`certs.write_material` wrote at startup, and the dial is verified against this app's own root
and presents this app's own client certificate. Two things about it are load-bearing. (a) The
credential travels in a **client built per dial**, because `httpx` resolves TLS material when a
client is *constructed* (there is no per-request `verify`/`cert`, and `verify=<str>`/`cert=` are
deprecated anyway — the supported form is an `ssl.SSLContext`, which is also the honest shape:
trust in one named CA, hostname checks on) while one shared client dials the plain agents and
both engines — a `verify` on *that* would make every engine read depend on a CA nobody issued
their cert with. (b) `status` is a **callable** doing a fresh `certs.agents` read, so a
revocation clicked in the browser bites on the next poll with no restart, and a revoked host is
refused **before any socket opens** — invariant 3's shape: a source the app knows it must not
talk to says so without a network round trip, and the poller's stale/last-good semantics do the
rest. `refusal_detail()` quotes the agent's own 403 sentence (`client certificate required` /
`client certificate revoked`) because those are two different operator actions, and the
certificate wording is used only for a dial where a certificate was presented — a 401/403 from a
token-mode host keeps the token/HTTP wording, or the operator is sent to `/certs` for a token
problem. `agent_tls_map()` builds one entry per configured host (the CA and client pair are
shared — the dashboard has one identity — only the revocation answer is per name, and an absent
entry is *active*, never-issued being a different fact from revoked); `build_hub` takes that map
and hands the **same** entry to the `host:`, `disks:` and `gateway` jobs of one host, so a
revocation fails every source reading that agent with one sentence and none of them dials
(`tests/test_api.py`'s revoked-gateway pin is the three-job version of the `host:`/`disks:` one).
`fetch_gateway_snapshot()` + `gateway_job()` (phase 29 task 02) are the third reader of one
`/snapshot`: the job is **one per deployment** (the `nut` shape, key `gateway`, `label` = the
`GATEWAY_HOST` name) on `GATEWAY_INTERVAL_SECONDS` with `GATEWAY_TIMEOUT_SECONDS`, it is handed
**no** `RateEstimator` (a connection count that goes through a Δ-window is a rate with a label and
no measurement behind it, rule 2), and `gateway_job` returns `None` when the name is not in
`HOST_TARGETS` — the `disks` configured gate, so an unconfigured source gets no task that could
look broken and `/api/health` says nothing about it. `fetch_snapshot()` keeps
`http://` targets on the shared client with their bearer token, unchanged from before the phase,
so a rollout is per host.
`NutMonitor` owns per-UPS transition bookkeeping (its
self-test event takes its severity from `nut.self_test_outcome()` and not its own word list,
so a passed test is never an amber line in the feed while the card beside it reads green).
Its per-UPS tolerance for a failed `LIST CMD`/`LIST RW` is for a *complete* error reply only:
one battery whose command list is denied loses its buttons and the poll goes on, while a
**stalled** reply fails the poll, because `NutClient` marks a socket it could not read to its
`END` as unusable (a reply still arriving would otherwise be taken for the *next* command's
answer — the real cause of the `expected 'BEGIN LIST VAR', got 'RW apc …'` lines this fleet's
event table used to fill with, and the same hazard on `INSTCMD`, where a late `OK` read as a
different command's success would be a gate that lies).
Failure messages name the **redacted** URL (`strip_userinfo` / `Target.display_url`), never the one
that was dialled, because they become health records and event rows. `fetch_snapshot()` is
the one agent `/snapshot` read that `system` and `disks` share, so a dead host says the
same sentence on both pages and only the `timeout` differs (the disks poll has to outlast
the agent's smartctl pass, and `None` is translated into httpx's sentinel because a
literal `None` means *no timeout at all*). Agent `errors`
become the payload's `degraded` **except** the add-on lines named by
`ADDON_ERROR_PREFIXES` (`smart`, `haproxy`, `nginx`, `sockets`, `resolver`) — a host without
smartmontools,
without device read access, without a proxy config, whose kernel socket table did not answer, or
whose resolver had nothing to resolve / stopped mid-pass is unconfigured, not broken (invariant
3), and `degraded` is `alerts.py`'s input, so a fleet-wide
agent upgrade would otherwise put an overview WARN on every box that never ran haproxy or nginx.
Those lines stay verbatim in `agent.errors` for the page that owns the add-on — `/disks` explains
the `smart:` ones (and since phase 22 a host whose agent said `smartctl` is *missing* is not
listed on that page at all, `api._disk_visible`), phase 29's gateway page reads the
`haproxy:`/`nginx:`/`sockets:`/`resolver:` ones the same way.
**A new agent add-on section adds its prefix here in the same commit**, or its absence-report
becomes a red banner on a page that has nothing to say about it.
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card
(`gpu.busy_pct@<index>`). `agent_tls_fields()` is the phase-15 passthrough: a TLS agent's
`tls`/`bundle_rev` ride into the payload's existing `agent` block, and a token-mode snapshot
grows nothing (no `tls: false`, no `bundle_rev: null` — rule 2), so the fields are how the API
distinguishes a stale bundle from an agent that never had one.
- `sources/disks.py` — the same snapshot projected onto **physical** drives for `/disks`
(phase 09): one row per whole-`/sys/block` disk (never a partition, never a mount), the
agent's `smart` block carried through verbatim, `health` the drive's own word normalised to
`ok`/`failed`/`None`, and `smartctl_available` three-way on purpose — `True` (a disk
answered), `False` (the agent said the binary is not installed), `None` (an agent predating
the pass, or `--no-smart`: nobody knows, so `n/a` plus the agent's own `smart_errors` line
and never a guessed verdict). Since phase 22 the flag's *page-side* consequence is not only
the why-line: `api._disk_visible` reads `False` as "not a host on `/disks` at all", while
`None` keeps its dimmed row — the computation here is unchanged and the agent still reports
all three arms whatever the API decides to list. **Nothing here
is a rate**: power-on hours and the sector counters are stored counters-as-reported, so the
source takes no `RateEstimator` and its payload carries no `window_seconds` (rule 2 — a rate
with no estimator, no `Δ<window>` label and no `source` string is an invented number).
`disk.health_ok@dev` is not a series either: a verdict is not a number, so the drive's word
travels in the payload only. Each row does carry a `severity` dict (`SEVERITY_FIELDS` = the
two rated numbers, `temp_c`/`percentage_used`, plus `health` through `HEALTH_SEVERITY`,
which translates the drive's own word instead of grading it), and `UNGRADED_FIELDS` names
the numbers that must never be graded. `fleet_disk_summary()` sums the per-host `summary`
counts for the page header and drops a host with no payload from the sums while still
counting it in `hosts_total` — the `/llm` fleet-block rule, so the page never does
arithmetic and a dead agent never shrinks a fleet total.
- `sources/gateway.py` — the same snapshot projected onto the **connection topology** of the one
host `GATEWAY_HOST` names (phase 29): `parse(snapshot, host)` joins agent 1.2.0's `haproxy`
frontends and `nginx` `listen`s into **`listeners`** (one per listen port, carrying the
`server_name` domains and the live ESTABLISHED count + distinct client addresses — the key was
`inbound` until phase 30 task 01 renamed it, because "a port a config names" is the thing the
page draws a trunk for, not a direction) and haproxy
and the config-declared routes into **`connections[]`**. **The join is two attributions and
nothing else** (phase 30 task 02 — the operator called the phase-29 version overengineered, and it
was). Inbound: a socket's **local** port names a configured listener, which is a fact (`/proc`
states it, no config reading needed). Outbound: a socket's remote `(address, port)` **equals a
target written as an address** in one of the two config files — the only outbound match that
claims nothing, because the target of a `server` line, a `proxy_pass` or an `upstream` member *is*
the endpoint the gateway opened. `join_sockets()` still partitions the table in **one** pass, so a
socket classified both ways would be counted twice — which is exactly what `127.0.0.1:8443` is:
the stream `map`'s `default` sends `:443` to that port and the box's own http vhosts listen on it,
so the loopback hop counts **once as one inbound and once as one outbound**, a pipe on each card,
neither claiming the other's number. Everything else is **ignored**: not counted, not totalled,
not warned about. What that retires is the guessing and the bookkeeping it existed to explain —
the port-only guess for a name target, `unmatched` / `unmatched.groups` / `partial` /
`unmatched_listeners`, and `by_port` with its `MAX_GROUP_HOSTS` chip. A **name** target
(`reeseapps-immich.reeselink.com:443`) is credited **only through an address the gateway itself
published** (phase 32, below): crediting it any other way needs DNS, which the module still never
calls (spy-pinned), and every name here resolves back through this same router, so a vantage of
ours would attribute a socket to a route on the strength of a query the router never made. Two
consequences the
page and the tests both depend on: **`live_out` is not the bandwidth** and the socket table never
reconciles to it — a pipe count means "this route has N live connections" and nothing more — and
several `(listen, endpoint)` lines reaching one endpoint carry one count **repeated** with
`shared: true` (never a split nobody measured) — a shape that is real now that a name route is
counted: two vhosts writing one `(address, port)`, two **names** the resolver placed on one
address, or a multi-homed name sharing one address with a single-homed one — which is why the
API's totals and the tests sum **distinct endpoints**. A pipe is `null` (never `0`) whenever the socket table did not answer or no
line reaches its endpoint, and publishes **no sample** then, so a chart cannot draw a zero for an
unknown. `supported: false`
(a 1.1.0 agent, or `--no-proxy`) keeps every key with empty lists and null counts plus
`WHY_PREDATES`, `available: false`/`null` per section becomes a `why` sentence and the agent's
own line in `section_errors`, and nothing here carries a `severity` (`net.*` precedent: a busy
proxy is the job) or a `window_seconds` (a count is not a rate). **`connections[]`** (phase 30
task 01) is the graph the pipe lists never stated: one row per route the two config files
actually declare, from `config_connections()`, each carrying its `in.port`, its `out`
`{host, port, name, kind, selectors, address}` (`kind` is `backend` / `server_name` / `pool` /
`map`, `selectors` the config's own word(s) — a haproxy backend name, an nginx `server_name`, a
stream `map` key, the literal `default` kept verbatim), its own `metric`, and `connections: null`
until task 02 re-points the socket join at it. Four derivations (haproxy `frontend → routes →
backend → server`, an nginx vhost's concrete `proxy_pass` per `listen` port of **its own block**
— which is why `nginx_server_blocks()` replaced the per-port `nginx_listen_ports()` — a
port-less `proxy_pass` naming an `upstream`, and a `proxy_pass $var` fanning out through the
`map` whose `variable` matches), one merge rule (identical `(source, in_port, host, port)` is one
line whose selectors union in first-seen order; **never** merged across `in_port`) and two silent
refusals (a `proxy_pass` with neither target nor variable, and a haproxy frontend that routes by
a fetch expression, which the agent reports as no name — that trunk is drawn alone, because "no
name" and "no routing directive" are one value in this payload). **A name-written endpoint is
joinable through the address the gateway published for it** (phase 32 task 03): `resolution_index()`
keys agent 1.3.0's `resolver.entries` by `(name, port)` — `{}` when the section is absent or its
`available` is not `true`, which is phase 30 byte-for-byte, and never an entry with no addresses —
and `apply_resolution()` stamps every `out` with `resolved` (the address list, **never `[]`**: an
entry with no addresses and no entry at all are one fact, so the page's dash arm is one `is null`
test), `via` (the agent's own `"libc"`, copied, `null` where nothing was resolved) and
`resolved_ts` (**that entry's** stamp, falling back to the section's, because a name refreshed five
minutes ago and one answered on this poll are two claims). All three stay `null` on an
`address: true` row: the config wrote that address, and copying the agent's `literal` echo of it
into `resolved` would report a `getaddrinfo` that never ran. With an address and a port the line
becomes countable, so **name-keyed gauges are back** — `gateway.out_conns@<domain>:<port>`, under
the config's own spelling so history continues under one name. That is the reverse of phase 30
task 02's retirement for exactly one reason: the retirement existed because the only figure such a
series could carry was the *port's*, and the figure here is the sockets to the address this domain's
own libc resolution produced **on the box that dials**. An unresolved name keeps `metric: null`,
`connections: null`, no sample and the dash the footer explains. `gateway.in_conns@<port>` is
unchanged, and the port **stays** in the outbound name, because this fleet's two haproxy Minecraft
backends share `minecraft.reeselink.com` on 25565 and 25566 and two gauges on one name would
overwrite each other in silence; lines sharing an endpoint write the same value to the same name
(they are one count, not two) and `samples()` de-duplicates by metric.
**The join key is a normalised address, never a spelling** (`_address_key`, `endpoint_of`,
`_endpoint_groups`): `ipaddress` canonicalises and folds `/proc`'s `::ffff:10.1.2.64` to the
`10.1.2.64` `getaddrinfo` returned — one kernel endpoint, one count, however many spellings or names
reach it — while a zero-padded `010.1.2.64` stays **no one's address** (decimal versus octal is a
guess, and a guess on the far end of a pipe is rule 2), so `out.resolved` keeps the agent's bytes and
only the key is canonical. `_endpoint_groups()` is a union-find over those keys (a name route and an
address route reaching one endpoint are one group; overlapping multi-homed sets merge; a route with
no endpoint is no group) and `distinct_endpoint_counts()` is what `api._endpoint_values` sums for
`live_out`, which is the only reason the head tally and the pipes cannot disagree. Matching stays
exact equality on `(address, port)` — no prefix, no range — and this module still resolves nothing:
`socket.getaddrinfo` is spy-pinned absent over a name-only synthetic, a resolver-full synthetic and
the captured `resolved` ingress, plus `samples()` (a gauge name is built from a host string and is
the other place a resolver could creep in). The payload's own `resolver` block is
`{available, via, entries}` with each entry **projected** to `name`/`port`/`status`/`addresses`/`ts`
— every entry including `nxdomain` and `error`, because the index filters *within* the published
list and a page that says *this name has no record* needs the entry that says so — and a section
that did not answer publishes none. `parse_gateway_snapshot()` is the
`Result` the poller consumes; there are **no events** (a connection count has no transition
worth a line — a proxy that stops routing shows up as its host's own poll failing). The
never-answered arm is `stub(host, ts)` — the **same skeleton** `parse` fills in, plus
`unreachable: true`, `supported: null` and `WHY_NO_ANSWER` — because the route that serves a host
with no payload yet must not hand the page a shape with keys missing, and `WHY_NO_ANSWER` is not
`WHY_PREDATES`: an agent that has not replied has not said it has no sections (invariant 3's two
facts staying two sentences). **`warnings()`** (phase 30 task 03) is the payload's other half: the
two config files judged against **themselves**, seven kinds and nothing else —
`haproxy_route_dangling` (a `default_backend`/`use_backend` name no `backend` declares, one row per
frontend→backend line, `subject` `"<frontend>-><backend>"`), `haproxy_backend_orphan` (a `backend`
no frontend routes to, its server count from the config), `haproxy_backend_empty` (a `backend` with
no `server` — deliberately *not* also an orphan: `orphan` needs a server to have nowhere to go),
`nginx_server_unlistened` (a `server` block with a `server_name` and no `listen`, in both its
proxying and non-proxying arms), `nginx_pool_undeclared` (a `proxy_pass` whose value has no port —
the shape task 01's `out.kind: "pool"` already draws no line for — naming no `upstream`; a block
with two such `listen`s is **one** warning, `subject` `"<pool>@<port>+<port>"`, because the repair
is one line), `nginx_variable_unmapped` (a `proxy_pass $var` no `map` defines — the `map` *is* this
fleet's router, so that is a broken route and not a style note) and `nginx_map_route_missing` (an
entry of a **referenced** `map` whose value is not an address). Every row is
`{id, kind, subject, message}` with `id = "<kind>:<subject>"`, `message` complete on its own because
`alerts.py` quotes `warnings[0]` verbatim, `kind` the vocabulary the page groups by and `subject`
the config's own name for the thing, never a paraphrase of the message. Three things it refuses to
do, each pinned from the real capture and not only from a fake section: it never **joins** a warning
to `connections[]` or `listeners[]` (a page indexes one row by its own `id`; a boolean beside a
hundred rows is a second source of truth about the same route), it never guesses a **name
target**'s address (the module still never calls `socket.getaddrinfo`, spy-pinned), and it stays
silent about the three shapes that are legal or unreadable rather than broken — a vhost with a
`listen` and no `proxy_pass` (a `:80` `return 301` is identical here), a frontend whose only routing
is a fetch expression (agent 1.2.0 reports *no name*, the same value "no routing directive" has, so
distinguishing them is an agent change and not a source-level guess), and an entry of a `map` no
`proxy_pass` names (the `$connection_upgrade` map's `upgrade`/`close` are header words, not missing
routes). Like the socket half, a section whose `available` is not `True` is **not judged** — a host
that cannot be read has not been found guilty of anything — so `warnings` is `[]` for an absent,
unreadable or pre-1.2.0 config and the page cannot tell "clean" from "unreadable" by that list
alone; it reads `sections` for that, the same pair `supported`/`why` already form. Order is
deterministic (haproxy kinds in config order, then nginx kinds) and rows are deduplicated by `id`,
because the page keys its DOM on it and the alert quotes the first.
- `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`,
`VLLM_HISTOGRAMS`), measured Δ rates **diffed per label set** (`rate_key()` keys the
estimator on the resolved name plus its labels; the cell is the Σ of the per-series
rates over the widest window one of them actually used — phase 13, so a joining
`{engine="1"}` series can never lend its carried counter to one interval's delta),
prefill speed as `mean(prompt tokens)/mean(prefill time)`, session list from gufo
`/slots`. A refused window (the estimator's floor) is still an answer: the cell keeps
its `computed:Δ<series>/Δt` source and its honest `Δ<window>` beside a `null` value.
- `sources/nut.py` — upsd client, `UpsReading`, `VAR_MAP`, `assess()` (explicit checks, not
a mystery score), `SELF_TEST_OUTCOMES` + `self_test_outcome()` (the `Last self-test` verdict
follows the *outcome* of the driver's `*.test.result` word, not its phrasing: pass words →
`ok` (including `Done and passed`, what the CyberPower here answers), fail words → `fail`,
`No test initiated` / empty / no variable → `unknown` printing `NEVER_TESTED_DETAIL` so both
"never tested" states say one sentence, and an unrecognised word → `warn`, because a driver
inventing a verdict is drift worth noticing and the check prints its own word to be added),
`COMMANDS` catalog with risk tiers, `COMMAND_ALIASES` + `resolve_command()`
(drivers answer two names for the same action — `beeper.enable`/`beeper.on`,
`beeper.disable`/`beeper.off` — so one entry is offered per action and any of its names hits
that entry's verdict, never a gate the rendered name missed), `available_commands()` filtering,
kelvin-vs-celsius normalisation, and `charger_status` normalised to the `CHARGER_STATUS`
keys (the same lowercase token `states()` and `on_battery` already compare), and `states()`
publishes one chip per fact — a driver exporting both `battery.charger.status` and the
legacy flag that says the same word gets "charging, online", not "charging, online,
charging". `battery_runtime`
comes from `battery.runtime` **only** — `battery.runtime.low` is a shutdown threshold, not
time left, so it lives at `battery_runtime_low` and never becomes the hero's "runtime left".
- `sources/nut_sim.py` — a **labelled** simulator so the UPS page is developable without a
battery. It must never be the default, and every payload it produces says `simulated`.
Its modelled `battery.mfr.date` is computed relative to today (not a fixed string that
ages), so a fresh simulator reads healthy and amber keeps meaning something. Its status
follows the real drivers: a below-float pack on mains reports `OL CHRG`/`chg` (the glyph's
bolt state is keyed on that token, and the model has to be able to produce it) and a battery
being drawn from is `OB`/`dchg`, never `ol`. Every command it advertises is implemented and
visible in a variable a real driver publishes — the beeper ones set `ups.beeper.status`
(`enabled`/`muted`/`disabled`), and a test window that ends writes `ups.test.result: PASSED`
while an aborted one writes nothing; the load/shutdown commands answer `OK` without modelling
an outlet, because a faked power cut in the history is indistinguishable from a real one.
`from_env()` is the single place the `NUT_SIMULATE` truthy rule lives — `app.py` builds the
monitor through it, so the assembly and the model cannot disagree about what is enabled.
- `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own).
`scalar(agg="sum")` stays right for **gauges**; `values_by_series()` is the per-label-set
view **counter** deltas need, because a summed counter mixes whatever label sets exist now.
- `rates.py` — Δ/window estimator, keyed by resolved series name + label set. Its effective
window is `now - base[0]` **bounded on both sides** (phase 13): under
`MIN_WINDOW_SECONDS` (the poll loop's 0.2 s catch-up gap after any slow fetch) it answers
`(None, dt)` — one `n/a` with the window it refused — and drops that base so the polls
*after* the stall cannot re-diff it; the window-empty fallback accepts a base no older
than `MAX_WINDOW_RATIO × span` (the `2 × span` ceiling the docstring always claimed, now
enforced). A rate ≥ 8× the previously published one is held for one
poll (the previous value repeats, label unchanged) and publishes only if the next interval
confirms it (`JUMP_CONFIRM_RATIO`) — and **confirmation means a different anchor**: a held
jump remembers the sample its window was diffed from, and stays held while that same sample
still anchors the window. A stalled fetch stamps its body with the time the poll *finished*,
so that sample is the stall older than its own timestamp, and every later window off it is
the same skewed delta re-diffed over a longer span — measured twice from one sample is one
measurement, and the hold simply continues until the stale anchor ages out of the span.
`min_window=0.0` exists for tests that scrape faster than any real loop. **The span belongs to
the cadence that fills it** (phase 23): `app.build_hub` builds one estimator per measuring
cadence — the host's at `RATE_WINDOW_SECONDS` (10), the LLM's at `LLM_INTERVAL_SECONDS` (2) —
because a window wider than the poll averages bursts across the polls the operator is watching
and a window narrower than it has nothing inside it to diff. Nothing else about the estimator
moved: the floor, the ceiling, the jump gate and the label are the same code, and a 2 s window
is 4× above the floor, so a healthy 2 s cadence is never refused.
- `auth.py` — every authentication decision as a pure function, stdlib only (phase 11):
`hash_password`/`verify_password`/`needs_rehash` over a self-describing
`pbkdf2_sha256$<iterations>$<salt_hex>$<hash_hex>` verifier (so the cost can rise without a
migration), `mint_session`/`read_session` for an HMAC-signed stateless cookie (no session
table and nothing to prune; rotating the stored secret signs every cookie ever issued out,
which is how changing the password evicts every other browser), `mint_csrf`/`verify_csrf`
bound to the session token instead of stored, `LoginThrottle` (in-process exponential
backoff), and `origin_matches`/`sec_fetch_site_allowed` as string functions so the gate is
testable without a `Request`. The primitives **never raise and record nothing**: a malformed
verifier, a tampered cookie, and a rotated secret all answer `False`/`None`, because the
reason an input was rejected is the credential (invariant 6). No HTTP, no SQLite driver, no
new dependency — `app.py`'s middleware and `api.py`'s `/api/auth/*` are the callers, so a
decision made here is made once and is unit-testable without a `Request`. It also carries the
two strings the gate and the pages share (`LOGIN_PATH`, `LOGIN_NEXT`) and `safe_next()`, the
only judge of a `?next=` value: kept verbatim (query string and all, because the range chips
live there) when it stays on this app, dropped otherwise — scheme-prefixed, scheme-relative,
the percent-encoded spellings of both, anything with a control character (not a path but a
second `Location` header), and anything without a leading `/`.
The **verifier bootstrap** (task 02) is the one deliberate exception to "never raises":
`load_auth_state(store, admin_password=…, iterations=…, session_hours=…, db_path=…)` reads the
`meta` rows `auth.verifier` / `auth.session_secret` through a three-method `MetaStore`
`Protocol` (so this module still imports nothing in the package, and its tests use a dict)
and returns a frozen `AuthState` — the environment **seeds** a password once and the stored
verifier wins thereafter (`env_ignored` and one fixed-line warning when both are present, so
`/settings` is the real answer and no redeploy can roll a changed password back); no row and
no seed raises a `RuntimeError` naming `DASH_ADMIN_PASSWORD`, the meta key, and the
`sqlite3 … DELETE FROM meta …` reset command, because "start open with a warning" is the
failure this phase exists to prevent; a stored secret that is not 256 bits of hex is
regenerated and persisted (the effect is one login prompt, never a lockout). Nothing here
reads a clock, so the state carries no expiry — a session's is inside its signed payload.
`update_verifier(store, state, new_password)` writes the new verifier **and** a fresh secret
(secret first, so a crash cannot leave sessions outliving the password they were issued
under) and refuses an empty/whitespace password; it is what `POST /api/auth/password` calls,
because rotating the secret **is** this app's revocation list — a stateless cookie has no row to
delete, so without it a stolen session would outlive the password that "revoked" it for another
`DASH_SESSION_HOURS`. Its two raises, like the loader's, carry
fixed text — `tests/test_auth.py` pins the file's raise sites and its one logging call
structurally, so no future edit can interpolate a credential into either.
- `certs.py` — the fleet's certificate authority as the same shape of module (phase 15): a pure
lifecycle over the three-method `MetaStore` protocol, no clock it cannot be handed, no `logging`,
and one fixed-text `RuntimeError` (`tests/test_certs.py` pins all three structurally, the way
`tests/test_auth.py` pins `auth.py`). Five `meta` rows and nothing else:
`certs.root.key`/`certs.root.cert` (self-signed `CN=dashboards-root`, 10 years),
`certs.client.key`/`certs.client.cert` (`CN=dashboards`, 1 year — the identity every agent will
check), and `certs.agents`, one JSON object mapping a `HOST_TARGETS` name to
`{serial, not_after, status, bundle_rev, bundle, san}`. **A stored row always wins:** the root is
generated on the first boot of a database and never on a later one, because a CA that rotates
with the process untrusts every bundle already installed on a host — and a newly generated root
takes its client certificate with it, since a leaf signed by the dead root is a credential
nothing can verify. `issue_agent_cert`/`reissue_agent_cert` (the API's "rotate") sign a fresh
`CN=<host>` leaf whose SAN is the addresses that host is dialled by — the name is always in the
list, and an IP goes in as an `iPAddress` or hostname verification of an IP dial fails — bump
`bundle_rev`, and reset `status` to `active`. `revoke` marks the host and **re-seals every
bundle**, because the `# REVOKED` denylist lives inside each of them and an agent that cannot see
a serial keeps accepting that client; there is no "unrevoke", and revoking a never-issued host
still records the revocation (what a revoke enforces is the dashboard refusing to dial). The
**bundle** is the install artifact — agent cert, agent key, the root *cert* (never the root key,
asserted by header count and by parsing the one key against both keypairs), `# BUNDLE-REV <n>`,
then the denylist — stored verbatim so what an operator downloads is byte-identical to what the
row holds. `rotate_root` mints a new root plus a new client and reissues every agent
(`bundle_rev` +1 each, denylist reset with the generation); the running process keeps the
material it loaded at startup, which is the honest `restart_required` task 04 answers with.
Every certificate carries `SubjectKeyIdentifier` and every leaf an `AuthorityKeyIdentifier`
naming it, non-critical: OpenSSL's **strict** mode refuses a leaf with no AKI
(`CERTIFICATE_VERIFY_FAILED: Missing Authority Key Identifier`) and Python's
`ssl.create_default_context()` — the context the dashboard's own https dial builds — turns that
mode on. The agent's server-side check is not strict, so nothing on the agent side notices;
`tests/test_certs.py` pins the identifiers instead, and a bundle recorded without them is one a
strict client rejects. `write_material` puts the CA and the client pair in a `0700` temp dir with
`0600` files for the dialing — the root key is never written to disk, so its only copy stays the
`meta` row — and the lifespan removes the directory on shutdown. Unreadable PEM, a key that does
not match its certificate, or a leaf the stored root did not sign is a **fail-closed startup** (the
`ctx.auth` rule), and the message is the `sqlite3 "<db>" "DELETE FROM meta WHERE key LIKE
'certs.%'"` reset. `cryptography` is this module's dependency and the app's only new one;
`agent/host_agent.py` stays stdlib and loads the PEMs with `ssl`. There are **two reads** and the
difference is a side effect: `ensure_certs` is the startup one (it mints what the table is missing,
which is why it is what `app.build_certs` calls and nothing else may), and `read_state` is the
request-time one task 04's `GET /api/certs` uses — same validation, same fixed-text refusal, never
a write, so a page refresh cannot bootstrap an authority the operator deleted. The material is
read with (task 03): `collectors.agent_tls_map` turns `AppContext.certs` into the per-host dial
pair, and because the revocation answer is `agent_status()` called per poll, a revoke in this
module is what stops the dial — no restart, no cache, no second list to keep in sync. There is a
**third door and it is deliberately outside this module** (phase 24):
`scripts/dev-certs-export.py` is the dev loop's exporter — stdlib `urllib`, signs in through
`POST /api/auth/login`, reads the token out of the `Set-Cookie`, and downloads each
`HOST_TARGETS` name's bundle into gitignored `data/certs/` at `0600`. It cannot mint because it
imports neither read; it dials the route the `/certs` page downloads from, which is what makes the
exported file byte-identical to the row. `scripts/dev-server.sh` calls it after the readiness
probe and treats a failure as a warning line — the server being up is the thing the probe proved.
- `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric,
`classify()` → `ok`/`warn`/`fail`/`None`, plus `attach()`/`with_severity()` to put the verdict
beside a value. No measurement, no verdict: `None` in → `None` out, never a green pass for
an unknown. The verdicts travel **additively** in payloads — `cpu.severity`,
`memory.severity`, `disks.severity`, each UPS card's `severity` (charge/load/runtime),
`severity` inside rated LLM cells (KV pressure, classified on a
percent scale because the engines publish a 0-1 ratio), and `severity` per host/fleet/UPS
row in `/api/summary`. Pages render the class; they never compare a number to a threshold.
There is **no `gpu.*` row and no `gpus[].severity`** (phase 06): these are LLM servers,
a busy GPU is the job at every level, so the GPU card and the fleet GPU column render
uncoloured — like `net.*`, whose colourlessness predates it. Removing the verdict is not
removing the data: `gpu.busy_pct@N` / `gpu.mem_used_pct@N` are still stored and charted.
The `/disks` rows (phase 09) are `disk.temp_c` (50/70) and `disk.percentage_used` (80/100)
— an SSD at 100 % of its rated wear is a replacement, not a warning — attached per row by
`sources/disks.py`; the sector counters and `disk.power_on_hours` deliberately have **no
row**, because a non-zero counter is an alarm chip on presence and zero is not an
achievement (the `/llm` preemption-chip rule), so `classify("disk.reallocated", 1)` is
`None` and the page shows a chip instead of a colour.
- `alerts.py` — `build(settings, job_states, host_payloads, engine_payloads, ups_payload,
recent_events)` → the "what is wrong right now" list `/api/summary` serves as `alerts`
(`{id, severity, source, message, href, at}`, fail first then warn, then by source). A
pure projection: no I/O, no store, no thresholds of its own — it reads job states for
reachability, the `severity` dicts for breaches, `health.checks[]` for UPS conditions.
Rules: an **unconfigured** source never alerts (invariant 3); a source that is ok but
older than 3× its own poll interval is `warn` while a source that stopped answering is
`fail` (the interval is the job's own dashboard's, so `disks` goes amber after 15 quiet
minutes rather than 30 seconds); a dead source gets exactly one line (its stale payload
adds nothing) that names it once — the poller's `"UpstreamError: "` wrapper and the
duplicated host name are stripped, because a message that says `gateway` twice reads as two
problems, and the same rule silences the **`disks` echo** of a dead agent (the disk job
dials the same `/snapshot`, so its failure carries no new information — while a `disks` job
failing *while its host answers*, SMART wedged inside the handler, is the only voice that
fault has); `unknown` and missing numbers are silence; one alert per (source, metric) and at most 12, the last being a marker
that says how many it hid. This is why the overview can be two panels. There is **no
disk-health rule and none is wanted**: a drive the hardware itself called `FAILED` is a red
chip on its `/disks` card and a line in that page's event feed, not an overview alert —
`severity.py` rates no sector counter and this module may not have a threshold of its own, so
grading a drive here would be the bug, not the fix (phase 09). Its `HREF` table does carry
`/disks`, so the one alert a disk job *can* raise — SMART failing while its host answers —
links to the page that names the drive, and since phase 29 `/gateway` for the same reason: a
`gateway` job failing **while its host answers** (the proxy configs or the socket table went
quiet inside a healthy agent) has no other voice. The echo suppression is now a table
(`AGENT_ECHO_DASHBOARDS` = `disks`, `gateway`) because three jobs read one agent, and it finds the
partner differently for each: `disks:<host>` names the host in its key, while the single `gateway`
job names it in its **label** (its key is one word, and `GATEWAY_HOST` is what an alert prints).
A missing `host:<name>` record suppresses nothing — "I cannot find the host's job" is not evidence
the host is fine. The gateway source has **no verdict rule and none is wanted**, the same shape as
the disk rule: its payload publishes no `severity`, `severity.THRESHOLDS` has no `gateway.*` row,
so 400 connections on an ingress is the job, not a fault. What phase 30 task 03 added is the one
non-reachability voice it *can* have without a threshold — **config integrity** — because "a proxy
config that does not connect to itself" is a structural fact, not a graded number: `alerts.build`
gained a keyword-only `gateway_payload`, and one `warn` row (`id` `gateway:config`, `href`
`/gateway`, `at` the payload's own timestamp so the line does not reset every 30 s) exists exactly
when that payload carries `warnings` **and** its job is configured, `ok` and not stale. One row
whatever the count — seven dangling routes are one config to fix and the page lists all seven —
reading `"{n} proxy config {problem|problems} to address — {first message}"`, with the message
built by `sources/gateway.py` and only quoted here. It is suppressed whenever the job's own
reachability row is on screen (`ok: false` or stale past 3× its interval), because a router that is
not answering is the operator's problem and re-listing warnings from a reading they cannot refresh
is noise beside it; the page keeps showing them beside the last good numbers the whole time. So
`alerts.build`'s signature is now the boundary that keeps the *counts* out: no `severity` reaches
it and no `gateway.*` threshold can be consulted, which is what
`test_a_gateway_payload_with_a_fabricated_verdict_alerts_nothing` re-pins after it replaced the
older "takes no gateway payload at all" wording.
- `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning. The
`meta` table is the only durable key/value state in the app and is reached through three
named helpers — `get_meta`/`set_meta`/`delete_meta` (the last returns its rowcount, because
the password-reset procedure has to be able to say whether a row was there, and one key per
call, because a reset that deletes a row it did not name deletes the operator's credential).
`stats()` reads `last_maintain` through `get_meta()` too — outside its own lock, which
`get_meta` takes itself and `threading.Lock` is not reentrant — while `maintain()`'s write
stays inline inside that method's transaction. **No per-request path may pay a whole-table
scan** (phase 14 task 02): `series_names()` (1.4 s of `SCAN sample` on a production-sized store,
behind `/api/metrics`) and the two `COUNT`s behind `stats()` (1.25 s, run by `app.js`'s
`/api/health` poll on every tab every 15 s) answer from short-TTL caches — 60 s and 30 s, the
series list additionally invalidated **only by a write that can change it** (task 04: an insert
cannot remove a name, so `_note_series_write()` ends the cache on a `(source, host, metric)` the
store has never written and leaves a poll of an existing series alone — invalidating on every
sample was task 02's defect, and it put the 572 ms scan back on nearly every `/api/metrics` call
on a store every poller writes to, behind a cache that never served; `delete_series()`/`maintain()`
invalidate unconditionally because a `DELETE` can take a name, `delete_series()` also forgetting
the triples it emptied so a re-created series counts as new again), through a generation counter so
a write racing a read cannot park a stale answer under a fresh timestamp, with the name cache
bounded to 256 `(source, host)` keys because `/api/metrics` takes both as free text and the
existence tracking capped at `SERIES_TRACKED_SERIES_MAX`, below which it falls back to
invalidating on every write. A tracking set made wrong by *another* process deleting rows costs at
most one TTL window, which is the reason the cache has a TTL and not only the tracking. `last_maintain` and `size_bytes` are never
cached: a `last_maintain` that stops moving is how an operator spots the invariant-4 failure
mode, and it costs a three-row lookup. The resolution decision is one compound statement of
per-`(host, metric, res)` branches whose `COUNT` is capped by `LIMIT max_points + 1` — exact,
because the decision only ever asks whether a count is `0`, `≤ max_points`, or neither — and
`pick_res_many()`/`history_many()` are the batched door the N × M endpoints go through. The rows
stay a per-series seek **measuredly**: a `ROW_NUMBER()` window or a compound of the 27 tail
selects both cost more than 27 index seeks on the seeded store (`data/logs/NOTES-store-perf.md`
records the table), so batching the *decision* is the fix and batching the *fetch* would be a
round trip traded for work.
**The write side is windowed too** (phase 14 task 03): `maintain()` reads `last_maintain` through
`get_meta()` *before* its transaction (same non-reentrant-lock rule as `stats()`) and bounds each
rollup step to what became due since, instead of averaging every row the resolution retains —
3.97 M rows of 1-minute history per minute, a 5.4 s write transaction that made a poller's
`add_many` outlive `busy_timeout=5000` (gap 4's `store-error` line). Two properties make that a
cost bound and not a data change, and both are pinned: the lower bound reaches **past** the
previous pass (one bucket plus slack, so ten quiet minutes are a catch-up scan rather than ten
minutes of empty charts), and it is snapped **down to a bucket boundary** — the step's ceiling is
a row bound, so the newest bucket is partial and a later pass re-rolls it as its rows land, and a
floor that cut into a bucket would overwrite the fuller average with the mean of its tail rows
(2.5 becoming 4.5 on a six-sample minute: rule 2 with a chart's clothes on). A window is only a
cost bound if the plan can seek to it, and two things were in the way. The prune keeps its full
retention bound — a rollup may window because it only adds rows, a prune must reach anything that
expired — but it is **one `DELETE` per resolution**, because one
`WHERE (res=? AND ts<?) OR (res=? AND ts<?) OR (…)` is unreadable to the planner and read the
whole 6.4 M-row table to delete nothing (554 ms of a 562 ms pass; 0.3 ms per statement now). And
the store had no query statistics, so every window was priced as a scan however few rows were
inside it: `maintain()` ends with `PRAGMA optimize` and `__init__` sets `automatic_analyze=0`,
because SQLite's own refresh lands inside whichever poller's commit trips its counter — the exact
path this shortens — while a tick a minute costs ~100 ms once and 0.0 ms afterwards. Measured on
the seeded store: **5,368 ms → 2.7 ms** steady, 48 ms for a ten-minute catch-up, and the
two-connection race at the app's cadence goes from 5 of 9 inserts raised to 100 of 100 landing
with a 21 ms worst wait. `__init__` also sets `wal_autocheckpoint` (SQLite's own 1 000 pages,
written down so the knob is visible) plus `journal_size_limit = 64 MB`, the cap that truncates a
runaway WAL at the next checkpoint. `busy_timeout=5000` is unchanged: the fix is a shorter write,
not a longer wait.
- `api.py` — GETs everywhere plus the one POST that dials an upstream, and the `auth_router()`
that creates, describes, **re-credentials**, and ends a session (`POST /api/auth/login` public;
`GET /api/auth/status`, `POST /api/auth/logout` and `POST /api/auth/password` protected). Those
three, and the `certs_router()` of phase 15 (`GET /api/certs`, `GET /api/certs/<name>/bundle`,
`POST /api/certs/<name>/rotate`, `POST /api/certs/<name>/revoke`,
`POST /api/certs/root/rotate`), touch this process and its `meta` table and dial
nothing — invariant 1 is about the fleet, and `tests/test_api.py` pins the three lists separately
rather than widening the one that names `/api/ups/command`.
The **`/api/certs` surface** (phase 15 task 04) is the CA's own admin half. The list reads the
**stored** state through `certs.read_state` — the request-time twin of `ensure_certs`, which
mints and must only ever be called at startup, because a `GET` that bootstrapped a root CA would
replace the authority nobody asked it to replace — and its rows are the union of `HOST_TARGETS`
and `certs.agents`, so a configured-but-uninstalled host appears as `status: "unissued"` with
nulls (invariant 3, rule 2) and an entry whose host left the config is listed but not writable.
`deployed_bundle_rev` comes from that host's `system` payload `agent` block (task 03's
passthrough), never from what the CA hopes. `bundle_rev` is what was issued; the two numbers side
by side are the "install the new bundle" sentence, and neither carries a `severity` — an expiry
and an identity counter are not graded numbers. A `<name>` outside `HOST_TARGETS` is refused in
the route's first line with `404 {"configured": false}` **before** any signing, dial, or store
write (spy-pinned both ways, and the same gate makes `GET …/bundle` answer
`{"configured": true, "issued": false}` for a host that is configured and has nothing issued —
two different facts, two different bodies). The bundle download is the stored row **verbatim** as
`application/x-pem-file` with an `attachment` filename, because it is `install-agent.sh`'s input
and it carries that host's private key (session-gated, `no-store`). `POST /api/certs/root/rotate`
answers `restart_required: true` rather than pretending a hot-swap: the CA and the client pair are
startup material, so the table is new at once while the process keeps dialling with the old one.
Every write logs one fixed-text `certs`-source event naming host and action, and serials travel as
**decimal strings** — a 64-bit certificate serial through a JS number is a serial a page cannot
match against a bundle's `# REVOKED` line (invariant 6: no PEM reaches an event, a health record,
or any payload but the bundle route's).
The login route is where a *sign-in* password is compared: it
throttles before it hashes, refuses an absent/empty/non-object body with `422` so `""` never
reaches `verify_password`, answers every failure with the one sentence `wrong password`, and
answers `429` + `Retry-After` while the per-address backoff runs (the only extra fact it adds,
because a wait is something the caller has to act on). After a good sign-in it also upgrades a
verifier minted at a lower `DASH_PBKDF2_ITERATIONS` (`auth.needs_rehash`), keeping the signing
secret — that knob has to actually buy something, and a login is the one moment the plaintext is
in hand. A refused upgrade write is swallowed with one fixed-text `logging` line (this module's
only one, carrying no credential: invariant 6) because `database is locked` on a bookkeeping row
must not turn a correct password into no login at all. `POST /api/auth/password` is task 04: it
validates `new` first (`MIN_PASSWORD_LENGTH = 8`, non-empty once stripped, different from
`current` — length is the whole policy, because composition mandates produce sticky notes),
throttles the *current* password on its own key `password:<address>`, refuses a wrong one with
`403` + a `password-denied` event, and on the right one calls `auth.update_verifier` and answers
from the **new** state with a cookie and CSRF token minted from the rotated secret. Building that
response from the old `ctx.auth` is the bug the order exists to prevent: the operator would be
signed out by their own form.
`AppContext` carries `auth` and the one
`LoginThrottle` per process — a throttle built per request is a throttle that never trips. A
refused sign-in writes a `login-denied` **event** and nothing else, and so do the two password
outcomes (`password-denied`, `password-changed`): `alerts.py` turns only
`kind: "unreachable"` into an overview row and every feed filters by `source`, so
`/api/events?source=auth` is the audit trail and the overview stays about outages (do not
"fix" that by hiding the event). Neither fact appears on `/api/health` — which is public — or
`/api/config`. `/api/config` is a redaction boundary.
`/api/disks/*` (phase 09) is the drive-health pair: `/snapshot` answers one entry per
configured host **with drives to show** (disk rows + `smart` + `smartctl_available` +
per-row `severity` + a `fleet` block summed by the API, with a configured-but-never-answered
host as an `unreachable` stub whose counts are `None`, not `0`), and `/history` serves the
`disk.*@dev` series with unknown names answering as empty series like the other
history endpoints. The configured gate runs **before** anything that could reach the
network — `HOST_TARGETS` empty is `404 {"configured": false}` with no client, no fetch and
no `disks` job in the hub at all (invariant 3), and `tests/test_api.py` proves it with
spies on `httpx.AsyncClient.get` and `collectors.fetch_snapshot`. Configured-ness and
visibility are two different gates, and only one helper decides the second:
`api._disk_visible(host)` drops a host whose payload's `smartctl_available` is **explicitly**
`False` ("no smartmontools, no drives to report" — the VM case) from `hosts` and therefore
from `fleet` (`fleet_disk_summary` sums whatever it is handed, so `hosts_total` is the row
count), while `True`, `None` and an `unreachable` stub all stay — an agent that cannot say is
not an agent that said *no*, and a dead agent must not read as "never configured"
(invariant 3). The check is `is not False`, not falsy, and it runs in `/api/summary`'s disks
block too, before both the fleet sum and the per-host rows, so the two endpoints cannot list
different machines; `states` still names every `disks:<host>` job (hidden included, still
polling, still `ok`) and the source, the store and the health records see no change at all —
which is also why an overview alert from a hidden host's `disks` job still names that host
(`alerts.py` reads job states, not this list): hiding a card must never hide a fault
(phase 22). The new surface grows no
write of its own — a fourth dashboard must not quietly become a second one, so the app's
only POST that **dials an upstream** stays `/api/ups/command` (invariant 1, pinned by enumerating
every served route's verbs and sorting the writes into fleet / session / certificate-authority
lists — the third is phase 15's, and it writes `meta` rows and nothing else).
`/api/gateway/*` (phase 29 task 03) is the ingress pair, and it is the same two rules on a
single-host source. `/snapshot` answers `sources/gateway.parse`'s payload for the one
`GATEWAY_HOST` entry — the haproxy frontends and the nginx `listen`/`map` echo, the `listeners`
(port, domains, live count, up to `MAX_CLIENTS` client addresses), the `connections` graph (one row
per route the configs declare, its `in.port` always one of those listeners, its own `metric`,
`shared` when more than one line reaches its endpoint), and the `warnings` list (phase 30 task 03:
seven kinds of config that does not connect to itself, each `{id, kind, subject, message}` with
`id = "<kind>:<subject>"`, empty for a pre-1.2.0 agent and for a section that did not answer) —
with `states` appended and the payload's own
`ts`
left alone, because that is when the agent took the reading and it is what the page ages. Three
arms reach it and none of them is red: `404 {"configured": false}` **in the route's first line**
when `GATEWAY_HOST` names no `HOST_TARGETS` entry (unset and wrong-name are both spy-pinned to
`httpx.AsyncClient.get` and `collectors.fetch_snapshot`, and neither arm has a `gateway` job to
age out either), `supported: false` for a 1.1.0/`--no-proxy agent, and `gateway_source.stub()`
for a configured host that has never answered — the payload's key set plus `unreachable: true`
and minus only the `agent_version` nobody published, every count `null` because an agent that has
not spoken has not reported an idle gateway (rule 2). Nothing in the payload carries a `severity`
or a `window_seconds`, at any depth (pinned by a recursive walk: a busy proxy is the job and no
Δ was ever measured). Phase 30 task 02 took `outbound[]`, `unmatched` and `by_port` out of it: the
flat list duplicated the graph with a worse count, and the residual existed only to explain a
guessing rule that no longer runs. Phase 32 task 03 added one top-level key, `resolver` — the
agent's own block, `{available, via, entries}`, each entry projected to the five fields the agent
publishes — so a dash on one line is answerable from the document the page already holds; `stub()`
grows it with nulls, which is what the stub/answer key-parity pin compares. `/history`
serves the `gateway.in_conns@<port>` /
`gateway.out_conns@<host>:<port>` gauges with `hosts=` defaulting to the ingress **alone** (the
name the samples are stored under — defaulting to every target would draw empty plots for
machines nobody asked to route), and it carries no configured gate because it dials nothing:
like `/api/disks/history`, an unconfigured source answers no hosts and no series. The pair adds
no write (the verb enumeration pins the two GETs and that the app's one upstream POST is still
`/api/ups/command`), and `tests/test_auth_api.py` lists both paths as session-required.
All five `history` routes reach the store through one door, `api._history_batch()` →
`store.history_many()` (phase 14 task 02). That is a cost change and nothing else: the series
list is host-major and metric-minor in request order, each entry the same
`{source, host, metric, range_seconds, points}` document the per-series loop built, with the
same empty-series answer for a name that has no rows. `tests/test_api.py` pins it for all five
routes against points read straight out of the store with `query()` — a third path, in neither
the loop nor the batch — so a future "optimisation" that moves a point, an order, or a key
fails a test instead of redrawing a chart wrong.
`/api/summary` is the overview's single call and the cheapest whole-fleet probe there is:
the page renders only `alerts` plus `system.fleet`/`system.hosts`, while `disks.hosts`,
`llm.engines`, `ups.ups`, the `states` maps and `recent_events` stay for programmatic
consumers (live tests, an external check, a notifier). `system`/`llm`/`ups`/`recent_events`
are unchanged and `alerts` (phase 02) and `disks` (phase 09) are additive — the `disks`
block is `fleet_disk_summary` verbatim plus one row per host (`disks`/`with_smart`/
`health_failed`/`alarm_disks`/`smartctl_available`/`failed_devices`), so a notifier learns
*which* drive the hardware called `FAILED` without a second call, and
`tests/test_api.py` pins every one of those key paths, with and without targets configured,
so retiring a panel can never shrink the payload. Since phase 22 those rows are the **visible**
hosts — the same `_disk_visible` predicate `/api/disks/snapshot` applies, over the same
entries, before the fleet sum — so `disks.fleet` equals `/api/disks/snapshot`'s `fleet` and a
host with no smartmontools is missing from both, while `system.hosts` still lists it (a missing
binary is a fact about the drive page and nothing else) and `disks.states` still lists its job.
A configured
job with no payload yet is listed as a stub (`unreachable: true`, every value `null`) in
`system.hosts`/`llm.engines` exactly as `/api/system/snapshot` does, so `hosts_total`
counts the machines there are and `hosts_live` the ones answering.
The **`gateway` block** (phase 29 task 02) is additive in the same way: `{configured, host, href,
supported, unreachable, listeners, connections, live_in, live_out, map_targets, warnings, states}`
— twelve keys, `warnings` added by task 03. The middle six were renamed by phase 30 task 02 from
`inbound`/`outbound`/`unmatched`/`unmatched_listeners`/`partial`, because the first pair now counts
a graph and not a flat pipe list and the residual no longer exists; `tests/test_api.py` pins the
set, so a thirteenth field is a decision and not an accident. `listeners` and `connections` are **row counts** ("how many trunks / routes does
this gateway declare", which is what the overview's one line prints) and `live_in`/`live_out` are
the two socket-derived totals; Task 02 wrote it for programmatic consumers (the page itself reads
`/api/gateway/snapshot`, task 03); task 05 gave it one page-side reader, the single line `/`
renders under the fleet table, so `href` joined the block — derived from `alerts.HREF`, the one
table that writes a dashboard's route, so the overview's link and an alert about the same source
cannot spell the page differently (`test_the_gateway_block_links_to_the_page_the_alert_row_links_to`,
and the page half reads only block keys: `test_the_overview_line_reads_the_gateway_block_it_claims`
resolves the served `GATEWAY_FIELDS` against a live document). `live_in` sums the listeners and
`live_out` the **distinct outbound endpoints** — never the connection lines — because several
`(listen, endpoint)` lines carry one socket count repeated with `shared: true`, and summing the
lines would book the same sockets once per line and print a router busier than it is (the same
rule that keys the outbound gauges by `(address, port)`, with the address normalised so two names
— or two spellings — resolving to one endpoint book it **once**); `api._endpoint_values()` is the one
reader of that (it delegates to `gateway_source.distinct_endpoint_counts()`, the same union-find the
join builds its counters with, so the block's total and the pipes cannot disagree), and a line with
no `metric` (a name nothing resolved) is absent rather than an unmeasured endpoint holding the total
hostage. `map_targets` counts the
stream map's routing entries. Every figure is `null` — never `0` — in the arms that did not
measure (unconfigured, configured-but-never-answered, a 1.1.0/`--no-proxy agent, a silent socket
table), and a total is `null` rather than a part of itself when any countable endpoint is unknown
(`page-gateway.js`'s `sumLive()` mirrors it for the one tally the page paints — the page has no
outbound total of its own, so only the inbound half is summed there). Task 30.03 added the
twelfth field, `warnings` — the **count** of `sources/gateway.warnings()` rows, which is what the
page's `warnings` head tally and the overview's ` · N to address` suffix both read (the overview
prints it only above zero; a permanent `0 to address` is the ink the empty-state rules exist to
avoid). Nothing in the
block carries a `severity`. `/api/config` grew the matching row (`enabled`,
`host`, `url`, `interval_seconds`, `timeout_seconds`), where `url` is the named entry's
`Target.display_url` and `null` when no target carries the name — invariant 6's boundary applies
to a name-only source exactly as it does to a URL-bearing one, because this body is what a browser
renders.
- `app.py` — assembly, and the **session gate** (`session_gate` middleware, invariant 12) whose
`PUBLIC_PATHS`/`PUBLIC_PREFIXES` are the whole anonymous surface. It answers in this order and
the order is the design: no `AuthState` → `503` for everything; no session and no such route →
the router's `404` (a typo is not a sign-in problem, and "sign in required" would otherwise be
the only answer anyone outside ever gets for a wrong URL); public → through, except that the
origin check still applies to a public write (a cross-site login post spends the throttle of
whoever is typing) and `GET /login` for someone already signed in goes to the dashboard, not
back to the form; no session → `303` to `/login?next=…` for a page and the JSON refusal for
everything else; a write → origin, then this session's CSRF token when it arrived on the
cookie. `_own_origin` reads `X-Forwarded-Proto` for the same reason the cookie's `Secure` flag
does, so behind nginx a real same-origin write is not a cross-site refusal — the bug that only
appears in production, which is why it has a test. `create_app()` takes injectable settings/store so tests build an app
with an in-memory DB and no targets. The lifespan builds `ctx.auth`
(`auth.load_auth_state`) after the context exists and **before** the first poll, for the
owned-store and injected-store branches alike — one call site, because two copies drift — and
a `RuntimeError` from it leaves the process non-zero with the operator's fix in the output.
That failure is never caught here: a dashboard that cannot say who its admin is does not start
the loops either (`__main__.py` passes `SETTINGS` to `create_app()` explicitly so the served
app is configured by the same object the bind address came from). `build_certs(store)` is the
same rule for phase 15: one helper, called **before** `build_hub` in the owned-store
(`build_context`) and injected-store (lifespan) branches, whose `CertMaterial` (the CA + client
paths in a private temp dir) hangs off `AppContext.certs` and whose
directory the lifespan's `finally` removes. What hands that material to the loops is
`agent_tls_map(material, store, targets)` → `build_hub(..., tls=...)`, called in **both**
branches the same way: no material (or no targets) means no TLS map, which is the plain
`http://`/token path rather than an error, and one map covers all of a host's jobs — including the
`gateway` job, which is why one revoke in `/certs` silences the pipes with no restart. A corrupt stored certificate raises out of the same
place — a dashboard that cannot present its own client certificate would otherwise spend the
fleet's TLS history reporting an upstream problem it caused itself.
`build_hub` also builds **two** `RateEstimator`s, because a window is only a claim about the
polls that fill it (phase 23, `TODO.md:8`'s "ramp up/ramp down"): the `host` jobs get
`RateEstimator(span=settings.rate_window)` (10 s — what a 10 s cadence can fill) and the `llm`
jobs `RateEstimator(span=settings.llm_interval)` (2 s, one poll interval, so a slot that starts
reads steady on the next poll instead of climbing one fifth of the true rate per poll), and the
`gateway` job (phase 29) is handed **neither** — it is added to the hub between the `disks` and
`llm` jobs, only when `settings.gateway_enabled`, because a gauge needs no window and one job
cannot be right about two cadences' worth of them.,
there is no third knob — the LLM window **is** `LLM_INTERVAL_SECONDS`, which is the only way
the pair cannot drift. One estimator for both could only be right about one of them: a 2 s span
over a 10 s poll sits the window-empty fallback against the `2 × span` ceiling and answers
`n/a` every poll, and a 10 s span over a 2 s poll smooths the work the operator is watching.
The estimator's own gates (the `MIN_WINDOW_SECONDS` floor, the ceiling, the 8× jump
confirmation with its independent anchor) are unchanged and apply to the smaller window, whose
`Δ2.0s` label the payload carries with the value.
- `web/` — six dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`, `/gateway`) plus the
login page (`/login`, `pages/login.html` + `static/page-login.js`), one stylesheet, `charts.js`.
No framework. The tab nav is those six and nothing else, in that order, on **all eight**
signed-in pages (phase 29 task 05): `tests/test_api.py::test_the_disks_tab_is_in_the_masthead_of_every_page`
compares the nav's hrefs to `TAB_ORDER` (which **is** `PAGES`) and asserts the `aria-current`
count per page — one, naming itself, on the six dashboards and zero on `/settings` and `/certs`.
`/gateway`'s own tab arrived with that task; until then the page existed behind a route with no
way to click to it. `/login` renders the masthead's **brand link and theme toggle** and nothing else
from it: no tabs (every tab bounces straight back here, and a nav of links that all redirect is
a lie about what is available), no status dots and no clock (there is nothing signed in for them
to say). It posts through `Dash.fetchJSON` like every other script, which is only safe because
`bounceToLogin()` does nothing when the browser is already on `LOGIN_PATH` — without that guard
a `401` would reload the form instead of printing `Wrong password.`, and the page that exists to
end a bounce loop would be one. So the form branches on `err.status`: `401` is the inline
sentence plus `field.select()` (fixing a typo without retyping is the whole UX here), `429`
shows the server's `detail` and **counts down its `Retry-After`** with the button disabled until
the wait is over — `fetchJSON` therefore carries `response.headers` on the `Error` it throws,
because a wait this page made up would be an invented number (rule 2) — and anything else prints
the server's `detail` verbatim. Every refusal goes out through `textContent`, never
`innerHTML`: this is the one page a stranger reaches first. Its own copy of `auth.safe_next`
guards the one redirect followed on client strength (already-signed-in `?next=`), and
`Dash.ensureAuth()` up front stops a flash of the form for a browser that is already in. The
login page is audited for orphan `id`s and CDN references too (`tests/test_auth_api.py`), fetched
unsigned because a signed-in `GET /login` is a `303` and would audit `/`'s markup under the wrong
name. `/settings` (`pages/settings.html` + `static/page-settings.js`) is a **page and not a
seventh dashboard**: same `<head>` and masthead, the six tabs with **no** `aria-current` (it is
not one of them), the same five `#nav-status` badges as every dashboard (phase 25 task 01 —
`TODO.md:3`'s "hosts disks engines UPS disappear when in the certs/account page" was the
missing container, and `Dash.initChrome()` fills it from `/api/health` with no JS change), and
it is reached from the `⚙ Account` link in
`.masthead-right` of every dashboard **including itself** — outside `.tabs`, so the pinned tab
structure is untouched (`tests/test_api.py::test_every_page_links_to_both_admin_pages`). It prints
`/api/auth/status` (name, expiry as a local timestamp *and* as time left), signs out through
`POST /api/auth/logout`, and changes the password through `POST /api/auth/password` after
checking locally what it can judge itself (non-empty current, ≥ 8 new characters —
`tests/test_api.py` pins that number against `api.MIN_PASSWORD_LENGTH` so the two cannot drift —
new matching confirm), printing `403`/`422`/`429` through `textContent` as the server words them,
then re-reads the status so the panel cannot show a session the rotation ended. Its static
sub-line is the precedence rule (env seed → stored verifier wins → reset by deleting
`auth.verifier`), because that sentence is what stops the "I edited the env file and nothing
happened" call. So `app.js` exports **`resetAuth()`**: it clears the memoised `authStatus` and its
in-flight promise, and must be called after a password change (the secret rotated, so the cached
`csrf` is now invalid — every later write, the UPS buttons included, 403s until a reload) and on
any `401` bounce.
`/certs` (`pages/certs.html` + `static/page-certs.js`) is the second of those admin pages, and it
is the same shape for the same reasons: six tabs with **no** `aria-current`, the five fleet
badges in `#nav-status` like every other masthead (phase 25 task 01, same reasoning as
`/settings` above), and both masthead links — `⚷ Certs` beside `⚙ Account` — exactly as the six
dashboards render them. **Both links are on both admin pages** (phase 20 task 04, `TODO.md:4`),
which retires phase 15's "no self-link" precedent: chrome that changes shape depending on where you already are reads
as a broken button, while a link to the page you are on is a no-op that answers with the page you
were looking at (`data/logs/20-task04-admin-links.cjs` measures exactly that — one document
response, no redirect, the same cluster afterwards). The two links therefore sit on all eight
signed-in pages (the six dashboards plus `/settings` and `/certs`) in one order
(`⚷ Certs` → `⚙ Account` → theme toggle → clock — phase 25 task 03 moved the clock to the end so
it is the farthest-right item on a `flex-end` row), which is what lets
the pinned audit be one loop over `MARKUP_PAGES` instead of two over different tuples; `/login`
is the ninth page and the one with neither. Everything on it is
`GET /api/certs`: the Root CA block, the display-only Dashboard client line (that certificate is
minted with the root and has no route of its own), and one row per host with its status chip,
serial, expiry (local timestamp *and* `Dash.duration` left), issued-vs-deployed `bundle_rev`,
`san` list, and the three controls. Two rules it exists to keep. (1) **No colour the API did not
send**: `/api/certs` publishes no `severity` — an expiry is a date and a revision count is not a
graded number — so the page has no `severity` token at all, the one chip class map is
`status` word → `ok|fail` with `unissued` mapping to *nothing* (a certificate that does not exist
never borrows green), and a behind bundle is a neutral `stale` chip. (2) **No control the API
would refuse**: a row with `configured: false` prints why (`not in HOST_TARGETS`) instead of
three buttons that all answer `404`, a never-issued host has no **Revoke** and no **bundle** link
(that route answers `issued: false`), and the download is a plain `GET` link because a session
cookie is all a signed `GET` needs. Writes are `confirm()`d, not typed-`DANGER`d — that gate is
for cutting power and this page cannot — and the outcome line prints the route's own `hint`, so
`restart_required`'s restart sentence lives in `api.py` and nowhere else. Its `COLUMNS` table
names every payload key it indexes, and `tests/test_api.py` resolves that table against a live
`GET /api/certs` row in both directions (the `SLOT_FIELDS` device), so a rename on either side
fails a test instead of printing `n/a` beside a healthy fleet. The two CNs are markup, and pinned
to `certs.ROOT_CN`/`certs.CLIENT_CN` the way `/settings` pins its password floor. The agents
table is `table.data.scroll` — the class-level scroll mechanism (`TODO.md:9`: at phone width the
serial / bundle / `san` / controls tail was clipped by `.card { overflow: hidden }` with no way
to reach it), and the class on the served table and the served `table.data.scroll` rule are
pinned **together** in `tests/test_api.py`, because the mechanism exists only while both name
it (measured at 360 px and 1280 px by `data/logs/21-task03-scroll.cjs`). Since phase 27 task 01
it is no longer the only thing that scrolls — at ≤640px every `table.data` does — and `.scroll`
out-ranks that rule (two classes to one), so this table keeps its both-axis box and its 320 px
height cap on a phone exactly as on a desktop (both halves measured at both widths by
`data/logs/27-task01-scroll.cjs`).
`app.js`'s masthead status dots are one per dashboard job —
including `disks`, because the same agent can answer the 10-second utilisation poll and
have a wedged SMART pass, and one dot for both would be right about neither, and including
`gateway` (phase 29 task 05), which is one dot rather than one per host because the ingress is
one job and a dot for a source that stays `idle` when `GATEWAY_HOST` names nothing is how an
unconfigured source reads grey rather than red. The badge table is a literal in `initChrome()`,
so a new dashboard's job means a new entry there — `NAV_STATUS_BADGES` in `tests/test_api.py`
is the pin that says so. `app.js` owns the
colour vocabulary: `Dash.severityFrom(cellOrSeverity)` turns a payload verdict into an
`ok|warn|fail` class (`''` for anything unrated, so unknown never looks green) and
`Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
`.hero` big number (plus an optional `.bar`) from that class — it never compares a number
to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for
values the API does not rate, and **no page script calls it any more** — `/` was converted
in phase 02, `/system` in phase 03, `/llm` in phase 04, `/ups` in phase 05 and `/disks`
shipped that way from its first commit, so every colour on those five pages comes from a
`severity` verdict the API sent
(`tests/test_api.py` fails if a page script names it again). The sixth dashboard colours nothing
by verdict at all — `/gateway`'s only colour pair is existence (`--gate-on`/`--gate-off`), and it
carries no `severity` token anywhere. A value the API should rate
gets a row in `severity.py`, never a bound in JS.
**The masthead is three rows at every width, the two control rows flush right** (phase 20 task
01 made the rows, `TODO.md:5-6`; phase 25 task 02 re-justified them, `TODO.md:4` — "I made a
mistake earlier, I did want hosts disks engines UPS badges and certs/account/dark right
justified"): row 1 the brand, `#nav-toggle` and the six tabs; row 2 the `#nav-status`
per-source dots; row 3
the `.masthead-right` controls (`⚷ Certs`, `⚙ Account`, theme, then the clock — phase 25 task 03
moved the clock last for `TODO.md:5`'s "the time should show up as the farthest right item";
phase 21 task 01 deleted the `⏸ Pause` that used to sit beside the links, button, `Space` shortcut and
`setPaused`/`_paused` plumbing alike: it froze the numbers on screen without growing a `stale`
label, and a backgrounded tab already idles its polls, which is the only stop this app needs).
That was CSS and
no markup change, which decides the mechanism: `#nav-status` is a **child** of
`.masthead-right`, so there is no wrapper to promote, and the rows are line breaks — `.masthead`
is `flex-wrap: wrap`, and both `#nav-status` and `.masthead-right` carry `flex-basis: 100%` with
`justify-content: flex-end` (line breaking uses a flex item's **base** size, before shrink, so
a 100 % item is alone on its line: the div's break starts row 2 of the bar and the dots' starts
row 2 inside it, leaving the controls to wrap to row 3). `.masthead-right` lost the
`margin-inline-start: auto` that pinned the cluster to the right edge of the one-row bar.
`flex-direction: column` was the other option the task allowed and is the wrong one: it puts the
brand and the tabs on separate rows. `height: var(--header-h)` became **`min-height`** (three rows
do not fit in 58 px and a fixed height clips the controls) and is declared **once** — the ≤640px
block no longer restates `height`/`min-height`/`flex-wrap`, and that pair drifting is exactly what
would clip the bar on a phone only — except that **`#nav-status` wraps at ≤640px since phase 29
task 05**: five badges are 352 px of chips against this bar's 320 px content box, and a `nowrap`
item that may not shrink does not get smaller, it widens the *document* (measured 372 px — the
sideways scroll phase 18 forbids). The fifth chip takes a second line of the same right-aligned
row and the bar grows by one line box; no label shrinks and nothing is hidden. Phase 18's
`nowrap` was right for four (267 px) and the pin now demands the opposite, with
`data/logs/29-task05-tabs.cjs` measuring 360 px documents on all eight pages.
Every page that renders the bar now renders three rows —
`/certs` and `/settings` grew the dots row in phase 25 task 01, where `#nav-status` is pinned as
the **first child** of `.masthead-right` on all eight signed-in pages (`app.js`'s *badge list* is
a literal, and phase 25's fix was markup because the container was what was missing; phase 29
task 05 added the fifth entry — `{ key: 'gateway', label: 'gateway', href: '/gateway' }` — because
the ingress has a job of its own and a job with no badge is a source that blinks in silence) —
and `/login` remains the one page with no dots, rendering two rows
(brand, then the theme toggle). Pinned on the served bytes by phase 20's three tests in
`tests/test_api.py` — the row declarations (task 02 renamed that one to
`test_the_masthead_is_three_rows_right_justified_at_every_width`), the mobile block's silence
about the bar's height, and the bar's `position` — and measured in a browser at 1280 px and 360 px by
`data/logs/20-task01-rows.cjs` (seven pages × two widths plus the dropdown; screenshots
`data/logs/masthead-rows*.png`). **Phase 25 task 02 moved the alignment of those two rows to
the other edge** (`TODO.md:4` — "I made a mistake earlier, I did want hosts disks engines UPS
badges and certs/account/dark right justified"): the two `justify-content` declarations are
`flex-end`, `margin-inline-start` is still absent from the cluster (a full-basis row aligned by
`flex-end` is the mechanism, not the old single-row right-pinning), and the ≤640px block still
restates neither the alignment nor the bar's height, so the phone's wrapped control row follows
the base rule. The pin is `tests/test_api.py::test_the_masthead_is_three_rows_right_justified_at_every_width`
(the row-break assertions are the same ones phase 20 wrote, deliberately: the re-justify must
not cost the rows), and the measured half is `data/logs/25-task02-justify.cjs` — seven pages ×
two widths, each demanding both rows compute `flex-end` and sit flush with the bar's right
padding edge, the four badges present (a four-badge claim — `29-task05-tabs.cjs` is the pin that
the fifth, `gateway`, joined them), three rows intact, the dropdown still under the whole
bar and no sideways scroll, plus `/login`'s two-row bar with no badges
(`data/logs/25-task02-justify.out`, screenshots `data/logs/25-task02-*.png`). **Phase 25 task 03
then put the clock at the end of the control row** (`TODO.md:5` — "The time should show up as the
farthest right item on the navbar"): the `<span class="clock" id="clock">` moved from the first
control to the last one on all seven templates then (eight since `/gateway` joined the nav), so
the row reads `⚷ Certs` → `⚙ Account` → theme
toggle → clock. It is a **markup** move and deliberately not a CSS `order:` declaration, because
on a `flex-end` row the last child *is* the farthest-right item and an `order:` override would be
something the served-byte pin cannot see — `test_every_page_links_to_both_admin_pages` therefore
asserts `html.index('id="clock"') > html.index('id="theme-toggle"')` on every page (the two
Certs/Account-before-theme assertions stay, and the `/login` half of the test is untouched).
The clock keeps its long-standing `@media (max-width: 900px) { .clock { display: none } }` hide:
it is a readout and not a control, so this is not the "no control is hidden to make the width
work" rule the masthead keeps. Measured by `data/logs/25-task03-verify.cjs` — seven pages × two
widths, each demanding three rows, both right-hand rows flush with the bar's right padding edge
and the chips' content (not the full-basis row box) measured to prove it is not flush left, the
clock the rightmost painted thing on the bar at 1280 px, the cluster's source order unchanged at
360 px where the clock is hidden, and the four badges populated on `/certs` and `/settings` as on
the dashboards; plus the 360 px dropdown and `/login` unsigned at both widths
(`data/logs/25-task03-verify.out`, screenshots `data/logs/masthead-25-*.png`).
**The bar is not sticky** (phase 20 task 02, `TODO.md:10` — "It shouldn't be sticky at all. We
want to maximize the amount of screen space for data viewing"): `position: sticky; top: 0` became
**`position: relative`** with no offset, so the whole masthead scrolls away with the page and the
phone's "sticky only part way down" quirk — the fixed-height sticky bar fighting the wrapped
rows — went with it. `relative` rather than the default `static` because `.masthead` is still the
positioned ancestor the ≤640px dropdown's `top: 100%` resolves against (a `static` bar moves the
menu to the top of the document), and `z-index: 40` stays so the menu's `41` still means one
above the bar; the consequence is that the open menu scrolls with the bar, which is what an
unpinned bar means. The only `position: sticky` left in the sheet is a table's own `thead`, inside
its own scroll container, and this task is not it. Proven by `data/logs/20-task02-unsticky.cjs`
(seven pages scrolled at 1280 px and 360 px — with a squeezed-height leg for the pages that fit
a tall viewport — plus the 360 px dropdown still landing directly under the bar and still hanging
off it after a scroll; screenshots `data/logs/masthead-unsticky*.png`). Note that a browser cannot
tell `relative` from `relative; top: 0` — Chrome computes `top: 0px` for either — so the absence
of the offset is a served-CSS pin, not a measurement.
**The tab nav collapses into a hamburger at ≤640px** (phase 18): the eight tabbed pages (the six
dashboards plus `/settings` and `/certs`; `/gateway` joined in phase 29 task 05) each
carry `#nav-toggle` (`class="ghost nav-toggle"`, `type="button"`, `aria-controls="nav-tabs"`,
`aria-label="Toggle navigation"`, `☰`) between `.brand` and `<nav class="tabs" id="nav-tabs">`,
and `/login` carries neither — it renders the brand link and the theme toggle only. Its base
rule is `#nav-toggle { display: var(--nav-toggle-display) }` with `--nav-toggle-display: none`
in `:root`, and that token is a specificity ladder, not decoration: the button also carries
`button.ghost { display: inline-flex }`, so a `.nav-toggle { display: none }` loses and ships a
visible button into the desktop masthead. The `@media (max-width: 640px)` block reveals it
through the token, hides `nav.tabs`, and tightens the bar's `row-gap` plus the cluster's gap (the
cluster still wraps inside its own row — hiding a control to make the width work is not the deal)
and pads the bar down by
`env(safe-area-inset-top, 0px)` (the block's `padding-block` value restated on the top side so
the two cannot drift), because a real device's top-edge zone — Android Chrome is edge-to-edge
by default, and iOS does the same once the page opts into `viewport-fit=cover` — does not
deliver taps to the page: the first-row button in it is *unreachable*, not unbound, and no
headless emulation has such a zone, so a fully green emulated pass still ships a dead button
(the same symptom brain of reese fixed in production; `env()` resolves to 0 on every non-notch
surface, which render byte-identical, and the dropdown's `top: 100%` follows the taller bar);
it renders the dropdown as the `nav` element itself — since phase 20 task 03 that element is
**one box in two states**: `nav.tabs` carries the geometry (`position: absolute; top: 100%;
z-index: 41`) under the **whole** bar — three rows since task 01, which is the intended look: a
menu below the masthead, not wedged between its rows — *and* the closed state
(`visibility: hidden; opacity: 0; transform: translateY(-8px); pointer-events: none`), while
`nav.tabs.is-open` flips only those four properties to their visible counterparts. That shape is
what the slide needs, not a stylistic choice: the block used to close the menu with
`display: none`, and a box that is not rendered has no intermediate state, so the 180 ms
`opacity`/`transform` transition the operator asked for (`TODO.md:11`, "a slide out and slide
back in", after `brain_of_reese`'s `.app-nav`) could never have run. `visibility` is delayed by
exactly the slide on the closed rule (so the box fades out before leaving the tree) and
undelayed on `.is-open` (or the menu would be invisible for the first 180 ms of its own
opening); `visibility: hidden` + `pointer-events: none` keep the closed menu out of the a11y
tree, out of the tab order and unhittable, exactly as `display: none` did, and the closed state
costs no scroll area (measured). With no second menu element, no `body` marker class, no colour
outside the tokens and **no JS change at all**, both states are named in the file's one
`@media (prefers-reduced-motion: reduce)` block — naming only the closed one would leave the
higher-specificity open transition animating (brain of reese's phase-46 lesson). One binding in
`app.js`'s `initChrome()` owns the state through a `setNavMenu(open)` funnel (`.is-open` on the
nav and `aria-expanded` on the button, always together) with **four close paths**: the button
again, a tab click delegated on the nav, `Esc` with focus returned to the button, and
`matchMedia` turning false on resize back above the breakpoint — which drops a stale `.is-open`
so the next shrink starts closed and `aria-expanded` never claims a menu that is not on screen.
The `if (navToggle && appNav)` guard is the whole `/login` story: the shared script loads there
and does nothing. The `640px` literal lives in two files — the stylesheet block and that media
query — and has to move in both, or the menu opens into a hidden nav. The contract is
pinned on the **served** bytes by three `tests/test_api.py` pins — the button on every page in
`MARKUP_PAGES` (eight since `/gateway` joined the nav) and nowhere on `/login`, and one test that
**compares** the CSS media query with the JS
`matchMedia` instead of grepping for `640px` twice, so a drift fails with both numbers in the
message — and proven in a browser at 360px by `data/logs/nav-task04-verify.cjs` (the `/` +
`/system` interaction, 24 checks) and `data/logs/nav-final-pass.cjs` (the phase-completion
sweep: all seven tabbed pages at 360px and at 1280px, `/login` at both, 45 checks — its desktop
leg now expects the three-row bar's height, not the old fixed 58 px, and still demands inline
tabs, no button and a dropped `.is-open`). Phase 20 task 03 adds three served-CSS pins (the
closed state's `visibility`/`opacity`/`transform`/`pointer-events` and its delayed-`visibility`
transition, the open state's visible counterparts with the delay dropped, and the both-states
reduced-motion rule inside the file's only such block) plus the guard that no `display: none`
comes back to that block, and the browser half is `data/logs/20-task03-slide.cjs` (the computed
closed state, a mid-transition frame at ≈60 ms in both directions, the four close paths with
`aria-expanded`, the snap under emulated `prefers-reduced-motion: reduce`, the untouched desktop
tabs and the no-phantom-scroll measurement; screenshots `data/logs/slide-*.png`). The
safe-area line is pinned in the same test's rule loop; it was verified on the operator's
mobile Chrome rather than in an emulator, because CDP's `safeAreaInsets` override does not
feed `env(safe-area-inset-top)` — the one half of this contract a headless pass cannot prove.
Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
`gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device
series — `/api/metrics` answers with exactly these names. The `sample.source` column is the
dashboard family — `host`, `llm`, `ups`, `disks` for the SMART series
(`disk.temp_c@nvme0n1`, `disk.power_on_hours@…`, `disk.reallocated@…`, `disk.pending@…`,
`disk.uncorrectable@…`, `disk.percentage_used@…`), and `gateway` for the phase-29 connection
gauges (`gateway.in_conns@<listen-port>`, `gateway.out_conns@<host>:<port>` — the port belongs
in the outbound name because two services on one host are two facts and one name would make the
second insert overwrite the first; the outbound gauge is written for an endpoint whose address the
join knows — one the config wrote as an **address**, or (phase 32 task 03) one the gateway's own
`resolver` section published for the name it wrote, under the config's own spelling, so the
name-keyed `gateway.out_conns@<domain>:<port>` series phase 30 task 02 stopped inserting continue
instead of restarting. A name nothing resolved is still never inserted, and its chart simply stops
growing points). Those keep their own family instead of
sitting beside `host`'s `disk.used_pct@/mount` because the cadences differ thirty-fold and
`GET /api/metrics?source=disks` has to answer "what does *this page* have history for".
That scheme is the history API's key — treat it as a public interface.
## The fleet (real environment, verified 2026-10)
| Host | ssh alias | What it gives |
| ----------------- | ------------------ | -------------------------------------------------------------------- |
| `driveripper` | `driveripper-ops` | Threadripper 24c/48t + Radeon AI PRO R9700, vLLM `:8000`, **upsd**, the dashboards |
| `deskwork` | `deskwork-ops` | Strix Halo (Radeon 8060S iGPU), gufo `:8731` |
| `gateway` | `gateway-ops` | router, no GPU, the fleet's ingress: haproxy (`/etc/haproxy/haproxy.cfg`, mode `0644`: six `frontend`s on `:::<port> v4v6` → six `backend`s, one `server <name> <domain>:<port> check inter 5s` each — the file the agent's `haproxy` section parses and `tests/fixtures/gateway_haproxy.cfg` captures) and nginx (`/etc/nginx/nginx.conf` plus `conf.d/http.conf` vhosts on `4443`/`8443` and `stream.d/stream.conf`'s `map $ssl_preread_server_name $targetBackend` → `proxy_pass $targetBackend` on `443` — the tree the agent's `nginx` section parses) |
- **Transport is direct dial: dev is the prod topology, repo-wide (phase 10 task 05).** The
designed shape is one `dashboards-agent` **systemd service per host**, installed by `ops`
through the `deploy/agent-sudoers` grant with `deploy/install-agent.sh`, bound to
the address the dashboard dials — since phase 24 registered **with a certificate bundle**
(`--fetch-bundle`, so its `HOST_TARGETS` URL is `https://` and carries no userinfo); what was
once the token in the URL is the compatibility path for a host nobody has bundled. driveripper's
upsd serves the LAN on 3051 with the host and laptop ranges allowed
by the **firewall**, never by a `upsd.conf` line. The old ssh-tunnel script is deleted and
neither `.env.dev` nor `.env.dev.example` declares a forwarded port — dev dials the same
addresses the deployed dashboard does, and `scripts/capture-fixtures.sh` reads
`HOST_TARGETS` from `.env.dev` instead of naming hosts of its own
(`tests/test_deploy_scripts.py` fails the day any tunnel name or port returns).
**Fleet transport, re-verified from the laptop on 2026-10-19.** The port-selective inter-VLAN
drop phase 10 recorded on the lab gateway `10.2.0.1` is **gone**: `gateway` (`10.1.1.27:9120`)
and `deskwork` (`10.1.0.51:9120`) both take a plain TCP connect and answer `/snapshot`, and
`driveripper`'s upsd answers `LIST UPS` on `10.1.0.126:3051` from this same laptop. All three
hosts now run the **installed** service — `systemctl is-active dashboards-agent` is `active` on
each, with `/usr/local/lib/dashboards-agent/host_agent.py` and a `--token-file` unit bound to
the address the dashboard dials. **`driveripper` joined during phase 13 on 2026-10-19, in two
operator steps.** It went from *no agent at all* (unit `inactive`, nothing listening →
`ConnectionRefused`, an `unreachable` stub row) to **`active`** with
`/usr/local/lib/dashboards-agent/host_agent.py` installed at 13:27, and then its bearer token
went into `HOST_TARGETS`' URL userinfo — so `host:driveripper` and `disks:driveripper` are
**`ok`** and the `live` suite is **11 passed / 0 failed** (it was 9/2 on the missing token that
morning, and 8/3 on the ConnectError before that; the suite has grown since — `uv run pytest -m
live --live` reads **12 passed / 0 failed** on 2026-10-23). The unit reads
`--host 10.1.0.126 --port 9120 --token-file /etc/dashboards-agent.token` (`640
root:dashboards-agent`, the mode `deploy/install-agent.sh` writes) and is still in the `disk`
group (`getent group disk` → `disk:x:6:dashboards-agent`) — but it does **not** run as that
user: gap 10's hand-fix set `User=root`, and re-read from the host on 2026-10-20 the installed
unit is `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` with no device filter.
**Those two extra lines are now template output, and the unit's rewrite is no longer
silent**: phase 17 took the
`DevicePolicy`/`DeviceAllow` lines out of the template (task 01), gave it `--as-root` and
`--caps` (task 02), and made a re-run whose generated unit differs from the installed one print
the diff and exit 1 unless `--force` was passed (task 03), so
`sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force`
reproduces it and a re-run that would quietly replace a hand-fixed unit is now a refusal with a
diff attached rather than a clobber (gap 10).
`/api/health` today reads **all ten sources `ok`**: `host:driveripper`, `host:deskwork`,
`host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `gateway`, `nut`,
`llm:gufo`, `llm:vllm`. The tenth is phase 29's: one job for the ingress host, dialling the same
`https://` agent as `host:gateway` with the same certificate entry, and re-verified live on the
dev server with no change to any pre-existing source.
**The fleet is mTLS since 2026-10-21 (phase 24 task 03), so the bearer-token transport is now
history rather than the designed shape.** `gateway` runs `agent/host_agent.py` **1.3.0** (phase 32
task 05, 2026-10-25, on its existing bundle and an unchanged unit — `data/logs/NOTES-32-rollout.md`)
and `deskwork`/`driveripper` still run **1.1.0**; the spread costs nothing,
because both upgrades were purely additive — the three proxy-config sections and the `resolver`
section are new keys, and a 1.1.0 snapshot simply has **none of them**: no `available` flag, no
error line, nothing for `collectors.ADDON_ERROR_PREFIXES` to keep out of `/system`'s `degraded`, and
`sources/gateway.py` answers that absence as `supported: false` + `WHY_PREDATES` rather than an empty
gateway — so those two move at their next re-stage rather than on a flag day. (The three-way
`available: false` arm belongs to a **1.2.0/1.3.0** host that has no proxy config to read, which is
a different fact from a host whose agent predates the section.)
The controls that prove "purely additive" are `data/logs/verify-28-final.py` and
`data/logs/verify-32-additive.py`: each stashes the whole
working tree, imports both `host_agent.py` files side by side, calls both `collect()`s against
the same `--root`, and demands the newer document equal the older one key-for-key with
`haproxy`/`nginx`/`sockets` (phase 28) and `resolver` (phase 32) added — and the newer one with
`--no-proxy` / `--no-resolve` equal it exactly.
All three are installed from a bundle, every unit carries `--cert-bundle /etc/dashboards-agent.certs` (plus
`RuntimeDirectory=dashboards-agent`, and still its `--token-file` line), and `HOST_TARGETS` is
`"driveripper=https://10.1.0.126:9120 deskwork=https://10.1.0.51:9120
gateway=https://10.1.1.27:9120"` — **no `http://` entry and no userinfo anywhere**: the
certificate is the credential (the pre-flip `.env.dev` is backed up in `data/env-backup/`). `agent.tls: true` + `agent.bundle_rev: 2` in
every payload, `deployed_bundle_rev == bundle_rev` for all three on `/certs`, and
`./scripts/dev-server.sh status` reads the nine sources `ok`. **Phase 28 moved the gateway to
1.2.0 on 2026-10-22 and only the agent file moved.** Task 04 read it read-only first
(`data/logs/verify-28-task04.py` → `28-task04-probe.out`): 1.1.0, `agent.tls: true` + `bundle_rev 2`,
every pre-1.2.0 section answering with live numbers and **no** `haproxy`/`nginx`/`sockets` key —
the control that made the install provably additive. Task 05 then ran the phase-24 procedure on
that one host (re-stage `/home/ops/agent-install/` first with `*-20261022-backup` snapshots, gap 10;
`sudo /home/ops/agent-install/deploy/install-agent.sh --bind 10.1.1.27 --port 9120` with the
arguments read back off `systemctl cat`; `systemctl restart dashboards-agent`) and verified it two
ways: the CA-only `--status` probe answers `version 1.2.0` on `/health` and still refuses
`/snapshot` with `client certificate required`, and the dashboard's own mTLS dial
(`data/logs/verify-28-task05.py` → `28-task05-verify.out`) reads `agent_version 1.2.0` with all
three sections `available: true` (6 frontends all routed / 6 backends, 11 nginx servers and the
`$targetBackend` map, 17 listeners including 443) beside every pre-1.2.0 section intact and
`host:gateway`/`disks:gateway` `ok`. That listener count is a reading, not a fact: the same probe
re-run on 2026-10-22 for this phase's completion pass (`data/logs/verify-28-final.py` →
`28-finalpass-live.out`) answered **19** listeners with the config halves unchanged (6/6, 11
servers) — `/proc/net/tcp` is what a busy gateway writes every second, which is exactly why no
test asserts a row of it. The one fact from that transcript worth carrying to the next
host: the bundle **on disk** — not the `--cert-bundle` flag — is what puts the TLS lines into the
unit render, so an upgrade re-run that omits the flag is the silent path (no guard refusal) and
`systemctl cat` came back byte-identical (`26bd812…`); only `host_agent.py` changed. The per-host order was the safety:
install agent 1.1.0 + bundle → `systemctl restart` → the CA-only `--status` probe answers →
*then* that host's URL flips, so no source was red longer than its own host's restart.
`data/logs/NOTES-24-rollout.md` is the transcript (sha256 of what was staged where, the unit
diffs the re-run guard printed, the before/after SMART tables, the screenshots). Two facts in it
are load-bearing for the next host: the fetch needed an **SSH reverse tunnel** (no monitored host
can route back to a dev box, so `ssh -R 18099:127.0.0.1:8090 <host>` +
`--fetch-bundle http://127.0.0.1:18099` is how registration reaches the dashboard), and
`--status` takes its address from `--bind`, so run bare it dials `127.0.0.1` and reports "no
answer" about a healthy host. **Rolling a host back is not just the URL**: a TLS-mode agent
serves https only, so the `--cert-bundle` line and `/etc/dashboards-agent.certs` both have to go
with it. And the fleet now trusts **this** dashboard's CA — a second deployment with its own
`certs.*` rows cannot dial these agents until it carries the same authority or re-issues and
re-installs every bundle.
Phase 10's proof that the old drop was not this repo's firewall stands as history in
`data/logs/NOTES-verify.md` — do not re-add a tunnel on the strength of it, and re-run the
probe rather than trusting either note.
The per-host record and the exact root commands are in `data/logs/NOTES-agent-install.md`
(its firewall lines are the phase-10 state, not today's), how far the NUT wire reaches from each
network is in `data/logs/NOTES-upsd-acl.md`, and the end-to-end direct-dial verification
(health table, no-tunnel-port proof, live-suite tally, screenshots, and the probes that put
the block on the lab gateway's port allow-list rather than in this app) is in
`data/logs/NOTES-verify.md`, and the phase-11 login verification (the `curl` refusal codes, the
six screenshots, the two `meta` keys the database actually holds, and the operator's
deployment-repo actions) is in `data/logs/NOTES-auth.md`. Those notes are local evidence, not
tracked files: `data/` is gitignored (invariant 10's `data/logs/*.png` included), so a fresh
clone has the conclusions in the prose above and the captures live on the box that ran them —
re-run the probe rather than trusting either.
- Engines are reached directly: gufo `deskwork.reeselink.com:8731`, vLLM `driveripper.reeselink.com:8000`.
- NUT 2.8.5 on driveripper: **configs live in `/etc/ups/`, not `/etc/nut/`**, and **upsd is
on port 3051, not 3493**. Nothing listens on 3493 at all, so a default client reports
*Connection refused* and it looks like a dead service — `ss -ltnH` plus `printf 'LIST UPS\n'
| nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down.
**NUT 2.8 has no `ACL` directive** (this Fedora build is `--without-wrap`; upsd logs
`ACL in upsd.conf is no longer supported - switch to LISTEN` once per line and then ignores
it), so per-source restriction is a **firewall** rule and never an `upsd.conf` line — and
2.8 does not refuse non-local clients the way older docs suggest: `LISTEN 0.0.0.0 3051` +
`LISTEN :: 3051` with `3051/tcp` in the `FedoraServer` zone answers any client routed to it
(reads are unauthenticated; `SET`/`INSTCMD` still need an `upsd.users` account). The host
range `10.1.0.0/21` reaches it, and **since 2026-10-19 so does the laptop**: a raw
`LIST UPS` to `10.1.0.126:3051` from the dev box answers `BEGIN LIST UPS`, which is why
`nut` reads `ok` in `./scripts/dev-server.sh status` with two real UPSes and no
`NUT_SIMULATE`. Phase 10 recorded `10.2.0.1` dropping 3051/3493 for this laptop; that is
history now, and the probes behind both statements are in `data/logs/NOTES-upsd-acl.md`.
Two UPSes are exported over USB and dialled directly as `NUT_HOST` + `NUT_PORT=3051`, and that
reads live from every vantage the dashboard has ever used — the host itself, the deployed
dashboard, and since 2026-10-19 the laptop (`/ups` shows both boxes at 100 % charge, `OL`,
with no `SIMULATED` badge anywhere):
`apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U (that is
`ups.description`, the operator's label — the driver's own `identity.model` answers
`GX1500U`, so the description is what identifies the box).
Real-hardware quirks the parsers already handle: `battery.mfr.date` is the useful date
(`battery.date` is often an unset placeholder — `2001/09/25`, slashes and all, on the APC),
CyberPower reports `battery.mfr.date: CPS` (not a date → `unknown`, never a fake age),
**neither driver reports a battery temperature or the NUT 2.8 `battery.charger.status`**
(so the battery glyph's state has to come from `on_battery`, and `charger_status` must stay
optional), and **`ups.test.result` is where the two boxes disagree**: the APC still answers
the factory `No test initiated` while the CyberPower answers `Done and passed` (a test has
run on it since the phase-05 capture; both words re-read from the wire on 2026-10-19 and
pinned as `LIVE_SELF_TEST_WORDS` in `tests/test_nut.py`, which fails if the fixture and the
capture ever drift).
Both UPSes are recorded verbatim (`LIST UPS`/`LIST VAR`/`LIST CMD`, serials scrubbed) as
`tests/fixtures/nut_<ups>.vars` by `./scripts/capture-fixtures.sh`, and the wire quotes a
field only when it must — `VAR apc ups.status "OL"`, not three quoted fields.
- Host access is via ssh keys as `ops`, and **passwordless sudo is different on every host** —
that is what decides how much of the fleet an install can finish unattended. `gateway` grants
`(ALL) NOPASSWD: ALL`, which is why its agent installed in phase 10 with no
`deploy/agent-sudoers` line present; `driveripper` grants a command-scoped set (`smartctl`,
`upsc`, `udevadm`, the `/etc/ups` reads, and the `tee /etc/ups/upsd.users|upsd.conf` +
`systemctl restart|reload nut-server` pair documented in `deploy/nut-sudoers`); `deskwork`
grants an unrelated set (`btrfs scrub`, `snapper`, `setfacl /srv/*`, `osbuild-auto.sh`).
Neither `deskwork` nor `driveripper` grants `visudo`, so a sudoers change is the operator's
action. **`deskwork` and `driveripper` both grant the installer** —
`(root) NOPASSWD: /home/ops/agent-install/deploy/install-agent.sh` — and both also grant
`(root) NOPASSWD: /usr/bin/systemctl restart *`. The earlier record
here said deskwork's `ops` had no installer grant and that its install went through the
operator's root; that is out of date — `sudo -n -l` on 2026-10-21 lists the grant, and phase 24
task 03 installed deskwork's bundle as `ops`. The grant names a **path**, and until phase 24
that path held an Oct-18 snapshot of the script which carried neither the later flags, nor the
filter removal, nor the re-run guard — running it would have replaced driveripper's `User=root`
+ `CapabilityBoundingSet` unit with the snapshot's capability-less one, **silently**. That trap
is now closed on all three hosts: `/home/ops/agent-install/` is `ops`-owned, so the repo's
`deploy/install-agent.sh` (sha256 `ffbcd91f…`) and `agent/host_agent.py` (**1.1.0**, re-staged as
**1.2.0** on `gateway` in phase 28 and **1.3.0** in phase 32) were staged over it without root, and
each snapshot was left
beside the grant as
`install-agent.sh.oct18-backup`, which sudo will not run. **Re-staging is therefore the first
step of any install, not an optional one** — the grant is only as current as the file it points
at. The `systemctl restart *` grant on those two hosts is the other half of an unattended
install: a bundle changes the unit, and the unit only takes effect on a restart that
`systemctl enable --now` will not perform.
The template now writes the live unit's shape: no device filter at all
(phase 17 task 01 — systemd's default `auto`, with `PrivateDevices=false`) and `--as-root`
plus `--caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` for the `User=root` and
`CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` lines (task 02). The old allow-list's
missing nodes are no longer a concern for the next install: with no filter, `/dev/ng*n*` and
`/dev/bsg/*` are reachable the same way `/dev/nvme*` and `/dev/sd*` are, subject to DAC and
SELinux.
- **SMART: real captures, and one live host finally reads them.** `smartctl` 7.5 is installed on
deskwork and driveripper. The `ops` account (which is who runs a hand-started dev agent) is in
no `disk` group, driveripper grants it `(root) NOPASSWD: /usr/sbin/smartctl*` and deskwork
grants nothing, and `host_agent.py` shells the binary **as itself** — it never calls sudo — so
an agent run as `ops` still answers *Permission denied* (exit 2) and reports `smart: null`.
What changed by 2026-10-19 is that `deskwork`'s **installed** service runs as
`dashboards-agent`, which is in the `disk` group (`getent group disk` → `disk:dashboards-agent`)
on an agent installed after the SMART pass: `/disks` shows real numbers there —
`smartctl_available: true`, `nvme0n1` healthy at 55 °C that day — amber, past `disk.temp_c`'s
50 °C threshold, and a plain 48 °C `ok` on the 2026-10-20 re-read — 19 166 power-on hours (an
odometer, so that figure is a capture date, not a fact about the drive), 11 % of rated
wear — so the wear bar and the alarm chips are drawn from live hardware, not only fixtures.
`gateway` answers `smartctl_available: false` (no smartmontools in its VM: the *install it*
half of the three-way flag — and, since phase 22, the arm that keeps it off the `/disks` page
entirely, so `/api/disks/snapshot` there lists deskwork + driveripper and `fleet.hosts_total`
is 2 while `host:gateway`/`disks:gateway` both stay `ok` — re-verified live read-only on
2026-10-21, where the two cards still draw their pre-phase numbers (`deskwork nvme0n1` 48 °C /
19 201 h / 11 %, `driveripper` 16 drives with the `nvme5n1` `13 uncorrectable` chip, `/`
unchanged at three fleet rows; `data/logs/NOTES-22-disks-live.md`), and **`driveripper` is the fourth
case the flag was built for:
`smartctl_available: true` with almost no numbers.** Its agent answers (16 whole-disk rows,
`nvme0n1`…`nvme7n1` + `sda`…`sdh`) and `smart` is non-null for the 8 NVMes, but every one of
them carries only identity — `health`, `temp_c`, `power_on_hours`, `percentage_used` and the
counters are all `null` — while the 8 spinning drives come back `smart: null` each with the
agent's own line `smart: sda: Try an additional '-d ata' or '-d sat' argument.` Pinned on
2026-10-19 with read-only probes (`data/logs/NOTES-driveripper-agent.md`): the `disk`-group
**open of the block nodes works** (agent user, `O_RDWR`), there is no SELinux denier (zero
AVCs with `auditd` running), and the install's `DevicePolicy=strict`/`DeviceAllow` lines were
removed from the unit (they matched nothing useful anyway — systemd resolves `DeviceAllow`
globs by name against `/proc/devices`, where this kernel lists NVMe as `blkext`, so
`nvme*` matches nothing). What locks the agent out is **kernel 7.2's unprivileged
passthrough gate**: NVMe admin via the block node is unprivileged-only for five identify
types; the SMART-log read needs the drive's ACEL to mark it supported (the Samsung 980
PRO's 2020-era `5B2QGXA7` does not — `NVME_IOCTL_ADMIN_CMD: Permission denied`) or
`CAP_SYS_ADMIN`, and SATA SMART behind a SAT layer is ATA pass-through, which the
unprivileged `SG_IO` allow-list (`scsi_cmd_allowed()`: read-safe opcodes only) does not
include — explicit `-d sat`/`-d ata` fails identically, so no `host_agent.smart()` retry
form can reach it. `deskwork` reads live with the same user/unit because its drive's
firmware does advertise the page. So `/disks` shows driveripper's drive inventory with
`n/a` values and a why-line — the designed degradation, **resolved the same day (gap 10):
the unit now runs `User=root` with
`CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`, and all 16 drives read live** (the
980 PRO's 59 °C / 6328 h / 14 % row, `nvme1n1` at 58 % wear, the red `13 uncorrectable`
alarm chip on `nvme5n1`; verified in `data/logs/disks-driveripper-live.png`). A root
process with an **empty** capability bounding set is still capability-less (`CapEff: 0`)
because the kernel gates check capabilities, not uid — the named bounding set is what
unblocked it.
**Real data is in the repo now:** `tests/fixtures/smartctl_driveripper_<dev>.txt` is all 16
drives (8 NVMe, 8 spinning) read through that sudo rule, and `tests/test_agent.py` parses
every one — which is what found the parser's one real bug: this binary annotates the
attribute table's RAW_VALUE column (`35 (Min/Max 34/44)` on every Seagate, and one Western
Digital writes its hours as `43894h+43m+32.519s`), so reading the *end* of the row lost the
temperature of all eight HDDs and one power-on count. The captures are also the first
honest look at this fleet's drives: `nvme5n1` reports 13 media-and-data-integrity errors and
`nvme1n1` is at 58 % of its rated wear, so the alarm chip and the amber wear bar on `/disks`
are real features with real data behind them, not demo furniture.
**Lighting up the rest of it was the same two operator actions that travel together** — install
the agent on `driveripper`, and give its agent user device read access: the `disk` group
(`sudo usermod -aG disk <agent-user>`, which is what `deploy/install-agent.sh` does on install),
*not* a sudo rule, because the agent must read the device itself — on an `agent/host_agent.py`
from phase 09 or later (never redeploy without the grant: 16 permission-denied lines would age
nothing but are pure noise, while with the grant they are 16 drives' worth of temperature,
hours and sector counters). **The agent and the `disk` group are both in place on
`driveripper`** — the `disk` group alone is not enough on kernel 7.2 (the passthrough gate, see above), so the unit additionally runs as root with the two gate capabilities bounded (gap 10) — all 16 drives now read live. `gateway` needs smartmontools first and will still read no SMART
after that, being a VM with no physical disk behind `vda` — and it is the host phase 22's
visibility gate exists for: its `smartctl_available: false` means `/disks` draws **no gateway
card at all** and `/api/disks/snapshot` lists two hosts, not three — while its `disks:gateway`
job still polls, still reads `ok`, and its transitions still reach that page's drive-events
feed, because the gate hides a host row and not the job's history. That is a different
designed degradation from a host that cannot *read* its drives for any other reason (a denied
device, `--no-smart`, an agent predating the pass): that one keeps its dimmed cards, every
number `n/a`, and the one-line why — **no host here is in that `None` arm today** (all three
agents post-date the SMART pass), so the arm that stays visible is proven by
`tests/test_api.py`'s pins and by nothing on the fleet.
`tests/fixtures/smartctl_reference_*.txt` stay as what they are: **reference shapes** (labels
verbatim from the deployed binary, values synthetic), kept beside the real captures so the
synthetic edge cases stay pinned. Re-record the real ones with
`SMART_HOSTS="driveripper-ops" ./scripts/capture-fixtures.sh`: it tries plain `smartctl`
first (so a fixture can never read healthier than the agent does), falls back to `sudo -n`,
refuses to overwrite good fixtures with unreadable output, and scrubs
Serial/WWN/EUI-64 (`test_smartctl_captures_carry_no_unscrubbed_serials` fails the day it
stops).
## Commands
```bash
uv sync --python 3.14 # deps + editable install
cp .env.example .env # production-shaped config (DASH_ADMIN_PASSWORD seeds
# the login on the first boot of a database — with no
# seed and no stored verifier the server exits non-zero)
cp .env.dev.example .env.dev # dev-only: the real hosts (this fleet's URLs are
# `https://` with no userinfo — the bundle is the cred)
set -a; . ./.env.dev; set +a # dev dials the fleet the prod way, no tunnels
./scripts/dev-server.sh restart # keep http://127.0.0.1:8090 up; sign in with the
# DASH_ADMIN_PASSWORD from .env.dev (`testpass`), then
# export every issued HOST_TARGETS bundle to data/certs/
./scripts/dev-server.sh status # per-source ok/DOWN — unauthenticated: /api/health
./scripts/dev-server.sh bundles # what that export left: names, sizes, modes
./scripts/dev-server.sh logs
uv run python scripts/dev-certs-export.py --url http://127.0.0.1:8090 \
--password "$DASH_ADMIN_PASSWORD" --names <the HOST_TARGETS names>
# the export on its own (phase 24): reads
# GET /api/certs/<name>/bundle and writes
# data/certs/<name>.bundle at 0600 — mints nothing
uv run pytest # hermetic: 1796 passed + 16 skipped (12 live,
# 4 store_perf), no network
uv run pytest --cov # coverage gate (98.9%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
uv run pytest -m store_perf --store-perf # opt-in: the store hot-path timings over the seeded
# store, priced against `store_perf.BUDGETS` (the cached
# reads report cold *and* warm, so a before/after never
# books a cache hit as a speed-up)
uv run python scripts/watch-store-perf.py
# ... and the live proof: 25 signed-in minutes against the
# running dev server, timing the page paths while counting
# `store-error` rows and the WAL size (DASH_WATCH_SECONDS
# shortens it; writes store-perf-watch.{log,tsv,json})
uv run python scripts/watch-llm-window.py
# ... and the phase-23 proof: signed-in reads of
# /api/llm/snapshot at the app's own cadence, judging every
# slot start/end against "settled within two polls", the
# Δ<window> labels and the 1 m rollups — read-only, it
# starts no workload (DASH_LLM_WATCH_SECONDS shortens it;
# DASH_LLM_WATCH_REPLAY=<tsv> re-derives the verdict from a
# captured pass over the same samples, sending nothing)
uv run python scripts/store_perf_probe.py # ... as a report (data/logs/store-perf-baseline.{md,json},
# then store-perf-task03.* for the write side and
# store-perf-task04.* for the budget table);
# --rebuild re-seeds, --dev-db <file> also times a real
# store read-only, DASH_PERF_STORE_DIR moves the cache, and
# every pass that *writes* runs on a scratch copy so the
# seeded baseline never moves under the next measurement
# the bundle for one host: the stored bytes (that host's private key included) and
# `install-agent.sh --cert-bundle`'s only input — signed-in GET, see README's rollout
# curl -s -H "Authorization: Bearer $SESSION" -o /tmp/agent-<host>.bundle \
# http://127.0.0.1:8090/api/certs/<host>/bundle
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh # re-record ground truth (agents + UPS + gufo /slots)
SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh
# ... plus smartctl per physical disk (needs device read
# access: plain smartctl, else `sudo -n`; scrubs serials)
./scripts/capture-gateway-config.sh # ... plus the gateway's proxy tree, socket tables and
# resolved targets: haproxy.cfg, nginx.conf + its glob
# includes, /proc/net/tcp{,6}, and gateway_resolver.json
# (the names `target_names()` derives from those configs,
# read back with `getent ahostsv4/6` — no DNS client, no
# sudo) over $GATEWAY_SSH (gateway-ops); a failed or empty
# read never replaces a good fixture, and a vantage where
# not one name answers is refused rather than recorded
bash .agents/validate.sh # the phase gate: the four checks above (pytest
# --cov --cov-fail-under=94, ruff, pyright, plus a
# node --check of web/static) + repo hygiene.
# Always `uv run` — a bare `python3`/`ruff` on PATH
# tests an interpreter nobody deploys.
```
Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on `0.0.0.0:8090`,
SQLite under `/var/lib/dashboards` — put a volume there) and `deploy/install-agent.sh`
per monitored host (systemd unit, own user, `--bind`/`--port`/`--token`/`--rotate-token`/
`--cert-bundle`/`--fetch-bundle`/`--cert-name`/`--session`/`--as-root`/`--caps`/`--force`/
`--status`/`--uninstall`; it stores the token
`0640 root:dashboards-agent` — the
unit hands the agent the *path* and the agent opens it as its own user, so that user must be able
to read it and no other account may, and a `--cert-bundle` file is installed to
`/etc/dashboards-agent.certs` at that same mode after a shape check that runs before anything is
touched (with a local file this script dials nothing; the fetched path is the one dial, below), so
0600 would fail the unit exactly like it fails the token file; the task file's `0600` is the
intent — *root and the agent's own user only* — and 0640 is the mode that achieves it) — never
puts a token on argv, and runs under `ProtectSystem=strict` with **no device
filter** — SMART device access is DAC via the `disk` group + SELinux, because kernel 7.2 matches
a device-filter glob by name against `/proc/devices` (where NVMe is `blkext`), so the old
allow-list denied the whole namespace; `PrivateDevices=false` stays explicit because a private
`/dev` with no device list kills the unit at spawn (`status=208/STDIN`). Where DAC is not enough,
`--as-root` writes `User=root` and `--caps CAP1,CAP2` writes that `CapabilityBoundingSet` line
(space-separated, as systemd spells it): kernel 7.2's unprivileged-passthrough gate checks
capabilities rather than uid, so `--as-root` alone is capability-less (`CapEff 0`) and prints a
note saying so, while `--caps` without `--as-root` is refused with exit 1 before `need_root` and
before any write. Both default off, and neither touches the token file's mode or the `--status`
probe. A bundle-installed
unit also gets `--cert-bundle` plus `RuntimeDirectory=dashboards-agent` (mode 0700 — the bundle unpacks its
PEM blocks there and the agent deletes the key once the SSL context holds it), keeps the
`--token-file` line for a host-by-host rollout, and `--status` then probes `https` on the
bundle's CA with **no** client certificate via a `python3` `ssl` one-liner — task 02's
`CERT_OPTIONAL` + `/health` exemption is what makes that probeable, and there is no external TLS
binary. A re-run without the flag leaves an installed bundle in place, and `--uninstall` removes
it with the token — and neither is guarded, because both are deliberate. **Registration can fetch
that bundle** (phase 24 task 02): `--fetch-bundle <dashboard url> --cert-name <HOST_TARGETS name>
--session <signed-in token>` dials `GET /api/certs/<name>/bundle` with `python3` `urllib` (no
`curl` assumption on a monitored host) in the install run and hands the bytes to that same
`--cert-bundle` path, so the shape check, the `0640` install, the unit's TLS lines and the
`--status` probe behave identically for a fetched and a downloaded bundle. The three flags are
all-or-nothing, are refused together with a local `--cert-bundle` and with `--status`/`--uninstall`,
and the fetch plus the shape check run **before** `need_root`: a 401, an unissued name, a proxy's
200-with-HTML or a dashboard that never answers is one line naming the status and no partial
install. The session is a **fetch credential only** — it travels to python in the environment (argv
is world-readable through `ps`, which is why the agent's token is a file), is `unset` as soon as
the bytes are in hand, appears in no written artefact and not even in the sudoers hint's re-quoted
command, and the fetched temp file is removed by an EXIT trap on every path. What is guarded is the
unit itself (phase 17 task 03): it is rendered to a temp file first, and if an installed `$UNIT`
differs the run prints `refusing to overwrite …` plus the `diff -u` on stdout and exits 1 before
the unit write, the `daemon-reload` and the `enable`, so a redeploy cannot undo a hand-fix and an
identical re-run stays the silent zero-noise path. `--force` prints the same diff and applies it —
which is what the TLS rollout's `--cert-bundle` step needs on a token-mode host, because that step
*is* a unit change. What it applies is the **file**: `systemctl enable --now` only *starts* a unit,
so a run that rewrote `ExecStart` leaves the already-running agent on its old settings until
`systemctl restart dashboards-agent` — and `--status` chooses http from https by reading the unit
file, so probing before that restart reports "no answer" for a change that landed correctly (the
rollout in README restarts for exactly this reason). It takes its **address** from the flags rather
than the unit, so a bare `--status` probes `127.0.0.1:9120` and answers "no answer" about an agent
bound to a LAN address that is healthy — always pass the same `--bind`/`--port` the install did. No root shell? The one sudoers
line that lets `ops` run the installer is `deploy/agent-sudoers`; run the script without root and
it prints that line with its own resolved path.)
The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other
service here. The source itself is `git@git.reeseapps.com:services/dashboards.git` (`origin`,
branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only gate.
## Testing policy
**Real captures for shape, fakes for behaviour, `live` for drift.**
- Anything that must match an upstream — `/proc` layouts, vLLM series names, NUT framing —
is asserted against a verbatim capture in `tests/fixtures/`. If the schema drifts, a
fixture test fails before a dashboard does. `smartctl_driveripper_<dev>.txt` is the whole
16-drive fleet read through a `smartctl` sudo rule (Serial/WWN/EUI-64 scrubbed), and
`test_the_real_drives_answer_every_number_the_disks_page_prints` asserts every number
`/disks` prints comes out of real drive output — that test is what caught the parser
reading the end of an attribute row instead of its annotated RAW_VALUE column.
- **A join is pinned by the rules it applies and the one question it may never ask** (phase 29,
simplified by phase 30 task 02). `tests/test_gateway_source.py` runs the captured gateway configs
and the recorded `/proc/net/tcp{,6}` tables through the **real** `host_agent` parsers, so the join
is compared with the agent's document and not with a copy of its field names, and then demands the
two attributions and nothing beyond them: a socket whose **local** port names a listener is
inbound, a socket whose remote `(address, port)` equals a configured target is outbound, and
**every other socket is ignored** — the capture's `established` and the two totals are therefore
both printed and deliberately never reconciled, because the identity this suite used to demand
(in + out + unmatched = the table) was the arithmetic of a guessing rule, and because most of a
busy router's sockets are nobody's pipe, which is why **`live_out` is not the bandwidth**. The
rules that must never come back are pinned as absences: no `by_port`/`partial`/`unmatched`
vocabulary in the payload or on the page, no `0` for a pipe the join could not count, and
`socket.getaddrinfo` spied so the day the join resolves a name the run fails — a name is never
credited with a connection (rule 2).
**Config-derived counts are literals in these tests; socket-derived ones are computed from the
same capture.** `tests/test_api.py` asserts 10 listeners, 33 connections and 18 `map_targets`
as numbers because only the operator moves those, and derives `live_in`, `live_out`, each pipe's
count and `established` from `gateway_snapshot("full")` because a second capture of a running
router lawfully changes them — pinning the live totals as literals is how a re-capture turns a
green suite red over nothing. The rule is the assertion; the reading is not.
Which is the same discipline that closed the drift phase 32 task 02 folded in: that capture's
`stream.conf` gained a `map` entry (a domain was added to the router after phase 30 recorded it),
so 17 became 18 and 32 connections became 33 **in the commit that re-ran
`./scripts/capture-gateway-config.sh`** — re-tallying the config-derived counts, never editing a
test to match a live reading while the fixture stays where it was.
- Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host
rebooting mid-poll, a counter reset — is a fake in `tests/conftest.py`.
- **A verdict the operator sees is pinned on the surface that shows it, in both directions.**
`tests/test_nut.py` grades the word; `tests/test_api.py`'s `self_test_app` harness holds the
`nut` job's own payload (its collector is swapped, so a poll cannot overwrite what the test
mutated) and rewrites the `Last self-test` check in the object `/api/ups/snapshot` serves:
a pass (`Done and passed`) and a never-tested pack produce **no** `/api/summary` alert row,
a fail produces exactly one `fail` row per UPS linking to `/ups`. The two directions ship
together on purpose — an absence-only assertion is satisfied by deleting the check, which is
not the fix the report asked for (phase 12). The same file pins that the `/ups` check list
needs **no** markup for a new verdict: `<li class="${check.status}">` plus
`ul.checks li.<verdict> .mark` in `dash.css` is the whole render path, and no page asset or
stylesheet rule names one check (a second opinion about the driver's word is the bug).
- **A measurement window is pinned from both sources, through one contract module** (phase 23).
`tests/rate_window_contract.py` names the two Δ windows (`HOST_WINDOW` = `RATE_WINDOW_SECONDS`,
`LLM_WINDOW` = `LLM_INTERVAL_SECONDS`), the `delta_label()` rounding the pages use to print
`Δ2.0s` / `Δ10.0s`, and `assert_window()` — the estimator's own bounds (`MIN_WINDOW_SECONDS` up
to `2 × span`, because a span equal to the cadence means the base comes from the window-empty
fallback and a real window is the poll gap: 2.009-2.026 s live on `/llm`) **plus** the printed
label, so raising a cadence knob moves the window and the pin together instead of rotting it.
Four files read it: `test_api.py` spies the estimator object each source's jobs receive and pins that
`/api/config`'s LLM window is the LLM's own span; `test_llm_source.py` walks both throughput
cells of **both** engines at 2 s and re-runs the phase-13 floor refusal through the production
parse path at that span; `test_host_source.py` keeps the host payload's `window_seconds` at 10 s
and shows the same 10 s cadence starved to `n/a` by a 2 s span (why one estimator cannot serve
both); `test_rates.py` runs the floor, the `2 × span` ceiling and the 8× jump gate at both
spans, because those are properties of the estimator and not of the window size. A pin naming
one source's window is what this prevents — the two payloads are the same shape and only the
number differs, so half the pin is how a quiet regression would look.
- **Page payloads that a rewrite must not break are pinned as a table.**
`tests/card_contract.py` holds the dotted `/api/system/snapshot` paths the `/system` hero
cards read (`CARD_METRICS`, `GPU_METRICS`, `HOST_SHAPES`); `tests/test_system_contract.py`
asserts it against the three captures (discrete GPU, APU, no GPU) and over HTTP, and
`test_every_host_serves_the_hero_card_contract` asserts the same table against the live
agents — so a rename fails a test, and a card that shows `n/a` cannot be "fixed" by
inventing a field. `network` carries no `severity` on purpose (throughput has no good/bad)
and neither does a GPU row (phase 06: a busy GPU is the job on an LLM server) — the
contract pins both absences. `tests/ups_contract.py` is the same device for the `/ups`
hero row and battery glyph (`HERO_FIELDS`, `UPS_SHAPES`, `assert_ups_hero_contract`):
`tests/test_ups_contract.py` drives it from `tests/fixtures/nut_<ups>.vars` — verbatim
`LIST UPS`/`LIST VAR`/`LIST CMD` replies from the two real UPSes — and
`test_ups_hero_fields_are_present_on_the_real_batteries` drives it against live upsd. The
captures are what make the driver disagreements testable: one of them answers
`battery.mfr.date: CPS`, neither exports `battery.temperature` or
`battery.charger.status`. `tests/slots_contract.py` is the same device for gufo's `/slots`
rows, which the `/llm` slot grid (phase 07) reads (`assert_slot_shape`, required/optional
field tables, duplicate slot ids, the `next_token[0]` entries): hermetically
from `tests/fixtures/slots_gufo.json` — the verbatim `/slots` reply, idle slots and all,
captured whenever `GUFO_BASE_URL` is set — and live via
`test_gufo_slots_answer_the_pinned_shape`. That fixture exists because its absence is how
the deleted slot table rendered dashes forever: its field names (`slot.state`,
`slot.n_prompt`, `slot.decode_per_second`) were a memory of an upstream, not a capture.
**The contract names one canonical key per fact and the grid renders that one**: the capture
answers `id_task` *and* `task_id` for the same request, so `CANONICAL_TASK_ID_FIELD` decides
and the alias is captured but rendered by nothing — a rename fails a test instead of becoming
a silent `a ?? b` chain in the page. `tests/test_api.py` reads the served `SLOT_FIELDS` table
and resolves every key it names against the capture, in both directions.
A busy slot's `prompt` is the user's in-flight request text, so the capture script blanks
it and the page never reads it.
- `live` (`tests/test_live_fleet.py`) re-runs the production code paths against the real
fleet, engines, and upsd. Skipped unless `--live`; each test skips when its source is
unset, so partial fleet availability is normal. **Live tests must not write** — a battery
test on hardware something is plugged into is not a test.
**An `https://` agent is dialled with the stored authority, through `fetch_snapshot` and never a
hand-rolled `client.get`** (phase 24 task 03). The module-scoped `agent_tls` fixture reads the
`certs.*` rows out of the same SQLite file the app uses — through a read-only `sqlite3` URI whose
`set_meta`/`delete_meta` **raise**, because "live tests do not write" includes not minting or
repairing an authority from a test — and hands `collectors.agent_tls_map` the material
`certs.write_material` laid down, exactly as `app.build_hub` does. The failure this removes is the
misleading kind: a raw `get` against a healthy mTLS agent dies with `CERTIFICATE_VERIFY_FAILED:
self-signed certificate in certificate chain`, which reads like a dead host and is really a test
that forgot to present a certificate. No store file, or no authority in it, skips — the same
partial-environment rule as an unreachable host.
- The agent's real-`/proc` test (`tests/test_agent.py`) touches only the box running pytest,
so it stays in the default run and is *not* marked `live`.
- `tests/auth_contract.py` is why the HTTP suite can start a server at all: every app in a test
is seeded with `admin_password="testpass"` (the local-testing password — a server with no
stored verifier and no seed refuses to start, so an app under test needs a credential exactly
like a deployment does) and hashed at `pbkdf2_iterations=1000`, because the real 600 000 costs
~0.3 s per client and the suite starts a hundred-odd of them. The real figure stays pinned in
`test_auth.py` and `test_config.py`, so the cheap cost here cannot become the default there.
- **The perimeter is its own file, and it is paranoid in two specific ways.**
`tests/auth_client.py::sign_in(client)` is the one way a test gets a session: it posts
`testpass`, asserts the sign-in worked (so a suite cannot silently start from a 401 and fail
three lines later on something that looks like an unrelated bug), and puts the returned CSRF
token on the client's default headers — which is what let the ~200 pre-existing HTTP assertions
in `tests/test_api.py` stay untouched, asserting UPS behaviour rather than the new gate.
`tests/test_auth_api.py` then runs the gate through a real ASGI request with
`follow_redirects=False` on every anonymous call: `TestClient` follows redirects by default, so
an unsigned `GET /api/config` would fetch the login page and report `200` — the way a gated
suite ends up passing while testing nothing. Two of its tests are the ones to keep if you keep
none: `test_every_route_that_is_not_public_requires_a_session` walks the **served** route table
(`walk_routes` flattens pages, `include_router` wrappers and the `/static` mount, and raises on
anything it does not recognise, so the audit cannot silently shrink) and demands `401`/`303` for
every path outside `PUBLIC_PATHS`, GET and POST — the test that protects the next phase rather
than this one; and the UPS pair prove the CSRF refusal on **the wire** (a scripted
`nut.NutClient` records nothing for a tokenless write and does record it once the header is
there, and `driver.killpower` still needs `DANGER` behind the session), because 403 after
`instcmd` would be a power cut with an error message attached.
- **The setup surface has its own pins, because prose rots quietly.**
`tests/test_auth_docs.py` reads the artefacts an operator actually holds when a dashboard will
not come up: both tracked env templates (every `DASH_*` auth setting declared; every
secret-shaped *assignment* a placeholder — the same allow-list `.agents/validate.sh` 2b uses,
so `uv run pytest` catches a committed password on the working tree and not only in the gate;
the local-testing word allowed in prose and forbidden as a value), `scripts/dev-server.sh` (its
probe and its `status` ride `/api/health`, no `dash_session`/`/api/auth/`/`Authorization:`
anywhere, the password named but never assigned), and `README.md`/`AGENTS.md` (the *Signing in*
section before *Configuration*, the four public entries, the reset `DELETE FROM meta …` command
verbatim, the four config rows, the `DASH_COOKIE_SECURE=1` deployment note, the three auth
gaps). No server starts and no request is made: a doc that stops naming the reset command is a
locked-out admin, and that is worth a test even though it asserts nothing about Python.
- **A module that holds a private key is pinned structurally, not only behaviourally**
(phase 15). `tests/test_certs.py` reads `src/dashboards/certs.py` as text and fails if it grows a
`logging` import, a raise carrying an interpolated value, or an import from the rest of the
package — the same three pins `tests/test_auth.py` holds over `auth.py` — because this app turns
log lines and error strings into `event` rows and `/api/health` fields that a browser reads, and a
stored PEM is a key. Its behavioural half runs the whole lifecycle over a dict `MetaStore` (five
rows on first boot, no byte changed on the second, rotate/revoke/root-rotate and their denylists,
a bundle that carries the agent key and never the root key), and one test completes a real
client-certificate handshake through the material `write_material` lays down. The other two halves
run real code rather than a model: `tests/test_agent.py` drives the real `host_agent` handler over
a TLS-wrapped loopback port (the handshake matrix, the live denylist re-read, token mode unchanged)
and `tests/test_deploy_scripts.py` runs `install-agent.sh` in a sandbox whose `install`/`chown` are
faked, which is how a `0640` mode, a byte-identical bundle and the https `--status` probe are proven
without a root shell or a host. **A `0600` in a task file is not a fact about this repo**: the agent
opens its own bundle and re-reads it for the denylist, so the group bit is load-bearing.
- **A bundle source is validated once, whichever source it is** (phase 24 task 02). The
installer's `--fetch-bundle` is tested through the same `served_dashboard` socket as the export,
because the claim is the same claim: the file a host installs is the route's bytes. The pins that
go with it are the ones a second install path would break — byte-identical `0640` install with
the unit's TLS lines; `issued: false`, a 401 and a 200-with-HTML each refused with their own
sentence, no stub call logged and no file written (the stub log is the proof the refusal came
*before* the root boundary); a temp `TMPDIR` left empty on the success path and on every refusal
path, including the one where the shape check is what stopped the run; the three flags
all-or-nothing and refused beside a local `--cert-bundle`, beside `--status`/`--uninstall`; and
one label-aware validator, pinned by demanding that the local path still reads `--cert-bundle …`
and never names the flag that was not used.
- **A script that is an HTTP client is tested over a real socket** (phase 24). The dev bundle
exporter is run as a subprocess against a real `create_app(...)` served by uvicorn on
`127.0.0.1:0` in a thread, not through `TestClient`: what has to be proven is that it signs in
through the public route, reads the session token back out of a `Set-Cookie` header, and sends it
as `Authorization: Bearer` — none of which a transport-level test client has to serialise. The
pins that go with it are the ones that keep the artifact honest: the file byte-identical to the
route's bytes, `0600` (including the re-`chmod` of a file somebody loosened), no file and one
note for `issued: false` and for `configured: false`, exit 2 with nothing written for a refused
sign-in, **no mint** (the serials, revisions and root are the same after the run), and no PEM,
token or password in the output. The dev-server composition is pinned on the served text too —
export after the readiness probe, failure as `|| echo` and no `return 1` — because "it runs
later, and never fails `start`" is the part a refactor loses.
- **A live measurement is re-derivable, so a verdict can be re-checked without re-watching**
(phase 23). `scripts/watch-llm-window.py` writes the series it judged (value, the `Δ<window>`
the payload carried, the engine's own busy count, the raw token counter) beside its verdict, and
`DASH_LLM_WATCH_REPLAY=<tsv>` re-computes that verdict from the file with no traffic sent. The
pass line is read on the same samples re-diffed over the estimator's own span *and* on the
published value, because the phase-13 jump gate can legitimately hold a correct number for a
poll — judging only the published column would call a designed hold a ramp.
- **A cost claim is pinned as a shape and a control, never as a wall clock** (phase 14). The default
run keeps the seeded-store tests hermetic and asserts **shape and determinism** — counts,
retention windows, the statements a plan is taken from, idempotence — while the timings live behind
`-m store_perf --store-perf` and in `scripts/store_perf_probe.py`, because a budget asserted on
first writing is a number fitted to the machine it was written on. Two rules came out of it.
(1) A cheaper query is proven *equal*, not just faster:
`test_windowed_rollups_answer_exactly_what_one_full_window_rollup_answers` compares every stored
row of a windowed store against a second store rolled the old way, and
`test_a_re_rolled_bucket_only_ever_gains_rows` pins the one bug that equivalence alone misses (a
window floor that cuts into a bucket re-rolls it from its tail rows). (2) A race that comes back
clean has to be shown capable of going dirty: `store_perf` runs the pre-fix whole-window
`maintain()` as a **control** beside the windowed one and `tests/test_store.py` fails if the control
stops contending, because "0 `database is locked`" from a race that never met is the invariant-4
failure mode with a green checkmark attached. (3) **A budget is a table, not a test body**
(task 04): the 14 ceilings live in `store_perf.BUDGETS` with the measured figure and the margin in
each row's `basis`, the probe renders that table, and two default-run tests fail if a budget stops
being priced or a ceiling contradicts its own ratio partner — a ceiling nobody measures is a
comment. (4) A cache's cost claim needs the *write* cadence in the measurement: task 02's series
cache reported 0.0 ms on a quiet seeded store and served almost nothing on a live one, because
every sample write invalidated it. The pinned version is statement-counting, not timing — a poll of
existing series issues no `SELECT`, a write of a name nobody wrote ends the cache — and the live
version is `scripts/watch-store-perf.py`, which is where the defect showed up.
- Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces
94% (`--cov-fail-under`), and ~99% is where we are; keep it there. New/modified Python
still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB,
never a real target, and never a real upstream: `/api/ups/command`'s send is proven against
a scripted `nut.NutClient` (`ScriptedNutClient` in `tests/test_api.py`) because the live
suite is read-only — that fake is the only place the write reaches a "driver".
## UI conventions
Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ<window>` labels on
measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status
dots in the header. Every chart names itself for assistive tech: pages pass `ariaLabel` to
`LineChart`, which puts `role="img"` + `aria-label` on the `<svg>` (an unlabelled plot is
silence, and the legend is a sibling that labels nothing). Every `LineChart` on every page has
one — `/system`'s three per-host slots, `/disks`'s two per-host plots, `/llm`'s four trend
plots, `/ups`'s four battery plots — each pinned by `tests/test_api.py`, so a chart that ships
anonymous fails a test.
Window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups` and `/disks`; the API accepts
5m…90d — SMART lands every 300 s, so a 5-minute window on `/disks` is one dot). `/gateway` has
**no** chips and one window: a connection trend is read as "the last day of this pipe", so each
card's plot is a fixed 24 h named by the `Charts` chip beside its toggle rather than a page control
that would re-fetch every card at once. A
panel that cannot get data says why in the panel — the operator should never have to open
devtools to learn that a source is unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
**Flavour text is not helpful information.** A page does not narrate its own data pipeline or
its own design; copy exists only where the operator needs it to act — a why-this-panel-is-empty
sentence, a provenance tag, a measured `Δ<window>` label, a chart's unit, a `SIMULATED` badge,
an error that names what failed. The audit is one question per string: *would the operator act
differently without it?* No → delete it, container and all. Deleted in phase 08 (the cited
examples, from `TODO.md:7`): "every number is scraped from the box it belongs to · `null`
renders as `n/a`" (`/`'s footer), "agent-reported values; rates are measured over the Δ window
in the footer, and each panel's charts cover only that host" (`/system`'s *Per host* sub-line),
and the same genre — `/system`'s "numbers are measured, not inferred", `/llm`'s "each throughput
number carries its provenance", `/ups`'s "commands are the only write path in this app", each
event feed's content caption (the empty-state sentence already says it), and the design
clauses inside empty-state prose. What survived the audit did so because it is a measurement or
a why: every `Δ<window>` footnote, `provenance()` on each rate, `fleet.seams`, *sums only
engines answering now*, the queued-request note under the slot grid, the SIMULATED banners, and
the setup instructions in every empty state. `tests/test_api.py` pins the deleted sentences
**absent** from every served page and script (a sentence that migrates to another page is the
same slop), the kept functional strings **present**, and that no removed sub-line left an empty
`.sub`/`.faint` container behind.
The overview is exactly two panels — **Warnings & errors** (`ul.alerts`, one row per
`/api/summary` alert: verdict class, source linked to its dashboard via the alert's own
`href`, message, age) and **Fleet**. A non-empty list floats the alerts panel above the
fleet table (`#overview-panels[data-state]` swaps them); an empty one is an explicit
"all clear" state below it, because an empty list is a message, not a blank box. A host that
is not answering keeps its row, dimmed with the shared `.stale` class and reading `n/a`, so
the table never implies last good numbers are live and never silently loses a machine.
No summary cards and no raw event feed survive on `/` — `tests/test_api.py` pins the
two-panel shape against the served HTML.
Under the fleet table sits the **ingress's one line** (phase 29 task 05): `#fleet-gate`, filled by
`renderGateway()` from `/api/summary`'s `gateway` block with the block's own `href`, so the page
invents neither a route nor a number (`gateway 10 listeners · 33 connections · 17 live in · 2 live
out` as the live ingress answered on 2026-10-23 — those four figures are a *reading*, the same way a
listener count is, and only the field names behind them are a contract — since phase 30 task 02
reshaped the block to say what the configs declare as well as what
the sockets are doing, `n/a` for a count the source did not measure, the shared `.stale` dim when
the agent stopped answering). A non-zero `warnings` appends ` · N to address` through its own
one-row table, `GATEWAY_WARNING_FIELD`, and **only** when `Number.isInteger` and `> 0`: a clean
ingress and an ingress that never answered both print nothing, because a permanent `0 to address`
is the ink the empty-state rules exist to avoid and a permanent sentence about checks that did not
fire trains the eye to skip the one that did (phase 30 task 03 — the same file also pins that the
tally is read through the table, so a rename fails there rather than printing nothing forever).
`configured: false` renders **nothing at all** — the `disks`
block's invariant-3 rule, no row and no red one to learn to ignore — and the row is a `<p>` inside
the fleet panel because the page is two panels, not three (the pin counts `<section>`s).
Nothing waits for a writer that never came: `tests/test_api.py` fails if a page keeps an
`id` that no page script, no shared script, no stylesheet rule and no `aria-*`/`for`
attribute references — the orphan a deleted renderer leaves behind, caught on served bytes
because there is no JS test runner (a locked anchor). Same rule for `dash.css`: delete the
rules a removed panel made dead (`.grid.wide` went with the overview's fleet+events row);
component vocabulary the pages are still being converted to (`.hero-bar`, the grid and bar
variants) stays even while only one page uses it.
**A wide table scrolls sideways inside its own card, and only inside its card** (phase 27 task 01,
`TODO.md:7`). At ≤640px `table.data` is `display: block; overflow-x: auto` — no `max-height`,
because `.scroll`'s 320 px cap is that modifier's vertical-use contract and a phone does not need
a height-capped table on top of a horizontal one — and `.body.stack > *` carries `min-width: 0`,
without which the first rule is inert: a card body is a one-column grid, a grid item's automatic
minimum size is its content's min-content width, and every cell is `white-space: nowrap`, so a
wide table grew the column to 520 px inside a 320 px card instead of overflowing its own box
(measured on the live `/system` disclosures, where `scrollWidth === clientWidth` said there was
nothing to scroll). One class rule reaches all seven tables — `#fleet-table`, `#agents-table`, the
four `page-system.js` tables inside "More detail" and the NUT-variables table in `page-ups.js` —
which is why no table carries a hand-placed `.scroll`; `tests/test_api.py` therefore pins the pair
in the served ≤640px block, pins the all-widths `table.data`/`table.data.scroll` rules unchanged,
and pins that every `<table>` on every served page and in every page script still carries `data`
(a table outside the selector would clip again in silence). The scroll container is the table,
never `main`: phase 18's "the document does not scroll sideways at a phone width" still holds, and
`data/logs/27-task01-scroll.cjs` measures both halves at 360 px and 1280 px.
**Nothing is allowed to be painted past a card edge and eaten** (phase 27 task 03, the audit's
findings). `.card { overflow: hidden }` is what makes three separate layout facts fatal, and the
three fixes are one declaration each — all of them *at every width*, because a clipped fact is a
bug at 1280 px too and each is inert where the content already fits:
- **`.card > header { flex-wrap: wrap }`** (+ `min-width: 0; max-width: 100%` on the header's `h3`
and chips, `white-space: normal` on the chip). A header is a flex line of `nowrap` facts, and a
line that cannot fit *shrinks* its one shrinkable child instead of wrapping: measured at 1280 px
`/system`'s `.updated` stamp was squeezed to 67 px and rendered `updated` / `4s` on two torn
lines, while at 360 px the OS/model chip pushed the stamp 18-191 px past the card so the age of
every number disappeared. Wrapping costs lines (74 px instead of 62 px on a host header, 104 px
on the `apc` card at 360 px) and costs no content. `.chip` itself keeps `white-space: nowrap` —
this is an exception for header chips, not a change to the chip vocabulary.
- **`.card > .body.stack > details { overflow-x: auto }`**. A disclosure is a stack item whose
automatic minimum size is its content's min-content width, so one wide table grew the whole item
to 520 px inside a 404 px card at 1280 px — and because the card clips, the GPU table's last
column *and* the `charts-box` item stretched to the same 520 px were painted under the next card
(tail right 554 px, next card at 438 px), reachable by no gesture. The disclosure is the
scroller rather than the table because hoisting task 01's `display: block` would make a
block-level table shrink-to-fit, un-stretching every table that fits its card.
- task 01's `table.data` / `.body.stack > *` pair at ≤640px, unchanged.
The pair is pinned on served bytes by
`tests/test_api.py::test_a_card_header_wraps_its_facts_instead_of_losing_its_tail` and
`::test_a_card_disclosure_scrolls_what_it_reveals_at_every_width` (the second also demands no
`max-height`/`overflow-y`/`overflow` shorthand — a height cap is `.scroll`'s contract — and that
the rule is *not* inside the ≤640px block, since the defect it repairs is at the control width);
`data/logs/27-task03-verify.cjs` measures both fixes and states what each one costs, and
`data/logs/27-task03-control.cjs` re-creates every first-run offender with the declarations
neutralised, so neither rule is decorative and the audit's reachability check is not vacuous.
A big number is `Dash.hero()` on `.hero` / `.hero-bar` (`dash.css`): 2.4rem tabular numerals,
muted unit, mono `.sub` footnote (`Δ2.0s` on `/llm`'s rates since phase 23, "at the current load"
on `/ups`), coloured by the payload's `severity` through the existing `.ok` / `.warn` / `.fail`
classes — there is no `.sev-*` family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS
rule reaches for a token that `:root` does not declare.
**The `/disks` card is one panel per *physical* drive, led by identity and verdict, not by a
hero** (phase 09): a drive has no one-number story the way a UPS has charge %, so the card opens
with the device path (`/dev/nvme0n1`) and the drive's own word (`OK` / `FAILED` / `unknown`, a
`.chip` coloured only by `severity.health` — an unreadable drive gets grey `unknown`, never a
green pass), then the identity line (`model · size · SSD|HDD`, nothing invented where a field is
absent) and three equal `.metric` cells: **Temp**, **Power-on**, **Wear**. Colour comes from
`severity.temp_c` / `severity.percentage_used` and nowhere else; `power_on_hours` is an odometer,
written with a `null` verdict, so it cannot read green at 100 h or red at 60 000 h. The sector
counters are **`.chip.fail` alarm chips that exist only while non-zero** (the `/llm` preemption
rule, and why `severity.py` rates no counter): a drive at zero draws no chip row at all, because a
permanent "0 pending sectors" line trains the eye to ignore it and a green one would call a quiet
drive an achievement. Per host the three-way `smartctl_available` decides one `.banner.info`
why-line and nothing else — `null` prints the agent's own `smart_errors` line or names both fixes
without claiming to know which half is missing, and an agent that is not answering says that
instead — and in either case the drive grid takes the shared `.stale` dim while the host card
stays uncoloured (invariant 3: unreadable drives are unconfigured, not broken, and a red panel
here would train the operator to ignore red on the drives that do report). The `false` arm — the
agent said `smartctl` is not installed — no longer reaches the page at all: `api._disk_visible`
drops that host from `/api/disks/snapshot`, from the fleet sums and from `/api/summary`'s disks
rows (phase 22), because a card whose only content is *install smartmontools and grant the disk
group* is not a host on a drive page; `whyLine()`'s `false` sentence stays as the renderer for a
payload that says it, so relaxing the gate later cannot turn those hosts into silent blank cards.
Nothing on the page is per
partition or per mount: the rows are the agent's whole-`/sys/block` list, and a filesystem field
naming its way into `page-disks.js` fails a test. The fleet header renders
`fleet_disk_summary()` — hidden below two hosts (one box's sums duplicate its own cards) and
coloured nowhere, because a count of drives is not a graded number — and the per-host `Charts`
disclosure follows `/system` exactly: collapsed, lazy-built, **one** `/api/disks/history` request
per host per refresh naming `disk.temp_c@dev` and `disk.percentage_used@dev` for every drive (the
endpoint answers unknown names as empty series on purpose), the wear plot created only once some
drive publishes the field, legends naming drives because the card heading already says who, and
`pruneHosts` taking the cards, the toggle and the plots off the page together.
A cell-scale trend is `DashCharts.sparkline()` — the `llm_monitor` `drawSpark` idiom, so a
percent metric is plotted against `fixedMax` (**never** autoscaled: an 8 % KV window that once
held a 92 % spike must not read flat, and the axis gets top/bottom scale marks so "a third of
the way up" says a third of what). `align: 'right'` pins the newest sample right and grows the
line leftwards inside a real `span` in seconds — x stays time, not sample index. Stroke colour
comes from the payload `severity` via the `--ok`/`--warn`/`--fail` tokens only; an unrated
series draws a faint `--text-faint` stroke with no fill, because a shape is not a verdict.
Below `minPoints` (2) it prints `no history` rather than drawing a confident line through one
point, `labels: {current, max}` renders the `now / max` foot through the caller's `format`, and
there is no animation at all (`prefers-reduced-motion` holds by construction). `minSpan` is the
other half of the same honesty for count metrics: history arrives as rollups, so eight seconds
of one queued request averages to 0.14 and an axis fitted to that draws a full-height spike for
a queue nobody was in. `/llm`'s count rows moved to the phase-07 stats strip (numbers, not
sparks), so no page passes `minSpan` today and `drawSparks()` still plumbs it through — a count
sparkline that comes back must set it, and the component's own contract test keeps the option
honest (`test_sparkline_ships_the_fixed_scale_and_verdict_options`).
A battery is `DashCharts.battery(container, {charge, severity, state, label})` — one inline SVG
whose **fill *is* the charge**: 1 viewBox unit per percent, clamped to 0–100 and never rounded up
(a 2 % cell must look nearly empty, because a reassuring battery above a flat one is a lie),
clipped to the plate so 0 % leaves an empty shell rather than a stray pixel. Colour comes only from
the payload `severity` token; a measured charge with no verdict draws faint grey at half strength
(`sparkline`'s rule — a shape is not a verdict) and never an empty shell ("0 %" and "not rated" are
different sentences). Every state has a **shape** as well as a colour, so the glyph survives
monochrome: solid shell `online`, bolt `charging`, dashed outline `onbattery`, no fill with a dash
pattern *through* the body `unknown`; a state word it does not know degrades to `unknown`, never to
a confident "on line". Accessible **by default** — `role="img"` + `<title>` + `aria-label` built
from the inputs ("apc: Charge 100 percent, on line"), because the glyph carries information the
markup does not say in words — and a page that prints that sentence next to it passes
`decorative: true` to make the `<svg>` `aria-hidden` rather than say it twice. No animation and no
`ResizeObserver`: one `viewBox` + `preserveAspectRatio` scaled by `.battery` in `dash.css`, so the
hero row gets a big cell on a desktop and a small one on a phone.
**The `/ups` card is led by the glyph and three heroes** (phase 05): the battery on the left, then
**Charge %**, **Load %** and **Runtime left** on `.heroes` at the shared 2.4rem, coloured from
`severity.charge_pct` / `severity.load_pct` / `severity.battery_runtime` and nothing else. Charge
carries **no** `.bar` — the glyph beside it is that value drawn as a shape, and one value drawn
twice in one row is the pattern this app avoids (Load keeps its bar; nothing else draws it). Its
`.sub` lines are the context each number needs and nothing more: `24 V nominal · 27.3 V` for the
pack, `128 W of 900 W` when the driver publishes both `realpower` figures (a wattage inferred from
a percent against another model's rating would be an invented number), `at the current load` for
runtime. Runtime prose is `runtimeText()`'s (`1h 32m`, `10m 21s`) and an estimate the driver never
made goes into `Dash.hero()` as `null`, so it prints faint and uncoloured rather than as `0m`. The
readings the hero row displaced — line in, output, battery volts and amps, battery °C — stay on the
card, demoted to `.ups-electrical`: muted, `.95rem`, never folded away, because they are what you
reach for *while* the UPS is on battery, and battery °C reads `n/a` on both real drivers here.
`state` is derived in the page from `on_battery` → `charger_status` (`chg` charges; `dchg`/`rec` are
the server's own on-battery tokens; `full` is **on line, not charging** — a rested floating battery
reports it) with one fallback: a state chip the API rated `ok`, because `nut.states()` reserves
`ok` for on-line/fully-charged and neither real driver here publishes `battery.charger.status` at
all. `ups.status` itself is never re-parsed in JS (`tests/test_ups_contract.py`). `/ups` passes
`decorative: true`, because the charge hero and the state chips already say the sentence. The glyph
repaints only when charge / verdict / state change (`drawGlyph()` keys a `data-drawn` triple): the
charge float jitters every `NUT_INTERVAL_SECONDS`, and rebuilding the `<svg>` on that re-announces
an accessible name for a difference no eye can see.
**The `/gateway` page is the config's own tree, and a line's only claim is a count** (phase 29;
joined differently by phase 30 task 02; laid out as trunk→leaf panels by phase 30 task 04; made
readable — whole-row colour, inbound-first rows, a moving trunk, a filter — by phase 31): **exactly
two panels**, one per config file, in the order `nginx` then `haproxy` (`GATEWAY_CARD_KEYS`), each
shipping `hidden` and appearing on the poll that has rows for it — a gateway with no nginx in it
spends no panel saying so, and a section that did not answer at all is the page's one `why` line
rather than a card repeating it in a smaller font. There is **no `map` panel and no `unmatched` one**:
the stream `map` is nginx config, so its fan-out is the nginx panel's `:443` leaves
(`out.kind: "map"`, its SNI key as the selector) read beside the port they reach, and a residual was
the left-over of a rule the join stopped applying.
Inside a panel the rows are **groups** (`groupsFor()` → `renderGroups()` → `renderTrunk()` +
`renderLeaves()`): a *trunk* is one port that file binds (`:<port>`, its live count, up to
`MAX_CLIENTS` client addresses, its `server_name` count with the whole list on `title`, and — since
phase 31 task 03 — a **short `.pipe.pipe-trunk` of its own**, appended through the DOM as the trunk's
last child because an `<svg>` needs `createElementNS` and cannot come out of `groupElement()`'s static
`innerHTML` template), and the *leaves* hanging off it are the routes that port's config declares,
each with its own `metric`. That stub is the port's live count drawn as a line: `renderTrunk` puts
`live` on it when `listeners[].connections >= 1`, `still` when it is `0` **or** when it is `null` —
never `live` for an unknown, and never a `0` in `.pipe-count` for one either — and `on` always, because
`listeners[]` contains only ports a config names, so the row existing *is* the existence fact. Its
claim is **the port and never a leaf**: `trunkAria()` writes
`:4443 — 9 connections are open on this port right now` / `… no connections are open …` /
`… connections on this port are unknown: the socket table did not answer` into both `aria-label` and
`title` — the latter with `setAttribute`, because `title` is a reflected IDL attribute of
`HTMLElement` and **not** of `SVGElement`, so `svg.title = …` is a JavaScript property no tooltip ever
reads (task 05's browser pass found the served code doing exactly that) — and it indexes nothing but
`listeners[].connections`: no `endpointText`, no `out.*`, no client address, because attributing one
port's sockets to one of the routes hanging under it is a claim `sources/gateway.py` does not make.
`margin-inline-start: auto` on `.pipe-trunk` is measured, not polish: the trunk wraps, and
`.pipe-count`'s own auto margin flushes only the item that *starts* a
line, so without it a two-line trunk (`:80` on the live ingress) dropped the dash alone at the far
left — ink stranded on the opposite side of the trunk from the spine.
**A leaf reads inbound first** (phase 31 task 02, correcting phase 30's rank): line 1 is
`.pipe-name` and carries the config's word for **who reaches** the endpoint — a vhost's
`server_name`, a stream `map` key (the literal `default` verbatim), or a haproxy `backend` name —
from `inboundText()`, which joins **every** `out.selectors` entry, falls back to `out.kind`, and
never to `out.host`; line 2 is `.pipe-end` and carries `counterpartText()`'s
`→ host:port · kind`, the counterpart it flows to. Both lines are mono so a fan-out of eighteen rows reads
down its columns, `.pipe-name` stays the primary line of its row for the trunk too (a trunk prints
its port there), and neither helper cuts a string: CSS clips, and `leafTitle()` / `leafAria()` carry
**both ends whole** (gap 13). Phase 30 had the two reversed, which buried the domain the operator
types into a browser under the address they never type. The page draws what the config declares and
nothing else: the deleted `0`-padding
(listeners with no connection, endpoints no line reaches) is gone because a group *is* a listener and
an unrendered endpoint has no row to sit in. **A trunk whose config names no target is a fact and not
a gap in the drawing** — this fleet's nginx `:80` is a `return 301`, so its leaf list renders the
script's own "no target named in the config" rather than an empty box (the payload publishes no row
for it; `sources/gateway.py` refuses to warn about that shape for the same reason). Groups are keyed
on the port, and rows on the connection's own `id`, in per-panel `Map`s (`state.groups`,
`state.rows`):
built once, patched afterwards, reordered with `appendChild` of the existing node and pruned once per
panel by id, because rebuilding a fan-out the size of this ingress's `:443` every 30 s re-announces
all of it to assistive tech
for a difference of one connection (the `/llm` slot-grid rule). `pruneCards()` wiping a panel's whole
map is the only bulk clear, for the case where the family itself left the payload.
A count is printed by `countText()`, which answers `null` for anything that is not a finite number —
**never `0`** (rule 2). Phase 32 task 04 made the missing count say **which** missing count it is:
`countCause()` reads the same two fields `sources/gateway.py` joins on (`out.address === true` or a
non-empty `out.resolved`) and returns one of two fragments, so `no number: the gateway could not
resolve this name in this reading` is printed only when nothing is known for the endpoint, and
`no number: the socket table did not answer this reading` only when an address **is** known and
nobody measured — because those are two operator actions, and either one drawn as the other sends
someone to the wrong box. Both go out through `countWhy()` on the `title` and through the same
`countCause()` in `leafAria()` (`connections not attributed: …`), so a row's tooltip and its
accessible name cannot disagree, and a dash is never the trunk's `n/a`, which is the trunk's own
fact about its own port. **A `0` is not an unknown at all**: a resolved endpoint with no socket is a
measurement, keeps `0 open connection(s) on this endpoint`, and never borrows either sentence
(`test_an_unresolved_name_and_an_idle_route_say_two_different_sentences`, and
`data/logs/32-task04-sentences.cjs`, which serves all four arms through route interception because
the live 1.2.0 ingress can only produce one of them). Every **counted** line gets its provenance on
the same `title` rather than on a column of ink: `countTitle()` appends
`· address from <via>, resolved <age>` — `via` copied from the payload (`libc`) and the age through
`duration(ageSeconds(out.resolved_ts))` — and prints neither fragment on a row the config wrote as an
address, because `out.address` is already the whole story and echoing the agent's `literal` back as a
lookup would report a `getaddrinfo` that never ran; a missing `resolved_ts` costs the age and nothing
else, so no row prints `n/a` or `NaN` where an age belongs. Phase 32 task 03 left the rendering alone
and changed what lands in the arms: a name-written route is now counted through the address the
**gateway** resolved, so on the **captured** ingress all 33 lines carry a number (26 of them a
measured `0`) and a dash has become the exception that means *this name has no record*
(`nxdomain`/`error` in `resolver.entries`, or no entry beside it — including no section at all, which
is what a 1.2.0 agent or `--no-resolve` answers (the live ingress has run 1.3.0 since phase 32 task 05
on 2026-10-25, where **33 of 33** rows carried a number and none dashed; the same pass recorded
`resolver.available: true`, 32 entries, `via: "libc"`, `config_directive: "10.1.0.1"`, 30 `ok` + 2
`literal` targets). The footer's second paragraph says the same thing once for the whole page — a dash means the
gateway could not resolve that name in this reading, hover a counted line for the address it matched
and when that answer was taken — and the footer is still four `<p>`s (phase 31 task 04's filter
footnote is the fourth, which is why the count that phase's pin demands is four and not three).
The line itself is hand-rolled SVG (`role="img"` + an `aria-label` naming the inbound port, the
endpoint, the selector and the count) and the row carries **two independent class pairs** (phase 30
task 05, moved onto the `<li>` by phase 31 task 01), because the operator's green and the socket count
are two different facts:
* `live` / `still` — "≥ 1 ESTABLISHED socket on this pipe right now" — decides **motion only**, and it
stays on the `<svg>` alone. One `pipe-flow` keyframe whose sole property is `stroke-dashoffset`; the
live rule declares no `stroke`, so a busy line and a resting line are the same colour and motion can
never become a verdict. Motion may not spread: the spine segment between two rows is **shared** by
both leaves, so a dash travelling down it could not be attributed to one route, and animating it
would put a per-route claim on a per-list structure. There are **two** moving pipes and they read two
different payload paths (phase 31 task 03): a leaf's run from `connections[].connections` and a
trunk's stub from `listeners[].connections`. One `.pipe` component serves both — `buildPipe(trunk)`
adds `pipe-trunk` and nothing else — which is also why the file's single
`@media (prefers-reduced-motion: reduce)` block stills the stub without naming it: the stilling rules
are descendant selectors (`.pipe.live .pipe-path`), and
`test_the_pipe_animation_stills_in_both_states_under_reduced_motion` pins that no `.pipe-trunk` rule
may style `animation`/`transition` itself and quietly step outside that block.
* `on` / `off` — "a proxy config declares this route" — decides **colour for the whole row**, through
the sheet's own existence pair `--gate-on` / `--gate-off` (declared once in `:root` as a substitution
of the theme colours, so both themes follow and no gateway rule names `--ok`). `renderLeaf` toggles
it on the `<li>` **and** the `<svg>` off the same value in the same pass, and four rules carry it to
the two spine halves and the tick, so a route is green from its trunk to its endpoint — phase 30
coloured `.pipe-path` only and the operator read that as two thirds of every route being inactive
(`test_the_whole_route_takes_one_existence_class` pins the row class, the four connector rules and
that no gateway rule names the structure token any more). `routeInConfig()` answers `true`
unconditionally, which is honest rather than a stub: `connections[]` **is** the config's declaration
list, so `off` is the styled seam the write phase will fill and never a second state in use. Colour
still only ever *arrives* from a state class: the base `.pipe-path` declares no `stroke` and the base
`.gate-link::before` / `::after` / `.gate-elbow` rules declare their border `transparent` — an SVG's
default stroke is `none` for free, a border's default colour is `currentColor`, so "a row built and
never rendered draws nothing" has to be written down for a border.
So the page is **green because the route exists**, never because a number is good: `severity.py` has
no `gateway.*` row, the payload carries no `severity` at any depth, and
`test_no_gateway_count_is_coloured_by_a_verdict` fails the day a `severity`/`ok|warn|fail` token
appears in the page or the script — the one `warn` in the file is the class `renderWarnings` takes
from `alerts.py`, pinned by count. The **switch** at each line's head (`buildSwitch()`) is the
control from the operator's sketch and it is **built, not wired**: a real `<button type="button"
role="switch">` pill with a knob, keyed on `[aria-checked]` (never `:enabled`, which would paint every
one of them off) and coloured from the same existence pair, carrying a full accessible name
(`endpoint, reached by <selector>, in the nginx config, on`) because a colour cannot say that to a
screen reader. It is `aria-disabled="true"` — **never** the `disabled` attribute, which would take the
reading out of the a11y tree — with a `title` naming the missing write path, no listener (the file's
`addEventListener` calls are exactly four — the charts' `toggle`, `DOMContentLoaded`, and the filter's
`input`/`keydown` — and `test_the_switch_is_built_and_explicitly_unwired` pins that list as an
allow-list whose missing member is the point: **no `click` is bound anywhere on this page**) and no
`:hover` or `cursor: pointer` anywhere in the map, because invariant 1 means no browser can disable a
route:
`haproxy.cfg` and the nginx `server` block are the only switches on this fleet. The one `transition`
is the knob's, and the reduced-motion block names it beside both pipe states.
The branch is four CSS rules of geometry plus four of colour, and no JavaScript: each `.gate-link`
carries its own upper and lower half of the spine at the gutter's left edge (`::before` / `::after`,
each reaching half a leaf gap past its row so consecutive halves meet), the first and last rows drop
theirs so the spine starts at the first branch and stops at the last with no mask, no measured height
and no background-coloured eraser — which is also why it stays right on the poll that adds a leaf — and
`.gate-group.solo`, set by the script from the group's own leaf count, takes the vertical away so a
one-route port runs **straight across** (every haproxy frontend here). The ≤640px block restates the
reaching half's geometry and **no colour**, because a per-width copy of the existence pair is how a
pair drifts from its theme. `.gate-group { column-gap: 0 }` is what makes the trunk's
right edge and the spine the same x, so each tick starts *at* its trunk, and at ≤640px the stacked
trunk is reconnected by bringing the first row's upper half back at `top: calc(var(--gate-row-gap) * -1)`.
The head's **four** tallies are the payload's own figures and nothing is summed twice: `ports` =
`listeners.length`, `connections` = `connections.length`, `live` = `sumLive()` over the listener rows
that publish a `metric` (`null` — never `0` — when none measured, dimmed `stale` with a tooltip naming
the port when some did not), and `warnings` = `warnings.length`. `#gate-warnings` sits **above** the
panels and ships `hidden`: a non-empty `warnings` list raises it, rendered through
`ul.alerts#warnings-list` in **the overview's own row vocabulary** (`li.warn` with a `.src` from
`familyWord(kind)` and a `.msg` carrying `warning.message` verbatim) because "a backend no frontend
routes to" is the same kind of sentence as an alert and must not grow a second look; back to clean it
hides and its rows go with it (both directions pinned, and the live ingress is clean, so the panel is
off). Charts follow `/system` exactly: one collapsed `Charts` `<details>` per panel, **no** window
chips (a fixed 24 h named by the chip beside the toggle), the request
fired **once per visible panel per refresh** naming that panel's own gauges and cached so the first
expand paints from data in hand, while the `LineChart` is built and painted only while the disclosure
is open (`drawCardCharts()` returns at `chartsOpen(key)` — a `display:none` container measures 0 px),
and the legend names pipes because the panel heading already says which config file this is. Every
degradation arm uses the payload's own words — the `why` sentence for a pre-1.2.0/`--no-proxy` agent or
a silent section, the `unreachable` dim for a host that has never answered, and `#gate-empty` (naming
`GATEWAY_HOST` and agent **1.2.0**) for `404 {configured: false}` — and a failed poll re-draws nothing,
so the last topology stays on screen. The footer is **four** paragraphs and no more: why a line is
drawn and what its motion means; why a **name** endpoint carries no number; that the switch is not
wired; and what the filter box does to the map and what it leaves alone — the sentence is there because
the field is the one control that could be read as a query, and it is not one.
**`#gate-filter` is an eye, not a query** (phase 31 task 04). 33 routes in config order is not a
lookup, and the operator's question is always about one domain, so one field above the panels hides the
rows whose **own payload text** does not contain what was typed: `in.port`, `out.host`, `out.name`,
`out.kind`, every `out.selectors` entry and every `out.resolved` address for a route (`haystack()`;
the sixth field is phase 32 task 04's, and it is what makes typing `10.1.2.64` find the route whose
name target resolves there — a haystack of names could not answer a question asked about a socket),
`:${port}` plus
`listeners[].domains` and `.clients` for a trunk (`trunkHaystack()`). Nothing else is readable and
nothing else is matched — no DNS, no re-ranking, no rule about `.reeselink.com`, and no page-level
split by domain suffix, because "`reeseapps.com` vs `stackexpected.com`" is a fact neither config file
states and grouping by it would be inventing a taxonomy the ingress never declared (rule 2); the
operator types the suffix and the page hides the rest. A **trunk** that matches keeps its whole
fan-out: typing `4443` asks for the port, and its routes are the answer. Visibility is applied inside
the patch pass (`renderLeaves` sets `li.hidden` by index into `group.connections`, `renderGroups` sets
`el.hidden` and the panel's `[data-k="no-match"]`), so a poll cannot undo the filter, a hidden row
keeps its identity and its `<svg>`, and only the *config* removing a route ever deletes a node. The
repaint is `renderPanels(state.snapshot, state.row)` — **no** `fetchJSON` and no `refresh()` on the
filter path, which is what the type of the feature is (`refresh()` would add a history pair per
keystroke and change nothing), `Escape` clears the field because a filter you cannot leave with the
keyboard is a trap on a page whose other shortcuts are all keys, `#gate-filter-count` reads
`N of M routes` (leaf routes, both panels, and hidden while the field is empty), and
`renderHeader()`/`renderWarnings()` hold no reference to `state.filter` at all: `ports`,
`connections`, `live` and `warnings` describe the ingress, not the operator's current view of it, and
the warnings panel is a different claim rather than a shorter version of this one. The field is the
page's own markup wearing the sheet's shared field skin (`input[type="search"]` joined the
`input[type="text"], input[type="password"]` rule — the only three field types the app has), it is
`autocomplete="off" spellcheck="false"` with an `aria-label`, `.gate-filter` caps it at `18rem` on a
desktop and goes `width: 100%` at ≤640px, and it declares no `:hover`, no `cursor` and no
`transition` because the connection map owns one interactive-looking thing (the unwired knob) and this
is not it. `data/logs/31-task04-filter.cjs` measured it on the live ingress (transcript
`31-task04-filter.out`, screenshots `31-task04-filter-{stackexpected,no-match,cleared,360}.png`):
`stackexpected` → 6 of 33 rows with `nginx:443` and `haproxy:2000` the only trunks standing, a client
address → that port's whole fan-out, `no-such-domain-anywhere` → every group hidden with both panels
saying `no route matches "…"` and neither the tallies (`10 / 33 / 12 / 0`, byte-identical either side
of every keystroke) nor the warnings panel moving, **zero** requests across five keystroke states, the
same filtered view after a real 30 s poll as before it, and 0 px of sideways document scroll at 360 px
where the field is 320 px wide. `34 PASS / 0 FAIL`.
Phase 31's own pass measured the whole map rather than its parts, and is the pin that the three
readability changes landed together: `data/logs/31-route-readability.cjs` (transcript
`31-route-readability.out`, which opens with `data/logs/verify-31-page-only.py`'s payload proof,
screenshots `31-gateway-*.png`, twelve frames — 1440, 360, light, filtered, no-match, trunk-live,
phone, and one per degradation arm: predates, unconfigured, unreachable, no-nginx, sockets-quiet) —
signs in and reads **computed** colour, not classes — every segment of all 33 rows (both spine
halves, the tick, the run, the knob) computes to `--gate-on`, `rgb(63, 185, 80)` in dark and `rgb(26, 127, 55)` after the real
theme toggle, and **no dashed border anywhere inside `.gate-map` computes `--border-strong`**
(`rgb(51, 59, 77)` dark / `rgb(195, 202, 214)` light — the grey plumbing the operator read as
inactivity), with **0** boxes in the map carrying a verdict class in either theme. The motion truth
table comes out of the DOM against the payload's own numbers: every trunk the kernel reported ≥ 1 for
animated, every one it reported 0 for drew a motionless line, and each `0 live` string corresponded to
a port the payload carried 0 for — a reading of traffic, never a stored fact. A second, deterministic
arm settles the direction of the claim: with `:443` handed 7 sockets and **every** leaf's count set to
`null`, the trunk animated and 0 of its 18 leaves did, so a port's number provably does not leak into
its fan-out. The arm where the socket table
answers nothing draws **ten motionless `n/a` stubs** reading `connections on this port are unknown: the
socket table did not answer`, with **no** `0 live` ink anywhere, every leaf keeping its `—`, the header's
`live` tally `n/a` — and no panel taking the shared `stale` class, because the configs *did* answer. It
then holds that the **leaf** keeps its own claim: with a trunk live and the leaf under it at `—`, the
trunk animates and that leaf does not, because a port's sockets are not one route's. The filter pass is
zero requests across five keystroke states with the four tallies byte-identical either side of every
keystroke and `Escape` clearing the same way; `prefers-reduced-motion: reduce` stills all 43 paths (10 of
them trunk stubs, through the block's descendant selectors) and every knob; the degradation arms each
show their own sentence (`404 {configured: false}`, a `supported: false` agent, a silent socket section,
a never-answered host); and the document scrolls **0 px** sideways at 1440 px and at 360 px with both
charts open and the widest fan-out on screen. It also found, and fixed, the one defect of the phase that no class-based check could see:
`svg.title = aria` assigned a JavaScript property and produced **no tooltip**, because `title` is a
reflected IDL attribute of `HTMLElement` and not of `SVGElement` — the two map tooltips are
`setAttribute('title', …)` calls now, which is what the browser can show and what
`tests/test_api.py` demands. Its mobile-audit sibling is `31-task05-mobile-audit.out` (the
`27-task02-mobile-audit.cjs` sweep, 371 checks, 0 failures, 0 findings, exit 0).
The layout is three load-bearing declarations (`minmax(0, 9.5rem) minmax(0, 1fr)` per group so
a fan-out lines its leaves up instead of re-flowing them to the widest target, `minmax(0, …)` on every
track so a 40-character hostname ellipsises instead of growing the card — phase 27's rule — and one
column with the trunk above its leaves at ≤640 px). Task 04 measured that layout with
`data/logs/30-task04-panels.cjs` (transcript `30-task04-panels.out`, screenshots
`gateway-30-task04-{1440,360,reduced}.png`) and task 06 re-measured it, with the switches and the
pipe states in place, in `data/logs/30-connection-map.cjs` (transcript `30-connection-map.out`,
screenshots `30-connection-map-{desktop,light,warnings,predates,phone}.png` — the phase's own pass:
two panels, straight haproxy rows, one branched nginx `:443`, green in both themes, motion only where
a socket is live, dead switches, injected warnings, both degradation arms, 63 checks, 0 px of sideways
document scroll at 1440 px and 360 px). What it printed on 2026-10-23: 10
listeners / 33 connections, nginx 4 ports · 27 routes (`:443` 18 leaves, `:4443` 6, `:8443` 3, `:80`
the refusal sentence), haproxy 6 groups of one, 31 dashes beside 2 numbered lines, every line keeping
its full text on `title`, and **0 px** of sideways document scroll at both widths with the widest
fan-out and both charts open.
An event row names its subject **once**. Every feed (`/system`, `/disks`, `/llm`, `/ups`) renders the
message through `Dash.eventText(event)`, which drops the `UpstreamError: ` wrapper and a
*leading* duplicated subject the way `alerts._clean_error()` does for the overview — `poller.py`
messages start with their own label, and each row already prints that label as a chip, so
"gufo gufo is answering again" reads as two problems. A name in the middle of a message stays:
the UPS rows used to `replace()` it anywhere and mangled "Quick battery test sent to apc: OK".
This is presentation only — the stored event and `/api/events` keep every word, and the raw
error string still surfaces in `/api/health`.
An event feed is a **window**, not a printout (`Dash.eventsPager(el, events, opts)` in
`app.js`, one helper with **five** call sites over four pages (`/system`, `/disks`, `/llm`, and
**both** halves of `/ups` since phase 26) — a pager that
works on one page is the bug this
shape prevents). `ul.events` is a bounded 320px scroll container (`overscroll-behavior:
contain`) and the pager puts `EVENT_PAGE_SIZE` = **50** events in it, with a `.events-pager`
strip under the list: `1–50 of 200` between `‹ prev` and `next ›`, both `aria-label`led for
direction (the feed is newest-first, so `prev` walks towards now) and `disabled` at the ends.
The strip is `hidden` when the feed fits on one page — every row on screen *is* the whole
story, and a disabled arrow says "more history" when there is none. Each page keeps its own
`row(event)` (so `/ups` keeps its longer timestamp and runtime footnote — one `eventRow` const
shared by both of its lists since phase 26 — and escaping stays
the page's job — the pager has no default renderer and says so in a row when a caller forgets
one) and its own `empty` sentence, so the `loading…` and empty states read exactly as they
did before paging existed. Two numbers are load-bearing and pinned together: the fetch
(`EVENT_FEED_LIMIT` = 200, `/api/events?limit=…`) must exceed one page, because a limit at or
below `EVENT_PAGE_SIZE` is a pager that silently never appears; `/api/events` itself is
**unchanged** — no `page`/`limit` vocabulary grew for one page's layout. Paging state lives in
a `WeakMap` keyed by the `<ul>`: it survives a poll (so page 3 is still there two seconds
later — resetting every refresh would make page 2 unreadable) and resets to page 1 when the
event *set* changes (an operator reading page 3 of yesterday's history must not land on page 3
of a list that just changed). Rows are re-written only when their markup changed, which is
what keeps the reader's scroll position inside the container. The `/ups` feed is also
**expandable** (phase 19, `TODO.md:4`): its card body is a `details.events-box` disclosure with
no `open` attribute, so that card is one `Power events` summary line until the operator opens it
— a closed `<details>` renders no content, which is `/system`'s Charts rule applied to a feed,
and the `#events` id stays *inside* the disclosure so the pager and its bounded scroll keep their
target (pinned on the served bytes). What the disclosure gates is **visibility only** (phase 26,
`TODO.md:6`): `refresh()` fetches the feed on every poll and `refreshEvents()` splits the one
response on the stored `kind` — `status` (an `ups.status` transition, power loss included,
`collectors.py`), `test` (a self-test verdict) and `action` (a command this page sent, `api.py`)
are the power events that belong behind the summary, and **every other kind** is the
connection/error half (the poller's `unreachable`/`reachable`/`recording`/`store-error`, i.e.
`NUT server NutError: cannot reach upsd … (timed out after 4s)`), which must never be the list
behind a summary. Phase 19's lazy gate went out with that decision — no `state.events`, no
`toggle`-handler fetch, one unconditional `refreshEvents()` call — because a visible list that
ages is worse than one more 200-event fetch per interval, and the stale-last-good rule is now the
empty `catch` in `refreshEvents()`: nothing is stamped and nothing is re-drawn, so a failed fetch
leaves each `<ul>` holding the rows it already had. `/api/events` grew no vocabulary for the
split (`limit` + `source` only, pinned), and the kind set is pinned on the served script —
**parsed out of the `POWER_EVENT_KINDS` literal and compared for equality**, not matched as one
line — by `tests/test_api.py::test_ups_events_are_fetched_on_every_poll`, so a silent rename in
`collectors.py`/`api.py`, or a kind quietly added to or dropped from the hidden list, fails a
test instead of moving errors back into the collapsed list. The error half is a card of its own
(phase 26 task 02): `section.card#ups-errors-card`
sits **above** the disclosure with a `ul.events#events-errors` inside its `.body`, ships
`hidden` (the resting state — a permanent empty "Connection & errors" card is the ink the
empty-state rules exist to avoid) and `refreshEvents()` keeps it that way while the split has no
error rows, so its first `unreachable` line is what makes it appear and the `loading…` seed never
flashes. Both lists go through the **same** `Dash.eventsPager` with the **same** `eventRow` const
(one table split on `kind` cannot grow two row shapes — the pin reads each call's options through
`_pager_call()`, so a list that loses its renderer fails by name rather than turning a whole-file
count off by one, and the row template itself may not read `event.kind`), and because the pager's
state is a `WeakMap` keyed by the `<ul>` the two page independently. Nothing about the second list
needed a stylesheet: `ul.events` and `.events-pager` are class rules and the card carries the
page's own `.85rem` section gap inline, which is pinned as the *absence* of a
`#ups-errors-card`/`#events-errors` selector (`test_the_ups_error_card_adds_no_stylesheet_selector`,
and the same guard inside the disclosure's own CSS pin) alongside the card's shape, position and
`hidden = errors.length === 0` line
(`test_the_ups_connection_errors_are_a_card_of_their_own_above_the_disclosure`); the card also
ships outside **every** `<details>` on the page, checked over the disclosure blocks rather than
one known disclosure (`test_ups_power_events_feed_sits_behind_a_collapsed_disclosure`) — the
operator's complaint was that a `NutError` line needed a click, so "not inside the disclosure I
happened to name" is not the claim. The layout, the gap in both states and the first-appear are
measured in `data/logs/26-task02-layout.cjs`; the phase's own pass is
`data/logs/26-task03-verify.cjs` (transcript `26-task03-verify.out`, re-run green on the
phase-completion pass as `26-final-verify.out`, 31 checks: the disclosure
shut with its list unpainted while the error card holds 50 of the 199 error kinds and the power
list the one power kind of a direct `/api/events?source=ups&limit=200` read, every row equal to
the page's own `Dash.eventText` of that event, both lists paging independently, the card hidden
on a power-only feed and appearing on the poll that carries the first `unreachable` row, and 360 px
with no sideways scroll; screenshots `data/logs/ups-event-split-{visible,expanded,hidden,appears,phone}.png`).
The **disclosure's** styling is `details.charts-box`'s shape with one structural
difference: that toggle sits *inside* a card body, while this `<summary>` **replaces** the card's
`header` — so it carries the header's type, padding, sunken ground and bottom rule (a closed card
reads as its heading, an open one puts the feed under the same rule every other card draws) and
no `margin-bottom` when open, because the `.body` below carries that spacing. That is also why
`details.events-box > .body` restates the `.8rem`: `.card > .body` is a **child** selector, so a
body moved inside a disclosure stops being padded until something says so — while `ul.events`
and `.events-pager` are class-based and needed no new rule to keep the 320px scroll and the
`1–50 of N` strip working inside it.
The `/system` host card is one row of six hero slots — CPU, MEM, TEMP, DISK, GPU, NET (the
`.heroes` grid; `tests/test_api.py` pins the slots and their order on the served script).
Colour comes from the payload's `severity` dicts and nowhere else, and a missing **number or
verdict** prints `n/a` **uncoloured** — gateway's CPU temp, a driver with no temperature
input, a rate with no second sample yet. What a machine does not **have** is a different
fact and prints nothing at all: the GPU slot ships `hidden` and `renderGpuSlot()` un-hides
it only for a host whose payload carries a card, so gateway's card is **five** heroes with
no GPU slot in the row (`TODO.md:12` — "Boxes without a GPU should not report N/A but
instead should omit these values"), and the sheet's `[hidden] { display: none !important }`
is what takes the wrapper out of the grid so the row reflows instead of leaving a wide gap.
The two rules are one sentence each in the page and one test each; collapsing them into "any
missing thing is `n/a`" is what made a router look broken over the GPU it never had
(invariant 3). NET is one slot with two lines (`.hero-lines`, `↓` in / `↑` out) and is
deliberately colourless: throughput has no good/bad, so `network` carries no severity.
The GPU slot is one busy hero per card plus **four labelled values** under it — **Model**
(mono, `gpus[].model`), **VRAM** (`used / total`), **Pool** (`mem_used_pct`, the honest APU
view a Strix Halo needs because its model lives in GTT) and **Temp** (`gpus[].temp_c` — the
agent's max over that card's hwmon labels, the same value stored as `gpu.temp_c@N`, so the
cell and the history cannot disagree; the per-label breakdown stays in "More detail"). They
replaced one ` · `-joined 11px `.sub` — `TODO.md:12`: "`Radeon 8060S Graphics · VRAM 180.2
MiB / 512.0 MiB · pool 79% · 69 °C` is too much info for such small text and should be its
own values" — and they are `.metric` cells in a `.gpu-values` grid, so a 20-character model
string wraps instead of truncating. The four cells come from one `GPU_VALUE_FIELDS` table,
which `tests/test_api.py` resolves against every capture **and** against the served
`/api/system/snapshot` (`SLOT_FIELDS`' device): a rename on either side fails a test instead
of printing `n/a` beside a healthy fleet. A field the agent did not publish prints `n/a` in
its own cell (the card is real, so it is a number the card is supposed to have) and never
`0 °C`. Nothing in this slot is coloured and nothing draws a bar — no `severity`, no
`ok|warn|fail` token in the markup or the grid's rules (phase 06: a 95 %-busy card on an LLM
server is the job, and a bar under a verdictless number would be a verdict by shape).
Everything not glance-worthy — load averages, clock, disk read/write/busy, uptime, GPU power
and the per-label temp breakdown, filesystems, processes — folds into a collapsed "More
detail" `<details>`: a simpler surface, not a less complete page (the table keeps its Temp
column; this was a promotion, not a de-duplication).
A `Δ<window>` label is a measurement, never a config value: `sources/host.py` sends the window
the estimator actually used as the payload's `window_seconds`, and `/system`'s footnote prints
those (a range when the hosts disagree) instead of the poll interval, which only coincides with
them at the defaults (since phase 23 the `/llm` rates are measured over one poll interval, so
their tag reads `Δ2.0s` while `/system`'s stays `Δ10.0s` — each page naming the window its own
estimator used). `/llm`'s hero `.sub` prints the rate cell's own `window` through the same
`provenance()` tag, and the hero's `title` carries the exact `source` string — "78 tok/s" must
always say whether it was scraped, measured `Δcounter/Δt`, or estimated (`llm_monitor`'s
`setThroughput` idiom; dropping the tag to look cleaner is a regression, not a simplification).
**The `/llm` page reads per engine** (phase 04, decided in
`.agents/phases/complete/04_llm_chart_readability/01_chart_set_decision.md`, laid out by phase
07): a row of two `Dash.hero()` throughput numbers (decode, prompt — deliberately **uncoloured**,
because 0 tok/s on an idle server is healthy and `severity.py` has no throughput row) and the
latency rows, each a mean with a mono `p95 …` foot (no hover; p95 needs no stored series) and a
`sparkline()` fixed at 160px so rows line up. Rows are engine-scoped by the `only` flag on both
components: gufo never grows a TTFT mystery, vLLM never a spec-accept hole.
**The stats strip (`#strip`, phase 07 task 03) is the summary; the panels are the detail, and no
row is printed in both** — the UPS hero row's "one value drawn twice" rule, widened to two
components. One `.card.stat` per *configured* engine between the fleet block and the panels: the
reachability dot, the engine name and the age stamp in the head (same distinction every card
draws — the dot speaks reachability, the numbers never do), then the **KV-cache bar** — the one
cell on the page with a verdict, so the fill and a breach's text take their colour from the
payload's `severity` through `.bar.ok/.warn/.fail` and the ratio→percent conversion the panel's
cell used to make, with `ok` staying quiet ink and a missing gauge an empty bar plus `n/a` — then
`prompt tok/s` / `decode tok/s` with their `Δ<window>` tags, `total tokens` (lifetime, so its
source is the tooltip), the sessions this engine reports, `prefix cache hit`, `spec accept`
(gufo) and `queue time` with its `p95 …` foot (vLLM). `STRIP_ROWS` is the whole table and its
`key` **is** the `cellValue()` key, so every number on it is a cell of `/api/llm/snapshot`: no
field was invented in `sources/llm.py` for this card, and there is deliberately **no peak-ctx
row** (phase assumption). The session row is engine-shaped — vLLM publishes `running`/`waiting`
counts, gufo's is `slots busy`, counted from the `sessions.list` rows `/slots` answered
(`is_processing`, the shape pinned by `tests/slots_contract.py`); when `/slots` did not answer
there are no rows to count, so it falls back to the published `sessions.active` carrying the
API's own `sessions.detail` sentence rather than reporting 0 busy slots from no data (rule 2).
`preemptions` / `device_lost` are **alarm chips that exist only while non-zero** (`.chip.fail`,
container emptied and `hidden` at zero — the rule is the presence of an incident, never a graded
threshold) and they live on this card, not in a permanent cell; the lifetime token counter left
the panel's footnote line for the strip's row. **One `/api/llm/history` request per refresh**
carries every chart *and* spark series (`HISTORY_SERIES` = the four chart series +
`SPARK_CELLS`; the endpoint answers unknown names as empty series on purpose), and the two series
phase 04 added (`llm.tpot_seconds` vLLM-only, `llm.spec_accept_rate` gufo-only) stay stored
additively in `sources/llm.py` HISTORY_METRICS. Above the strip sits **the fleet block**
(`#fleet`, phase 07: "I'm missing the aggregate throughput at the top") — `ALL ENGINES`, a
reachability dot, an
`N of M engines` tag, the note *sums only engines answering now*, then running / waiting /
prompt tok/s / decode tok/s / total tokens. `FLEET_FIELDS` in `page-llm.js` is a list of
`fleet_totals()` keys and nothing else: the page sums no number of its own, a dead engine is
dropped from every total presented as current (its own stale-labelled panel still shows its
last good values), and the tag counts the answering engines, names the missing one in its
tooltip and takes the shared `.stale` dim while the sums are partial. It **hides below two
engines** — a cross-engine sum of one engine duplicates that engine's own panel — and it
carries **no severity colour anywhere**: `fleet_totals` has no `severity` to render (the
summed-KV bar is gone for that reason) and `severity.py` rates no throughput or queue depth,
so the numbers stay ink and only the dot speaks reachability. `fleet.seams` — the API's own
note on how the two engines count prompt tokens differently — is the prompt sum's tooltip
rather than another line of ink. **The block, the strip and the panels keep the page's shared
`.85rem` gap** (phase 13, `TODO.md:4`: fleet and strip rendered flush into one wall):
`#strip { margin-top: .85rem }` is unconditional because below two engines `#fleet` is `[hidden]`
and the strip *is* the first visible section, and `#banners:empty ~ #fleet { margin-top: .85rem }`
gives the block its top gap only when the banner slot renders nothing, so a banner's own
`margin-bottom:.85rem` never stacks a second gap under it. `tests/test_api.py` pins both rules on
the served stylesheet, pins that **exactly one** rule styles each of the two ids (a second,
unguarded one is the double gap), and pins that `#banners`/`#fleet`/`#strip` are still direct
siblings of `<main>` — the `~` combinator resolves between siblings, so re-flowing that markup has
to rewrite the selectors instead of losing the gap. The
four trend charts stay but at height 110 *below* the panels, each with an `ariaLabel`. Panels
sit at their own height (`#engines` carries `.items-start`): a two-hero gufo card next to a
latency-row vLLM one otherwise grows a blank bottom, and a blank panel reads as "no data". An
engine that leaves the payload leaves both components (`pruneEngines`), never a stale card in
either grid.
**The slot grid (`llm_monitor`'s, phase 07 task 04) belongs to the gufo panel, under its
`sessions (GET /slots)` label.** One card per row of `sessions.list`, **keyed by the row's own
`id`** in `state.slots`: built once by `slotCard()`, patched in place afterwards, and reordered
by `appendChild` of the existing node so DOM order stays slot order — rebuilding a busy slot's
card every 2 s poll re-announces the whole card to assistive tech for a difference of a few
tokens. `SLOT_FIELDS` is the whole table and each cell names **exactly one** `/slots` key
(`path: 'next_token'` reads `next_token[0]`, where the capture puts `n_remain`); `slotValue()`
is the only place a row is indexed, so `tests/test_api.py` can resolve every printed field
against `tests/fixtures/slots_gufo.json` and a rename fails there rather than printing dashes
or a plausible wrong number. No per-slot speed cell: the capture publishes no per-slot rate key
and a page-side Δn_decoded/Δt would be a measurement with no estimator, no `Δ<window>` label and
no `source` string (rule 2) — the engine's rate lives on its strip card and panel. llama.cpp's
`-1` sentinels print a word or nothing: `id_task: -1` is **`idle`**, `n_remain: -1` is **`n/a`**,
never a negative countdown and never the `0` that reads as one token from done. The busy marker
is a **border, a glyph (`○`/`●`) and the word `processing`** in the brand `--accent`, never an
`ok|warn|fail` token — a slot generating tokens is the job (phase 06's GPU reasoning), so
`severity.py` rates no slot and the stylesheet section contains no verdict token. The empty
state is a sentence, never a card of nothing, and it distinguishes the two cases: no `/slots`
answer prints the API's own `sessions.detail`, an answer with no rows prints *engine reports 0
slots*; `sessions.source` is the gate. The note under the grid stays because a **queued
(deferred) request is not in `/slots`** — the waiting row counts those. vLLM's panel gets one
line (*exposes no per-slot JSON*) with `sessions.detail` as its tooltip and **no grid**: a
component of permanent `n/a`s for a server that cannot answer is the dead-cell anti-pattern.
The gate is the engine name (`SLOT_ENGINES`), not row count, so an engine that has not answered
yet still shows its label and its honest empty sentence.
**Charts belong to the host they chart.** `/system` has two sections — the host panels and the
transition events — and nothing else: the fleet-wide chart stack, the nine-figure fleet-totals
panel and the host/metric detail picker are deleted, along with the `#chart-*` / `#detail-*` ids and
the `select` / `label.field` styling only those pickers used. Each panel stacks its own
`.host-charts` — utilisation (CPU, memory, worst disk), network (in/out), then GPU busy — where
the GPU **chart** slot is created and destroyed with `host.gpus` (its own thing from the hero
slot above it, which is hidden instead), so gateway shows neither a phantom card nor a hole. **That stack is collapsed behind a per-host `Charts` `<details>`** (phase 06: "only
visible if expanded… there will be lots of hosts"): the default `/system` view is the hero row,
the summary carries the window label the stack will draw (the chips themselves stay in the page
head, one control per page), and `drawHostCharts()` is the single place that decides whether to
paint — so nothing is built or drawn while the stack is shut (`LineChart` inside a
`display:none` container measures 0 px, and rebuilding three charts per host every poll for
plots nobody sees is the waste this removes). The **history request is deliberately unchanged**:
one per host per refresh, naming every series the panel could draw, and its response is cached
per host so the first expand paints from data in hand rather than waiting a poll; a cached
response from another window is re-fetched instead of being drawn under the wrong label.
Expanding one panel must not stretch its collapsed siblings into blank cards, so `#hosts`
carries the existing `.grid.items-start`. **One `/api/system/history` request per host per refresh**
(`hosts=<that host>&metrics=<that panel's series>&range=…`, pinned by
`test_every_system_chart_is_drawn_inside_its_own_host_panel` and the collapse by
`test_system_history_charts_are_collapsed_behind_a_toggle`): unknown metrics come back as
empty series on purpose, so a panel asks for every series it could draw and asks once. A legend
here names metrics (or one GPU's model), never hosts — the panel heading already says who this
is, and a legend that grew a second host is the bug this replaced. A host that stops reporting
loses its panel, its `Charts` toggle *and* its charts (`pruneHosts` deletes the card plus the
payload/chart/history maps), never a chart of nothing or an orphan control in the grid. Per-core
visuals are gone from `/system` and `renderCores` is **deleted** from `charts.js` with its last
caller.
`drawHostChart()` shades a chart only when one series has points — two or three filled areas
stack into one block and bury the quieter line, which is how a idle CPU would hide behind a busy
memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryable from
`/api/system/history` — removing the plot is not removing the data.
## Known gaps / next work
1. **UPS commands work, and the account can now cut power — the app's gates are the control.**
Three defects were cleared on 2026-10-17, each verified against real hardware. (a) `NutClient.login()` sent `USER`/`PASS`, which the deployed
upsd 2.8.5 does not implement — each answered `ERR UNKNOWN-COMMAND`, so `_read_line()`
raised and `POST /api/ups/command` could only 502 before `instcmd()`, credentials or not.
It now sends `USERNAME`/`PASSWORD`, the pair the server's own `HELP` advertises
(`HELP VER PROTVER GET LIST SET INSTCMD LOGIN LOGOUT USERNAME PASSWORD STARTTLS`), pinned
by `test_login_uses_the_verbs_upsd_2_8_5_answers_to` — the fake upsd has no `USER`/`PASS`
branch, so a regression answers `ERR UNKNOWN-COMMAND` exactly like the real thing.
(b) 12 of the 23 commands the two real UPSes advertise had **no `COMMANDS` entry**, and
`available_commands()` drops anything the catalog misses — the page silently offered less
than the hardware does. Every advertised command is catalogued now, each with a risk tier:
`test_every_command_the_hardware_advertises_is_catalogued` fails if a capture gains a
command the catalog does not know, and
`test_every_command_that_can_cut_power_is_in_the_dangerous_tier` keeps `load.off`,
`driver.killpower` and the `shutdown.*` sequence in the tier that types `DANGER` and is
refused on battery. `shutdown.stop` is the exception the name hides — it *cancels* a
shutdown, so it is `safe`, and the test says so in words.
(c) `[dashboards]` in `/etc/ups/upsd.users` is `instcmds = all` at the operator's request,
so this app may send every advertised command on both UPSes, power-cutting ones included.
`.env.dev` (mode `0600`, gitignored) carries the credentials.
What remains to know:
- **Proven live, and reversibly.** `beeper.enable` returned `OK` and `ups.beeper.status`
read `enabled` on the APC afterwards; `driver.killpower` without the word `DANGER` still
answers 422 and never reaches the wire. A battery test has deliberately not been run —
it puts real load on a battery things are plugged into, which stays the operator's call.
- **`enabled: true` cannot mean "this account may send it".** `available_commands()`
filters on `LIST CMD`, which upsd serves **unfiltered**, and NUT has no call for a
user's privileges — so the wire cannot tell the page what the account is granted. With
`instcmds = all` the two agree by configuration rather than by check, which is precisely
why the risk tiers and the typed `DANGER` are load-bearing and not decoration.
- **Pre-existing upsd log noise:** every upsd start logs `Unknown upsmon type =` (20
occurrences over 12 starts, oldest before this work). Harmless — `upsmon` logs into both
UPSes right after it — but unexplained.
- **One connection every few minutes answers nothing for >4 s.** Live upsd here stalls a
single poll past `NUT_TIMEOUT_SECONDS` roughly every 2–4 minutes and then answers normally
again; `nut` reads `ok` throughout, because the poller's job is one amber line plus the
recovery, not a frozen page (invariant 4). What phase 12's verification pass fixed on our
side was the **cascade** that used to follow it — the event table filling with
`expected 'BEGIN LIST VAR', got 'RW apc battery.charge.low "10"'`, which blamed the wire
protocol for a stall two commands earlier (see the `NutClient` bullet above). The stall
itself is an upsd/driver-side question with no repo bug in it, and a battery test is not
how to chase it.
Editing `upsd.users` at all: never grant `actions = -`, and never write it blind — the
running `upsmon` authenticates as its own account from that same file for both UPSes, so
dropping it means no orderly power-down. Copy the untouched sections from the live bytes
and `cmp` them before writing; `tee` truncates in place, so owner and mode survive.
2. **GPU coverage is AMD only.** `host_agent.gpus()` reads amdgpu DRM sysfs
(`gpu_busy_percent`, `mem_info_vram_*`, hwmon temps, clocks). That covers the current
fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at
all — needs a `nvidia-smi`/NVML or DCGM path, and a fixture from a real card before the
field map is written.
3. **UPS history depth.** Events and the numeric series are stored; long-window battery
health (capacity fade over months) has the rollups but no dedicated view.
4. **The dev DB's `database is locked` line is fixed (phase 14); a cold-cache tail is left.** The dev store (~640 MB on
2026-10-19, its WAL grown past 140 MB at its worst) logged `store-error: OperationalError:
database is locked` a few times an hour across `system`/`llm`/`ups` through 2026-10-18; the
poller handles it exactly as invariant 4 requires — the source goes red with the real reason,
ages, and writes `recording` when the next poll lands — so nothing freezes, but those polls'
samples are lost. The one cause this repo can prove is two servers on one file, which is why
`scripts/dev-server.sh stop` waits for the old pid to die (its comment records the incident).
A controlled 25 s external read transaction during the phase-09 pass produced none, and
phase 12's re-check on 2026-10-19 found the 190 stored `store-error` rows all older than
2026-10-18 06:33 with none in the previous 24 h of the running dev store — the absence of a
sample, not the absence of the pressure, since the file only keeps growing.
Retention/`wal_checkpoint` tuning for a long-running store was its own phase rather than a
phase-09 follow-up — that phase is `14_store_performance`, and the two paragraphs below are
what it measured and changed.
**Phase 14 has since split this in two, and the read half is closed.** The in-process half was
never a SQLite lock at all: `SampleStore._lock` serialises everything, so a 1.25 s `stats()`
from a browser tab's `/api/health` poll was a poller waiting up to 1.45 s to insert 30 rows
(measured, `data/logs/NOTES-store-perf.md`). Task 02 took the two whole-table reads off the
per-request path, and `/api/health` now answers in ~1.2 ms instead of ~1.27 s, so the store
mutex is no longer held by a reader for seconds at a time. The **write half is closed too**
(task 03): `maintain()` rolls up only what became due since `last_maintain`, its prune is one
`DELETE` per resolution instead of one `OR` the planner could not use, and `PRAGMA optimize` runs
there (with SQLite's automatic ANALYZE switched off, so no poller's commit pays for it). On the
seeded 6.4 M-row store that is 5,368 ms → **2.7 ms** steady / 48 ms for a ten-minute catch-up, and
the two-connection race at the app's cadence is **100 of 100 inserts, zero `database is locked`,
max wait 21 ms** where the pre-fix control still waits 4.76 s (task 04's re-run of the same
budget: 6.8 ms steady / 46 ms catch-up / 8.5 ms worst wait, so the row is a range, not a
digit). The WAL follows the same shape: a steady pass leaves ~0 MB and a catch-up tick 4 MB
instead of 38 MB that never comes back down, and `journal_size_limit=64 MB` caps what a
long-lived reader can make it grow to (`data/logs/NOTES-store-perf.md` has the tables).
**The live proof is in** (task 04, 25 minutes / 74 passes / 296 signed-in requests against the
running dev server on the operator's own 636 MB store): **0 new `store-error` rows** (still 190,
newest 2026-10-18 06:33:30, i.e. 39 h before the watch), **WAL 6.5 → 7.77 MB** against the 67.1 MB
cap and a main file that did not move, 0 non-200 responses, and page paths at 1.8-3.3 ms median
(`/api/health`, `/api/summary`, `/api/metrics`, a 3-host × 9-metric 90-day `/api/system/history`).
The watch also caught a defect in task 02's fix — the series cache was invalidated by *every*
sample write, so on a store the hub writes to every few seconds it never served and `/api/metrics`
paid the 572 ms scan on nearly every call (measured: 535 ms per call pre-fix, 1.8 ms median
after) — and that is what the store bullet above now describes.
What is **left** is the cost of the two cold answers themselves, measured on that store: 599 ms
for `series_names('host')`, 729 ms for `stats()`' counts, now once per cache window (~1.3 s of
store mutex per minute) instead of once per call — which is why the watch's medians are 1.8-3.3 ms
and its p95s are 620/729 ms: the p95 is the window boundary, and 37 of 74 passes had one request
over 250 ms, every one of them a cold re-derivation. No cache policy reaches *that*; the fix is
the bounded `(source, metric)` structure or a counts table named in task 03's note. Two cheaper
knobs exist and were deliberately not pulled, because both change what `/api/health` answers and
move rather than remove the mutex hold: raising `STATS_CACHE_TTL_SECONDS`, and refreshing both
caches from the maintenance tick so the miss lands on a poller instead of a page load. That is a
phase, not a cleanup. Two servers on one file can still contend, and `busy_timeout=5000` is
unchanged on purpose so a second one stays visible as a slower write; what they contend with is
now milliseconds instead of 5.4 s.
One consequence worth naming, because two other gaps pointed here for it: the **~10 s
process-wide stall** (gap 9's `Δ`-window dilution, `poller`'s catch-up gaps) is gone with the
mutex holds — coincident multi-second gaps in the 2 s `llm` and 5 s `ups` raw streams went from
393 and 198 in the two halves of 2026-10-19 to **0** across the 110 minutes the phase-14 store
has run (re-measured in this phase's verification pass from the `sample` table; the signature
separates a process stall from one slow engine).
5. **The login throttle keys on the socket peer address** (phase 11), and the one
`LoginThrottle` instance now holds two keys per address: the bare one for
`POST /api/auth/login` and `password:<address>` for `POST /api/auth/password`. Behind nginx
every login shares one bucket, so anyone who can reach the login form can lock the operator out
of the *dashboard* — not the UPS, not the data, not a host — for at most
`LoginThrottle.max_seconds` (1 h by default) at a time, and a process restart clears the
counters. The two keys are separate on purpose, so neither form can be hammered at the other's
rate and a stolen cookie plus the change form is not an unthrottled oracle for the old
password; the cost of that choice is that one address can be blocked twice over. Accepted for a
single-admin box rather than becoming an `X-Forwarded-For` knob:
that alternative is spoofable by anyone the moment somebody forgets to configure which proxy
to trust, which would hand the attacker the throttle key instead of leaving it on the
network layer. PBKDF2-HMAC-SHA256 at `PBKDF2_ITERATIONS` (600 000) stays the real cost of
every guess; the throttle only makes hammering the form slower than useless.
6. **One admin, one factor — no second factor and no per-user accounts, by design** (phase 11).
The operator asked for a single admin, and that is what the model is: one verifier row, one
username in every signed session, one name in every `password-changed` event, no read-only
role and no per-viewer view. The consequence is worth stating out loud, because it is the
whole security budget of a public domain: the PBKDF2 cost plus the backoff is the only thing
between a stranger and the fleet's hostnames and the UPS command buttons, so there is no
defence in depth behind it — no TOTP, no WebAuthn, no per-IP allow-list, no "this login is
from somewhere new" notice. Adding a second human means more than a second row: the gate has
no notion of scope, so a colleague would be able to cut power unless somebody designs a
read-only session first. That is a phase, not an edit.
7. **The throttle is per-process, so a second uvicorn worker multiplies it** (phase 11).
`LoginThrottle` is a dict in one process's memory: `uvicorn --workers 4` hands anyone who
reaches the form four independent buckets and four times the attempts before a `429`, and a
restart clears all of them (the gap-5 note above). Nothing else in the auth design has that
problem — the session is an HMAC-signed cookie and the CSRF token derives from it, so both
are worker-safe by construction with no store to share and nothing to prune. The reason this
stays unfixed is that the app is single-process by nature anyway: every worker would run its
own poller loops against the same SQLite file (AGENTS.md invariant 4, and the `database is
locked` incident in gap 4 is what two processes on one store already looks like). If this is
ever revisited, the fix is a shared counter (the `meta` table would do, at login rates), not
a bigger number in one process.
8. **A logout ends this browser's cookie, not a copy of it** (phase 11). The session is an
HMAC-signed cookie with no server-side record — that is the design invariant 12 and gap 7 rest
on — so `POST /api/auth/logout` can only tell *this* browser to drop the cookie (verified
against a running server: a replayed token from a logged-out jar still authenticates until it
expires). The same is true of an `Authorization: Bearer` token, which was never in a cookie to
begin with. The only revocation this app has is rotating `auth.session_secret`, which
`POST /api/auth/password` does on purpose, and the reset `DELETE FROM meta …` does as the
lockout escape hatch. Accepted for a single-admin box on a domain nobody else signs in to —
the operator's own machine is the threat model here, not a shared kiosk — and the reason a
lost laptop is a password change and not a sign-out. A real revocation list would be a row per
session with a prune rule, which is the thing this design traded away.
9. **The `/llm` `decode tok/s` spike is fixed, and verified live** (phase 13, 2026-10-19;
the diagnosis is in the git-ignored `data/logs/NOTES-decode-spike.md`, the live proof in
`data/logs/NOTES-decode-stable.md`). **Task 04's sustained watch passed the ceiling test:**
11 minutes / 285 signed-in `/api/llm/snapshot` polls against the live engines, worst value
**117.57 tok/s** (gufo) and **98.00** (vLLM) — both in the payload and in the stored raw
series, 2.6× under the 300 tok/s pass line, with **zero** values printed without their
`Δ<window>` label. The stall is still there (38 catch-up gaps < 1.5 s in 11 minutes, a
10.6 s gap between two samples) and the estimator now absorbs it: three refusals each
labelled with the sub-0.5 s window it refused (`Δ0.387s`, `Δ0.399s`, `Δ0.363s`), two ≥ 8×
steps held for one poll and published only when the next interval confirmed them, and — the
point of the whole fix — **1 m rollups topping out at 51.9 / 34.1 tok/s** where task 02 found
a 222.7 tok/s minute, so the trend chart's y-scale stops being set by arithmetic. Fixtures
were **not** re-captured: task 02 compared the live exposure with `tests/fixtures/` series
by series (gufo 15/15, vLLM 128/128, identical sets), task 13 changed no parsing, and the
page still reads `15 series · 0 unparsed` / `128 series · 0 unparsed`. Screenshots:
`data/logs/llm-decode-stable.png` (+ `-top`, the fleet-block gap and both `Δ` tags) and
`data/logs/llm-decode-burst.png` (a real two-slot burst: 62.1 tok/s `Δ8.5s`, fleet sum 102.4).
The confirmed mechanism was neither a counter bug nor a label-set sum: `poller.Job._loop`
sleeps `max(0.2, interval - elapsed)`, so any fetch slower than `LLM_INTERVAL_SECONDS`
is followed by a poll **0.2 s later**, and the estimator happily diffed a whole stall's
tokens over that gap — the 2026-10-18 09:16 gufo row read **3284.59 tok/s for an engine
doing 53**, with an inflated tail because the stale sample stayed `inside[0]`. Task 03
made that impossible by construction (see the `rates.py` bullet above): the window floor
turns the catch-up poll into one `n/a` **and drops the stale base** so the tail re-bases;
the fallback's `2 × span` ceiling is enforced (the honest 15.25 s diluted row still
publishes); jumps ≥ 8× hold one poll and publish only on confirmation; and
`values_by_series()` + `rate_key()` diff every counter
per label set, so the latent joining-series defect (candidate (a)) contributes 0 tokens
until it has its own baseline. The four task-02 `xfail(strict=True)` tests are green with
their markers removed and every assertion kept. Spike-free across two days of raw history
the fleet's real maximum is **119.9 tok/s** (gufo / deskwork) and **171.7**
(vLLM / driveripper) — anything above ~300 is arithmetic, not work, which was task 04's
pass line and what the watch above stayed under. The `Δ<window>` label stayed honest throughout —
both before and after the fix it names the window actually used; do not "fix" anything
here by hiding or rewriting it. The ~10–15 s process-wide **stall** that triggers a
refused poll was gap 4's and `14_store_performance`'s problem, and **phase 14 closed it**
(measured on the dev store from the sample table: coincident multi-second gaps in the 2 s `llm`
and 5 s `ups` raw streams — the signature of a process-wide stall, not a slow engine — were
393 in 00:00–12:00 and 198 in 12:00–21:36 on 2026-10-19, and **0** across the 110 minutes the
phase-14 store has been running, worst `llm` gap 7.1 s and that one the recorded gufo timeout).
The mechanism is the one phase 14 removed: a whole-table `stats()` count held the store mutex
for ~1.25 s on every browser tab's `/api/health` poll, and `maintain()` held the write lock for
5.4 s. The estimator no longer
turns a slow fetch into 0.2 s arithmetic, and one `n/a` per stall is the designed degradation
(the residual the verification pass measured is the paragraph below).
**The verification pass found one residual path and closed it** (same day, live rows in the
store the fix was running): a stalled fetch stamps its body with the time the poll
*finished*, so the sample it stores is the stall older than its own timestamp. The floor
cannot see that (the window is honest, the value is old), the sample then anchors every
window for the rest of the span, and the jump gate published the second one because it was
measured from the same anchor — gufo 12:42:10 read **232.28 tok/s** with exactly one request
running at ~50, followed by the tell-tale decaying tail 146.4 / 114.1 / 94.8 and the true
47.5 (a 13:40:58 row repeats it at 200.6). `rates.py` now requires an **independent anchor**
for a confirmation (`_pending` stores the base sample's timestamp and the hold continues
while the window is still diffed from it), so a skewed sample is held until it ages out and
the honest rate publishes — the reconstruction of both rows is pinned by
`test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor` (estimator) and
`test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine` (through
`parse_gufo_metrics`), each of which fails on the pre-fix gate and prints the live shape.
**That closes the gate's path, not the skew itself, and the verification pass measured the
difference.** A slow reply's counter is true somewhere between the request and the reply and
nothing in the payload says where, so when the resulting jump is *under* 8× the last published
rate the gate never engages: the post-fix watch (10 min / 540 reads, `data/logs/
llm-decode-verify-pass.tsv`) caught gufo at **203.05 tok/s labelled `Δ2.595s`** where two busy
slots were really doing ~110, then 127.4 / 100.2 / 89.6 as the window grew back to the span and
35.6 once the freed slot settled — every value labelled, **0 of 540 above the 300 pass line**,
and no stored raw row over it since the restart either. A collapsed window is what makes a
burst *look* like a rate, so the fix for this family is the ~10 s stall itself (gap 4 /
`14_store_performance` — which has since removed it, see gap 4 and the paragraph above, so what
is left of this family is the sub-8× skew itself), not a fresh threshold invented in the
estimator: task 03 pinned
`MIN_WINDOW_SECONDS` as an absolute floor and pinned that an in-bounds diluted row publishes
(`test_a_fallback_inside_the_documented_ceiling_still_publishes`), so a span-relative floor
would be a re-decision of that phase, not a bug fix. Do not "fix" the residual by hiding the
short window or its label.
**Phase 23 changed the size of the window, not the gates** (`TODO.md:8`: "a strange delay and a
'ramp up/ramp down' effect"). What was left after phases 13/14 was the span itself: one
10 s-window estimator served both cadences, so on the 2 s `llm` poll a slot's first seconds
were spread over five polls — the capture replay in
`tests/test_llm_source.py::test_gufo_the_window_the_llm_jobs_are_spanned_by_is_one_that_fills`
prints 12.5 → 20 → 30 tok/s for an engine steady at 50, and the same five polls backwards when
the slot ends. `build_hub` now builds **one estimator per measuring cadence** (the `llm` jobs
spanned by `LLM_INTERVAL_SECONDS`, the host jobs by `RATE_WINDOW_SECONDS`), which reads 50 on
the first poll after the change and 0 on the first after it ends. Every phase-13 protection
still applies to the smaller window — `Δ2.0s` is 4× above `MIN_WINDOW_SECONDS`, the ceiling is
relative to the span, and the 8× jump gate is the estimator's own — so this is the smoothing
dial turned down, not a second definition of a rate.
**The live half has landed** (task 03, 2026-10-21, read-only: nothing was written and no
workload was started — the busy periods were the operator's own requests). A 13-minute
signed-in pass of `/api/llm/snapshot` at the app's own 2 s cadence
(`scripts/watch-llm-window.py`; series `data/logs/llm-window-watch.tsv`, verdict
`data/logs/llm-window-watch.{json,log}`) recorded **zero** values printed without their
`Δ<window>` label across 780 rows, with every window in 0.61-3.38 s and never 10 s. It found
17 busy-edges on gufo and judged the 5 starts + 5 ends that held long enough (an edge followed
by the next request within a poll is not a slot ending, and prefill polls genuinely decode
~0 — both are reported, neither is judged): **every** one was inside the 8× pass line at poll 2
on the ungated Δ. Re-diffing the *same samples* over the span this phase replaced is the
comparison the operator's complaint was about: median |ln(value/steady)| at poll 2 was **0.037**
with this window against **0.924** with the 10 s one, whose median edge needed **poll 3** to
settle (the five starts read 0.98 / 1.04 / 0.95 / 1.00 / 1.27 of steady at poll 2 on the
ungated 2 s Δ where those same five read 0.25 / 0.34 / 0.96 / 0.40 / 0.40 on the 10 s one; ends
took polls 4-5 to fall away against poll 2 now).
One start had its first correct value held for a poll by the phase-13 jump gate and published
on the next — designed behaviour, and the reason the pass line is read on the ungated re-diff
as well as on the published column. The stored **1 m rollups** for the window topped out at
**42.3 tok/s** (raw max 201.6 — that row's own `Δ0.610s` label says so, gap 9's sub-8× skew,
far under the 300 pass line). A second pass over the same engines an hour later
(`data/logs/llm-window-watch-pass2.log`) reproduced it: 3 starts + 2 ends, 0 unsettled, 0
unlabelled, 1 m rollups max 24.7 tok/s.
The browser half (`data/logs/23-live-window.cjs`, transcript `data/logs/23-live-window.out`,
screenshots `data/logs/23-live-llm-{opening,burst-settled,midburst,post-free,gufo-idle,vllm-idle,final}.png`)
checked the same claims as rendered: **866** decode figures, **0** without a tag in either
component that prints the number (the panel hero's `.sub` and the strip card's `.prov`), every
tag a Δ inside the estimator's own bounds — `MIN_WINDOW_SECONDS` to `2 × span`, the modal one
`Δ2.0s` on both the page and the API (833 of 866) and never the host's `Δ10s`. Its own edge
checks (isolated only, for the same reason) read inside the 8× line at poll 2 on two starts and
exactly `0.0 tok/s` at poll 2 on two ends — the frame named post-free is only taken once the
engine is provably idle, because the busy signal flickers between back-to-back requests.
10. **`driveripper`'s 16 drives read `n/a`: kernel 7.2's unprivileged-passthrough gate
(2026-10-19, resolved the same day; the installer template that shipped the broken device
filter was cleared in phase 17).**
The agent is installed, the `disk` group grant is in place, the block nodes open, and the
page renders all 16 real drive rows — but kernel 7.2.11 blocks the *agent user's* `smartctl`
at the command level (not the open; the probe matrix is in
`data/logs/NOTES-driveripper-agent.md`): the 8 SATA drives' SMART is ATA pass-through
behind a SAT layer, and the kernel's unprivileged `SG_IO` allow-list
(`scsi_cmd_allowed()`, read-safe opcodes only) does not include it — `CAP_SYS_RAWIO` or
root unblocks it, with or without `-d sat`/`-d ata`; the 8 Samsung/WD NVMes' SMART-log
read needs the drive's ACEL to advertise the page (the 980 PRO's `5B2QGXA7`, 2020-era,
does not: `NVME_IOCTL_ADMIN_CMD: Permission denied`) or `CAP_SYS_ADMIN`; `deskwork` reads
live because its firmware does. Options, in order of narrowness: (a) accept the designed
degradation (what `/disks` rendered then — the identity is real, the values are `n/a` with a
why-line); (b) `AmbientCapabilities=CAP_SYS_RAWIO` (+ `CapabilityBoundingSet`) on the unit
— unblocks exactly the 8 HDDs, is the capability the kernel names for the gate, but a
compromised agent could then write raw block I/O — test it in a `systemd-run --scope`
before committing; (c) `User=root` with the empty capability set — unblocks all 16 but
hands a token-guarded process root's file-read DAC (not recommended); (d) a Samsung
firmware update — the only clean fix for the NVMe side. **Resolved the same day with a
tighter variant of (c): `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`
— all 16 drives read live** (root DAC is the price; the bounding set keeps the grant
named rather than full). Two empirical notes: a root process under the unit's *empty*
bounding set is capability-less (`CapEff: 0`), because the kernel gates check
capabilities, not uid — that is why the first `User=root` attempt still read `n/a`; and
`CapEff` came back `0000000000220000` (SYS_ADMIN + systemd's auto CHROOT; RAWIO in the
bounding set but not effective) — the 8 SATA drives' reads were unblocked by SYS_ADMIN
alone, so the effective grant is {SYS_ADMIN}; the RAWIO in the line stays as the named
intent for the `SG_IO` gate. The agent also gained the pass-budget scale-up — 20 s floor, 5 s ×
disk count, an explicit override still winning — because a 16-disk pass takes more than 20 s
when the drives answer, and truncating it would null the tail drives. Phase 16 pinned that
default in `tests/test_agent.py` (`test_the_pass_budget_scales_with_the_disk_count`,
`test_the_pass_budget_floor_is_twenty_seconds`,
`test_the_pass_budget_error_line_names_the_scaled_figure` — which demands the line read `80s`
for 16 disks, not the floor; the override test stays). Re-read live on 2026-10-20: the installed
agent answers 16/16 drives with temperatures and no `pass budget … exhausted` line.
The `deploy/install-agent.sh` template no longer ships the device-filter lines that
misbehaved on this host — phase 17 task 01 removed them (the `nvme*` glob matches nothing
on this kernel because `/proc/devices` lists NVMe as `blkext`, and a strict device policy
denies everything a list fails to match), and the unit pins in `tests/test_deploy_scripts.py`
now assert their absence. Task 02 added the two flags that reproduce this host's remaining
lines — `--as-root` → `User=root`, `--caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` →
`CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`, both off by default, `--caps` refused
without `--as-root`, `--as-root` alone accepted with the CapEff-0 note — pinned by
`tests/test_deploy_scripts.py` (exact `User=` / `CapabilityBoundingSet=` lines in a fake-root
sandbox, the refusal before any write, the `--help` recipe). Task 03 removed the unconditional
rewrite: the unit is rendered to a temp file, and a re-run whose render differs from the
installed unit prints the diff on stdout and exits 1 unless `--force` is passed, so the flag
line above now needs `--force` to land and a routine re-run can no longer clobber this host's
unit (pinned by the refuse/force/silent sequence in `tests/test_deploy_scripts.py`, including
the `--force` leg of this host's own recipe — the same
`--bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` line the
fleet record above prints). Task 04 verified the reproduction **read-only** rather
than by running it: the unit installed on `driveripper` (`systemctl cat`, 2026-10-20) differs
from what this template renders with that exact flag line in **comment lines only** — every
directive matches, `User=root`, `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`, no device
filter, `PrivateDevices=false`, the same `ExecStart` on the token file — and the host still
answers `active` with `CapEff: 0000000000220000` and 16/16 drives with temperatures in
`/snapshot`. So the operator's `--force` run lands the new rationale prose and changes nothing
systemd acts on (the three captures and the exact probe list are in
`data/logs/NOTES-17-installer-repro.md`; no live change was made while pinning this phase). Two facts that came out of reading the live host, both load-bearing for the
next install: the sudo-granted `/home/ops/agent-install` path held the Oct-18 file (no
`--as-root`, no `--caps`, no `--force`, no `--cert-bundle`, still nine `DeviceAllow` lines) and
it **refused** the recipe with `unknown argument: --as-root` and exit 2 before `need_root`, so
that path could not apply this recipe at all — while a re-run of it *without* the flags was
still the silent clobber the guard exists for. **Resolved on 2026-10-21:** phase 24 task 03
staged the repo's script and agent 1.1.0 into `/home/ops/agent-install/` on all three hosts
(the directory is `ops`-owned, so no root needed) before running the granted path, and left the
snapshot beside it as a file sudo will not execute. The rule that came out of it is in the
fleet record above: re-stage first, because a path grant is only as current as the file. (The other
fact from that read — an applied change reaches the file but not the running process until
something restarts it — is in the Deployment paragraph, and README's rollout now restarts.)
`tests/fixtures/smartctl_driveripper_*.txt` (captured via `sudo`) remain the ground truth
for the drives' shape.
11. **The fleet runs on installed bundles; what is left is rotation hygiene**
(phase 15 tasks 01-05 landed 2026-10-20; phase 24 task 03 rolled the fleet out 2026-10-21).
The rollout is done and verified: gateway, deskwork and driveripper each run `agent/host_agent.py`
**1.1.0** with `/etc/dashboards-agent.certs` installed `0640 root:dashboards-agent`, their units
carry `--cert-bundle` + `RuntimeDirectory=dashboards-agent` (and keep `--token-file`) — the
gateway has since moved to **1.2.0** (phase 28) and then **1.3.0** (phase 32) on the same unit and
the same bundle, see the fleet record above —
`HOST_TARGETS` is `https://` for all three with no userinfo, `agent.bundle_rev` reads `2` on
every host and equals what `/api/certs` issued, and the nine sources are `ok`. The transcript is
`data/logs/NOTES-24-rollout.md`, and task 04 re-read the whole end state **read-only** afterwards
(`data/logs/verify-24-mtls.py` → `24-task04-endstate.txt`, plus the rendered proof
`data/logs/24-task04-verify.cjs` → `24-task04-{certs,disks,system}.png`): nine sources `ok`,
`status: active` with `deployed_bundle_rev == bundle_rev` on all three rows, `agent.tls` +
`agent.bundle_rev` + `agent.version 1.1.0` in every host payload, driveripper's 16 drives and
deskwork's `nvme0n1` reporting temperature, hours and wear, deskwork's GPU card drawn on
`/system`, gateway still absent from `/disks` with `disks:gateway` still `ok`, no `http://` and
no userinfo in any URL the dashboard serves, and **no transport or certificate row in
`/api/summary`'s alerts**. It revoked nothing: the standing negative control for "a revoked
serial is refused" remains phase 15's `data/logs/verify-15-tls-e2e.py` (transcript
`verify-15-tls-e2e.out`) plus the two suite pins — `tests/test_collectors.py`'s
`test_a_revoked_agent_is_refused_before_anything_is_dialled` and `tests/test_api.py`'s
`test_a_revoked_agent_fails_both_of_its_sources_without_a_dial`.
The **docs** are part of this end state, because a fleet comes back to `http://` through a
setup file somebody copies: `tests/test_deploy_scripts.py` now pins that `README.md`,
`AGENTS.md` and both tracked env templates name the registration trio (`--fetch-bundle`,
`--cert-name`, `--session`), and that every `HOST_TARGETS` example line in the templates is
`https://` with no userinfo (an empty `HOST_TARGETS=` stays allowed — unconfigured is not
misconfigured). The token-mode dial itself is unchanged and still documented as the
compatibility path; what may not come back is it being the shape a template seeds.
Two things that gap learned belong with this record, because both bite before anybody reads a
dashboard:
* **A `certs.root.cert` row older than the SKI/AKI fix signs chains nobody can verify.** "A
stored row always wins" is the invariant that keeps a CA from rotating with the process, and
its cost is that a root minted before the extension keeps being used forever: a leaf issued
afterwards carries a correct AKI pointing at an identifier its issuer does not have, and
`ssl.create_default_context()` — strict mode, the context `AgentTls.ssl_context()` builds —
answers `CERTIFICATE_VERIFY_FAILED: Missing Subject Key Identifier`. The agent is healthy,
`--status` answers over TLS, and the dashboard calls the host unreachable. The repair is
`POST /api/certs/root/rotate`, a dashboard restart (the CA and client pair are startup
material) and a fresh bundle per host — never `DELETE FROM meta`, which throws away the
authority instead of replacing it. Pinned by
`tests/test_certs.py::test_a_stored_root_without_an_identifier_still_boots_and_root_rotate_repairs_it`,
which also pins the other half: reading such a table must **not** fail closed, because
`/certs` is the page that fixes it.
* **These bundles came from the dev-box authority.** The fleet trusts the `certs.*` rows in the
database `./scripts/dev-server.sh` runs against, and nothing else dials these agents (a
`ss` sweep on each host saw only the dev box on 9120). A second deployment with its own
authority cannot read this fleet until it carries the same root or re-issues and reinstalls;
that is a deployment-repo decision, and the honest version is one authority per fleet.
Everything below is the phase-15/24 history that led here.
`certs.py` mints the CA and the dashboard's client certificate into `meta` on the first boot of
a database — verified against the running dev store: the five `certs.*` rows exist
and did not change across two restarts — signs per-host server certificates into
`certs.agents`, and does reissue, revocation, and root rotation. Task 02 gave
`agent/host_agent.py` its TLS mode (see the agent bullet above: one installed bundle, the
per-path client-cert check, the live `# REVOKED` denylist, `bundle_rev` in the snapshot) and
task 03 gave the collector the matching half: an `https://` `HOST_TARGETS` entry is fetched
with this app's client certificate and verified against its own root, and a host marked
`revoked` fails its `host:` and `disks:` jobs with `agent certificate revoked` **before any
socket opens** — spy-pinned in `tests/test_collectors.py` and again through the real
`build_hub` wiring in `tests/test_api.py`, where the revocation is written into the same `meta`
table the running app reads. So both halves are proven against each other with no network and
no hardware. What phase 15 left missing — an **installed** bundle, which is what makes
`DELETE FROM meta` the expensive command the reset message warns about — is what task 03
supplied: deleting `certs.*` now costs the fleet's identity, and every host has to be
re-bundled. Phase 24
task 01 has since closed the **dev-side** half of that sentence: `./scripts/dev-server.sh
restart` exports every issued bundle into gitignored `data/certs/<name>.bundle` at `0600`, so
putting one on a host is an `scp` plus `install-agent.sh --cert-bundle` rather than a hand-run
signed `curl` (`scripts/dev-certs-export.py`, invariant 6's export rules, pinned by
`tests/test_deploy_scripts.py` against a real uvicorn-served `create_app`). Task 02 has since
removed even that hand-run from the rollout: `install-agent.sh --fetch-bundle <url> --cert-name
<name> --session <token>` fetches `GET /api/certs/<name>/bundle` during the install and feeds
the bytes to the unchanged validator + `0640` install + TLS-unit path, refusing a 401, an
unissued name or a non-bundle body before it writes anything, with the session token absent from
every artefact the run leaves (pinned the same way, over the same socket) — which is the path
all three hosts were installed through, over an SSH reverse tunnel because no monitored host
can route back to a laptop. Proving
the dial also found one defect in task 01's output: the leaves carried no
`AuthorityKeyIdentifier`, and OpenSSL's strict mode — which is what Python's
`ssl.create_default_context()` turns on, and that is the context this dial builds — refuses
such a chain with `CERTIFICATE_VERIFY_FAILED: Missing Authority Key Identifier`. Fixed in
`certs.py` (SKI on the root and every leaf, AKI on every leaf, non-critical), pinned in
`tests/test_certs.py`, and the reason a bundle recorded before it must not be installed. Task
04 put the same lifecycle behind the gate: `/api/certs` lists the **stored** authority (through
`certs.read_state`, so a page refresh cannot mint anything) with one row per configured host and
per existing entry, `GET /api/certs/<name>/bundle` serves the stored bytes as an `attachment`,
and the three writes rotate, revoke, and rotate the root with `restart_required: true` said out
loud — each leaving one fixed-text `certs`-source event and no PEM outside `meta` (pinned by a
grep of the stored rows and of every payload a browser reads). Task 05 put the page on top of it:
`/certs` (`⚷ Certs`, in every signed-in masthead — its own included, since phase 20 task 04)
renders that document — the root
CN/serial/expiry, the display-only client line, one row per host with its status chip and
issued-vs-deployed bundle rev and its **Rotate** / **Revoke** / **bundle** controls — and keeps
two rules the API already keeps: no colour the payload did not send (no `severity` token on the
page at all; `unissued` is a grey chip, a behind bundle a neutral `stale` one) and no control
the API would refuse (no buttons on a `configured: false` row, no revoke or download on a
never-issued host, the download a plain session `GET`). Writes are `confirm()`d rather than
typed-`DANGER`d, and the restart sentence is the route's own `hint`. Task 06 has since landed
the installer half: `install-agent.sh --cert-bundle <file>` validates the bundle by the
agent's own text shape *before* `need_root` (a bad file says "wrong file", not "add a sudoers
line"; exit 2), installs it `0640 root:dashboards-agent` byte-identically — the task file's
literal `0600` cannot work, because the agent opens the bundle as itself and re-reads the
denylist live (the exact incident the token file's comment records), and the task's own
intent is "only root and the agent's own service user may read it", which is what 0640
achieves — re-runs without the flag leave the installed bundle in place, the unit gains
`--cert-bundle` + `RuntimeDirectory=dashboards-agent` (`0700`) from one conditional branch
while keeping the token line, `--status` probes `https` with the bundle's CA and no client
certificate (`python3` `ssl`, never an external TLS binary; an unreadable bundle says *run
--status as root* instead of pretending), `--uninstall` sweeps the bundle with the token, and
`deploy/agent-sudoers` gained no line (the bundle travels through the same root invocation).
`tests/test_deploy_scripts.py` pins all of it, including the probe run end-to-end against a
real in-process TLS agent: CA-only reads `/health`, a foreign root is refused, token mode
keeps the plain probe. The final pass proved the composition rather than the halves: a real
`host_agent.py --cert-bundle` on loopback and a real dashboard process over a scratch database
(`HOST_TARGETS=loop=https://127.0.0.1:<port>`) answer 27 checks — `host:loop`/`disks:loop` `ok`
through the certificate dial, `agent.tls`/`agent.bundle_rev` in both snapshots, the bundle
download byte-identical to the `meta` row, an unsigned `/certs` a `303` with `?next=`, a revoke
followed by `agent certificate revoked` on both jobs with no dial, the dashboard's own client
identity untouched by a *host* revoke (the list is serials, not hosts), and a revoked **client**
serial refused 403 by the running agent on `/health` too the moment the re-sealed bundle is in
place (`data/logs/verify-15-tls-e2e.py`, transcript `verify-15-tls-e2e.out`).
What was left was the live rollout, and phase 24 task 03 ran it on 2026-10-21, host by host, in
the order that keeps every number honest: install **1.1.0 together with the bundle** (never the
bundle alone — the installed agents were 1.0.0 and do not know `--cert-bundle`), restart the
unit, let the CA-only `--status` probe answer, and only *then* flip that host's URL, because a
URL flipped before its bundle is installed turns the host red by construction. All three: the
trivial `gateway`, the drop-in-carrying `deskwork`, and `driveripper` with the gap-10 line
(`--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force`), whose dry-run diff moved comments and
the TLS lines only — `User=root` and `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` came
through byte-identical, and all 16 drives read live on both sides of the rewrite (same single
`nvme5n1` alarm chip). Every host ran the installer **twice**, once without `--force` to read
what task 17's guard would refuse before letting it apply. Staging the repo's script and agent
into the sudo-granted `/home/ops/agent-install/` path was step one on all three (gap 10), and
fetching was reached through an SSH reverse tunnel because no monitored host can route back to
the dev box. `data/logs/NOTES-24-rollout.md` is the transcript: commands, unit diffs, the
before/after SMART tables, the screenshots, and the two operational traps named at the head of
this gap. What the fleet now *never* does is answer an agent over plain `http` or accept a
bearer token on the certificate path — the TODO's "the client should never allow non-mtls
communication" is the shipped state, and a new host has to be registered the same way.
12. **Closed: the nav collapses on a phone, every table scrolls inside its own card, and no card
edge eats a fact** (the clipping is phase 18's verification measurement of 2026-10-20; the
phone half was closed by `27_mobile_tables_and_audit` task 01 and everything the audit found
after it by task 03, both on 2026-10-21).
What was wrong: at 360px `/`'s fleet table measured 450-510 px inside a 320 px `section.card`,
and `.card { overflow: hidden }` clipped the tail columns (`WORST DISK`, `GPU BUSY`, `TEMP`,
`UP`) with no scroll container to reach them — `table.data` scrolled only with its `.scroll`
modifier, which that table does not carry. It was measured as pre-existing by serving the
`git show HEAD:` bytes of both `dash.css` and `pages/index.html` over a live session: same
clipping (a clipped card never un-clips when the document scrolls sideways). Phase 18 had
already stopped the *document* scrolling sideways at a phone width — 360 px where it was
750 px — by breaking the bar into rows, so the masthead was complete and only the table tail
was out of reach.
**The phone half is closed**, and not by the shape gap 12 proposed (`.scroll` on the wide
tables, or a per-host card layout): one ≤640px rule makes every `table.data` a horizontal
scroll container inside the card, with no height cap, plus `min-width: 0` on the items of
`.body.stack` — the second declaration is what the first one needed, because a card body is a
one-column grid whose items may not be narrower than their `nowrap` content by default (before
it, `/system`'s "More detail" tables measured 520 px in a 320 px card with
`scrollWidth === clientWidth`: nothing to scroll, tail clipped). Pinned on the served
stylesheet by `tests/test_api.py::test_every_table_data_scrolls_sideways_at_the_phone_breakpoint`
and on every `<table>` still carrying `data`, and measured by `data/logs/27-task01-scroll.cjs`
(+ `data/logs/27-task01-*.png`): `/`, `/system` and `/ups` reach their tail columns at 360 px,
`/certs` keeps its `.scroll` box, no page scrolls sideways, and `table.data` is a plain table
with no scrollbar at 1280 px.
Task 02 wrote that audit: `data/logs/27-task02-mobile-audit.cjs`
(`NODE_PATH=<playwright node_modules> node …`, `DASH_BASE`/`DASH_PW`, transcript
`27-task02-mobile-audit.out`, screenshots `data/logs/mobile-27-<page>-<width>.png`) sweeps
**every page the app serves** — every signed-in page plus `/login` reached unsigned (nine of
them since phase 29 task 05 put `/gateway` in the list; the sweep is exactly the kind of page a
hand-written four-page loop would have missed) — at 360 px and 1280 px, and per page demands: no
sideways document scroll, no `table.data` clipped (its `overflow-x` must be `auto`/`scroll`
when its content is wider, and its tail column measured reachable), no element past the
viewport except inside a horizontal scroll container, the three-row masthead with its controls
on screen and the dropdown under the whole bar, plus each expanded disclosure opened at **both**
widths (`/system`'s "More detail", `/ups`'s NUT variables — a closed `<details>` still lays its
content out in Chrome, so a table behind one is measured only once revealed) and phase 26's
`/ups` two-card leg, whose `hidden` state is checked against the page's own
`POWER_EVENT_KINDS`. Its first run (2026-10-21, live fleet; 280 PASS / 6 FAIL / 1 FINDING,
identical on a re-run) is green on **every table claim at both widths** — task 01's rule holds,
disclosures included — and its findings list is the pre-existing header defect: `.card > header`
is `display: flex` with `nowrap` content, so at a 320 px card the header's tail is pushed past
the card's content edge and clipped — the `.updated` age stamp on `/system` (right 388-415 px),
`/llm` (374-393) and `/ups` (392-531) against a 360 px viewport, plus `.chip.model` (376) and
`.chip.health` (463) on the `apc` card. The one FINDING is the control-width half of the table
defect: all four of the opened host's `/system` "More detail" tables are 520 px inside a 404 px
card at 1280 px. The script exits non-zero on any FAIL, and task 03 brought it to **exit 0** at
both widths.
**Task 03 closed both findings**, with two CSS declarations and no markup change, and the
audit's sweep came out stricter rather than weaker.
*The 6 FAILs — a card header's tail was eaten at a phone width.* `.card > header` was a single
`nowrap` flex line, so the `.updated` age stamp stood 18-191 px past its card on `/system`,
`/llm` and `/ups` and the `apc` card's `.chip.model` / `.chip.health` did too, all of it
swallowed by `.card { overflow: hidden }`. The fix is the row, not the content:
`flex-wrap: wrap` plus `min-width: 0; max-width: 100%` on the header's `h3` and chips and
`white-space: normal` on a header chip (`.chip` elsewhere keeps `nowrap`). Wrapping costs
lines, which is stated rather than hidden: a host header is 74 px instead of 62 px, the `apc`
card's 104 px at 360 px, and at 320 px the 46-character model chip renders in full inside its
own pill. What the fix also exposed is that "inert at the control width" was the wrong
expectation: at 1280 px `/system`'s headers *did* move, because a nowrap line that cannot fit
does not overflow, it **shrinks** its one shrinkable child, so `updated 5s` was rendering as
`updated` / `5s` torn across two lines inside a 67 px box. `Range.getClientRects()` counts
those line boxes, which is how `data/logs/27-task03-verify.cjs` separates a header that gained
a line it needed (3 of 4 on `/system`) from one moved for nothing (0 on every other page).
*The 1 FINDING — the control-width half of the table defect, which was worse than clipped.* A
disclosure is a `.body.stack` grid item whose automatic minimum size is its content's
min-content width, so `/system`'s "More detail" grew to 520 px inside a 404 px card at 1280 px
and the GPU table's last column — `97.8 GiB`, tail right 554 px against a card content edge of
424 px — was painted **under the next card**, which starts at 438 px; the `charts-box` item
beside it was stretched to the same 520 px, so a chart's newest edge went with it. One
declaration fixes it: `.card > .body.stack > details { overflow-x: auto }` — the *disclosure*
is the scroll container, because hoisting task 01's `display: block` to every width would make
a block-level table shrink-to-fit and un-stretch every table that fits its card today
(measured: the three tables that fit are 377 px and fill 377 px, the one that does not is 520
px and scrolls, its tail landing at 411 px inside the card). It is deliberately **not** inside
the ≤640px block, because the defect it repairs is at 1280 px.
*The audit changed shape too, and only upward.* Check (2b) had asserted geometry ("no table box
past the card's content edge"), which is not what gap 12 is about and which a scroll container
satisfies only by accident; both widths now assert **reachability** — a table standing past its
card must have a scroll container between it and that card, and scrolling it must land the
tail cell inside the card — and an unreachable spill FAILs at either width. Two new checks
came with it: no card header's facts past the card (both widths; the viewport-only scan cannot
see a 1280 px clip) and every table that fits still fills its container (the no-layout-shift
half, measured rather than assumed). The final run over the live fleet is **332 PASS /
0 FAIL / 0 FINDING** (9 skips, each one a check that width cannot answer — the clock, hidden
below 900 px) on all eight served pages at both widths, exit 0
(`data/logs/27-task02-mobile-audit.out`, screenshots `mobile-27-*.png`; the first run is kept
as `27-task02-first-run.out`). Phase 29's completion pass re-ran the same sweep now that
`/gateway` is in it — **371 PASS / 0 FAIL / 0 FINDING**, 10 skips, nine served pages, exit 0
(`data/logs/29-final-mobile-audit.out`) — and phase 30 task 04 re-ran it once that page became two
trunk→leaf panels: **370 PASS / 0 FAIL / 0 FINDING**, 10 skips, exit 0, with only its `/gateway`
wait selector moved from the deleted `#t-inbound` tally to `#t-ports` (the clip counts inside the
new rows are `data/logs/30-task04-panels.out`, not this audit's business). Phase 30's completion
pass ran it again after the switches went in — **371 PASS / 0 FAIL / 0 FINDING**, 10 skips, nine
served pages, exit 0 (`data/logs/30-final-mobile-audit.out`) — so neither the two-panels rewrite,
nor the switch pill in every leaf row, nor the pipe classes reintroduced a clipped fact or a
sideways document anywhere on the app. So the two declarations
above still hold on a page this audit was written before, and `data/logs/27-task03-control.cjs` proves the new check is
not vacuous by neutralising both declarations and re-creating the whole first-run findings list
(3/4/2 header offenders at 360 px, 8 stranded disclosure tables at 1280 px). The served-byte
pins are `test_a_card_header_wraps_its_facts_instead_of_losing_its_tail` and
`test_a_card_disclosure_scrolls_what_it_reveals_at_every_width`, beside task 01's
`test_every_table_data_scrolls_sideways_at_the_phone_breakpoint` and
`test_every_table_on_every_page_carries_the_class_the_rule_targets`.
Of the two operator TODO items that sat
beside it, `20_masthead_nav_polish` has landed in full (now complete): the bar is three
rows at every width (flush right on the two control rows since phase 25 task 02), it is no
longer sticky, the phone menu slides, and both
`⚙ Account` and `⚷ Certs` render on all eight signed-in pages including the two they name
(task 04, reversing phase 15's "no self-link" rule at the operator's request). The other,
`24_dev_mtls_rollout` (never accepting a non-mTLS agent) is closed: its fleet half landed on
2026-10-21 and task 04 re-read the end state read-only the same day — gap 11. No `http://`
agent URL and no token-mode agent is left on this fleet, and the setup templates no longer
seed one (`tests/test_deploy_scripts.py` keeps it that way). What remains of phase 24 is
ordinary rotation hygiene, not a missing step.
13. **`/gateway`'s long endpoint labels are ellipsised on a phone — a column-width question, not a
lost fact, and the filter is what a clipped label used to block** (found by phase 29 task 06's
live pass, **closed at the desktop width by phase 30 task 04's trunk→leaf row, re-measured
after phase 31 re-ranked the row on 2026-10-26 and again after phase 32 put the count
provenance on the `title` on 2026-10-26**; `data/logs/32-counted-routes.out` is the current
measurement, `data/logs/31-route-readability.out`, `data/logs/30-connection-map.out`,
`data/logs/30-task04-panels.out`, `data/logs/29-task06-live.out` and
`data/logs/29-task06-card-haproxy.png` the history). Phase 32's re-check found the clip
counts **unchanged** by the DOM/provenance move — **19 of 33** leaf inbound lines and **33
of 33** counterpart lines at 360 px (the 10 trunk names, none clipped), 0 of 33 at 1440 px,
and the whole pair on **33 of 33** rows' tooltips — so the trade is the same and only the
tooltip's second sentence (the provenance) is new.
**Phase 30 task 04 closed the desktop half by changing the row, not the font.** The two
equal-width pipe columns of phase 29 became a group of `minmax(0, 9.5rem) minmax(0, 1fr)` — a
trunk that only ever says `:25566 · 4 live`, and a leaf list that gets everything else — so on the
live ingress at 1440 px the target column measures **361 px** (391 px when task 04 measured it,
before task 05's switch pill took 30 px out of the row) and **0 of 33** endpoint labels and
**0 of 33** selector lines are ellipsised. What used to be the whole gap is now one line in a
`FINDING` the script prints rather than an assertion, because it is arithmetic about a viewport
and not a bug: the longest target here is 42 characters
(`reeseapps-brain-of-code.reeselink.com:443`) and it fits.
**Phase 31 kept the desktop half closed and moved the phone's clipping to where it belongs.** Its
pass (`data/logs/31-route-readability.cjs`, both widths, both themes) reads **0 of 33** inbound
lines and **0 of 33** counterpart lines ellipsised at 1440 px in the same **361 px** leaf column,
with the longest inbound word 35 characters and the longest counterpart line 50 — the row got a
second line and a `→ … · kind` wrapper and still fits. At **360 px** a group stacks and the column
is **139 px**, so the two lines now clip at **19 of 33** inbound and **33 of 33** counterpart.
That is the intended direction of the trade: phase 31 task 02 made line 1 the config's own short
word for *who reaches* the endpoint (`server_name`, `map` key, `backend` name) and demoted the long
`host:port` to line 2, so what still fits is the part an operator reads and what clips is the part
they read once. Nothing is lost: `renderLeaf()` puts **both** ends whole on the row's `title` and
`leafAria()`/`leafTitle()` carry them on the line, which is what
`tests/test_api.py::test_a_clipped_pipe_label_keeps_its_full_text_on_the_element` pins, and the
browser pass measured 33 of 33 rows carrying their whole pair at both widths — as a **painted**
tooltip now, because that pass found `svg.title = aria` had been assigning a property
`SVGElement` does not reflect and showing nothing, and the fix (`setAttribute`) is what makes
"nothing is lost" true in a browser rather than only in the DOM. The declarations that decide *how
much* clips stay pinned (`min-width: 0` on `.gate-target`, `minmax(0, 1fr)` on the leaf track,
`max-content` on `.pipe-name`, `flex-wrap` on the trunk) so a future edit cannot squeeze a label
back to the seven characters phase 29 measured. What changed the gap's *cost* is task 04's filter:
the reason a clipped label mattered was that a long target could not be **found**, and typing
`brain` now hides 31 of 33 rows without one request, matching whole text CSS never showed. In order
of cost for what is left: (a) at ≤640 px let the leaf row put the count on the counterpart's second
line instead of its own column, which buys roughly two more characters per label and is a
stylesheet-only change; (b) accept it — the fan-out is read as a set, the part that fits is the
inbound word, and the number the row is about is the count. Whoever picks this up re-runs
`data/logs/31-route-readability.cjs` (which supersedes `data/logs/30-connection-map.cjs`): its two
`FINDING` lines are the counts at both widths, and the run **fails** if any row ever stops carrying
its full pair — 33 of 33 at 1440 px and at 360 px as this phase closed, which is the assertion the
gap is actually about.
How it got here, kept because the two rejected fixes are the reason the current shape looks the
way it does: at 1440 px phase 29's outbound name showed **seven characters** (`reeseapps-…`) for a
40-character `server_name → target` string, because `.pipe-label` was one `nowrap` flex row and the
name shrank to whatever the `by port` chip (retired since), the config's own `server` name and the
`flex: none` count left. Wrapping the *text* inside that box was measured and rejected — it tore
one hostname over eleven lines of a word each — so task 06 gave the label the line to itself
(`flex-wrap: wrap` plus `flex: 1 1 max-content`), which reached ~22 characters. Task 04 kept both
declarations and removed the reason they were needed: with the trunk in its own track, the leaf's
name is the widest thing in a column that is now wide enough for it.
14. **A name-written route is counted through the address the gateway published — closed, the
re-stage included** (recorded by phase 31 task 05, implemented by
phase 32 tasks 01-03). Until phase 32, 31 of this ingress's 33 routes ended in a
`server_name`/`map` target and carried an em dash where a live count would go, because
`sources/gateway.py` matches sockets by address and resolves nothing. That is still true — the
module calls no resolver, spy-pinned over the parse, the gauge names and the capture — but it no
longer has to, because agent **1.3.0** publishes `resolver`: every `(name, port)` its own two
config files write as a target, answered by libc **on the box that opens the connection**, with
the nginx `resolver` directive printed beside it as `config_directive` so a divergence between the
two vantages is visible instead of guessed. `sources/gateway.py` reads that document, stamps
`out.resolved` / `via` / `resolved_ts`, joins sockets through the normalised address, and writes
the `gateway.out_conns@<domain>:<port>` gauge under the config's own spelling — so the dash is gone
on the live page **because the gateway said what the name became**, which was the acceptance line,
and not because a column was hidden.
**The deployment half is done.** `gateway` ran **1.2.0** until phase 32 task 05 re-staged it on
2026-10-25 (`data/logs/NOTES-32-rollout.md`: unit sha256 unchanged, bundle untouched, the install
run guard silent, one file and one restart), and that pass read `/api/gateway/snapshot` going from
**2 of 33** rows carrying a number to **33 of 33**, with all 31 resolver-placed rows finite and
`test_a_resolved_target_on_the_live_ingress_is_never_a_dash` passing instead of skipping. What is
left of the gap is rotation hygiene, not a step: `deskwork` and `driveripper` stay on 1.1.0 (no
proxy configs, therefore nothing to resolve), and a host that later rolls **back** to a pre-1.3.0
file simply returns its name routes to dashes — `sources/gateway.py` treats a missing `resolver`
section as phase-30 behaviour. A name that stops
resolving is `nxdomain`/`error` and goes back to being a dash on the next poll (the agent caches
300 s and never publishes a stale address as current), which is the designed degradation rather
than a gap — and since task 04 the page says **which** degradation it is: `countCause()` separates
that dash (`the gateway could not resolve this name in this reading`) from the one an unreadable
socket table causes (`the socket table did not answer this reading`), while a resolved endpoint with
nothing open prints its `0` beside `· address from libc, resolved <age>`, so the sentence a line
carries names the box to go and look at. `data/logs/32-task04-sentences.cjs` measures all four arms,
serving the three a pre-1.3.0 host could not produce through route interception (43 checks, and
each stub leg
first proves its DOM equals the document that leg was served, so a leg that rendered the live page
instead fails by name rather than three checks later).
15. **`/gateway`'s switches are knobs with no write behind them — `33_gateway_route_switching` is the
phase that decides whether they ever get one** (recorded by phase 31 task 05, no design written).
Enabling or disabling a route is a second gated upstream write path in front of the proxy configs,
which **amends invariant 1** (today `POST /api/ups/command` is the app's only upstream write, and
the route enumeration in `tests/test_api.py` is the pin that says so) and needs the operator's
sign-off in `.agents/PLAN.md` **before** it is written as a phase. Until that decision exists the
knobs stay `aria-disabled`, unbound, and explained in the footer; the `PATCH`-shaped endpoint, the
config writer and the rollback are that phase's work and none of it is here.
## Locked decisions (and why)
- **SQLite, not Prometheus/Grafana.** Handful of boxes, 1–10s resolution, one file the
container owns. Rollups keep a year of charts at a few hundred MB.
- **Our own stdlib host agent, not node_exporter.** One JSON snapshot with exactly the
fields the page needs (GPU model strings, per-mount usage, temperature labels) versus
~1500 series and a re-labeling layer; no dependency to install, `ProtectSystem=strict`
friendly, and read-only by construction. `scripts/capture-node-exporter.sh` remains to
compare ground truth if that is ever revisited.
- **Our own NUT client, not `pynut`/`upsc`.** The protocol is ~10 line commands, the
third-party libs are subprocess-based or stalled, and we need instant commands with
explicit gating.
- **In-house Prometheus parser.** Only two upstreams use the format; keeps `prometheus_client`
out of the dependency set.
- **`cryptography` for X.509, dashboard-side only** (phase 15). Minting a CA and signing leaves is
not something to hand-roll, and there is exactly one place in this app that does it
(`certs.py`). The agent gets no new dependency: it loads the PEMs the dashboard signed with stdlib
`ssl`, which is why the bundle is plain text and why the agent still runs on a host's Python 3.10.
- **Port 8090**, inherited from the `llm_monitor` prototype this started from: 8000 is vLLM
and 8731 is gufo in this fleet.
- **`.env` is a seed, never an override** — container/systemd env must win.
## Definition of done (per change)
The first three are exactly what `bash .agents/validate.sh` (the phase gate) checks, so no
phase may be called done on a subset of them.
`uv run pytest` green · coverage not down · `ruff` + `pyright` clean · `uv pip list --outdated`
empty if deps moved · dev server restarted and `/api/health` sane · UI change screenshotted
in a browser against **live** data · fixtures re-captured if an upstream contract changed ·
this file and `README.md` updated when a rule, command, gap, or fleet fact changed.
Reference in New Issue View Git Blame Copy Permalink