From 0271387053fa766fc344b8112f945ed6f614e6fc Mon Sep 17 00:00:00 2001 From: ducoterra Date: Wed, 28 Oct 2026 00:02:17 -0400 Subject: [PATCH] compact AGENTS.md --- AGENTS.md | 3852 +++++------------------------------------------------ 1 file changed, 350 insertions(+), 3502 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 3938532..dcdcec0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,3575 +1,423 @@ # AGENTS.md — homelab dashboards -The plan, the rules, and the environment facts an agent needs to work on this repo without -re-deriving them. `README.md` is the operator-facing how-to; this file is the why, the -invariants, and the state of the fleet. +Orientation for an agent starting work in this repo: what the app is, the rules that keep +its numbers honest, where things live, the fleet it watches, and the commands. `README.md` +is the operator-facing how-to (quickstart, agent install, signing in, configuration, +deployment) — read it for those; this file does not repeat it. This file is **orientation, +not a changelog**: phase history lives in `.agents/phases/` and `data/logs/`. When a phase +closes, update the sections below only if a *rule, invariant, command, or fleet fact* +changed — never append the phase's narrative. ## Working protocol -This project runs the `.agents/phases/` phased-execution pipeline. - -1. Always read `.agents/PLAN.md` first to understand the project context and goals. -2. Follow the phased execution protocol in `.agents/phases/`. -3. Never modify `.agents/PLAN.md` or any files in `.agents/phases/complete/`. -4. If you need to update any file in `.agents/phases/todo/`, you must ask the user for permission first. -5. Strictly adhere to the **LOCKED DECISIONS** listed in `.agents/PLAN.md`. - -Everything below this section is the project's own content: what the app is, the rules that -keep its numbers honest, and the state of the fleet it watches. +1. Read `.agents/PLAN.md` first; follow the phased execution protocol in `.agents/phases/`. +2. Never modify `.agents/PLAN.md` or files in `.agents/phases/complete/`; ask the user + before updating anything in `.agents/phases/todo/`. +3. Strictly adhere to the **LOCKED DECISIONS** in `.agents/PLAN.md` and the section at the + bottom of this file. ## What this is Five dashboards for one homelab, behind one domain, in one process, backed by one SQLite -file — plus an overview page that says which of the five you need: +file — plus an overview page: -| Route | Question it answers | Source | -| ---------- | ------------------------------------------------------------------------------ | ------------------------------------------------- | -| `/` | Is anything wrong, and where do I look first? | all five, in one request | -| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent per host | -| `/disks` | Are any of my physical disks failing, and which one — health, temp, hours, wear | the same agents, `smartctl -i -A -H` per whole drive (never a partition) | -| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo `/metrics` + `/slots`, vLLM `/metrics` | -| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) | -| `/gateway` | Which domains come in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`), a trunk per bound port with its configured routes branching off it, each line reading the config's own **inbound** word (`server_name`, `map` key, `backend` name) over the counterpart it routes to, plus its live socket count, with one filter box above the map that hides the routes whose own labels do not contain what was typed | the same agent (1.2.0's three proxy sections, 1.3.0's `resolver`) on the one host `GATEWAY_HOST` names: its `haproxy` + `nginx` config sections joined to its `/proc/net/tcp{,6}` sockets through the addresses the gateway itself resolved | +| Route | Question it answers | Source | +| ---------- | -------------------------------------------------------------- | ------------------------------------------------------------- | +| `/` | Is anything wrong, and where do I look first? | all five, in one request | +| `/system` | CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent per host | +| `/disks` | Are any physical disks failing, and which — health, temp, wear | the same agents, `smartctl -i -A -H` per whole drive | +| `/llm` | Tokens/s, active sessions, queue depth, KV pressure, latency | gufo `/metrics` + `/slots`, vLLM `/metrics` | +| `/ups` | Battery health, runtime left, power events, test buttons | NUT `upsd` (our own wire client; port **3051**, not 3493) | +| `/gateway` | Which domains on which port, and where each goes | the same agent's `haproxy` + `nginx` sections joined to `/proc/net/tcp{,6}` on the one `GATEWAY_HOST` host | -History is a first-class feature, not a sparkline afterthought: every number worth graphing -is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`. +History is a first-class feature: every number worth graphing is stored, rolled up +(raw → 1m → 1h) and served from `/api/*/history`. None of it is public. One admin signs in at `/login` (the only page an anonymous visitor -reaches), and the two admin pages live behind it: `/settings` — `⚙ Account` in the masthead — -where that session is read, ended and re-credentialled, and `/certs` — `⚷ Certs`, next to it — -where the fleet's certificate authority is listed, rotated, revoked and downloaded. Every page -above, `/settings`, `/certs`, and every `/api` route but the four named in invariant 12 require -the session. +reaches); `/settings` (⚙ Account) and `/certs` (⚷ Certs) live behind it. Every page and +every `/api` route requires the session except the public entries in rule 12. ## Non-negotiable rules -These are the things that make the numbers trustworthy. Breaking one is a bug even if the -page still renders. +Breaking one is a bug even if the page still renders. -1. **Read-only by construction, with exactly one exception.** Nothing in this app writes to - an upstream except `POST /api/ups/command`. That endpoint is gated four times - (`NUT_ALLOW_ACTIONS`, not `DASH_READ_ONLY`, the upsd.users account actually having the - command, and a UI confirmation — `DANGER` for anything that can cut power) and the server - re-checks every gate on every request. Hiding a button is not a safety control. - Commands that would cut output are refused outright while the UPS is already on battery. Since - phase 11 there are two more in front of it — a session, and this session's CSRF token for a - cookie write (invariant 12) — and they are additive: `tests/test_auth_api.py` proves the - `DANGER` requirement and the wire send still behave exactly as they did without them. +1. **Read-only by construction, with exactly one exception.** Nothing writes to an upstream + except `POST /api/ups/command`, gated four times (env allow, not `DASH_READ_ONLY`, the + upsd account actually having the command, a UI confirmation — typed `DANGER` for + anything that can cut power) and re-checked on every request; power-cutting commands are + refused outright while on battery. Hiding a button is not a safety control. 2. **Never invent a number.** A value the upstream does not publish renders as `n/a`, never - as `0`. Every derived value carries the `source` string it came from and `derived: true` - so the UI can print *how* it was obtained (`computed:Δtokens/Δt` labelled `Δ2.0s`, the LLM's - own poll interval since phase 23). Aggregating - two different series into one rate is the failure mode this rule exists to catch — the - rate estimator is keyed by the *resolved* upstream series name **plus its label set** - for exactly that reason (phase 13: a label set joining a summed counter mid-window - contributes nothing until it has its own baseline, never its carried cumulative value), - and the estimator refuses a window too short or too stale to carry the claim. + as `0`. Every derived value carries its `source` string and `derived: true`; the rate + estimator is keyed by the *resolved* series name **plus its label set** and refuses a + window too short or too stale. Aggregating two different series into one rate is the + failure mode this rule exists to catch. 3. **Unconfigured is not broken.** A source with no env config answers `404 {"configured": false}` **without dialling**, and the page says "not configured". - A red panel for a server the operator never installed destroys the signal from the - panels that matter. -4. **One dead source must not blink another — and nothing may stop polling in silence.** Each - source is its own asyncio task with its own timeout and health record (`poller.py`). A - failed poll keeps the last good payload and marks it stale in the UI instead of zeroing it — - on `/` too, where a host that has never answered still gets a fleet row - (`unreachable: true`, dimmed, every number `n/a`) because dropping the row would turn a dead - agent into "never configured". Transitions write events; repeated failure writes one line, - not 500. **The task is the source's only poller, so no exception may end it**: a store that - refuses an otherwise good poll is a *reported* failure (`ok: false`, the real reason in - `error`, `last_ok` frozen so the page ages the numbers, one `store-error` event then one - `recording` event on recovery), and anything that escapes the poll body is logged and the - loop continues. Four dev-server sources once froze behind an unretrieved SQLite exception - while `/api/health` went on calling them `ok` — a frozen source that looks healthy is the - one failure mode worse than an outage. + A red panel for a server the operator never installed destroys the signal. +4. **One dead source must not blink another — and nothing may stop polling in silence.** + Each source is its own asyncio task with its own timeout and health record. A failed + poll keeps the last good payload and marks it stale instead of zeroing it; a host that + never answered still gets its row (`unreachable: true`, dimmed, `n/a`), because dropping + it would turn a dead agent into "never configured". **The task is the source's only + poller, so no exception may end it**: a store that refuses an otherwise good poll is a + *reported* failure (`ok: false`, the real reason, `last_ok` frozen, one event then + `recording` on recovery), and anything that escapes the poll body is logged and the loop + continues. A frozen source that looks healthy is the one failure mode worse than an + outage. 5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app; - charts are hand-rolled SVG (`src/dashboards/web/static/charts.js`). The engines live on a network with - no outbound access — a CDN reference is a request that never comes back. + charts are hand-rolled SVG (`src/dashboards/web/static/charts.js`). The engines live on + a network with no outbound access — a CDN reference is a request that never comes back. 6. **Never leak a credential.** Target URLs may carry a userinfo token (`parse_targets` - moves it into an `Authorization` header, so `Target.url` ends up clean). Nothing that leaves - the process may carry one: `/api/config` serves `Target.display_url` and - `config.strip_userinfo`-redacted engine URLs, and every URL inside an `UpstreamError` message - is redacted too — those strings go into the health record *and* the `event` table, which - `/api/events` serves to the browser and `page-system.js` prints. No token, password, or - personal hostname goes into tracked files, fixtures, or the browser. **The admin's own - credential is the same case** (phase 11): `auth.py` records nothing and answers - `False`/`None` to every rejection, because the *reason* a login was refused is a verifier, a - salt, or a session token — and this app turns reasons into health records and `event` rows - that a browser reads. Its one `logging` call and its two `raise` sites (the fail-closed - startup and the empty-new-password refusal) carry **fixed** operator text with nothing - interpolated, not even a prefix of a hash; `tests/test_auth.py` pins that structurally, so - the day someone adds `logger.warning(f"bad verifier {v}")` a test fails instead of a log line. - **A private key is the same case** (phase 15): the root CA key and the dashboard's client key - live only in `meta`, `certs.py` has no `logging` import and one fixed-text raise naming the rows - to delete (pinned structurally by `tests/test_certs.py`, which also asserts the refusal quotes - no line of the stored key), the root key is never written to the temp dir the client material - lives in, and a bundle carries the root *certificate* — never the key — because an agent only - has to verify its clients and a stolen agent install must not be a stolen CA. - **A session token used to fetch a bundle is the same case** (phase 24): the installer's - `--fetch-bundle --cert-name --session ` sends that token on exactly one - request, hands it to its `python3` child through the **environment** rather than argv, prints - only the URL in its log line, `unset`s it before anything is written, and never repeats it in - the sudoers hint's re-quoted command (which names `--bind`/`--port` and nothing else). - `tests/test_deploy_scripts.py` signs in for a real token and then greps every file the run - could have touched plus both output streams. - **A bundle exported for the dev loop is the same case** (phase 24): - `scripts/dev-certs-export.py` writes only into the gitignored `data/certs/` tree, creates each - file with `os.open` so `0600` is its *birth* mode (and re-chmods a file that already existed - looser), dials `GET /api/certs//bundle` over HTTP instead of importing `ensure_certs` — so - it cannot mint even by accident — and prints one line per host with no PEM, no token and no - password in it (`tests/test_deploy_scripts.py` greps the output for all three). - **A captured socket table is the same case** (phase 28): `scripts/capture-gateway-config.sh` - keeps every config byte (a `server_name` and its `proxy_pass` are the ingress map the page draws, - and a key *path* is a filename, not a key) and rewrites **only** the two address columns of - `/proc/net/tcp{,6}`, and only when `ipaddress` calls the address globally routable — a Linux - global IPv6 is EUI-64, so it encodes this box's NIC, and an inbound visitor's address was never - ours to publish. Documentation space (RFC 5737 / RFC 3849), first-seen order, same width, same - `::ffff:` spelling, idempotent; ports, states and every other column stay verbatim. - **A captured resolver answer is the same case too** (phase 32): what `getent` returns for - `reeseapps-immich.reeselink.com` on this router is as much an endpoint as a socket row is, so the - same substitution runs over `tests/fixtures/gateway_resolver.json`, and the name itself is kept - verbatim because the name is the thing the page draws. `tests/test_agent.py` fails the day either - fixture grows a routable endpoint back. + moves it into `Authorization`, so `Target.url` ends up clean), but nothing that leaves + the process may: URLs in `UpstreamError` messages, health records, `event` rows + (`/api/events` serves them to the browser) and `/api/config` are redacted. **The admin's + credential, the private keys, and session tokens are the same case**: `auth.py` records + nothing and answers `False`/`None` to every rejection, `certs.py` has no `logging` + import, and both modules' `raise` sites and `auth.py`'s one `logging` call carry + **fixed** text with nothing interpolated — pinned structurally by `tests/test_auth.py` + and `tests/test_certs.py`. No token, password, or personal hostname goes into tracked + files or fixtures; captured socket tables and resolver answers substitute globally + routable addresses for documentation space (RFC 5737 / RFC 3849), idempotently. 7. **Verify against the deployed upstream; never guess a metric name.** Field maps in - `sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT - 2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with - `scripts/capture-fixtures.sh` (and `scripts/capture-node-exporter.sh` when comparing - exporters, `scripts/capture-gateway-config.sh` for the agent's `haproxy`/`nginx`/`sockets`/ - `resolver` ground truth) rather than trusting a README or a memory of an upstream. -8. **Keep dependencies current on purpose.** LLM-written version floors are usually stale. - Check after touching dependencies: `uv pip list --outdated` must come back empty. - Python is 3.14 (`requires-python >=3.14`, ruff `py314`, pyright `3.14`, Containerfile - `python3.14`) — do not lower these to make something else work. + `sources/llm.py` and `sources/nut.py` were captured from the running engines and pinned + by `tests/fixtures/`. Re-capture with `scripts/capture-fixtures.sh` (and + `scripts/capture-gateway-config.sh` for the gateway's proxy tree, socket tables and + resolved names) rather than trusting a README or a memory of an upstream. +8. **Keep dependencies current on purpose.** After touching dependencies, `uv pip list + --outdated` must come back empty. Python is 3.14 everywhere (`requires-python`, ruff + `py314`, pyright, Containerfile) — do not lower it to make something else work. 9. **SQLite owns the history and must stay bounded.** Writes go through `store.py`; new metrics need a rollup path and a retention story, or a year of charts becomes a disk emergency. Put the DB on a volume in deployment. 10. **The dev server stays up while work happens** (`./scripts/dev-server.sh restart`, http://127.0.0.1:8090), and UI changes are verified in a real browser with a screenshot - before they are called done. `data/logs/*.png` is where those land. -11. **The dashboard never starts open, and the stored password beats the environment** - (phase 11). The admin verifier and the session-signing key are durable rows in the SQLite - `meta` table (`auth.verifier`, `auth.session_secret`) and are the only place either secret - lives; `DASH_ADMIN_PASSWORD` is a **seed** that hashes into the row on the first boot of a - database and is ignored thereafter (one warning line, value never quoted), so a password is - changeable without a redeploy and no container env can roll it back. A server with neither a - stored verifier nor the seed **refuses to start** — a `RuntimeError` out of the lifespan, a - non-zero exit, and a message naming the variable, the meta key, and the - `sqlite3 "DELETE FROM meta WHERE key IN ('auth.verifier','auth.session_secret')"` - reset command. "Start open with a warning" is not an option: the domain is public and the - fleet's hostnames and UPS controls are behind that login. `testpass` is the local-testing - password (dev config and the hermetic suite) and never a production fallback. Invariant 12 is - the request-time half; this one is why there is always a credential to check against. -12. **The gate is default-deny and its public surface is four entries** (phase 11). `app.py`'s - `PUBLIC_PATHS` (`/login`, `/api/auth/login`, `/api/health`) plus `PUBLIC_PREFIXES` - (`/static/`) are the only things an anonymous caller can reach — every page, every `/api` - data route, and `/api/auth/status`/`logout` require a session read from the - `dash_session` cookie or an `Authorization: Bearer` token. `/api/health` is the one data - route that stays open on purpose (the Containerfile `HEALTHCHECK` and - `./scripts/dev-server.sh status` poll it unsigned, and it leaks nothing); its shape is now a - contract with deployment, which means **nothing may be added to it without re-deciding that** - — a new field there is a field the whole internet reads. Adding a path to that list is a - security decision, and - `tests/test_auth_api.py::test_every_route_that_is_not_public_requires_a_session` walks the - **served** route table so a route that forgets the gate fails a test instead of shipping — - the middleware guards by default, which is why no new route has to remember to. Rules the - gate keeps and each has a test: a page `GET` gets `303` to `/login?next=…` (query string - included) and everything else the one JSON refusal, because a redirect a script cannot render - is not an answer; every answer carries `no-store`; `?next=` passes `auth.safe_next` and is - never followed raw (an open redirect on a login page is a phishing kit with a homelab theme); - a write additionally needs a matching `Origin`/`Sec-Fetch-Site` and — when it came in on the - **cookie** — this session's `X-CSRF-Token`, because a bearer caller is something no other site - can forge. A process whose lifespan never produced `ctx.auth` answers `503` for **everything, - public paths included**: "healthy" from a server that cannot say who its admin is is the - invariant-4 failure mode, and a login form that cannot verify a password is a worse promise - than no form. `POST /api/ups/command` keeps all four of its own gates behind this one (the - `DANGER` word is still what cuts power), and `POST /api/auth/logout` is gated like any other - write because a sign-you-out link is a CSRF. `POST /api/auth/password` (task 04) is the third - gated write and the app's only revocation: guesses at the current password spend their own - `password:
` bucket, and a right one replaces `auth.verifier` **and** - `auth.session_secret` so every browser that is not in front of the form dies on its next - request — which is why the response is minted from the new state and never the one being - replaced. + (`data/logs/*.png`) before they are called done. +11. **The dashboard never starts open, and the stored password beats the environment.** The + admin verifier and the session-signing key are durable `meta` rows + (`auth.verifier`, `auth.session_secret`) and the only place either secret lives; + `DASH_ADMIN_PASSWORD` is a **seed** that hashes into the row on the first boot of a + database and is ignored thereafter. A server with neither a stored verifier nor the seed + **refuses to start** (a `RuntimeError` out of the lifespan, non-zero exit). `testpass` + is the local-testing password (dev config and the hermetic suite), never a production + fallback. +12. **The gate is default-deny and its public surface is four entries** (`/login`, + `/api/auth/login`, `/api/health`, `/static/`). A page `GET` gets `303` to + `/login?next=…` (query string included, checked by `auth.safe_next`); everything else + the one JSON refusal; every answer carries `no-store`; a write additionally needs a + matching `Origin`/`Sec-Fetch-Site` and — when it came in on the **cookie** — this + session's `X-CSRF-Token`. A process whose lifespan never produced `ctx.auth` answers + `503` for everything, public paths included. `tests/test_auth_api.py` walks the + **served** route table so a route that forgets the gate fails a test instead of + shipping — the middleware guards by default. Nothing may be added to `/api/health` + without re-deciding that: it is public, and a new field there is a field the whole + internet reads. ## Architecture ``` agent/host_agent.py (each host) gufo / vLLM upsd (NUT) - │ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD - │ │ + /slots (gufo) │ + │ GET /snapshot │ GET /metrics │ LIST UPS / VAR / CMD ▼ ▼ ▼ -collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command ──┐ - │ parse + Δcounter/Δt │ - ▼ ▼ -sources/host.py · disks.py · llm.py · nut.py poller.py (task per source) - │ Result(payload, samples, events) │ - ▼ ▼ +collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command + │ parse + Δcounter/Δt + ▼ +sources/host.py · disks.py · llm.py · nut.py · gateway.py poller.py (task per source) + │ Result(payload, samples, events) + ▼ store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ──► api.py ──► web/ ``` -Responsibilities, one line each: +One line each — the load-bearing behaviour is in the bullet, the rest is in the module: -- `agent/host_agent.py` — stdlib-only (3.10+), reads `/proc`, `/sys/block`, `/sys/class/drm`, - `statvfs` — plus, for the proxy add-on, the haproxy and nginx configs and `/proc/net/tcp{,6}` - on the host that has them; serves `GET /snapshot` + `GET /info` + `GET /schema` + `GET /health`, plus an - HTML status page at `GET /`. Refuses to bind off-loopback without a token **or** a certificate - bundle (exit 2). Counters stay counters: the dashboard computes - rates so the agent never remembers who polled it last. - **TLS mode (phase 15 task 02, `AGENT_VERSION` 1.1.0):** `--cert-bundle`/`AGENT_CERT_BUNDLE` - loads the one PEM the dashboard signs (`resolve_bundle`, the `resolve_token` idiom: server cert - + server key + root CA + `# BUNDLE-REV ` + `# REVOKED ` lines, grepped by text - because the agent has no `cryptography`), unpacks it into the unit's `RuntimeDirectory` when - there is one else a temp dir, builds a `PROTOCOL_TLS_SERVER` context with - `verify_mode = CERT_OPTIONAL`, and **deletes the PEMs once loaded** (the key then lives only in - this process, so a `kill -9` has no cleanup to forget). `CERT_OPTIONAL` is the assumption that - keeps `/health` probeable: the *handler* enforces, refusing a peer with no verified certificate - 403 `client certificate required` on every path but `/health`, and refusing a serial on the - bundle's denylist 403 `client certificate revoked` on **every** path including `/health` (the - exemption is for a probe holding the CA and no certificate, not for an identity the operator - withdrew). The denylist is re-read when the bundle's `(mtime_ns, size)` stamp changes — that is - the whole reason the `# REVOKED` lines live in the bundle — while a new certificate needs a - restart, because `ssl` cannot swap a chain under a live socket. Serial matching accepts the hex - `getpeercert()` renders *and* the decimal `certs.py` writes, because comparing one spelling to - the other accepts a revoked client forever. On the TLS path the bearer token is **not** read (the - handshake is the auth, and a second credential that outlives a revocation defeats the list), - `/snapshot` gains additive `tls: true` + `bundle_rev` only in TLS mode, and token mode is - byte-identical to before. Task 03 has since flipped the collector (see the `collectors.py` - bullet), so those two fields are how the dashboard answers "is this host running the bundle - `/certs` issued?" — `sources/host.py` and `sources/disks.py` carry them into both payloads as - `agent.tls` / `agent.bundle_rev`, present only when the agent published them. - SMART (phase 09): `smart()` runs `smartctl -i -A -H` per whole `/sys/block` disk - (`parse_smart_output` covers ATA + NVMe; NVMe media errors map to - `offline_uncorrectable`, hex-packed vendor raws are never decoded, and an attribute row is - read from its **RAW_VALUE column** rather than the end of the line because this binary - annotates that column — `35 (Min/Max 34/44)`, `43894h+43m+32.519s` — so its last token is - not the value), 5 s per device / a pass budget whose 20 s floor scales to 5 s × the disk count - / 240 s cache, attached as `disk_io.devices[].smart` (gap 10: a 16-drive shelf takes longer - than the floor, and truncating the pass would null the tail drives). Missing binary → - **one** `errors` line for the pass (the word `not found` in it is what - `sources/disks.py` reads to answer "is smartctl installed", and - `tests/test_agent.py` pins that the line still says it); one denied/wedged device → that - disk's `smart` is `null` with its own line; `--no-smart` opts out silently. Never fails - the snapshot. **An expired cache refreshes on a background thread while the request is - served the previous result** — only the first pass after startup runs inline — because a wedged - pass on the request path blows `HOST_TIMEOUT_SECONDS` and would turn the - *system* source red every four minutes over an add-on that page never draws. The refresher - is single-flight and keeps the old cache on failure, so a bad pass retries on the next - expiry instead of hammering the disk. - **The proxy-config add-on (phase 28 task 01)** is the `haproxy` section: `Collector.haproxy()` - reads `--haproxy-config` (default `/etc/haproxy/haproxy.cfg`, resolved under `--root` so the - fake-tree tests read the fake tree, one size-bounded read per snapshot) and - `parse_haproxy_config()` turns its text into `{frontends: [{name, binds: [{addr, port}], - default_backend, backends}], backends: [{name, servers: [{name, target, port}]}]}`. Three rules - keep it honest. (a) `available` is `smartctl_available`'s three-way answer — `true` read and - parsed, `false` no file at that path, `null` a file that could not be read or holds no haproxy - section — and each non-`true` arm costs exactly **one** `haproxy:` line for the pass, the missing - one saying **not found** (pinned wording, the same cross-module contract as smartctl's, because - the dashboard reads that word to answer "is haproxy installed?"). (b) The section reports the - config's own text and infers nothing about a request: a bind keeps the address as written - (`:::22` splits at the **last** colon into `"::"` and `22`, so an IPv6 bind survives, and - `bind :8080` keeps `addr: null` because `*` is a spelling the config did not use), a `server` - with no port keeps `port: null` and never `0`, a `use_backend` whose target is a fetch - expression (`%[req.hdr(host),map_dom(...)]`) contributes **no name** rather than an expression - where a backend belongs, and everything outside `bind`/`default_backend`/`use_backend`/`server` - is ignored without a word. The frontend→backend routing is therefore the names, which the - dashboard joins. (c) A `listen` section is reported as the frontend+backend pair haproxy itself - defines it as, so a config written in that shorthand cannot read as "this gateway routes - nothing". The section is absent from the document with `--no-proxy` (silence, the `--no-smart` - precedent — "not collected" stays distinguishable from "collected and empty"), it is additive in - the payload, and nothing in it can fail `/snapshot` or `/health`. - `tests/fixtures/gateway_haproxy.cfg` is the verbatim capture the parser is pinned to. - **The same add-on's `nginx` section (phase 28 task 02)** reads `--nginx-config` (default - `/etc/nginx/nginx.conf`) **plus the files its own `include` lines name, one level deep** — the - gateway keeps its vhosts in `conf.d/*.conf` and its SNI routing in `stream.d/*.conf`, so the - main config alone is not the picture. `parse_nginx_config(main_text, included)` is the pure - half over text, and it needs a real scanner rather than haproxy's indent loop because nginx - nests by `{}`: `_nginx_tokens` keeps a `;` inside a quoted value as data (the real config writes - `add_header … "max-age=31536000; includeSubDomains" always;`), `_nginx_statements` builds the - block tree and raises only for a file nginx itself would refuse to start on, and the walk fills - `server` blocks (`listen`, `server_name`, `proxy_pass`), `upstream` blocks and `map` blocks. - Same three rules as `haproxy`: `available` is three-way with exactly one `nginx:` line per arm - (the missing one saying **not found**, the pinned wording the dashboard reads), `--no-proxy` - removes the key in silence, and nothing in the section can fail `/snapshot` or `/health` — - including a config that will not tokenize, which is `available: null` plus its one line with - every other section intact. What it refuses to do is interpret: `listen` keeps the address as - written (`[::]:8443` → `"::"`, a bare `80` → `addr: null`) and ignores the `ssl`/`proxy_protocol` - flags, `server_name _` stays the literal `_`, a map's value becomes a `target` only when it is - an address (the `$connection_upgrade` map's `upgrade`/`close` are header words, and calling one - a host is inventing a fact to fill a field), and a `proxy_pass $var` travels with `variable` - (the `$` dropped once here, so both sides of the join spell it identically) and a **null - `target`** — the map decides per connection, so a merged target would be rule 2 with a page's - clothes on. Include-following is one level and bounded (8 files / 1 MB of included text, the - smart pass-budget shape): the excess is dropped with **one** line naming the count and the - section stays `available: true`; `files` is the main config then the documents actually merged, - spelled the way the config spells them (`/etc/nginx/conf.d/http.conf`, never the path under - `--root`, which is also what lets the parser match its own `include` lines); a literal include - naming a file this host does not have costs nothing and no line; and an `include` inside an - included file is not followed at all — `root=False` makes that structural, not a convention. - **The same add-on's `sockets` section (phase 28 task 03)** is the live half: `read_proc_table` - reads `/proc/net/tcp` and `/proc/net/tcp6` (resolved under `--root`, the config rule) through at - most `PROC_NET_MAX_LINES` = **4096 lines per table** — `/proc`'s own header among them, so 4095 - sockets — because a gateway's file is one line per socket and the read must not cost the poll - somebody else's connection count; the excess is reported in **one** line naming the bound and the - section stays `available: true`. `parse_proc_net_tcp(text, family)` keeps two kernel states only: - `0A` LISTEN → `{family, addr, port}` and `01` ESTABLISHED → - `{family, local: {addr, port}, remote: {addr, port}}`; every other state is a socket in transit - (`TIME_WAIT` above all) and is skipped in silence, since drawing it would put two pipes on one - connection and bury the live pairs. `_proc_net_addr` unrolls `/proc`'s per-32-bit-word - little-endian spelling through `socket.inet_ntop`, so a wildcard keeps its literal `0.0.0.0` / - `::` — the value the inbound join compares against — and a word that is not an address is `addr: - null` with the port kept, never a guess and never `0`. The two tables are read **independently**: - `available` is `true` when either answered (a kernel built without IPv6 has no `tcp6`, which is - absence, so no line — the nginx include rule), `null` with one combined `sockets:` line when - neither did, and a table that is there and denied costs one line even beside a good one. This - section has **no `false` arm on purpose** — `/proc/net/tcp` is not something a host leaves - uninstalled, and `false` beside an empty list would read as "no listeners", which is rule 2 — and - **no join with the config**: a socket does not know its domain, so phase 29's page joins these - ports to the `haproxy`/`nginx` sections and an unclaimed socket stays visible as one. - **The resolver section (phase 32 task 01) is the fourth and the only one that leaves the host.** - `target_names(haproxy, nginx)` is the whole allow-list — haproxy `backends[].servers[].target`, - then nginx `upstream` members, `server`-block `proxy_pass` targets and `maps[].entries[].target`, - deduplicated keeping first-seen order — and nothing else may ever enter it: not a name from - `/proc/net/tcp`, not a client address, not anything a request supplied. That is decision 3 and - `tests/test_agent.py` pins it by asserting the spied `getaddrinfo` call set **equals** this - function's output minus the address literals, in config order, with no duplicates, and that no - address from the recorded socket table ever appears in it. The lookup is - `socket.getaddrinfo(name, port, proto=SOCK_STREAM)` — `via: "libc"`, which is the resolution - haproxy's `init-addr last,libc,none` performs **on the box that dials**; the dashboard resolving - the same name through its own `resolv.conf` would be a different vantage answering a different - question, so `sources/gateway.py`'s resolver spy pin survives this phase on purpose and the - agent does **not** hand-roll UDP queries to the config's own `resolver` server either. That line - is published instead, verbatim, as `config_directive` (`"10.1.0.1"` on this fleet) — and it lives - inside this section rather than inside `nginx` so the `nginx` document stays **byte-stable**, which - keeps phase 32's additivity control a strict top-level key comparison with no exception list - (`Collector.nginx_document()` is the seam that returns the pair; `Collector.nginx()` is its - one-line wrapper). Entry shape is `{name, port, status: ok|nxdomain|error|literal, addresses, - ts}`: `addresses` is the unique answer sorted v4-ascending then v6-ascending (libc's order is not - stable and `entries` is a list a page renders), an address literal is `literal` with **no** lookup - (it is already an address), and `nxdomain`/`error` carry `addresses: []` — never a stale guess, - never a `0`, and `EAI_AGAIN` is `error` and not `nxdomain`, because "the server did not answer" - and "the name has no record" are two fixes. The pass is SMART's idiom for SMART's reason - (`_resolver_lock`/`_resolver_cache`/`_resolver_refreshing`, `RESOLVE_MAX_NAMES` 64 / - `RESOLVE_CACHE_SECONDS` 300 / `RESOLVE_PASS_BUDGET_SECONDS` 3.0): the **first** pass inline, every - later expiry refreshed on a single-flight daemon thread while the request is served the previous - result with its **original per-entry `ts`**, a failed refresh keeping the old entries and - re-stamping so the retry lands on the next expiry rather than on every poll, and the budget/name- - bound excess each reported in **one** line (`resolver: pass budget 3.0s exhausted, 41 of 58 - targets looked up`) with `available` still `true` — a black-holing resolver must never blow - `HOST_TIMEOUT_SECONDS` and blink `/system` over an add-on it never draws. `available` is the - family's three-way answer (`false` = nothing to resolve because no proxy config was read, one - line saying **not found** or **unreadable** per section; `null` = a pass that could not run, - which `snapshot()`'s `_safe` turns into the empty section plus its line), and **`--no-resolve` - removes the key in silence** the way `--no-smart` does — while `--no-proxy` removes it too, since - a box that collects no proxy has no declared names. - **Task 04 made the three sections a versioned contract with captured ground truth:** - `AGENT_VERSION` is **1.3.0** (`tests/test_agent.py` pins the constant and asserts the - `GET /schema` block names `haproxy`/`nginx`/`sockets` — each with its `available` flag — plus the - `resolver` section, its own `available`, and `--no-resolve`, beside `--no-proxy`, - `--haproxy-config` and `--nginx-config`), and `scripts/capture-gateway-config.sh` - re-records the seven fixtures the parsers and the join are pinned to over `GATEWAY_SSH` - (`gateway-ops`): the haproxy config, the nginx main config **plus every document its own glob - `include` names** (so the fixture tree cannot drift from the config's tree while the parser still - follows one level), `/proc/net/tcp{,6}`, and `gateway_resolver.json` — the resolver section - itself, captured the way the agent will produce it (task 02, below). Three rules it exists to - keep. (a) **No privilege and no copy tool**: one `ssh cat ` per document and one - `getent` per name per family, because `/proc/net/tcp` reports size 0 and `scp` would copy an - empty file, and every path and binary it uses is world-readable and world-runnable — the same - reads the agent performs as its own user, so a capture that needed sudo would be recording - something the agent cannot see. - (b) **A bad read is never a fixture**: a nonzero `cat`/`ls`, an empty body or a missing glob include - exits non-zero having written nothing, and a half-written file is removed — the pin is that the - recorded bytes survive a dead alias and an empty read both. (c) **Configs verbatim, socket tables - identifier-free**: the configs keep every `server_name`, `proxy_pass` and key path (a domain is the - map the page draws and a key path is a filename, not a key, while an agent-only bundle carries no - key at all), but a globally routable address in a socket table becomes a documentation one (RFC - 5737 `203.0.113.0/24`, RFC 3849 `2001:db8::/32`) through an idempotent column substitution — ports, - states, every other column and every space are verbatim, loopback/private/link-local/wildcard - addresses are kept, and the IPv4-mapped `::ffff:` spelling survives — because a global IPv6 on - Linux is EUI-64 (derived from the NIC MAC) and an inbound visitor's address is not this repo to - publish: the same rule that scrubs a drive's serial (invariant 6). Those two tables are volatile by - nature, so `tests/test_agent.py` asserts their **shape**, the ports the recorded configs claim, and - the absence of a globally routable endpoint — never a particular row, which a re-run may lawfully - change. - **The seventh fixture is the resolver section, and it is derived rather than listed (phase 32 - task 02).** `tests/fixtures/gateway_resolver.json` is the agent's own `resolver` document — same - keys, `via: "libc"`, entries in `target_names()` order — so a source test can exercise the - name→address join hermetically instead of wishing for DNS. Three rules keep it honest. (a) **The - name list is the parser's, never the script's**: a `plan` step rebuilds the agent's `--root` tree - from the config fixtures in this repo, imports `agent.host_agent`, and asks `target_names()` + - `address_family()` what to look up — so a route added to the gateway moves the fixture with it, an - address literal costs no remote read, and `tests/test_deploy_scripts.py` fails the day a domain is - hardcoded into the script. (b) **`getent ahostsv4` and `ahostsv6`, one name at a time**, because - that is libc on the box that dials; `rc 2` is "no record" and becomes `status: "nxdomain"` — a - fact, kept beside its successful neighbours — while `rc 255` (the ssh died), `rc 127` (no `getent`) - or **not one name answering** is a broken vantage, so the run exits 1 having left the recorded - document byte-identical: a fixture of nothing but `nxdomain` would make every live route on - `/gateway` a permanent dash and every join test a test of nothing. The `ts` is a constant and the - answer set is deduplicated and sorted with the agent's own `sort_resolver_addresses`, so - re-capturing a gateway nobody changed is a no-op. (c) **A resolver answer is an identifier**, so the - socket scrub runs over it too (`8.8.8.8` → `203.0.113.1`), and the `::ffff:` spelling `getent - ahostsv6` synthesises for a name with no AAAA record is dropped when the A record answered — the - agent's one `getaddrinfo` never publishes it. This fleet's targets all answer private LAN addresses, - which is why the recorded document is scrub-free; `tests/test_agent.py` pins that absence as a - warning sign, because a documentation-space address there would not necessarily match the one the - socket fixtures gave the same endpoint, and `tests/test_gateway_source.py`'s join depends on their - agreeing. The version's additivity is pinned two ways: permanently, by - `SNAPSHOT_KEYS_BEFORE_1_3_0` (1.2.0's top-level keys) in `tests/test_agent.py`, whose only legal - successor is that set plus `resolver` — and value-for-value, by `data/logs/verify-32-additive.py`, - which imports `HEAD`'s agent file beside the working tree's and calls both against one `--root`. - One defect only the second kind of control can find: the lookup is - `getaddrinfo(name, port, type=SOCK_STREAM)`, and `proto=SOCK_STREAM` — the spelling that reads as - "the TCP addresses" to anyone who has not read the man page — is `EAI_SERVICE` on real libc for - **every** name, because `proto` takes a protocol number (`IPPROTO_TCP` = 6) and `SOCK_STREAM` is 1. - A stubbed `getaddrinfo` accepts any keyword, so the whole resolver suite stayed green while the - agent reported `error` on a healthy gateway; `test_the_lookup_the_agent_performs_is_one_real_libc_accepts` - removes the module-wide stub and asks real libc about `localhost`, the one name it answers from - `/etc/hosts` with no network. What phase 28 stopped at was the wire: `sources/host.py` still passes **none** of the - three into `/api/system/snapshot`, and `store.py` still records no series for a config — the join - and the two gauges it publishes are phase 29's `sources/gateway.py`, which reads the same - `/snapshot` through its own `gateway` job. What the *system* page does read today is their - **error lines**, through `collectors.ADDON_ERROR_PREFIXES`, so a host with no proxy config stays - out of `/system`'s `degraded` (invariant 3). +- `agent/host_agent.py` — stdlib-only (3.10+), one per host; serves `GET /snapshot`, + `/info`, `/schema`, `/health`. Reads `/proc`, `/sys/block`, amdgpu DRM for GPUs, + `statvfs`; `smartctl -i -A -H` per whole `/sys/block` disk with a scaled pass budget + (20 s floor, 5 s per disk) whose refresh runs on a background thread — never inline on + a later poll. Refuses to bind off-loopback without a token **or** a certificate bundle + (exit 2). **TLS mode (≥ 1.1.0)**: the bundle is the auth (the bearer token is the + compatibility path), the `# REVOKED` denylist in the bundle is re-read when the bundle + changes, and a serial on it is refused on every path including `/health`. **The proxy + add-on (≥ 1.2.0)**: the `haproxy`, `nginx` (plus its one-level `include`s) and `sockets` + sections; **the resolver section (1.3.0)** looks up *only* the names the two config + files declare, through libc on the box that dials, cached and single-flight. All four + sections are additive, `available` is three-way (`true`/`false`/`null`) with exactly one + error line per non-`true` arm — the missing one saying **not found**, the pinned wording + the dashboard reads to answer "is it installed?" — and `--no-proxy` / `--no-resolve` + remove their keys in silence. Nothing in a section can fail `/snapshot` or `/health`. + Counters stay counters: the dashboard computes rates. - `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real - env var always wins. Defaults are safe except the sources, which default to unset. The auth - block (phase 11) is `DASH_ADMIN_PASSWORD` (the **seed**, and `_env()` strips, so `VAR=` means - *unset* rather than "the password is empty"), `DASH_SESSION_HOURS` (720 = 30 days: a dashboard - that logs the operator out mid-outage gets bypassed), `DASH_COOKIE_SECURE` (off, because dev is - `http://127.0.0.1`; task 03 also treats `X-Forwarded-Proto: https` as secure), and - `DASH_PBKDF2_ITERATIONS` (default `auth.PBKDF2_ITERATIONS`, imported so the primitive and the - setting cannot drift) — that knob exists so the hermetic suite hashes in microseconds instead - of 0.3 s per client and is not a production tuning parameter. The gateway block (phase 29) is - `GATEWAY_HOST` (default `gateway`, and **a name, never a URL**: `gateway_enabled` is true only - when a `HOST_TARGETS` entry has it, so the page is unconfigured rather than red for a - deployment with no ingress — the `nut_enabled` shape, and the reason `any_source_enabled` - includes it), `GATEWAY_INTERVAL_SECONDS` (30 — a connection count is the shape of half a - minute, not of ten seconds) and `GATEWAY_TIMEOUT_SECONDS` (10, the plain agent fetch with - headroom for a handler that may be mid-smartctl pass for `/disks`). + env var always wins. Safe defaults except the sources, which default to unset. The + gateway source is enabled only when `GATEWAY_HOST` names a `HOST_TARGETS` entry. - `poller.py` — job loop, health records, stale/last-good semantics, transition events. - `/disks` is a **source of its own** (`disks:`, `DISKS_INTERVAL_SECONDS` 300 s, - `DISKS_TIMEOUT_SECONDS` 30 s, events under the dashboard name `disks`) reading the same - agents the `system` jobs read: the separate task is what keeps a SMART poll out of the - 10-second utilisation loop and lets `/disks` age on its own instead of blinking `/system`. - `gateway` is a **job of its own** for the same reason (phase 29 task 02) — `GATEWAY_INTERVAL_SECONDS` - 30 s, `GATEWAY_TIMEOUT_SECONDS` 10 s, health and events under the dashboard name `gateway` — but - **one** job rather than one per host (the `nut` shape), dialling the `HOST_TARGETS` entry - `GATEWAY_HOST` names with its `label` set to that host's name, because the page is about the single - ingress and a record per machine would ask every host a question only one of them can answer. It exists only - when that name is a target: an unconfigured source has no task that could look red - (invariant 3), and it is handed no `RateEstimator` because a count is not a rate (rule 2). - The job list is built once (`app.build_hub()`) because it is wired twice — production and - the injected-store test path — and two hand-written copies drifting is how a source ends up - polling in tests and not in the container. The - success path records (payload → samples → events) and only *then* moves `last_ok`, so a - refused SQLite write ages the source and alerts instead of leaving it current with charts - that have silently stopped drawing. - Events are stored under the **dashboard** name (`system`/`disks`/`gateway`/`llm`/`ups`), not the sample - source (`host`) — a page filtering `/api/events` by `host` gets an empty feed forever - (found on `/system` during phase 03 browser verification). -- `collectors.py` — the fetch half. `AgentTls` (phase 15 task 03) is what makes an `https://` - `HOST_TARGETS` entry a **certificate-authenticated** dial: `verify` and `cert` are the paths - `certs.write_material` wrote at startup, and the dial is verified against this app's own root - and presents this app's own client certificate. Two things about it are load-bearing. (a) The - credential travels in a **client built per dial**, because `httpx` resolves TLS material when a - client is *constructed* (there is no per-request `verify`/`cert`, and `verify=`/`cert=` are - deprecated anyway — the supported form is an `ssl.SSLContext`, which is also the honest shape: - trust in one named CA, hostname checks on) while one shared client dials the plain agents and - both engines — a `verify` on *that* would make every engine read depend on a CA nobody issued - their cert with. (b) `status` is a **callable** doing a fresh `certs.agents` read, so a - revocation clicked in the browser bites on the next poll with no restart, and a revoked host is - refused **before any socket opens** — invariant 3's shape: a source the app knows it must not - talk to says so without a network round trip, and the poller's stale/last-good semantics do the - rest. `refusal_detail()` quotes the agent's own 403 sentence (`client certificate required` / - `client certificate revoked`) because those are two different operator actions, and the - certificate wording is used only for a dial where a certificate was presented — a 401/403 from a - token-mode host keeps the token/HTTP wording, or the operator is sent to `/certs` for a token - problem. `agent_tls_map()` builds one entry per configured host (the CA and client pair are - shared — the dashboard has one identity — only the revocation answer is per name, and an absent - entry is *active*, never-issued being a different fact from revoked); `build_hub` takes that map - and hands the **same** entry to the `host:`, `disks:` and `gateway` jobs of one host, so a - revocation fails every source reading that agent with one sentence and none of them dials - (`tests/test_api.py`'s revoked-gateway pin is the three-job version of the `host:`/`disks:` one). - `fetch_gateway_snapshot()` + `gateway_job()` (phase 29 task 02) are the third reader of one - `/snapshot`: the job is **one per deployment** (the `nut` shape, key `gateway`, `label` = the - `GATEWAY_HOST` name) on `GATEWAY_INTERVAL_SECONDS` with `GATEWAY_TIMEOUT_SECONDS`, it is handed - **no** `RateEstimator` (a connection count that goes through a Δ-window is a rate with a label and - no measurement behind it, rule 2), and `gateway_job` returns `None` when the name is not in - `HOST_TARGETS` — the `disks` configured gate, so an unconfigured source gets no task that could - look broken and `/api/health` says nothing about it. `fetch_snapshot()` keeps - `http://` targets on the shared client with their bearer token, unchanged from before the phase, - so a rollout is per host. - `NutMonitor` owns per-UPS transition bookkeeping (its - self-test event takes its severity from `nut.self_test_outcome()` and not its own word list, - so a passed test is never an amber line in the feed while the card beside it reads green). - Its per-UPS tolerance for a failed `LIST CMD`/`LIST RW` is for a *complete* error reply only: - one battery whose command list is denied loses its buttons and the poll goes on, while a - **stalled** reply fails the poll, because `NutClient` marks a socket it could not read to its - `END` as unusable (a reply still arriving would otherwise be taken for the *next* command's - answer — the real cause of the `expected 'BEGIN LIST VAR', got 'RW apc …'` lines this fleet's - event table used to fill with, and the same hazard on `INSTCMD`, where a late `OK` read as a - different command's success would be a gate that lies). - Failure messages name the **redacted** URL (`strip_userinfo` / `Target.display_url`), never the one - that was dialled, because they become health records and event rows. `fetch_snapshot()` is - the one agent `/snapshot` read that `system` and `disks` share, so a dead host says the - same sentence on both pages and only the `timeout` differs (the disks poll has to outlast - the agent's smartctl pass, and `None` is translated into httpx's sentinel because a - literal `None` means *no timeout at all*). Agent `errors` - become the payload's `degraded` **except** the add-on lines named by - `ADDON_ERROR_PREFIXES` (`smart`, `haproxy`, `nginx`, `sockets`, `resolver`) — a host without - smartmontools, - without device read access, without a proxy config, whose kernel socket table did not answer, or - whose resolver had nothing to resolve / stopped mid-pass is unconfigured, not broken (invariant - 3), and `degraded` is `alerts.py`'s input, so a fleet-wide - agent upgrade would otherwise put an overview WARN on every box that never ran haproxy or nginx. - Those lines stay verbatim in `agent.errors` for the page that owns the add-on — `/disks` explains - the `smart:` ones (and since phase 22 a host whose agent said `smartctl` is *missing* is not - listed on that page at all, `api._disk_visible`), phase 29's gateway page reads the - `haproxy:`/`nginx:`/`sockets:`/`resolver:` ones the same way. - **A new agent add-on section adds its prefix here in the same commit**, or its absence-report - becomes a red banner on a page that has nothing to say about it. -- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card - (`gpu.busy_pct@`). `agent_tls_fields()` is the phase-15 passthrough: a TLS agent's - `tls`/`bundle_rev` ride into the payload's existing `agent` block, and a token-mode snapshot - grows nothing (no `tls: false`, no `bundle_rev: null` — rule 2), so the fields are how the API - distinguishes a stale bundle from an agent that never had one. -- `sources/disks.py` — the same snapshot projected onto **physical** drives for `/disks` - (phase 09): one row per whole-`/sys/block` disk (never a partition, never a mount), the - agent's `smart` block carried through verbatim, `health` the drive's own word normalised to - `ok`/`failed`/`None`, and `smartctl_available` three-way on purpose — `True` (a disk - answered), `False` (the agent said the binary is not installed), `None` (an agent predating - the pass, or `--no-smart`: nobody knows, so `n/a` plus the agent's own `smart_errors` line - and never a guessed verdict). Since phase 22 the flag's *page-side* consequence is not only - the why-line: `api._disk_visible` reads `False` as "not a host on `/disks` at all", while - `None` keeps its dimmed row — the computation here is unchanged and the agent still reports - all three arms whatever the API decides to list. **Nothing here - is a rate**: power-on hours and the sector counters are stored counters-as-reported, so the - source takes no `RateEstimator` and its payload carries no `window_seconds` (rule 2 — a rate - with no estimator, no `Δ` label and no `source` string is an invented number). - `disk.health_ok@dev` is not a series either: a verdict is not a number, so the drive's word - travels in the payload only. Each row does carry a `severity` dict (`SEVERITY_FIELDS` = the - two rated numbers, `temp_c`/`percentage_used`, plus `health` through `HEALTH_SEVERITY`, - which translates the drive's own word instead of grading it), and `UNGRADED_FIELDS` names - the numbers that must never be graded. `fleet_disk_summary()` sums the per-host `summary` - counts for the page header and drops a host with no payload from the sums while still - counting it in `hosts_total` — the `/llm` fleet-block rule, so the page never does - arithmetic and a dead agent never shrinks a fleet total. -- `sources/gateway.py` — the same snapshot projected onto the **connection topology** of the one - host `GATEWAY_HOST` names (phase 29): `parse(snapshot, host)` joins agent 1.2.0's `haproxy` - frontends and `nginx` `listen`s into **`listeners`** (one per listen port, carrying the - `server_name` domains and the live ESTABLISHED count + distinct client addresses — the key was - `inbound` until phase 30 task 01 renamed it, because "a port a config names" is the thing the - page draws a trunk for, not a direction) and haproxy - and the config-declared routes into **`connections[]`**. **The join is two attributions and - nothing else** (phase 30 task 02 — the operator called the phase-29 version overengineered, and it - was). Inbound: a socket's **local** port names a configured listener, which is a fact (`/proc` - states it, no config reading needed). Outbound: a socket's remote `(address, port)` **equals a - target written as an address** in one of the two config files — the only outbound match that - claims nothing, because the target of a `server` line, a `proxy_pass` or an `upstream` member *is* - the endpoint the gateway opened. `join_sockets()` still partitions the table in **one** pass, so a - socket classified both ways would be counted twice — which is exactly what `127.0.0.1:8443` is: - the stream `map`'s `default` sends `:443` to that port and the box's own http vhosts listen on it, - so the loopback hop counts **once as one inbound and once as one outbound**, a pipe on each card, - neither claiming the other's number. Everything else is **ignored**: not counted, not totalled, - not warned about. What that retires is the guessing and the bookkeeping it existed to explain — - the port-only guess for a name target, `unmatched` / `unmatched.groups` / `partial` / - `unmatched_listeners`, and `by_port` with its `MAX_GROUP_HOSTS` chip. A **name** target - (`reeseapps-immich.reeselink.com:443`) is credited **only through an address the gateway itself - published** (phase 32, below): crediting it any other way needs DNS, which the module still never - calls (spy-pinned), and every name here resolves back through this same router, so a vantage of - ours would attribute a socket to a route on the strength of a query the router never made. Two - consequences the - page and the tests both depend on: **`live_out` is not the bandwidth** and the socket table never - reconciles to it — a pipe count means "this route has N live connections" and nothing more — and - several `(listen, endpoint)` lines reaching one endpoint carry one count **repeated** with - `shared: true` (never a split nobody measured) — a shape that is real now that a name route is - counted: two vhosts writing one `(address, port)`, two **names** the resolver placed on one - address, or a multi-homed name sharing one address with a single-homed one — which is why the - API's totals and the tests sum **distinct endpoints**. A pipe is `null` (never `0`) whenever the socket table did not answer or no - line reaches its endpoint, and publishes **no sample** then, so a chart cannot draw a zero for an - unknown. `supported: false` - (a 1.1.0 agent, or `--no-proxy`) keeps every key with empty lists and null counts plus - `WHY_PREDATES`, `available: false`/`null` per section becomes a `why` sentence and the agent's - own line in `section_errors`, and nothing here carries a `severity` (`net.*` precedent: a busy - proxy is the job) or a `window_seconds` (a count is not a rate). **`connections[]`** (phase 30 - task 01) is the graph the pipe lists never stated: one row per route the two config files - actually declare, from `config_connections()`, each carrying its `in.port`, its `out` - `{host, port, name, kind, selectors, address}` (`kind` is `backend` / `server_name` / `pool` / - `map`, `selectors` the config's own word(s) — a haproxy backend name, an nginx `server_name`, a - stream `map` key, the literal `default` kept verbatim), its own `metric`, and `connections: null` - until task 02 re-points the socket join at it. Four derivations (haproxy `frontend → routes → - backend → server`, an nginx vhost's concrete `proxy_pass` per `listen` port of **its own block** - — which is why `nginx_server_blocks()` replaced the per-port `nginx_listen_ports()` — a - port-less `proxy_pass` naming an `upstream`, and a `proxy_pass $var` fanning out through the - `map` whose `variable` matches), one merge rule (identical `(source, in_port, host, port)` is one - line whose selectors union in first-seen order; **never** merged across `in_port`) and two silent - refusals (a `proxy_pass` with neither target nor variable, and a haproxy frontend that routes by - a fetch expression, which the agent reports as no name — that trunk is drawn alone, because "no - name" and "no routing directive" are one value in this payload). **A name-written endpoint is - joinable through the address the gateway published for it** (phase 32 task 03): `resolution_index()` - keys agent 1.3.0's `resolver.entries` by `(name, port)` — `{}` when the section is absent or its - `available` is not `true`, which is phase 30 byte-for-byte, and never an entry with no addresses — - and `apply_resolution()` stamps every `out` with `resolved` (the address list, **never `[]`**: an - entry with no addresses and no entry at all are one fact, so the page's dash arm is one `is null` - test), `via` (the agent's own `"libc"`, copied, `null` where nothing was resolved) and - `resolved_ts` (**that entry's** stamp, falling back to the section's, because a name refreshed five - minutes ago and one answered on this poll are two claims). All three stay `null` on an - `address: true` row: the config wrote that address, and copying the agent's `literal` echo of it - into `resolved` would report a `getaddrinfo` that never ran. With an address and a port the line - becomes countable, so **name-keyed gauges are back** — `gateway.out_conns@:`, under - the config's own spelling so history continues under one name. That is the reverse of phase 30 - task 02's retirement for exactly one reason: the retirement existed because the only figure such a - series could carry was the *port's*, and the figure here is the sockets to the address this domain's - own libc resolution produced **on the box that dials**. An unresolved name keeps `metric: null`, - `connections: null`, no sample and the dash the footer explains. `gateway.in_conns@` is - unchanged, and the port **stays** in the outbound name, because this fleet's two haproxy Minecraft - backends share `minecraft.reeselink.com` on 25565 and 25566 and two gauges on one name would - overwrite each other in silence; lines sharing an endpoint write the same value to the same name - (they are one count, not two) and `samples()` de-duplicates by metric. - **The join key is a normalised address, never a spelling** (`_address_key`, `endpoint_of`, - `_endpoint_groups`): `ipaddress` canonicalises and folds `/proc`'s `::ffff:10.1.2.64` to the - `10.1.2.64` `getaddrinfo` returned — one kernel endpoint, one count, however many spellings or names - reach it — while a zero-padded `010.1.2.64` stays **no one's address** (decimal versus octal is a - guess, and a guess on the far end of a pipe is rule 2), so `out.resolved` keeps the agent's bytes and - only the key is canonical. `_endpoint_groups()` is a union-find over those keys (a name route and an - address route reaching one endpoint are one group; overlapping multi-homed sets merge; a route with - no endpoint is no group) and `distinct_endpoint_counts()` is what `api._endpoint_values` sums for - `live_out`, which is the only reason the head tally and the pipes cannot disagree. Matching stays - exact equality on `(address, port)` — no prefix, no range — and this module still resolves nothing: - `socket.getaddrinfo` is spy-pinned absent over a name-only synthetic, a resolver-full synthetic and - the captured `resolved` ingress, plus `samples()` (a gauge name is built from a host string and is - the other place a resolver could creep in). The payload's own `resolver` block is - `{available, via, entries}` with each entry **projected** to `name`/`port`/`status`/`addresses`/`ts` - — every entry including `nxdomain` and `error`, because the index filters *within* the published - list and a page that says *this name has no record* needs the entry that says so — and a section - that did not answer publishes none. `parse_gateway_snapshot()` is the - `Result` the poller consumes; there are **no events** (a connection count has no transition - worth a line — a proxy that stops routing shows up as its host's own poll failing). The - never-answered arm is `stub(host, ts)` — the **same skeleton** `parse` fills in, plus - `unreachable: true`, `supported: null` and `WHY_NO_ANSWER` — because the route that serves a host - with no payload yet must not hand the page a shape with keys missing, and `WHY_NO_ANSWER` is not - `WHY_PREDATES`: an agent that has not replied has not said it has no sections (invariant 3's two - facts staying two sentences). **`warnings()`** (phase 30 task 03) is the payload's other half: the - two config files judged against **themselves**, seven kinds and nothing else — - `haproxy_route_dangling` (a `default_backend`/`use_backend` name no `backend` declares, one row per - frontend→backend line, `subject` `"->"`), `haproxy_backend_orphan` (a `backend` - no frontend routes to, its server count from the config), `haproxy_backend_empty` (a `backend` with - no `server` — deliberately *not* also an orphan: `orphan` needs a server to have nowhere to go), - `nginx_server_unlistened` (a `server` block with a `server_name` and no `listen`, in both its - proxying and non-proxying arms), `nginx_pool_undeclared` (a `proxy_pass` whose value has no port — - the shape task 01's `out.kind: "pool"` already draws no line for — naming no `upstream`; a block - with two such `listen`s is **one** warning, `subject` `"@+"`, because the repair - is one line), `nginx_variable_unmapped` (a `proxy_pass $var` no `map` defines — the `map` *is* this - fleet's router, so that is a broken route and not a style note) and `nginx_map_route_missing` (an - entry of a **referenced** `map` whose value is not an address). Every row is - `{id, kind, subject, message}` with `id = ":"`, `message` complete on its own because - `alerts.py` quotes `warnings[0]` verbatim, `kind` the vocabulary the page groups by and `subject` - the config's own name for the thing, never a paraphrase of the message. Three things it refuses to - do, each pinned from the real capture and not only from a fake section: it never **joins** a warning - to `connections[]` or `listeners[]` (a page indexes one row by its own `id`; a boolean beside a - hundred rows is a second source of truth about the same route), it never guesses a **name - target**'s address (the module still never calls `socket.getaddrinfo`, spy-pinned), and it stays - silent about the three shapes that are legal or unreadable rather than broken — a vhost with a - `listen` and no `proxy_pass` (a `:80` `return 301` is identical here), a frontend whose only routing - is a fetch expression (agent 1.2.0 reports *no name*, the same value "no routing directive" has, so - distinguishing them is an agent change and not a source-level guess), and an entry of a `map` no - `proxy_pass` names (the `$connection_upgrade` map's `upgrade`/`close` are header words, not missing - routes). Like the socket half, a section whose `available` is not `True` is **not judged** — a host - that cannot be read has not been found guilty of anything — so `warnings` is `[]` for an absent, - unreadable or pre-1.2.0 config and the page cannot tell "clean" from "unreadable" by that list - alone; it reads `sections` for that, the same pair `supported`/`why` already form. Order is - deterministic (haproxy kinds in config order, then nginx kinds) and rows are deduplicated by `id`, - because the page keys its DOM on it and the alert quotes the first. -- `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`, - `VLLM_HISTOGRAMS`), measured Δ rates **diffed per label set** (`rate_key()` keys the - estimator on the resolved name plus its labels; the cell is the Σ of the per-series - rates over the widest window one of them actually used — phase 13, so a joining - `{engine="1"}` series can never lend its carried counter to one interval's delta), - prefill speed as `mean(prompt tokens)/mean(prefill time)`, session list from gufo - `/slots`. A refused window (the estimator's floor) is still an answer: the cell keeps - its `computed:Δ/Δt` source and its honest `Δ` beside a `null` value. -- `sources/nut.py` — upsd client, `UpsReading`, `VAR_MAP`, `assess()` (explicit checks, not - a mystery score), `SELF_TEST_OUTCOMES` + `self_test_outcome()` (the `Last self-test` verdict - follows the *outcome* of the driver's `*.test.result` word, not its phrasing: pass words → - `ok` (including `Done and passed`, what the CyberPower here answers), fail words → `fail`, - `No test initiated` / empty / no variable → `unknown` printing `NEVER_TESTED_DETAIL` so both - "never tested" states say one sentence, and an unrecognised word → `warn`, because a driver - inventing a verdict is drift worth noticing and the check prints its own word to be added), - `COMMANDS` catalog with risk tiers, `COMMAND_ALIASES` + `resolve_command()` - (drivers answer two names for the same action — `beeper.enable`/`beeper.on`, - `beeper.disable`/`beeper.off` — so one entry is offered per action and any of its names hits - that entry's verdict, never a gate the rendered name missed), `available_commands()` filtering, - kelvin-vs-celsius normalisation, and `charger_status` normalised to the `CHARGER_STATUS` - keys (the same lowercase token `states()` and `on_battery` already compare), and `states()` - publishes one chip per fact — a driver exporting both `battery.charger.status` and the - legacy flag that says the same word gets "charging, online", not "charging, online, - charging". `battery_runtime` - comes from `battery.runtime` **only** — `battery.runtime.low` is a shutdown threshold, not - time left, so it lives at `battery_runtime_low` and never becomes the hero's "runtime left". -- `sources/nut_sim.py` — a **labelled** simulator so the UPS page is developable without a - battery. It must never be the default, and every payload it produces says `simulated`. - Its modelled `battery.mfr.date` is computed relative to today (not a fixed string that - ages), so a fresh simulator reads healthy and amber keeps meaning something. Its status - follows the real drivers: a below-float pack on mains reports `OL CHRG`/`chg` (the glyph's - bolt state is keyed on that token, and the model has to be able to produce it) and a battery - being drawn from is `OB`/`dchg`, never `ol`. Every command it advertises is implemented and - visible in a variable a real driver publishes — the beeper ones set `ups.beeper.status` - (`enabled`/`muted`/`disabled`), and a test window that ends writes `ups.test.result: PASSED` - while an aborted one writes nothing; the load/shutdown commands answer `OK` without modelling - an outlet, because a faked power cut in the history is indistinguishable from a real one. - `from_env()` is the single place the `NUT_SIMULATE` truthy rule lives — `app.py` builds the - monitor through it, so the assembly and the model cannot disagree about what is enabled. -- `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own). - `scalar(agg="sum")` stays right for **gauges**; `values_by_series()` is the per-label-set - view **counter** deltas need, because a summed counter mixes whatever label sets exist now. -- `rates.py` — Δ/window estimator, keyed by resolved series name + label set. Its effective - window is `now - base[0]` **bounded on both sides** (phase 13): under - `MIN_WINDOW_SECONDS` (the poll loop's 0.2 s catch-up gap after any slow fetch) it answers - `(None, dt)` — one `n/a` with the window it refused — and drops that base so the polls - *after* the stall cannot re-diff it; the window-empty fallback accepts a base no older - than `MAX_WINDOW_RATIO × span` (the `2 × span` ceiling the docstring always claimed, now - enforced). A rate ≥ 8× the previously published one is held for one - poll (the previous value repeats, label unchanged) and publishes only if the next interval - confirms it (`JUMP_CONFIRM_RATIO`) — and **confirmation means a different anchor**: a held - jump remembers the sample its window was diffed from, and stays held while that same sample - still anchors the window. A stalled fetch stamps its body with the time the poll *finished*, - so that sample is the stall older than its own timestamp, and every later window off it is - the same skewed delta re-diffed over a longer span — measured twice from one sample is one - measurement, and the hold simply continues until the stale anchor ages out of the span. - `min_window=0.0` exists for tests that scrape faster than any real loop. **The span belongs to - the cadence that fills it** (phase 23): `app.build_hub` builds one estimator per measuring - cadence — the host's at `RATE_WINDOW_SECONDS` (10), the LLM's at `LLM_INTERVAL_SECONDS` (2) — - because a window wider than the poll averages bursts across the polls the operator is watching - and a window narrower than it has nothing inside it to diff. Nothing else about the estimator - moved: the floor, the ceiling, the jump gate and the label are the same code, and a 2 s window - is 4× above the floor, so a healthy 2 s cadence is never refused. -- `auth.py` — every authentication decision as a pure function, stdlib only (phase 11): - `hash_password`/`verify_password`/`needs_rehash` over a self-describing - `pbkdf2_sha256$$$` verifier (so the cost can rise without a - migration), `mint_session`/`read_session` for an HMAC-signed stateless cookie (no session - table and nothing to prune; rotating the stored secret signs every cookie ever issued out, - which is how changing the password evicts every other browser), `mint_csrf`/`verify_csrf` - bound to the session token instead of stored, `LoginThrottle` (in-process exponential - backoff), and `origin_matches`/`sec_fetch_site_allowed` as string functions so the gate is - testable without a `Request`. The primitives **never raise and record nothing**: a malformed - verifier, a tampered cookie, and a rotated secret all answer `False`/`None`, because the - reason an input was rejected is the credential (invariant 6). No HTTP, no SQLite driver, no - new dependency — `app.py`'s middleware and `api.py`'s `/api/auth/*` are the callers, so a - decision made here is made once and is unit-testable without a `Request`. It also carries the - two strings the gate and the pages share (`LOGIN_PATH`, `LOGIN_NEXT`) and `safe_next()`, the - only judge of a `?next=` value: kept verbatim (query string and all, because the range chips - live there) when it stays on this app, dropped otherwise — scheme-prefixed, scheme-relative, - the percent-encoded spellings of both, anything with a control character (not a path but a - second `Location` header), and anything without a leading `/`. - The **verifier bootstrap** (task 02) is the one deliberate exception to "never raises": - `load_auth_state(store, admin_password=…, iterations=…, session_hours=…, db_path=…)` reads the - `meta` rows `auth.verifier` / `auth.session_secret` through a three-method `MetaStore` - `Protocol` (so this module still imports nothing in the package, and its tests use a dict) - and returns a frozen `AuthState` — the environment **seeds** a password once and the stored - verifier wins thereafter (`env_ignored` and one fixed-line warning when both are present, so - `/settings` is the real answer and no redeploy can roll a changed password back); no row and - no seed raises a `RuntimeError` naming `DASH_ADMIN_PASSWORD`, the meta key, and the - `sqlite3 … DELETE FROM meta …` reset command, because "start open with a warning" is the - failure this phase exists to prevent; a stored secret that is not 256 bits of hex is - regenerated and persisted (the effect is one login prompt, never a lockout). Nothing here - reads a clock, so the state carries no expiry — a session's is inside its signed payload. - `update_verifier(store, state, new_password)` writes the new verifier **and** a fresh secret - (secret first, so a crash cannot leave sessions outliving the password they were issued - under) and refuses an empty/whitespace password; it is what `POST /api/auth/password` calls, - because rotating the secret **is** this app's revocation list — a stateless cookie has no row to - delete, so without it a stolen session would outlive the password that "revoked" it for another - `DASH_SESSION_HOURS`. Its two raises, like the loader's, carry - fixed text — `tests/test_auth.py` pins the file's raise sites and its one logging call - structurally, so no future edit can interpolate a credential into either. -- `certs.py` — the fleet's certificate authority as the same shape of module (phase 15): a pure - lifecycle over the three-method `MetaStore` protocol, no clock it cannot be handed, no `logging`, - and one fixed-text `RuntimeError` (`tests/test_certs.py` pins all three structurally, the way - `tests/test_auth.py` pins `auth.py`). Five `meta` rows and nothing else: - `certs.root.key`/`certs.root.cert` (self-signed `CN=dashboards-root`, 10 years), - `certs.client.key`/`certs.client.cert` (`CN=dashboards`, 1 year — the identity every agent will - check), and `certs.agents`, one JSON object mapping a `HOST_TARGETS` name to - `{serial, not_after, status, bundle_rev, bundle, san}`. **A stored row always wins:** the root is - generated on the first boot of a database and never on a later one, because a CA that rotates - with the process untrusts every bundle already installed on a host — and a newly generated root - takes its client certificate with it, since a leaf signed by the dead root is a credential - nothing can verify. `issue_agent_cert`/`reissue_agent_cert` (the API's "rotate") sign a fresh - `CN=` leaf whose SAN is the addresses that host is dialled by — the name is always in the - list, and an IP goes in as an `iPAddress` or hostname verification of an IP dial fails — bump - `bundle_rev`, and reset `status` to `active`. `revoke` marks the host and **re-seals every - bundle**, because the `# REVOKED` denylist lives inside each of them and an agent that cannot see - a serial keeps accepting that client; there is no "unrevoke", and revoking a never-issued host - still records the revocation (what a revoke enforces is the dashboard refusing to dial). The - **bundle** is the install artifact — agent cert, agent key, the root *cert* (never the root key, - asserted by header count and by parsing the one key against both keypairs), `# BUNDLE-REV `, - then the denylist — stored verbatim so what an operator downloads is byte-identical to what the - row holds. `rotate_root` mints a new root plus a new client and reissues every agent - (`bundle_rev` +1 each, denylist reset with the generation); the running process keeps the - material it loaded at startup, which is the honest `restart_required` task 04 answers with. - Every certificate carries `SubjectKeyIdentifier` and every leaf an `AuthorityKeyIdentifier` - naming it, non-critical: OpenSSL's **strict** mode refuses a leaf with no AKI - (`CERTIFICATE_VERIFY_FAILED: Missing Authority Key Identifier`) and Python's - `ssl.create_default_context()` — the context the dashboard's own https dial builds — turns that - mode on. The agent's server-side check is not strict, so nothing on the agent side notices; - `tests/test_certs.py` pins the identifiers instead, and a bundle recorded without them is one a - strict client rejects. `write_material` puts the CA and the client pair in a `0700` temp dir with - `0600` files for the dialing — the root key is never written to disk, so its only copy stays the - `meta` row — and the lifespan removes the directory on shutdown. Unreadable PEM, a key that does - not match its certificate, or a leaf the stored root did not sign is a **fail-closed startup** (the - `ctx.auth` rule), and the message is the `sqlite3 "" "DELETE FROM meta WHERE key LIKE - 'certs.%'"` reset. `cryptography` is this module's dependency and the app's only new one; - `agent/host_agent.py` stays stdlib and loads the PEMs with `ssl`. There are **two reads** and the - difference is a side effect: `ensure_certs` is the startup one (it mints what the table is missing, - which is why it is what `app.build_certs` calls and nothing else may), and `read_state` is the - request-time one task 04's `GET /api/certs` uses — same validation, same fixed-text refusal, never - a write, so a page refresh cannot bootstrap an authority the operator deleted. The material is - read with (task 03): `collectors.agent_tls_map` turns `AppContext.certs` into the per-host dial - pair, and because the revocation answer is `agent_status()` called per poll, a revoke in this - module is what stops the dial — no restart, no cache, no second list to keep in sync. There is a - **third door and it is deliberately outside this module** (phase 24): - `scripts/dev-certs-export.py` is the dev loop's exporter — stdlib `urllib`, signs in through - `POST /api/auth/login`, reads the token out of the `Set-Cookie`, and downloads each - `HOST_TARGETS` name's bundle into gitignored `data/certs/` at `0600`. It cannot mint because it - imports neither read; it dials the route the `/certs` page downloads from, which is what makes the - exported file byte-identical to the row. `scripts/dev-server.sh` calls it after the readiness - probe and treats a failure as a warning line — the server being up is the thing the probe proved. -- `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric, - `classify()` → `ok`/`warn`/`fail`/`None`, plus `attach()`/`with_severity()` to put the verdict - beside a value. No measurement, no verdict: `None` in → `None` out, never a green pass for - an unknown. The verdicts travel **additively** in payloads — `cpu.severity`, - `memory.severity`, `disks.severity`, each UPS card's `severity` (charge/load/runtime), - `severity` inside rated LLM cells (KV pressure, classified on a - percent scale because the engines publish a 0-1 ratio), and `severity` per host/fleet/UPS - row in `/api/summary`. Pages render the class; they never compare a number to a threshold. - There is **no `gpu.*` row and no `gpus[].severity`** (phase 06): these are LLM servers, - a busy GPU is the job at every level, so the GPU card and the fleet GPU column render - uncoloured — like `net.*`, whose colourlessness predates it. Removing the verdict is not - removing the data: `gpu.busy_pct@N` / `gpu.mem_used_pct@N` are still stored and charted. - The `/disks` rows (phase 09) are `disk.temp_c` (50/70) and `disk.percentage_used` (80/100) - — an SSD at 100 % of its rated wear is a replacement, not a warning — attached per row by - `sources/disks.py`; the sector counters and `disk.power_on_hours` deliberately have **no - row**, because a non-zero counter is an alarm chip on presence and zero is not an - achievement (the `/llm` preemption-chip rule), so `classify("disk.reallocated", 1)` is - `None` and the page shows a chip instead of a colour. -- `alerts.py` — `build(settings, job_states, host_payloads, engine_payloads, ups_payload, - recent_events)` → the "what is wrong right now" list `/api/summary` serves as `alerts` - (`{id, severity, source, message, href, at}`, fail first then warn, then by source). A - pure projection: no I/O, no store, no thresholds of its own — it reads job states for - reachability, the `severity` dicts for breaches, `health.checks[]` for UPS conditions. - Rules: an **unconfigured** source never alerts (invariant 3); a source that is ok but - older than 3× its own poll interval is `warn` while a source that stopped answering is - `fail` (the interval is the job's own dashboard's, so `disks` goes amber after 15 quiet - minutes rather than 30 seconds); a dead source gets exactly one line (its stale payload - adds nothing) that names it once — the poller's `"UpstreamError: "` wrapper and the - duplicated host name are stripped, because a message that says `gateway` twice reads as two - problems, and the same rule silences the **`disks` echo** of a dead agent (the disk job - dials the same `/snapshot`, so its failure carries no new information — while a `disks` job - failing *while its host answers*, SMART wedged inside the handler, is the only voice that - fault has); `unknown` and missing numbers are silence; one alert per (source, metric) and at most 12, the last being a marker - that says how many it hid. This is why the overview can be two panels. There is **no - disk-health rule and none is wanted**: a drive the hardware itself called `FAILED` is a red - chip on its `/disks` card and a line in that page's event feed, not an overview alert — - `severity.py` rates no sector counter and this module may not have a threshold of its own, so - grading a drive here would be the bug, not the fix (phase 09). Its `HREF` table does carry - `/disks`, so the one alert a disk job *can* raise — SMART failing while its host answers — - links to the page that names the drive, and since phase 29 `/gateway` for the same reason: a - `gateway` job failing **while its host answers** (the proxy configs or the socket table went - quiet inside a healthy agent) has no other voice. The echo suppression is now a table - (`AGENT_ECHO_DASHBOARDS` = `disks`, `gateway`) because three jobs read one agent, and it finds the - partner differently for each: `disks:` names the host in its key, while the single `gateway` - job names it in its **label** (its key is one word, and `GATEWAY_HOST` is what an alert prints). - A missing `host:` record suppresses nothing — "I cannot find the host's job" is not evidence - the host is fine. The gateway source has **no verdict rule and none is wanted**, the same shape as - the disk rule: its payload publishes no `severity`, `severity.THRESHOLDS` has no `gateway.*` row, - so 400 connections on an ingress is the job, not a fault. What phase 30 task 03 added is the one - non-reachability voice it *can* have without a threshold — **config integrity** — because "a proxy - config that does not connect to itself" is a structural fact, not a graded number: `alerts.build` - gained a keyword-only `gateway_payload`, and one `warn` row (`id` `gateway:config`, `href` - `/gateway`, `at` the payload's own timestamp so the line does not reset every 30 s) exists exactly - when that payload carries `warnings` **and** its job is configured, `ok` and not stale. One row - whatever the count — seven dangling routes are one config to fix and the page lists all seven — - reading `"{n} proxy config {problem|problems} to address — {first message}"`, with the message - built by `sources/gateway.py` and only quoted here. It is suppressed whenever the job's own - reachability row is on screen (`ok: false` or stale past 3× its interval), because a router that is - not answering is the operator's problem and re-listing warnings from a reading they cannot refresh - is noise beside it; the page keeps showing them beside the last good numbers the whole time. So - `alerts.build`'s signature is now the boundary that keeps the *counts* out: no `severity` reaches - it and no `gateway.*` threshold can be consulted, which is what - `test_a_gateway_payload_with_a_fabricated_verdict_alerts_nothing` re-pins after it replaced the - older "takes no gateway payload at all" wording. -- `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning. The - `meta` table is the only durable key/value state in the app and is reached through three - named helpers — `get_meta`/`set_meta`/`delete_meta` (the last returns its rowcount, because - the password-reset procedure has to be able to say whether a row was there, and one key per - call, because a reset that deletes a row it did not name deletes the operator's credential). - `stats()` reads `last_maintain` through `get_meta()` too — outside its own lock, which - `get_meta` takes itself and `threading.Lock` is not reentrant — while `maintain()`'s write - stays inline inside that method's transaction. **No per-request path may pay a whole-table - scan** (phase 14 task 02): `series_names()` (1.4 s of `SCAN sample` on a production-sized store, - behind `/api/metrics`) and the two `COUNT`s behind `stats()` (1.25 s, run by `app.js`'s - `/api/health` poll on every tab every 15 s) answer from short-TTL caches — 60 s and 30 s, the - series list additionally invalidated **only by a write that can change it** (task 04: an insert - cannot remove a name, so `_note_series_write()` ends the cache on a `(source, host, metric)` the - store has never written and leaves a poll of an existing series alone — invalidating on every - sample was task 02's defect, and it put the 572 ms scan back on nearly every `/api/metrics` call - on a store every poller writes to, behind a cache that never served; `delete_series()`/`maintain()` - invalidate unconditionally because a `DELETE` can take a name, `delete_series()` also forgetting - the triples it emptied so a re-created series counts as new again), through a generation counter so - a write racing a read cannot park a stale answer under a fresh timestamp, with the name cache - bounded to 256 `(source, host)` keys because `/api/metrics` takes both as free text and the - existence tracking capped at `SERIES_TRACKED_SERIES_MAX`, below which it falls back to - invalidating on every write. A tracking set made wrong by *another* process deleting rows costs at - most one TTL window, which is the reason the cache has a TTL and not only the tracking. `last_maintain` and `size_bytes` are never - cached: a `last_maintain` that stops moving is how an operator spots the invariant-4 failure - mode, and it costs a three-row lookup. The resolution decision is one compound statement of - per-`(host, metric, res)` branches whose `COUNT` is capped by `LIMIT max_points + 1` — exact, - because the decision only ever asks whether a count is `0`, `≤ max_points`, or neither — and - `pick_res_many()`/`history_many()` are the batched door the N × M endpoints go through. The rows - stay a per-series seek **measuredly**: a `ROW_NUMBER()` window or a compound of the 27 tail - selects both cost more than 27 index seeks on the seeded store (`data/logs/NOTES-store-perf.md` - records the table), so batching the *decision* is the fix and batching the *fetch* would be a - round trip traded for work. - **The write side is windowed too** (phase 14 task 03): `maintain()` reads `last_maintain` through - `get_meta()` *before* its transaction (same non-reentrant-lock rule as `stats()`) and bounds each - rollup step to what became due since, instead of averaging every row the resolution retains — - 3.97 M rows of 1-minute history per minute, a 5.4 s write transaction that made a poller's - `add_many` outlive `busy_timeout=5000` (gap 4's `store-error` line). Two properties make that a - cost bound and not a data change, and both are pinned: the lower bound reaches **past** the - previous pass (one bucket plus slack, so ten quiet minutes are a catch-up scan rather than ten - minutes of empty charts), and it is snapped **down to a bucket boundary** — the step's ceiling is - a row bound, so the newest bucket is partial and a later pass re-rolls it as its rows land, and a - floor that cut into a bucket would overwrite the fuller average with the mean of its tail rows - (2.5 becoming 4.5 on a six-sample minute: rule 2 with a chart's clothes on). A window is only a - cost bound if the plan can seek to it, and two things were in the way. The prune keeps its full - retention bound — a rollup may window because it only adds rows, a prune must reach anything that - expired — but it is **one `DELETE` per resolution**, because one - `WHERE (res=? AND ts/bundle`, - `POST /api/certs//rotate`, `POST /api/certs//revoke`, - `POST /api/certs/root/rotate`), touch this process and its `meta` table and dial - nothing — invariant 1 is about the fleet, and `tests/test_api.py` pins the three lists separately - rather than widening the one that names `/api/ups/command`. - The **`/api/certs` surface** (phase 15 task 04) is the CA's own admin half. The list reads the - **stored** state through `certs.read_state` — the request-time twin of `ensure_certs`, which - mints and must only ever be called at startup, because a `GET` that bootstrapped a root CA would - replace the authority nobody asked it to replace — and its rows are the union of `HOST_TARGETS` - and `certs.agents`, so a configured-but-uninstalled host appears as `status: "unissued"` with - nulls (invariant 3, rule 2) and an entry whose host left the config is listed but not writable. - `deployed_bundle_rev` comes from that host's `system` payload `agent` block (task 03's - passthrough), never from what the CA hopes. `bundle_rev` is what was issued; the two numbers side - by side are the "install the new bundle" sentence, and neither carries a `severity` — an expiry - and an identity counter are not graded numbers. A `` outside `HOST_TARGETS` is refused in - the route's first line with `404 {"configured": false}` **before** any signing, dial, or store - write (spy-pinned both ways, and the same gate makes `GET …/bundle` answer - `{"configured": true, "issued": false}` for a host that is configured and has nothing issued — - two different facts, two different bodies). The bundle download is the stored row **verbatim** as - `application/x-pem-file` with an `attachment` filename, because it is `install-agent.sh`'s input - and it carries that host's private key (session-gated, `no-store`). `POST /api/certs/root/rotate` - answers `restart_required: true` rather than pretending a hot-swap: the CA and the client pair are - startup material, so the table is new at once while the process keeps dialling with the old one. - Every write logs one fixed-text `certs`-source event naming host and action, and serials travel as - **decimal strings** — a 64-bit certificate serial through a JS number is a serial a page cannot - match against a bundle's `# REVOKED` line (invariant 6: no PEM reaches an event, a health record, - or any payload but the bundle route's). - The login route is where a *sign-in* password is compared: it - throttles before it hashes, refuses an absent/empty/non-object body with `422` so `""` never - reaches `verify_password`, answers every failure with the one sentence `wrong password`, and - answers `429` + `Retry-After` while the per-address backoff runs (the only extra fact it adds, - because a wait is something the caller has to act on). After a good sign-in it also upgrades a - verifier minted at a lower `DASH_PBKDF2_ITERATIONS` (`auth.needs_rehash`), keeping the signing - secret — that knob has to actually buy something, and a login is the one moment the plaintext is - in hand. A refused upgrade write is swallowed with one fixed-text `logging` line (this module's - only one, carrying no credential: invariant 6) because `database is locked` on a bookkeeping row - must not turn a correct password into no login at all. `POST /api/auth/password` is task 04: it - validates `new` first (`MIN_PASSWORD_LENGTH = 8`, non-empty once stripped, different from - `current` — length is the whole policy, because composition mandates produce sticky notes), - throttles the *current* password on its own key `password:
`, refuses a wrong one with - `403` + a `password-denied` event, and on the right one calls `auth.update_verifier` and answers - from the **new** state with a cookie and CSRF token minted from the rotated secret. Building that - response from the old `ctx.auth` is the bug the order exists to prevent: the operator would be - signed out by their own form. - `AppContext` carries `auth` and the one - `LoginThrottle` per process — a throttle built per request is a throttle that never trips. A - refused sign-in writes a `login-denied` **event** and nothing else, and so do the two password - outcomes (`password-denied`, `password-changed`): `alerts.py` turns only - `kind: "unreachable"` into an overview row and every feed filters by `source`, so - `/api/events?source=auth` is the audit trail and the overview stays about outages (do not - "fix" that by hiding the event). Neither fact appears on `/api/health` — which is public — or - `/api/config`. `/api/config` is a redaction boundary. - `/api/disks/*` (phase 09) is the drive-health pair: `/snapshot` answers one entry per - configured host **with drives to show** (disk rows + `smart` + `smartctl_available` + - per-row `severity` + a `fleet` block summed by the API, with a configured-but-never-answered - host as an `unreachable` stub whose counts are `None`, not `0`), and `/history` serves the - `disk.*@dev` series with unknown names answering as empty series like the other - history endpoints. The configured gate runs **before** anything that could reach the - network — `HOST_TARGETS` empty is `404 {"configured": false}` with no client, no fetch and - no `disks` job in the hub at all (invariant 3), and `tests/test_api.py` proves it with - spies on `httpx.AsyncClient.get` and `collectors.fetch_snapshot`. Configured-ness and - visibility are two different gates, and only one helper decides the second: - `api._disk_visible(host)` drops a host whose payload's `smartctl_available` is **explicitly** - `False` ("no smartmontools, no drives to report" — the VM case) from `hosts` and therefore - from `fleet` (`fleet_disk_summary` sums whatever it is handed, so `hosts_total` is the row - count), while `True`, `None` and an `unreachable` stub all stay — an agent that cannot say is - not an agent that said *no*, and a dead agent must not read as "never configured" - (invariant 3). The check is `is not False`, not falsy, and it runs in `/api/summary`'s disks - block too, before both the fleet sum and the per-host rows, so the two endpoints cannot list - different machines; `states` still names every `disks:` job (hidden included, still - polling, still `ok`) and the source, the store and the health records see no change at all — - which is also why an overview alert from a hidden host's `disks` job still names that host - (`alerts.py` reads job states, not this list): hiding a card must never hide a fault - (phase 22). The new surface grows no - write of its own — a fourth dashboard must not quietly become a second one, so the app's - only POST that **dials an upstream** stays `/api/ups/command` (invariant 1, pinned by enumerating - every served route's verbs and sorting the writes into fleet / session / certificate-authority - lists — the third is phase 15's, and it writes `meta` rows and nothing else). - `/api/gateway/*` (phase 29 task 03) is the ingress pair, and it is the same two rules on a - single-host source. `/snapshot` answers `sources/gateway.parse`'s payload for the one - `GATEWAY_HOST` entry — the haproxy frontends and the nginx `listen`/`map` echo, the `listeners` - (port, domains, live count, up to `MAX_CLIENTS` client addresses), the `connections` graph (one row - per route the configs declare, its `in.port` always one of those listeners, its own `metric`, - `shared` when more than one line reaches its endpoint), and the `warnings` list (phase 30 task 03: - seven kinds of config that does not connect to itself, each `{id, kind, subject, message}` with - `id = ":"`, empty for a pre-1.2.0 agent and for a section that did not answer) — - with `states` appended and the payload's own - `ts` - left alone, because that is when the agent took the reading and it is what the page ages. Three - arms reach it and none of them is red: `404 {"configured": false}` **in the route's first line** - when `GATEWAY_HOST` names no `HOST_TARGETS` entry (unset and wrong-name are both spy-pinned to - `httpx.AsyncClient.get` and `collectors.fetch_snapshot`, and neither arm has a `gateway` job to - age out either), `supported: false` for a 1.1.0/`--no-proxy agent, and `gateway_source.stub()` - for a configured host that has never answered — the payload's key set plus `unreachable: true` - and minus only the `agent_version` nobody published, every count `null` because an agent that has - not spoken has not reported an idle gateway (rule 2). Nothing in the payload carries a `severity` - or a `window_seconds`, at any depth (pinned by a recursive walk: a busy proxy is the job and no - Δ was ever measured). Phase 30 task 02 took `outbound[]`, `unmatched` and `by_port` out of it: the - flat list duplicated the graph with a worse count, and the residual existed only to explain a - guessing rule that no longer runs. Phase 32 task 03 added one top-level key, `resolver` — the - agent's own block, `{available, via, entries}`, each entry projected to the five fields the agent - publishes — so a dash on one line is answerable from the document the page already holds; `stub()` - grows it with nulls, which is what the stub/answer key-parity pin compares. `/history` - serves the `gateway.in_conns@` / - `gateway.out_conns@:` gauges with `hosts=` defaulting to the ingress **alone** (the - name the samples are stored under — defaulting to every target would draw empty plots for - machines nobody asked to route), and it carries no configured gate because it dials nothing: - like `/api/disks/history`, an unconfigured source answers no hosts and no series. The pair adds - no write (the verb enumeration pins the two GETs and that the app's one upstream POST is still - `/api/ups/command`), and `tests/test_auth_api.py` lists both paths as session-required. - All five `history` routes reach the store through one door, `api._history_batch()` → - `store.history_many()` (phase 14 task 02). That is a cost change and nothing else: the series - list is host-major and metric-minor in request order, each entry the same - `{source, host, metric, range_seconds, points}` document the per-series loop built, with the - same empty-series answer for a name that has no rows. `tests/test_api.py` pins it for all five - routes against points read straight out of the store with `query()` — a third path, in neither - the loop nor the batch — so a future "optimisation" that moves a point, an order, or a key - fails a test instead of redrawing a chart wrong. - `/api/summary` is the overview's single call and the cheapest whole-fleet probe there is: - the page renders only `alerts` plus `system.fleet`/`system.hosts`, while `disks.hosts`, - `llm.engines`, `ups.ups`, the `states` maps and `recent_events` stay for programmatic - consumers (live tests, an external check, a notifier). `system`/`llm`/`ups`/`recent_events` - are unchanged and `alerts` (phase 02) and `disks` (phase 09) are additive — the `disks` - block is `fleet_disk_summary` verbatim plus one row per host (`disks`/`with_smart`/ - `health_failed`/`alarm_disks`/`smartctl_available`/`failed_devices`), so a notifier learns - *which* drive the hardware called `FAILED` without a second call, and - `tests/test_api.py` pins every one of those key paths, with and without targets configured, - so retiring a panel can never shrink the payload. Since phase 22 those rows are the **visible** - hosts — the same `_disk_visible` predicate `/api/disks/snapshot` applies, over the same - entries, before the fleet sum — so `disks.fleet` equals `/api/disks/snapshot`'s `fleet` and a - host with no smartmontools is missing from both, while `system.hosts` still lists it (a missing - binary is a fact about the drive page and nothing else) and `disks.states` still lists its job. - A configured - job with no payload yet is listed as a stub (`unreachable: true`, every value `null`) in - `system.hosts`/`llm.engines` exactly as `/api/system/snapshot` does, so `hosts_total` - counts the machines there are and `hosts_live` the ones answering. - The **`gateway` block** (phase 29 task 02) is additive in the same way: `{configured, host, href, - supported, unreachable, listeners, connections, live_in, live_out, map_targets, warnings, states}` - — twelve keys, `warnings` added by task 03. The middle six were renamed by phase 30 task 02 from - `inbound`/`outbound`/`unmatched`/`unmatched_listeners`/`partial`, because the first pair now counts - a graph and not a flat pipe list and the residual no longer exists; `tests/test_api.py` pins the - set, so a thirteenth field is a decision and not an accident. `listeners` and `connections` are **row counts** ("how many trunks / routes does - this gateway declare", which is what the overview's one line prints) and `live_in`/`live_out` are - the two socket-derived totals; Task 02 wrote it for programmatic consumers (the page itself reads - `/api/gateway/snapshot`, task 03); task 05 gave it one page-side reader, the single line `/` - renders under the fleet table, so `href` joined the block — derived from `alerts.HREF`, the one - table that writes a dashboard's route, so the overview's link and an alert about the same source - cannot spell the page differently (`test_the_gateway_block_links_to_the_page_the_alert_row_links_to`, - and the page half reads only block keys: `test_the_overview_line_reads_the_gateway_block_it_claims` - resolves the served `GATEWAY_FIELDS` against a live document). `live_in` sums the listeners and - `live_out` the **distinct outbound endpoints** — never the connection lines — because several - `(listen, endpoint)` lines carry one socket count repeated with `shared: true`, and summing the - lines would book the same sockets once per line and print a router busier than it is (the same - rule that keys the outbound gauges by `(address, port)`, with the address normalised so two names - — or two spellings — resolving to one endpoint book it **once**); `api._endpoint_values()` is the one - reader of that (it delegates to `gateway_source.distinct_endpoint_counts()`, the same union-find the - join builds its counters with, so the block's total and the pipes cannot disagree), and a line with - no `metric` (a name nothing resolved) is absent rather than an unmeasured endpoint holding the total - hostage. `map_targets` counts the - stream map's routing entries. Every figure is `null` — never `0` — in the arms that did not - measure (unconfigured, configured-but-never-answered, a 1.1.0/`--no-proxy agent, a silent socket - table), and a total is `null` rather than a part of itself when any countable endpoint is unknown - (`page-gateway.js`'s `sumLive()` mirrors it for the one tally the page paints — the page has no - outbound total of its own, so only the inbound half is summed there). Task 30.03 added the - twelfth field, `warnings` — the **count** of `sources/gateway.warnings()` rows, which is what the - page's `warnings` head tally and the overview's ` · N to address` suffix both read (the overview - prints it only above zero; a permanent `0 to address` is the ink the empty-state rules exist to - avoid). Nothing in the - block carries a `severity`. `/api/config` grew the matching row (`enabled`, - `host`, `url`, `interval_seconds`, `timeout_seconds`), where `url` is the named entry's - `Target.display_url` and `null` when no target carries the name — invariant 6's boundary applies - to a name-only source exactly as it does to a URL-bearing one, because this body is what a browser - renders. -- `app.py` — assembly, and the **session gate** (`session_gate` middleware, invariant 12) whose - `PUBLIC_PATHS`/`PUBLIC_PREFIXES` are the whole anonymous surface. It answers in this order and - the order is the design: no `AuthState` → `503` for everything; no session and no such route → - the router's `404` (a typo is not a sign-in problem, and "sign in required" would otherwise be - the only answer anyone outside ever gets for a wrong URL); public → through, except that the - origin check still applies to a public write (a cross-site login post spends the throttle of - whoever is typing) and `GET /login` for someone already signed in goes to the dashboard, not - back to the form; no session → `303` to `/login?next=…` for a page and the JSON refusal for - everything else; a write → origin, then this session's CSRF token when it arrived on the - cookie. `_own_origin` reads `X-Forwarded-Proto` for the same reason the cookie's `Secure` flag - does, so behind nginx a real same-origin write is not a cross-site refusal — the bug that only - appears in production, which is why it has a test. `create_app()` takes injectable settings/store so tests build an app - with an in-memory DB and no targets. The lifespan builds `ctx.auth` - (`auth.load_auth_state`) after the context exists and **before** the first poll, for the - owned-store and injected-store branches alike — one call site, because two copies drift — and - a `RuntimeError` from it leaves the process non-zero with the operator's fix in the output. - That failure is never caught here: a dashboard that cannot say who its admin is does not start - the loops either (`__main__.py` passes `SETTINGS` to `create_app()` explicitly so the served - app is configured by the same object the bind address came from). `build_certs(store)` is the - same rule for phase 15: one helper, called **before** `build_hub` in the owned-store - (`build_context`) and injected-store (lifespan) branches, whose `CertMaterial` (the CA + client - paths in a private temp dir) hangs off `AppContext.certs` and whose - directory the lifespan's `finally` removes. What hands that material to the loops is - `agent_tls_map(material, store, targets)` → `build_hub(..., tls=...)`, called in **both** - branches the same way: no material (or no targets) means no TLS map, which is the plain - `http://`/token path rather than an error, and one map covers all of a host's jobs — including the - `gateway` job, which is why one revoke in `/certs` silences the pipes with no restart. A corrupt stored certificate raises out of the same - place — a dashboard that cannot present its own client certificate would otherwise spend the - fleet's TLS history reporting an upstream problem it caused itself. - `build_hub` also builds **two** `RateEstimator`s, because a window is only a claim about the - polls that fill it (phase 23, `TODO.md:8`'s "ramp up/ramp down"): the `host` jobs get - `RateEstimator(span=settings.rate_window)` (10 s — what a 10 s cadence can fill) and the `llm` - jobs `RateEstimator(span=settings.llm_interval)` (2 s, one poll interval, so a slot that starts - reads steady on the next poll instead of climbing one fifth of the true rate per poll), and the - `gateway` job (phase 29) is handed **neither** — it is added to the hub between the `disks` and - `llm` jobs, only when `settings.gateway_enabled`, because a gauge needs no window and one job - cannot be right about two cadences' worth of them., - there is no third knob — the LLM window **is** `LLM_INTERVAL_SECONDS`, which is the only way - the pair cannot drift. One estimator for both could only be right about one of them: a 2 s span - over a 10 s poll sits the window-empty fallback against the `2 × span` ceiling and answers - `n/a` every poll, and a 10 s span over a 2 s poll smooths the work the operator is watching. - The estimator's own gates (the `MIN_WINDOW_SECONDS` floor, the ceiling, the 8× jump - confirmation with its independent anchor) are unchanged and apply to the smaller window, whose - `Δ2.0s` label the payload carries with the value. -- `web/` — six dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`, `/gateway`) plus the - login page (`/login`, `pages/login.html` + `static/page-login.js`), one stylesheet, `charts.js`. - No framework. The tab nav is those six and nothing else, in that order, on **all eight** - signed-in pages (phase 29 task 05): `tests/test_api.py::test_the_disks_tab_is_in_the_masthead_of_every_page` - compares the nav's hrefs to `TAB_ORDER` (which **is** `PAGES`) and asserts the `aria-current` - count per page — one, naming itself, on the six dashboards and zero on `/settings` and `/certs`. - `/gateway`'s own tab arrived with that task; until then the page existed behind a route with no - way to click to it. `/login` renders the masthead's **brand link and theme toggle** and nothing else - from it: no tabs (every tab bounces straight back here, and a nav of links that all redirect is - a lie about what is available), no status dots and no clock (there is nothing signed in for them - to say). It posts through `Dash.fetchJSON` like every other script, which is only safe because - `bounceToLogin()` does nothing when the browser is already on `LOGIN_PATH` — without that guard - a `401` would reload the form instead of printing `Wrong password.`, and the page that exists to - end a bounce loop would be one. So the form branches on `err.status`: `401` is the inline - sentence plus `field.select()` (fixing a typo without retyping is the whole UX here), `429` - shows the server's `detail` and **counts down its `Retry-After`** with the button disabled until - the wait is over — `fetchJSON` therefore carries `response.headers` on the `Error` it throws, - because a wait this page made up would be an invented number (rule 2) — and anything else prints - the server's `detail` verbatim. Every refusal goes out through `textContent`, never - `innerHTML`: this is the one page a stranger reaches first. Its own copy of `auth.safe_next` - guards the one redirect followed on client strength (already-signed-in `?next=`), and - `Dash.ensureAuth()` up front stops a flash of the form for a browser that is already in. The - login page is audited for orphan `id`s and CDN references too (`tests/test_auth_api.py`), fetched - unsigned because a signed-in `GET /login` is a `303` and would audit `/`'s markup under the wrong - name. `/settings` (`pages/settings.html` + `static/page-settings.js`) is a **page and not a - seventh dashboard**: same `` and masthead, the six tabs with **no** `aria-current` (it is - not one of them), the same five `#nav-status` badges as every dashboard (phase 25 task 01 — - `TODO.md:3`'s "hosts disks engines UPS disappear when in the certs/account page" was the - missing container, and `Dash.initChrome()` fills it from `/api/health` with no JS change), and - it is reached from the `⚙ Account` link in - `.masthead-right` of every dashboard **including itself** — outside `.tabs`, so the pinned tab - structure is untouched (`tests/test_api.py::test_every_page_links_to_both_admin_pages`). It prints - `/api/auth/status` (name, expiry as a local timestamp *and* as time left), signs out through - `POST /api/auth/logout`, and changes the password through `POST /api/auth/password` after - checking locally what it can judge itself (non-empty current, ≥ 8 new characters — - `tests/test_api.py` pins that number against `api.MIN_PASSWORD_LENGTH` so the two cannot drift — - new matching confirm), printing `403`/`422`/`429` through `textContent` as the server words them, - then re-reads the status so the panel cannot show a session the rotation ended. Its static - sub-line is the precedence rule (env seed → stored verifier wins → reset by deleting - `auth.verifier`), because that sentence is what stops the "I edited the env file and nothing - happened" call. So `app.js` exports **`resetAuth()`**: it clears the memoised `authStatus` and its - in-flight promise, and must be called after a password change (the secret rotated, so the cached - `csrf` is now invalid — every later write, the UPS buttons included, 403s until a reload) and on - any `401` bounce. - `/certs` (`pages/certs.html` + `static/page-certs.js`) is the second of those admin pages, and it - is the same shape for the same reasons: six tabs with **no** `aria-current`, the five fleet - badges in `#nav-status` like every other masthead (phase 25 task 01, same reasoning as - `/settings` above), and both masthead links — `⚷ Certs` beside `⚙ Account` — exactly as the six - dashboards render them. **Both links are on both admin pages** (phase 20 task 04, `TODO.md:4`), - which retires phase 15's "no self-link" precedent: chrome that changes shape depending on where you already are reads - as a broken button, while a link to the page you are on is a no-op that answers with the page you - were looking at (`data/logs/20-task04-admin-links.cjs` measures exactly that — one document - response, no redirect, the same cluster afterwards). The two links therefore sit on all eight - signed-in pages (the six dashboards plus `/settings` and `/certs`) in one order - (`⚷ Certs` → `⚙ Account` → theme toggle → clock — phase 25 task 03 moved the clock to the end so - it is the farthest-right item on a `flex-end` row), which is what lets - the pinned audit be one loop over `MARKUP_PAGES` instead of two over different tuples; `/login` - is the ninth page and the one with neither. Everything on it is - `GET /api/certs`: the Root CA block, the display-only Dashboard client line (that certificate is - minted with the root and has no route of its own), and one row per host with its status chip, - serial, expiry (local timestamp *and* `Dash.duration` left), issued-vs-deployed `bundle_rev`, - `san` list, and the three controls. Two rules it exists to keep. (1) **No colour the API did not - send**: `/api/certs` publishes no `severity` — an expiry is a date and a revision count is not a - graded number — so the page has no `severity` token at all, the one chip class map is - `status` word → `ok|fail` with `unissued` mapping to *nothing* (a certificate that does not exist - never borrows green), and a behind bundle is a neutral `stale` chip. (2) **No control the API - would refuse**: a row with `configured: false` prints why (`not in HOST_TARGETS`) instead of - three buttons that all answer `404`, a never-issued host has no **Revoke** and no **bundle** link - (that route answers `issued: false`), and the download is a plain `GET` link because a session - cookie is all a signed `GET` needs. Writes are `confirm()`d, not typed-`DANGER`d — that gate is - for cutting power and this page cannot — and the outcome line prints the route's own `hint`, so - `restart_required`'s restart sentence lives in `api.py` and nowhere else. Its `COLUMNS` table - names every payload key it indexes, and `tests/test_api.py` resolves that table against a live - `GET /api/certs` row in both directions (the `SLOT_FIELDS` device), so a rename on either side - fails a test instead of printing `n/a` beside a healthy fleet. The two CNs are markup, and pinned - to `certs.ROOT_CN`/`certs.CLIENT_CN` the way `/settings` pins its password floor. The agents - table is `table.data.scroll` — the class-level scroll mechanism (`TODO.md:9`: at phone width the - serial / bundle / `san` / controls tail was clipped by `.card { overflow: hidden }` with no way - to reach it), and the class on the served table and the served `table.data.scroll` rule are - pinned **together** in `tests/test_api.py`, because the mechanism exists only while both name - it (measured at 360 px and 1280 px by `data/logs/21-task03-scroll.cjs`). Since phase 27 task 01 - it is no longer the only thing that scrolls — at ≤640px every `table.data` does — and `.scroll` - out-ranks that rule (two classes to one), so this table keeps its both-axis box and its 320 px - height cap on a phone exactly as on a desktop (both halves measured at both widths by - `data/logs/27-task01-scroll.cjs`). - `app.js`'s masthead status dots are one per dashboard job — - including `disks`, because the same agent can answer the 10-second utilisation poll and - have a wedged SMART pass, and one dot for both would be right about neither, and including - `gateway` (phase 29 task 05), which is one dot rather than one per host because the ingress is - one job and a dot for a source that stays `idle` when `GATEWAY_HOST` names nothing is how an - unconfigured source reads grey rather than red. The badge table is a literal in `initChrome()`, - so a new dashboard's job means a new entry there — `NAV_STATUS_BADGES` in `tests/test_api.py` - is the pin that says so. `app.js` owns the - colour vocabulary: `Dash.severityFrom(cellOrSeverity)` turns a payload verdict into an - `ok|warn|fail` class (`''` for anything unrated, so unknown never looks green) and - `Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a - `.hero` big number (plus an optional `.bar`) from that class — it never compares a number - to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for - values the API does not rate, and **no page script calls it any more** — `/` was converted - in phase 02, `/system` in phase 03, `/llm` in phase 04, `/ups` in phase 05 and `/disks` - shipped that way from its first commit, so every colour on those five pages comes from a - `severity` verdict the API sent - (`tests/test_api.py` fails if a page script names it again). The sixth dashboard colours nothing - by verdict at all — `/gateway`'s only colour pair is existence (`--gate-on`/`--gate-off`), and it - carries no `severity` token anywhere. A value the API should rate - gets a row in `severity.py`, never a bound in JS. - **The masthead is three rows at every width, the two control rows flush right** (phase 20 task - 01 made the rows, `TODO.md:5-6`; phase 25 task 02 re-justified them, `TODO.md:4` — "I made a - mistake earlier, I did want hosts disks engines UPS badges and certs/account/dark right - justified"): row 1 the brand, `#nav-toggle` and the six tabs; row 2 the `#nav-status` - per-source dots; row 3 - the `.masthead-right` controls (`⚷ Certs`, `⚙ Account`, theme, then the clock — phase 25 task 03 - moved the clock last for `TODO.md:5`'s "the time should show up as the farthest right item"; - phase 21 task 01 deleted the `⏸ Pause` that used to sit beside the links, button, `Space` shortcut and - `setPaused`/`_paused` plumbing alike: it froze the numbers on screen without growing a `stale` - label, and a backgrounded tab already idles its polls, which is the only stop this app needs). - That was CSS and - no markup change, which decides the mechanism: `#nav-status` is a **child** of - `.masthead-right`, so there is no wrapper to promote, and the rows are line breaks — `.masthead` - is `flex-wrap: wrap`, and both `#nav-status` and `.masthead-right` carry `flex-basis: 100%` with - `justify-content: flex-end` (line breaking uses a flex item's **base** size, before shrink, so - a 100 % item is alone on its line: the div's break starts row 2 of the bar and the dots' starts - row 2 inside it, leaving the controls to wrap to row 3). `.masthead-right` lost the - `margin-inline-start: auto` that pinned the cluster to the right edge of the one-row bar. - `flex-direction: column` was the other option the task allowed and is the wrong one: it puts the - brand and the tabs on separate rows. `height: var(--header-h)` became **`min-height`** (three rows - do not fit in 58 px and a fixed height clips the controls) and is declared **once** — the ≤640px - block no longer restates `height`/`min-height`/`flex-wrap`, and that pair drifting is exactly what - would clip the bar on a phone only — except that **`#nav-status` wraps at ≤640px since phase 29 - task 05**: five badges are 352 px of chips against this bar's 320 px content box, and a `nowrap` - item that may not shrink does not get smaller, it widens the *document* (measured 372 px — the - sideways scroll phase 18 forbids). The fifth chip takes a second line of the same right-aligned - row and the bar grows by one line box; no label shrinks and nothing is hidden. Phase 18's - `nowrap` was right for four (267 px) and the pin now demands the opposite, with - `data/logs/29-task05-tabs.cjs` measuring 360 px documents on all eight pages. - Every page that renders the bar now renders three rows — - `/certs` and `/settings` grew the dots row in phase 25 task 01, where `#nav-status` is pinned as - the **first child** of `.masthead-right` on all eight signed-in pages (`app.js`'s *badge list* is - a literal, and phase 25's fix was markup because the container was what was missing; phase 29 - task 05 added the fifth entry — `{ key: 'gateway', label: 'gateway', href: '/gateway' }` — because - the ingress has a job of its own and a job with no badge is a source that blinks in silence) — - and `/login` remains the one page with no dots, rendering two rows - (brand, then the theme toggle). Pinned on the served bytes by phase 20's three tests in - `tests/test_api.py` — the row declarations (task 02 renamed that one to - `test_the_masthead_is_three_rows_right_justified_at_every_width`), the mobile block's silence - about the bar's height, and the bar's `position` — and measured in a browser at 1280 px and 360 px by - `data/logs/20-task01-rows.cjs` (seven pages × two widths plus the dropdown; screenshots - `data/logs/masthead-rows*.png`). **Phase 25 task 02 moved the alignment of those two rows to - the other edge** (`TODO.md:4` — "I made a mistake earlier, I did want hosts disks engines UPS - badges and certs/account/dark right justified"): the two `justify-content` declarations are - `flex-end`, `margin-inline-start` is still absent from the cluster (a full-basis row aligned by - `flex-end` is the mechanism, not the old single-row right-pinning), and the ≤640px block still - restates neither the alignment nor the bar's height, so the phone's wrapped control row follows - the base rule. The pin is `tests/test_api.py::test_the_masthead_is_three_rows_right_justified_at_every_width` - (the row-break assertions are the same ones phase 20 wrote, deliberately: the re-justify must - not cost the rows), and the measured half is `data/logs/25-task02-justify.cjs` — seven pages × - two widths, each demanding both rows compute `flex-end` and sit flush with the bar's right - padding edge, the four badges present (a four-badge claim — `29-task05-tabs.cjs` is the pin that - the fifth, `gateway`, joined them), three rows intact, the dropdown still under the whole - bar and no sideways scroll, plus `/login`'s two-row bar with no badges - (`data/logs/25-task02-justify.out`, screenshots `data/logs/25-task02-*.png`). **Phase 25 task 03 - then put the clock at the end of the control row** (`TODO.md:5` — "The time should show up as the - farthest right item on the navbar"): the `` moved from the first - control to the last one on all seven templates then (eight since `/gateway` joined the nav), so - the row reads `⚷ Certs` → `⚙ Account` → theme - toggle → clock. It is a **markup** move and deliberately not a CSS `order:` declaration, because - on a `flex-end` row the last child *is* the farthest-right item and an `order:` override would be - something the served-byte pin cannot see — `test_every_page_links_to_both_admin_pages` therefore - asserts `html.index('id="clock"') > html.index('id="theme-toggle"')` on every page (the two - Certs/Account-before-theme assertions stay, and the `/login` half of the test is untouched). - The clock keeps its long-standing `@media (max-width: 900px) { .clock { display: none } }` hide: - it is a readout and not a control, so this is not the "no control is hidden to make the width - work" rule the masthead keeps. Measured by `data/logs/25-task03-verify.cjs` — seven pages × two - widths, each demanding three rows, both right-hand rows flush with the bar's right padding edge - and the chips' content (not the full-basis row box) measured to prove it is not flush left, the - clock the rightmost painted thing on the bar at 1280 px, the cluster's source order unchanged at - 360 px where the clock is hidden, and the four badges populated on `/certs` and `/settings` as on - the dashboards; plus the 360 px dropdown and `/login` unsigned at both widths - (`data/logs/25-task03-verify.out`, screenshots `data/logs/masthead-25-*.png`). - **The bar is not sticky** (phase 20 task 02, `TODO.md:10` — "It shouldn't be sticky at all. We - want to maximize the amount of screen space for data viewing"): `position: sticky; top: 0` became - **`position: relative`** with no offset, so the whole masthead scrolls away with the page and the - phone's "sticky only part way down" quirk — the fixed-height sticky bar fighting the wrapped - rows — went with it. `relative` rather than the default `static` because `.masthead` is still the - positioned ancestor the ≤640px dropdown's `top: 100%` resolves against (a `static` bar moves the - menu to the top of the document), and `z-index: 40` stays so the menu's `41` still means one - above the bar; the consequence is that the open menu scrolls with the bar, which is what an - unpinned bar means. The only `position: sticky` left in the sheet is a table's own `thead`, inside - its own scroll container, and this task is not it. Proven by `data/logs/20-task02-unsticky.cjs` - (seven pages scrolled at 1280 px and 360 px — with a squeezed-height leg for the pages that fit - a tall viewport — plus the 360 px dropdown still landing directly under the bar and still hanging - off it after a scroll; screenshots `data/logs/masthead-unsticky*.png`). Note that a browser cannot - tell `relative` from `relative; top: 0` — Chrome computes `top: 0px` for either — so the absence - of the offset is a served-CSS pin, not a measurement. - **The tab nav collapses into a hamburger at ≤640px** (phase 18): the eight tabbed pages (the six - dashboards plus `/settings` and `/certs`; `/gateway` joined in phase 29 task 05) each - carry `#nav-toggle` (`class="ghost nav-toggle"`, `type="button"`, `aria-controls="nav-tabs"`, - `aria-label="Toggle navigation"`, `☰`) between `.brand` and `