services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 13_llm_page_and_decode_rate

Browse Source 
All green. Final verification summary:

**Verified / implemented (phase 13 final pass)**
- All 4 tasks already in `complete/`; this was the verification pass. Confirmed the CSS rhythm rules (`#strip`, `#banners:empty ~ #fleet`) are served and pinned, `data/logs/NOTES-decode-spike.md` names the mechanism, and the phase-07/04 surfaces still pass their pins.
- **Found and fixed a residual defect in this phase's own code**: a stalled fetch stamps its body with the poll's *finish* time, so that sample anchors every window for the rest of the span; the jump gate "confirmed" a jump measured from that same anchor → live gufo rows 232.28 tok/s (one request at ~50) + a decaying 146/114/95 tail, repeated at 200.6. `rates.py::_publish` now requires an **independent anchor** (`_pending` stores the base sample's ts; the hold continues until it ages out). Pinned by `test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor` + `test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine` (both fail on task 03's gate).
- Re-verified live after a dev-server restart: 10 min / 540 signed-in `/api/llm/snapshot` polls + read-only SQLite pass; **0 values > 300, 0 without a `Δ<window>` label**; 0 post-fix stored rows over the ceiling except the two pre-fix-shape rows above.

**Tests / lint (exact, all 100 % green)**
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1195 passed, 11 skipped, coverage 98.70 %**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; repo hygiene clean.
- `uv run pytest -m live --live` → **11 passed / 0 failed**. `/api/health` → **all nine sources ok**.

**Completion criteria**
- Fleet/strip + head/fleet gap with screenshots (`llm-fleet-spacing{,-top,-banner-case}.png`) — met (CSS unchanged this pass, pinned on served bytes).
- Task-02 findings file names the confirmed mechanism — met.
- `test_rates.py`/`test_llm_source.py` pin label-set, both window bounds and the always-honest `Δ` — met (+2 new pins).
- Sustained watch under the recorded ceiling (300 tok/s) — met (worst 203.05, all labelled).
- Full suite + validate green; no phase-07/04 behaviour change — met.

**Documentation**
- README: `RATE_WINDOW_SECONDS` row now states the hold/confirm rule (only change needed; rest verified consistent).
- AGENTS.md: `rates.py` bullet + gap 9 rewritten (residual closed *and* the remaining sub-8× skew measured, root-caused to gap 4, explicitly not fixed by inventing a new threshold).
- **Fleet facts changed under this pass and were corrected**: operator added driveripper's bearer token → `host:driveripper`/`disks:driveripper` `ok`, live suite 9/2→11/0; but driveripper reads **identity only, zero SMART numbers** (root `smartctl` gives 55 °C/14 %; `/dev/nvme0`, `/dev/ng0n1`, `/dev/bsg/*` are root-only; its unit is `DevicePolicy=auto`, not the installer's `strict`). Evidence appended to `data/logs/NOTES-agent-install.md` + `NOTES-decode-stable.md`.

**Notable decisions / deviations**
- Did **not** add a span-relative window floor: that would re-decide task 03's pinned `MIN_WINDOW_SECONDS` and its "in-bounds diluted row publishes" test. Recorded as the next phase's call, with the root cause left at gap 4 / `14_store_performance` (the ~10 s stall).
- No live fleet change made (read-only probes + one dev-server restart). Fixtures deliberately not re-captured (no parse change, task 02 proved no shape drift). Dependency-currency advisory left untouched (no dependency moved in this phase): certifi/fastapi/starlette/websockets have newer releases.

**Next pending phase:** `.agents/phases/todo/14_store_performance` (then `15_agent_mtls`).
This commit is contained in:
ducoterra
2026-10-19 14:53:58 -04:00
parent 04d934dd94
commit 1bbd868401
36 changed files with 1231 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+154 -24
View File
@@ -56,7 +56,10 @@ page still renders.
as `0`. Every derived value carries the `source` string it came from and `derived: true`
so the UI can print *how* it was obtained (`computed:Δtokens/Δt`, `Δ10.0s`). Aggregating
two different series into one rate is the failure mode this rule exists to catch — the
rate estimator is keyed by the *resolved* upstream series name for exactly that reason.
rate estimator is keyed by the *resolved* upstream series name **plus its label set**
for exactly that reason (phase 13: a label set joining a summed counter mid-window
contributes nothing until it has its own baseline, never its carried cumulative value),
and the estimator refuses a window too short or too stale to carry the claim.
3. **Unconfigured is not broken.** A source with no env config answers
`404 {"configured": false}` **without dialling**, and the page says "not configured".
A red panel for a server the operator never installed destroys the signal from the
@@ -253,8 +256,13 @@ Responsibilities, one line each:
counting it in `hosts_total` — the `/llm` fleet-block rule, so the page never does
arithmetic and a dead agent never shrinks a fleet total.
- `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`,
`VLLM_HISTOGRAMS`), measured Δ rates, prefill speed as
`mean(prompt tokens)/mean(prefill time)`, session list from gufo `/slots`.
`VLLM_HISTOGRAMS`), measured Δ rates **diffed per label set** (`rate_key()` keys the
estimator on the resolved name plus its labels; the cell is the Σ of the per-series
rates over the widest window one of them actually used — phase 13, so a joining
`{engine="1"}` series can never lend its carried counter to one interval's delta),
prefill speed as `mean(prompt tokens)/mean(prefill time)`, session list from gufo
`/slots`. A refused window (the estimator's floor) is still an answer: the cell keeps
its `computed:Δ<series>/Δt` source and its honest `Δ<window>` beside a `null` value.
- `sources/nut.py` — upsd client, `UpsReading`, `VAR_MAP`, `assess()` (explicit checks, not
a mystery score), `SELF_TEST_OUTCOMES` + `self_test_outcome()` (the `Last self-test` verdict
follows the *outcome* of the driver's `*.test.result` word, not its phrasing: pass words →
@@ -287,7 +295,23 @@ Responsibilities, one line each:
`from_env()` is the single place the `NUT_SIMULATE` truthy rule lives — `app.py` builds the
monitor through it, so the assembly and the model cannot disagree about what is enabled.
- `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own).
- `rates.py` — Δ/window estimator, keyed by resolved series name.
`scalar(agg="sum")` stays right for **gauges**; `values_by_series()` is the per-label-set
view **counter** deltas need, because a summed counter mixes whatever label sets exist now.
- `rates.py` — Δ/window estimator, keyed by resolved series name + label set. Its effective
window is `now - base[0]` **bounded on both sides** (phase 13): under
`MIN_WINDOW_SECONDS` (the poll loop's 0.2 s catch-up gap after any slow fetch) it answers
`(None, dt)` — one `n/a` with the window it refused — and drops that base so the polls
*after* the stall cannot re-diff it; the window-empty fallback accepts a base no older
than `MAX_WINDOW_RATIO × span` (the `2 × span` ceiling the docstring always claimed, now
enforced). A rate ≥ 8× the previously published one is held for one
poll (the previous value repeats, label unchanged) and publishes only if the next interval
confirms it (`JUMP_CONFIRM_RATIO`) — and **confirmation means a different anchor**: a held
jump remembers the sample its window was diffed from, and stays held while that same sample
still anchors the window. A stalled fetch stamps its body with the time the poll *finished*,
so that sample is the stall older than its own timestamp, and every later window off it is
the same skewed delta re-diffed over a longer span — measured twice from one sample is one
measurement, and the hold simply continues until the stale anchor ages out of the span.
`min_window=0.0` exists for tests that scrape faster than any real loop.
- `auth.py` — every authentication decision as a pure function, stdlib only (phase 11):
`hash_password`/`verify_password`/`needs_rehash` over a self-describing
`pbkdf2_sha256$<iterations>$<salt_hex>$<hash_hex>` verifier (so the cost can rise without a
@@ -530,15 +554,26 @@ That scheme is the history API's key — treat it as a public interface.
**Fleet transport, re-verified from the laptop on 2026-10-19.** The port-selective inter-VLAN
drop phase 10 recorded on the lab gateway `10.2.0.1` is **gone**: `gateway` (`10.1.1.27:9120`)
and `deskwork` (`10.1.0.51:9120`) both take a plain TCP connect and answer `/snapshot`, and
`driveripper`'s upsd answers `LIST UPS` on `10.1.0.126:3051` from this same laptop. Two hosts
run the **installed** service — `systemctl is-active dashboards-agent` is `active` on both,
with `/usr/local/lib/dashboards-agent/host_agent.py` and a `--token-file` unit bound to the
address the dashboard dials. `driveripper` is the one host with no agent at all (unit
`inactive`, no installed file, nothing listening on `:9120` → `ConnectionRefused`), and it
renders as an `unreachable` stub row: the designed degradation, recorded rather than hidden.
`/api/health` today reads `host:deskwork ok`, `host:gateway ok`, `disks:deskwork ok`,
`disks:gateway ok`, `nut ok`, `llm:gufo ok`, `llm:vllm ok` with the two `driveripper` jobs
`DOWN`, and the `live` suite is 8 passed / 3 failed with every failure that one transport.
`driveripper`'s upsd answers `LIST UPS` on `10.1.0.126:3051` from this same laptop. All three
hosts now run the **installed** service — `systemctl is-active dashboards-agent` is `active` on
each, with `/usr/local/lib/dashboards-agent/host_agent.py` and a `--token-file` unit bound to
the address the dashboard dials. **`driveripper` joined during phase 13 on 2026-10-19, in two
operator steps.** It went from *no agent at all* (unit `inactive`, nothing listening →
`ConnectionRefused`, an `unreachable` stub row) to **`active`** with
`/usr/local/lib/dashboards-agent/host_agent.py` installed at 13:27, and then its bearer token
went into `HOST_TARGETS`' URL userinfo — so `host:driveripper` and `disks:driveripper` are
**`ok`** and the `live` suite is **11 passed / 0 failed** (it was 9/2 on the missing token that
morning, and 8/3 on the ConnectError before that). The unit reads
`--host 10.1.0.126 --port 9120 --token-file /etc/dashboards-agent.token` (`640
root:dashboards-agent`, the mode `deploy/install-agent.sh` writes) with `User=dashboards-agent`,
and `getent group disk` → `disk:x:6:dashboards-agent`.
**Two things about that install are not what this repo writes, and both cost SMART data on
`/disks` (see the SMART block below): `systemctl show -p DevicePolicy dashboards-agent` answers
`auto` with no `DeviceAllow` lines** (the installer's `strict` + node allow-list is not what is
running there), **and its `smartctl` cannot read drives as that user.**
`/api/health` today reads **all nine sources `ok`**: `host:driveripper`, `host:deskwork`,
`host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `nut`, `llm:gufo`,
`llm:vllm`.
Phase 10's proof that the old drop was not this repo's firewall stands as history in
`data/logs/NOTES-verify.md` — do not re-add a tunnel on the strength of it, and re-run the
probe rather than trusting either note.
@@ -597,8 +632,13 @@ That scheme is the history API's key — treat it as a public interface.
`systemctl restart|reload nut-server` pair documented in `deploy/nut-sudoers`); `deskwork`
grants an unrelated set (`btrfs scrub`, `snapper`, `setfacl /srv/*`, `osbuild-auto.sh`).
Neither host grants `ops` the installer or `visudo` (re-checked 2026-10-19), so every install
that has happened on those two went through root by the operator — including the one still
outstanding, `driveripper`'s.
that has happened on those two went through root by the operator — `driveripper`'s included,
which is why its unit is not the one this repo writes (see the SMART block: `DevicePolicy=auto`,
no `DeviceAllow`). Worth knowing before the next install: the allow-list `install-agent.sh`
does write (`/dev/nvme*`, `/dev/sd*`, `/dev/vd*`, `/dev/sg*`) has **no `/dev/ng*n*` and no
`/dev/bsg/*`**, the nodes smartctl 7.5 reaches for NVMe admin and SATA pass-through, so a
`strict` unit there would deny exactly what a SMART read needs — whatever phase takes
driveripper's SMART on has to settle the node list and the grant together.
- **SMART: real captures, and one live host finally reads them.** `smartctl` 7.5 is installed on
deskwork and driveripper. The `ops` account (which is who runs a hand-started dev agent) is in
no `disk` group, driveripper grants it `(root) NOPASSWD: /usr/sbin/smartctl*` and deskwork
@@ -607,12 +647,25 @@ That scheme is the history API's key — treat it as a public interface.
What changed by 2026-10-19 is that `deskwork`'s **installed** service runs as
`dashboards-agent`, which is in the `disk` group (`getent group disk` → `disk:dashboards-agent`)
on an agent installed after the SMART pass: `/disks` shows real numbers there —
`smartctl_available: true`, `nvme0n1` healthy at 52 °C, 19 160 power-on hours (an odometer,
`smartctl_available: true`, `nvme0n1` healthy at 55 °C (amber, past `disk.temp_c`'s 50 °C
threshold), 19 166 power-on hours (an odometer,
so that figure is a capture date, not a fact about the drive), 11 % of rated
wear — so the wear bar and the alarm chips are drawn from live hardware, not only fixtures.
`gateway` answers `smartctl_available: false` (no smartmontools in its VM: the *install it*
half of the three-way flag), and `driveripper` answers nothing at all until its agent exists,
which is where `smartctl_available: null` — "nobody told us either way" — still comes from.
half of the three-way flag), and **`driveripper` is the fourth case the flag was built for:
`smartctl_available: true` with almost no numbers.** Its agent answers (16 whole-disk rows,
`nvme0n1`…`nvme7n1` + `sda`…`sdh`) and `smart` is non-null for the 8 NVMes, but every one of
them carries only identity — `health`, `temp_c`, `power_on_hours`, `percentage_used` and the
counters are all `null` — while the 8 spinning drives come back `smart: null` each with the
agent's own line `smart: sda: Try an additional '-d ata' or '-d sat' argument.` Measured
read-only on 2026-10-19: the same `smartctl -i -A -H /dev/nvme0n1` **as root** prints
`PASSED`, 55 °C and 14 % used, so the drives answer and the *agent user* is what is locked out
— `/dev/nvme0`, `/dev/ng0n1` and `/dev/bsg/*` (the nodes NVMe admin and SATA pass-through
actually go through) are `crw------- root root`, and only `/dev/nvme0n1` / `/dev/sg0` carry the
`disk` group the agent is in. So `/disks` shows driveripper's drive inventory with `n/a` values
and a why-line, which is the designed degradation, and **lighting it up is a new phase, not a
token**: a udev rule granting those nodes, and/or `host_agent.smart()` learning the
`-d sat` / `/dev/ngXnY` forms smartctl needs when auto-detection fails.
**Real data is in the repo now:** `tests/fixtures/smartctl_driveripper_<dev>.txt` is all 16
drives (8 NVMe, 8 spinning) read through that sudo rule, and `tests/test_agent.py` parses
every one — which is what found the parser's one real bug: this binary annotates the
@@ -622,13 +675,15 @@ That scheme is the history API's key — treat it as a public interface.
honest look at this fleet's drives: `nvme5n1` reports 13 media-and-data-integrity errors and
`nvme1n1` is at 58 % of its rated wear, so the alarm chip and the amber wear bar on `/disks`
are real features with real data behind them, not demo furniture.
**Lighting up the rest of it is the same two operator actions that travel together:** install
the agent on `driveripper`, and give its agent user device read access — the `disk` group
**Lighting up the rest of it was the same two operator actions that travel together** — install
the agent on `driveripper`, and give its agent user device read access: the `disk` group
(`sudo usermod -aG disk <agent-user>`, which is what `deploy/install-agent.sh` does on install),
*not* a sudo rule, because the agent must read the device itself — on an `agent/host_agent.py`
from phase 09 or later (never redeploy without the grant: 16 permission-denied lines would age
nothing but are pure noise, while with the grant they are 16 drives' worth of temperature,
hours and sector counters). `gateway` needs smartmontools first and will still read no SMART
hours and sector counters). **The agent and the `disk` group are now both in place on
`driveripper`, and it still reads no SMART numbers** — the group only covers nodes that carry
it (see above), so what is left is the node grant / `-d sat` work, not an install. `gateway` needs smartmontools first and will still read no SMART
after that, being a VM with no physical disk behind `vda`. A host that cannot read its drives
renders the designed degradation: one dimmed card per physical drive, every number `n/a`, and
the one-line why.
@@ -656,8 +711,8 @@ set -a; . ./.env.dev; set +a # dev dials the fleet the prod way, no
./scripts/dev-server.sh status # per-source ok/DOWN — unauthenticated: /api/health
./scripts/dev-server.sh logs
uv run pytest # hermetic: 1172 passed + 11 live-skipped, no network
uv run pytest --cov # coverage gate (98.6%; floor 94%)
uv run pytest # hermetic: 1195 passed + 11 live-skipped, no network
uv run pytest --cov # coverage gate (98.7%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh # re-record ground truth (agents + UPS + gufo /slots)
@@ -1036,7 +1091,16 @@ carries **no severity colour anywhere**: `fleet_totals` has no `severity` to ren
summed-KV bar is gone for that reason) and `severity.py` rates no throughput or queue depth,
so the numbers stay ink and only the dot speaks reachability. `fleet.seams` — the API's own
note on how the two engines count prompt tokens differently — is the prompt sum's tooltip
rather than another line of ink. The
rather than another line of ink. **The block, the strip and the panels keep the page's shared
`.85rem` gap** (phase 13, `TODO.md:4`: fleet and strip rendered flush into one wall):
`#strip { margin-top: .85rem }` is unconditional because below two engines `#fleet` is `[hidden]`
and the strip *is* the first visible section, and `#banners:empty ~ #fleet { margin-top: .85rem }`
gives the block its top gap only when the banner slot renders nothing, so a banner's own
`margin-bottom:.85rem` never stacks a second gap under it. `tests/test_api.py` pins both rules on
the served stylesheet, pins that **exactly one** rule styles each of the two ids (a second,
unguarded one is the double gap), and pins that `#banners`/`#fleet`/`#strip` are still direct
siblings of `<main>` — the `~` combinator resolves between siblings, so re-flowing that markup has
to rewrite the selectors instead of losing the gap. The
four trend charts stay but at height 110 *below* the panels, each with an `ariaLabel`. Panels
sit at their own height (`#engines` carries `.items-start`): a two-hero gufo card next to a
latency-row vLLM one otherwise grows a blank bottom, and a blank panel reads as "no data". An
@@ -1216,6 +1280,72 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
the operator's own machine is the threat model here, not a shared kiosk — and the reason a
lost laptop is a password change and not a sign-out. A real revocation list would be a row per
session with a prune rule, which is the thing this design traded away.
9. **The `/llm` `decode tok/s` spike is fixed, and verified live** (phase 13, 2026-10-19;
the diagnosis is in the git-ignored `data/logs/NOTES-decode-spike.md`, the live proof in
`data/logs/NOTES-decode-stable.md`). **Task 04's sustained watch passed the ceiling test:**
11 minutes / 285 signed-in `/api/llm/snapshot` polls against the live engines, worst value
**117.57 tok/s** (gufo) and **98.00** (vLLM) — both in the payload and in the stored raw
series, 2.6× under the 300 tok/s pass line, with **zero** values printed without their
`Δ<window>` label. The stall is still there (38 catch-up gaps < 1.5 s in 11 minutes, a
10.6 s gap between two samples) and the estimator now absorbs it: three refusals each
labelled with the sub-0.5 s window it refused (`Δ0.387s`, `Δ0.399s`, `Δ0.363s`), two ≥ 8×
steps held for one poll and published only when the next interval confirmed them, and — the
point of the whole fix — **1 m rollups topping out at 51.9 / 34.1 tok/s** where task 02 found
a 222.7 tok/s minute, so the trend chart's y-scale stops being set by arithmetic. Fixtures
were **not** re-captured: task 02 compared the live exposure with `tests/fixtures/` series
by series (gufo 15/15, vLLM 128/128, identical sets), task 13 changed no parsing, and the
page still reads `15 series · 0 unparsed` / `128 series · 0 unparsed`. Screenshots:
`data/logs/llm-decode-stable.png` (+ `-top`, the fleet-block gap and both `Δ` tags) and
`data/logs/llm-decode-burst.png` (a real two-slot burst: 62.1 tok/s `Δ8.5s`, fleet sum 102.4).
The confirmed mechanism was neither a counter bug nor a label-set sum: `poller.Job._loop`
sleeps `max(0.2, interval - elapsed)`, so any fetch slower than `LLM_INTERVAL_SECONDS`
is followed by a poll **0.2 s later**, and the estimator happily diffed a whole stall's
tokens over that gap — the 2026-10-18 09:16 gufo row read **3284.59 tok/s for an engine
doing 53**, with an inflated tail because the stale sample stayed `inside[0]`. Task 03
made that impossible by construction (see the `rates.py` bullet above): the window floor
turns the catch-up poll into one `n/a` **and drops the stale base** so the tail re-bases;
the fallback's `2 × span` ceiling is enforced (the honest 15.25 s diluted row still
publishes); jumps ≥ 8× hold one poll and publish only on confirmation; and
`values_by_series()` + `rate_key()` diff every counter
per label set, so the latent joining-series defect (candidate (a)) contributes 0 tokens
until it has its own baseline. The four task-02 `xfail(strict=True)` tests are green with
their markers removed and every assertion kept. Spike-free across two days of raw history
the fleet's real maximum is **119.9 tok/s** (gufo / deskwork) and **171.7**
(vLLM / driveripper) — anything above ~300 is arithmetic, not work, which was task 04's
pass line and what the watch above stayed under. The `Δ<window>` label stayed honest throughout —
both before and after the fix it names the window actually used; do not "fix" anything
here by hiding or rewriting it. The ~10–15 s process-wide **stall** that triggers a
refused poll is still gap 4's and `14_store_performance`'s problem; the estimator no longer
turns a slow fetch into 0.2 s arithmetic, and one `n/a` per stall is the designed degradation
(the residual the verification pass measured is the paragraph below).
**The verification pass found one residual path and closed it** (same day, live rows in the
store the fix was running): a stalled fetch stamps its body with the time the poll
*finished*, so the sample it stores is the stall older than its own timestamp. The floor
cannot see that (the window is honest, the value is old), the sample then anchors every
window for the rest of the span, and the jump gate published the second one because it was
measured from the same anchor — gufo 12:42:10 read **232.28 tok/s** with exactly one request
running at ~50, followed by the tell-tale decaying tail 146.4 / 114.1 / 94.8 and the true
47.5 (a 13:40:58 row repeats it at 200.6). `rates.py` now requires an **independent anchor**
for a confirmation (`_pending` stores the base sample's timestamp and the hold continues
while the window is still diffed from it), so a skewed sample is held until it ages out and
the honest rate publishes — the reconstruction of both rows is pinned by
`test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor` (estimator) and
`test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine` (through
`parse_gufo_metrics`), each of which fails on the pre-fix gate and prints the live shape.
**That closes the gate's path, not the skew itself, and the verification pass measured the
difference.** A slow reply's counter is true somewhere between the request and the reply and
nothing in the payload says where, so when the resulting jump is *under* 8× the last published
rate the gate never engages: the post-fix watch (10 min / 540 reads, `data/logs/
llm-decode-verify-pass.tsv`) caught gufo at **203.05 tok/s labelled `Δ2.595s`** where two busy
slots were really doing ~110, then 127.4 / 100.2 / 89.6 as the window grew back to the span and
35.6 once the freed slot settled — every value labelled, **0 of 540 above the 300 pass line**,
and no stored raw row over it since the restart either. A collapsed window is what makes a
burst *look* like a rate, so the fix for this family is the ~10 s stall itself (gap 4 /
`14_store_performance`), not a fresh threshold invented in the estimator: task 03 pinned
`MIN_WINDOW_SECONDS` as an absolute floor and pinned that an in-bounds diluted row publishes
(`test_a_fallback_inside_the_documented_ceiling_still_publishes`), so a span-relative floor
would be a re-decision of that phase, not a bug fix. Do not "fix" the residual by hiding the
short window or its label.
## Locked decisions (and why)