services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 23_llm_rate_responsiveness

Browse Source 
Phase 23 final verification complete — all green.

```
VERIFIED (criteria)
1  /api/llm/snapshot windows ≈2 s live (1.98–2.00), /api/system window_seconds ≈10 s (9.996–10.005);
   /api/config llm.rate_window_seconds=2=LLM_INTERVAL_SECONDS; browser re-check ALL CHECKS PASSED
   (data/logs/23-window-{llm,system}.png — /llm prints Δ2.0s, /system footnote Δ10.0s)
2  Phase-13 pins unchanged: floor, 2×span ceiling, 8× jump gate + independent anchor — all pass
   (only import/span-line diffs in tests/; git diff confirms no assertion rewritten)
3  Live proof re-derived from artifacts: 780 rows, 0 unlabelled, windows 0.61–3.38 s, 5 starts +
   5 ends settled by poll 2 (0 unsettled), 1 m rollups max 42.3 vs raw 201.6, 0 over 300 tok/s;
   counterfactual 10 s window: median settle poll 3, |ln| 0.924 vs 0.037
4  uv run pytest: 1494 passed / 15 skipped · coverage 98.94% (floor 94) · ruff clean · pyright 0
   errors · node --check clean · bash .agents/validate.sh exit 0 · live 11 passed · store_perf 4 passed
   Dev server up, all nine sources ok. No git add/commit run; everything left in the working tree.

DOCS PASS — 4 inconsistencies fixed in AGENTS.md (last two runs above include them)
- UI conventions hero example still said the `.sub` footnote reads `Δ10.0s`; the only hero that
  prints a Δ is /llm's → now `Δ2.0s` on /llm, "at the current load" on /ups
- gap 9: "found 10 busy-edges" contradicted its own artifact (17 edges, 10 judged, 7 reported-not-
  judged) → 17; and the poll-2 ratios were a 3-of-5 cherry-pick → full five values both windows
- gap 12: the responsive-table fix is now scheduled — names 27_mobile_tables_and_audit
README, config table, .env.example, rates/prom docstrings, app/api bullets and the testing-policy
pins all re-checked against code and found consistent (store pragmas, budgets, throttle, fleet).

NOTES
- Advisory only, not phase 23: fastapi 0.153.0 → 0.156.8 available; dependencies untouched here,
  so rule 8's "update on purpose" is left to the operator.
- NOTES.md is empty in the working tree (its items became phases 24–27); left as-is, untracked-file
  rules forbid me removing it.
- Next phase: 24_dev_mtls_rollout (25–27 follow).
```
This commit is contained in:
ducoterra
2026-10-21 08:04:21 -04:00
parent 895f309f8e
commit 5f3ed9bcd4
37 changed files with 1694 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+104 -8
View File
@@ -56,7 +56,8 @@ page still renders.
`DANGER` requirement and the wire send still behave exactly as they did without them.
2. **Never invent a number.** A value the upstream does not publish renders as `n/a`, never
as `0`. Every derived value carries the `source` string it came from and `derived: true`
so the UI can print *how* it was obtained (`computed:Δtokens/Δt`, `Δ10.0s`). Aggregating
so the UI can print *how* it was obtained (`computed:Δtokens/Δt` labelled `Δ2.0s`, the LLM's
own poll interval since phase 23). Aggregating
two different series into one rate is the failure mode this rule exists to catch — the
rate estimator is keyed by the *resolved* upstream series name **plus its label set**
for exactly that reason (phase 13: a label set joining a summed counter mid-window
@@ -374,7 +375,13 @@ Responsibilities, one line each:
so that sample is the stall older than its own timestamp, and every later window off it is
the same skewed delta re-diffed over a longer span — measured twice from one sample is one
measurement, and the hold simply continues until the stale anchor ages out of the span.
`min_window=0.0` exists for tests that scrape faster than any real loop.
`min_window=0.0` exists for tests that scrape faster than any real loop. **The span belongs to
the cadence that fills it** (phase 23): `app.build_hub` builds one estimator per measuring
cadence — the host's at `RATE_WINDOW_SECONDS` (10), the LLM's at `LLM_INTERVAL_SECONDS` (2) —
because a window wider than the poll averages bursts across the polls the operator is watching
and a window narrower than it has nothing inside it to diff. Nothing else about the estimator
moved: the floor, the ceiling, the jump gate and the label are the same code, and a 2 s window
is 4× above the floor, so a healthy 2 s cadence is never refused.
- `auth.py` — every authentication decision as a pure function, stdlib only (phase 11):
`hash_password`/`verify_password`/`needs_rehash` over a self-describing
`pbkdf2_sha256$<iterations>$<salt_hex>$<hash_hex>` verifier (so the cost can rise without a
@@ -690,6 +697,18 @@ Responsibilities, one line each:
`http://`/token path rather than an error, and one map covers both of a host's jobs. A corrupt stored certificate raises out of the same
place — a dashboard that cannot present its own client certificate would otherwise spend the
fleet's TLS history reporting an upstream problem it caused itself.
`build_hub` also builds **two** `RateEstimator`s, because a window is only a claim about the
polls that fill it (phase 23, `TODO.md:8`'s "ramp up/ramp down"): the `host` jobs get
`RateEstimator(span=settings.rate_window)` (10 s — what a 10 s cadence can fill) and the `llm`
jobs `RateEstimator(span=settings.llm_interval)` (2 s, one poll interval, so a slot that starts
reads steady on the next poll instead of climbing one fifth of the true rate per poll), and
there is no third knob — the LLM window **is** `LLM_INTERVAL_SECONDS`, which is the only way
the pair cannot drift. One estimator for both could only be right about one of them: a 2 s span
over a 10 s poll sits the window-empty fallback against the `2 × span` ceiling and answers
`n/a` every poll, and a 10 s span over a 2 s poll smooths the work the operator is watching.
The estimator's own gates (the `MIN_WINDOW_SECONDS` floor, the ceiling, the 8× jump
confirmation with its independent anchor) are unchanged and apply to the smaller window, whose
`Δ2.0s` label the payload carries with the value.
- `web/` — five dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`) plus the login page
(`/login`, `pages/login.html` + `static/page-login.js`), one stylesheet, `charts.js`. No
framework. `/login` renders the masthead's **brand link and theme toggle** and nothing else
@@ -1110,7 +1129,7 @@ set -a; . ./.env.dev; set +a # dev dials the fleet the prod way, no
./scripts/dev-server.sh status # per-source ok/DOWN — unauthenticated: /api/health
./scripts/dev-server.sh logs
uv run pytest # hermetic: 1477 passed + 15 skipped (11 live,
uv run pytest # hermetic: 1494 passed + 15 skipped (11 live,
# 4 store_perf), no network
uv run pytest --cov # coverage gate (98.94%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
@@ -1123,6 +1142,14 @@ uv run python scripts/watch-store-perf.py
# running dev server, timing the page paths while counting
# `store-error` rows and the WAL size (DASH_WATCH_SECONDS
# shortens it; writes store-perf-watch.{log,tsv,json})
uv run python scripts/watch-llm-window.py
# ... and the phase-23 proof: signed-in reads of
# /api/llm/snapshot at the app's own cadence, judging every
# slot start/end against "settled within two polls", the
# Δ<window> labels and the 1 m rollups — read-only, it
# starts no workload (DASH_LLM_WATCH_SECONDS shortens it;
# DASH_LLM_WATCH_REPLAY=<tsv> re-derives the verdict from a
# captured pass over the same samples, sending nothing)
uv run python scripts/store_perf_probe.py # ... as a report (data/logs/store-perf-baseline.{md,json},
# then store-perf-task03.* for the write side and
# store-perf-task04.* for the budget table);
@@ -1216,6 +1243,22 @@ branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only
needs **no** markup for a new verdict: `<li class="${check.status}">` plus
`ul.checks li.<verdict> .mark` in `dash.css` is the whole render path, and no page asset or
stylesheet rule names one check (a second opinion about the driver's word is the bug).
- **A measurement window is pinned from both sources, through one contract module** (phase 23).
`tests/rate_window_contract.py` names the two Δ windows (`HOST_WINDOW` = `RATE_WINDOW_SECONDS`,
`LLM_WINDOW` = `LLM_INTERVAL_SECONDS`), the `delta_label()` rounding the pages use to print
`Δ2.0s` / `Δ10.0s`, and `assert_window()` — the estimator's own bounds (`MIN_WINDOW_SECONDS` up
to `2 × span`, because a span equal to the cadence means the base comes from the window-empty
fallback and a real window is the poll gap: 2.009-2.026 s live on `/llm`) **plus** the printed
label, so raising a cadence knob moves the window and the pin together instead of rotting it.
Four files read it: `test_api.py` spies the estimator object each source's jobs receive and pins that
`/api/config`'s LLM window is the LLM's own span; `test_llm_source.py` walks both throughput
cells of **both** engines at 2 s and re-runs the phase-13 floor refusal through the production
parse path at that span; `test_host_source.py` keeps the host payload's `window_seconds` at 10 s
and shows the same 10 s cadence starved to `n/a` by a 2 s span (why one estimator cannot serve
both); `test_rates.py` runs the floor, the `2 × span` ceiling and the 8× jump gate at both
spans, because those are properties of the estimator and not of the window size. A pin naming
one source's window is what this prevents — the two payloads are the same shape and only the
number differs, so half the pin is how a quiet regression would look.
- **Page payloads that a rewrite must not break are pinned as a table.**
`tests/card_contract.py` holds the dotted `/api/system/snapshot` paths the `/system` hero
cards read (`CARD_METRICS`, `GPU_METRICS`, `HOST_SHAPES`); `tests/test_system_contract.py`
@@ -1303,6 +1346,13 @@ branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only
faked, which is how a `0640` mode, a byte-identical bundle and the https `--status` probe are proven
without a root shell or a host. **A `0600` in a task file is not a fact about this repo**: the agent
opens its own bundle and re-reads it for the denylist, so the group bit is load-bearing.
- **A live measurement is re-derivable, so a verdict can be re-checked without re-watching**
(phase 23). `scripts/watch-llm-window.py` writes the series it judged (value, the `Δ<window>`
the payload carried, the engine's own busy count, the raw token counter) beside its verdict, and
`DASH_LLM_WATCH_REPLAY=<tsv>` re-computes that verdict from the file with no traffic sent. The
pass line is read on the same samples re-diffed over the estimator's own span *and* on the
published value, because the phase-13 jump gate can legitimately hold a correct number for a
poll — judging only the published column would call a designed hold a ramp.
- **A cost claim is pinned as a shape and a control, never as a wall clock** (phase 14). The default
run keeps the seeded-store tests hermetic and asserts **shape and determinism** — counts,
retention windows, the statements a plan is taken from, idempotence — while the timings live behind
@@ -1385,9 +1435,9 @@ component vocabulary the pages are still being converted to (`.hero-bar`, the gr
variants) stays even while only one page uses it.
A big number is `Dash.hero()` on `.hero` / `.hero-bar` (`dash.css`): 2.4rem tabular numerals,
muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the payload's
`severity` through the existing `.ok` / `.warn` / `.fail` classes — there is no `.sev-*`
family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS
muted unit, mono `.sub` footnote (`Δ2.0s` on `/llm`'s rates since phase 23, "at the current load"
on `/ups`), coloured by the payload's `severity` through the existing `.ok` / `.warn` / `.fail`
classes — there is no `.sev-*` family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS
rule reaches for a token that `:root` does not declare.
**The `/disks` card is one panel per *physical* drive, led by identity and verdict, not by a
@@ -1570,7 +1620,9 @@ column; this was a promotion, not a de-duplication).
A `Δ<window>` label is a measurement, never a config value: `sources/host.py` sends the window
the estimator actually used as the payload's `window_seconds`, and `/system`'s footnote prints
those (a range when the hosts disagree) instead of the poll interval, which only coincides with
them at the defaults. `/llm`'s hero `.sub` prints the rate cell's own `window` through the same
them at the defaults (since phase 23 the `/llm` rates are measured over one poll interval, so
their tag reads `Δ2.0s` while `/system`'s stays `Δ10.0s` — each page naming the window its own
estimator used). `/llm`'s hero `.sub` prints the rate cell's own `window` through the same
`provenance()` tag, and the hero's `title` carries the exact `source` string — "78 tok/s" must
always say whether it was scraped, measured `Δcounter/Δt`, or estimated (`llm_monitor`'s
`setThroughput` idiom; dropping the tag to look cleaner is a regression, not a simplification).
@@ -1930,6 +1982,49 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
(`test_a_fallback_inside_the_documented_ceiling_still_publishes`), so a span-relative floor
would be a re-decision of that phase, not a bug fix. Do not "fix" the residual by hiding the
short window or its label.
**Phase 23 changed the size of the window, not the gates** (`TODO.md:8`: "a strange delay and a
'ramp up/ramp down' effect"). What was left after phases 13/14 was the span itself: one
10 s-window estimator served both cadences, so on the 2 s `llm` poll a slot's first seconds
were spread over five polls — the capture replay in
`tests/test_llm_source.py::test_gufo_the_window_the_llm_jobs_are_spanned_by_is_one_that_fills`
prints 12.5 → 20 → 30 tok/s for an engine steady at 50, and the same five polls backwards when
the slot ends. `build_hub` now builds **one estimator per measuring cadence** (the `llm` jobs
spanned by `LLM_INTERVAL_SECONDS`, the host jobs by `RATE_WINDOW_SECONDS`), which reads 50 on
the first poll after the change and 0 on the first after it ends. Every phase-13 protection
still applies to the smaller window — `Δ2.0s` is 4× above `MIN_WINDOW_SECONDS`, the ceiling is
relative to the span, and the 8× jump gate is the estimator's own — so this is the smoothing
dial turned down, not a second definition of a rate.
**The live half has landed** (task 03, 2026-10-21, read-only: nothing was written and no
workload was started — the busy periods were the operator's own requests). A 13-minute
signed-in pass of `/api/llm/snapshot` at the app's own 2 s cadence
(`scripts/watch-llm-window.py`; series `data/logs/llm-window-watch.tsv`, verdict
`data/logs/llm-window-watch.{json,log}`) recorded **zero** values printed without their
`Δ<window>` label across 780 rows, with every window in 0.61-3.38 s and never 10 s. It found
17 busy-edges on gufo and judged the 5 starts + 5 ends that held long enough (an edge followed
by the next request within a poll is not a slot ending, and prefill polls genuinely decode
~0 — both are reported, neither is judged): **every** one was inside the 8× pass line at poll 2
on the ungated Δ. Re-diffing the *same samples* over the span this phase replaced is the
comparison the operator's complaint was about: median |ln(value/steady)| at poll 2 was **0.037**
with this window against **0.924** with the 10 s one, whose median edge needed **poll 3** to
settle (the five starts read 0.98 / 1.04 / 0.95 / 1.00 / 1.27 of steady at poll 2 on the
ungated 2 s Δ where those same five read 0.25 / 0.34 / 0.96 / 0.40 / 0.40 on the 10 s one; ends
took polls 4-5 to fall away against poll 2 now).
One start had its first correct value held for a poll by the phase-13 jump gate and published
on the next — designed behaviour, and the reason the pass line is read on the ungated re-diff
as well as on the published column. The stored **1 m rollups** for the window topped out at
**42.3 tok/s** (raw max 201.6 — that row's own `Δ0.610s` label says so, gap 9's sub-8× skew,
far under the 300 pass line). A second pass over the same engines an hour later
(`data/logs/llm-window-watch-pass2.log`) reproduced it: 3 starts + 2 ends, 0 unsettled, 0
unlabelled, 1 m rollups max 24.7 tok/s.
The browser half (`data/logs/23-live-window.cjs`, transcript `data/logs/23-live-window.out`,
screenshots `data/logs/23-live-llm-{opening,burst-settled,midburst,post-free,gufo-idle,vllm-idle,final}.png`)
checked the same claims as rendered: **866** decode figures, **0** without a tag in either
component that prints the number (the panel hero's `.sub` and the strip card's `.prov`), every
tag a Δ inside the estimator's own bounds — `MIN_WINDOW_SECONDS` to `2 × span`, the modal one
`Δ2.0s` on both the page and the API (833 of 866) and never the host's `Δ10s`. Its own edge
checks (isolated only, for the same reason) read inside the 8× line at poll 2 on two starts and
exactly `0.0 tok/s` at poll 2 on two ends — the frame named post-free is only taken once the
engine is provably idle, because the busy signal flickers between back-to-back requests.
10. **`driveripper`'s 16 drives read `n/a`: kernel 7.2's unprivileged-passthrough gate
(2026-10-19, resolved the same day; the installer template that shipped the broken device
filter was cleared in phase 17).**
@@ -2090,7 +2185,8 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
the table tail is still out of reach. The fix is a responsive-table pass (`.scroll` on the wide
tables, or a per-host card layout at ≤640px) and it touches pinned page shape on `/`, so it is
a phase rather than a cleanup — `21_page_cleanups`' `.scroll` on the `/certs` table is the same
shape applied to one table and does not reach this one. Of the two operator TODO items that sat
shape applied to one table and does not reach this one; `27_mobile_tables_and_audit` is that
pass, still in `todo/`. Of the two operator TODO items that sat
beside it, `20_masthead_nav_polish` has landed in full (now complete): the bar is three
left-justified rows at every width, it is no longer sticky, the phone menu slides, and both
`⚙ Account` and `⚷ Certs` render on all seven signed-in pages including the two they name