services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 23_llm_rate_responsiveness

Browse Source 
Phase 23 final verification complete — all green.

```
VERIFIED (criteria)
1  /api/llm/snapshot windows ≈2 s live (1.98–2.00), /api/system window_seconds ≈10 s (9.996–10.005);
   /api/config llm.rate_window_seconds=2=LLM_INTERVAL_SECONDS; browser re-check ALL CHECKS PASSED
   (data/logs/23-window-{llm,system}.png — /llm prints Δ2.0s, /system footnote Δ10.0s)
2  Phase-13 pins unchanged: floor, 2×span ceiling, 8× jump gate + independent anchor — all pass
   (only import/span-line diffs in tests/; git diff confirms no assertion rewritten)
3  Live proof re-derived from artifacts: 780 rows, 0 unlabelled, windows 0.61–3.38 s, 5 starts +
   5 ends settled by poll 2 (0 unsettled), 1 m rollups max 42.3 vs raw 201.6, 0 over 300 tok/s;
   counterfactual 10 s window: median settle poll 3, |ln| 0.924 vs 0.037
4  uv run pytest: 1494 passed / 15 skipped · coverage 98.94% (floor 94) · ruff clean · pyright 0
   errors · node --check clean · bash .agents/validate.sh exit 0 · live 11 passed · store_perf 4 passed
   Dev server up, all nine sources ok. No git add/commit run; everything left in the working tree.

DOCS PASS — 4 inconsistencies fixed in AGENTS.md (last two runs above include them)
- UI conventions hero example still said the `.sub` footnote reads `Δ10.0s`; the only hero that
  prints a Δ is /llm's → now `Δ2.0s` on /llm, "at the current load" on /ups
- gap 9: "found 10 busy-edges" contradicted its own artifact (17 edges, 10 judged, 7 reported-not-
  judged) → 17; and the poll-2 ratios were a 3-of-5 cherry-pick → full five values both windows
- gap 12: the responsive-table fix is now scheduled — names 27_mobile_tables_and_audit
README, config table, .env.example, rates/prom docstrings, app/api bullets and the testing-policy
pins all re-checked against code and found consistent (store pragmas, budgets, throttle, fleet).

NOTES
- Advisory only, not phase 23: fastapi 0.153.0 → 0.156.8 available; dependencies untouched here,
  so rule 8's "update on purpose" is left to the operator.
- NOTES.md is empty in the working tree (its items became phases 24–27); left as-is, untracked-file
  rules forbid me removing it.
- Next phase: 24_dev_mtls_rollout (25–27 follow).
```
This commit is contained in:
ducoterra
2026-10-21 08:04:21 -04:00
parent 895f309f8e
commit 5f3ed9bcd4
37 changed files with 1694 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+27 -5
View File
@@ -478,9 +478,10 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
| ------------------------------------- | -------------------------------------------------------------- |
| `HOST_TARGETS` | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`). An `https://` URL is dialled as a *protected* agent instead: verified against the dashboard's own root CA and presented with its client certificate, with a revoked host refused before the dial (`certs.py`, `collectors.AgentTls`) |
| `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
| `RATE_WINDOW_SECONDS` | the Δ window behind every measured rate — tokens/sec, bytes/sec, jiffies→CPU % (default 10); each value carries the window it actually got, and `/system`'s footnote prints that, not this default. A window the poll loop cannot honour — the ~0.2 s catch-up after a fetch slower than the interval, or a base older than 2× the window — answers one `n/a` instead of a spike, and a rate 8× above the last one is held until a window measured from a **different** sample confirms it (phase 13) |
| `RATE_WINDOW_SECONDS` | the **host's** Δ window (default 10) — bytes/sec and jiffies→CPU %, measured over what a 10 s poll can fill; each value carries the window it actually got, and `/system`'s footnote prints that, not this default. A window the poll loop cannot honour — the ~0.2 s catch-up after a fetch slower than the interval, or a base older than 2× the window — answers one `n/a` instead of a spike, and a rate 8× above the last one is held until a window measured from a **different** sample confirms it (phase 13). The LLM rates do **not** use it: they are measured over `LLM_INTERVAL_SECONDS` (phase 23) |
| `LLM_INTERVAL_SECONDS` | the `/llm` poll cadence (default 2 s) **and** the Δ window behind every token rate — one estimator per measuring cadence, so a slot that starts reads steady within two polls instead of ramping over five. Slowing the poll slows the window with it; there is no separate LLM window knob, and a window narrower than the interval would answer `n/a` every poll |
| `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) |
| `HOST_INTERVAL_SECONDS` / `LLM_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often each source is polled (10 / 2 / 5 s) |
| `HOST_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often the host agents and upsd are polled (10 / 5 s) |
| `DISKS_INTERVAL_SECONDS` | how often the disk-health source re-reads each agent for SMART (default 300 s — SMART is not a per-second signal). No targets of its own: the `disks:<host>` jobs exist exactly when `HOST_TARGETS` does |
| `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others |
| `DISKS_TIMEOUT_SECONDS` | the disks fetch's own budget (default 30 s), because the agent runs `smartctl` inside its snapshot handler and a wedged SMART pass must not read as a dead host |
@@ -674,7 +675,7 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
the way out and the same on the way back in, with nothing hidden from a keyboard or a screen
reader while it is shut, and no motion at all when the system asks for none
(`prefers-reduced-motion`). No framework, no CDN, no bundler, and no narration: a page never
explains its own pipeline, so the words on screen are the ones you act on — a unit, a `Δ10.0s` provenance tag, a
explains its own pipeline, so the words on screen are the ones you act on — a unit, a `Δ<window>` provenance tag, a
`SIMULATED` badge, or the sentence beside an empty panel. The overview (`/`) is exactly two panels — a
**Warnings & errors** list built from `/api/summary`'s `alerts` (each row links to the
dashboard that explains it) and the **Fleet** table — and nothing else: an empty alert
@@ -751,8 +752,8 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
in the section stack — sits the **stats strip**: one card per configured
engine — reachability dot, name, age stamp, the **KV-cache bar** (the one rated cell: the
fill and a breach's text take their colour from the payload's `severity` and nothing else),
then `prompt tok/s` and `decode tok/s` with their provenance tag (`Δ10.0s`, scraped,
estimated), `total tokens`, the sessions this engine reports (`slots busy` counted from
then `prompt tok/s` and `decode tok/s` with their provenance tag (`Δ2.0s` at the default cadence —
the LLM window is one poll interval, phase 23 — scraped, estimated), `total tokens`, the sessions this engine reports (`slots busy` counted from
gufo's `/slots`, or `running`/`waiting` for vLLM, plus `waiting`), `prefix cache hit`,
`spec accept` (gufo), `queue time` with its `p95 …` foot (vLLM) — and red alarm chips for
preemptions / GPU-context losses that exist only while non-zero. Then the engine panels
@@ -811,6 +812,15 @@ uv run python scripts/store_perf_probe.py # the same measurements as a repo
DASH_WATCH_SECONDS=1500 uv run python scripts/watch-store-perf.py
# ... and the live watch: 25 signed-in minutes against
# the running dev server, counting store-error and WAL
DASH_LLM_WATCH_SECONDS=780 uv run python scripts/watch-llm-window.py
# ... and the LLM Δ-window watch (phase 23): signed-in
# reads of /api/llm/snapshot at the app's own cadence,
# judging every slot start/end against "settled within
# two polls", the Δ<window> labels and the 1 m rollups.
# Read-only: it starts no workload and writes nothing.
DASH_LLM_WATCH_REPLAY=data/logs/llm-window-watch.tsv \
uv run python scripts/watch-llm-window.py # ... or re-derive that verdict from a captured pass,
# over the same samples, with no traffic sent
uv run ruff check src tests agent && uv run pyright # lint + types
./scripts/capture-fixtures.sh # re-record ground truth: host agents, UPS replies, gufo /slots
./scripts/capture-fixtures.sh --with-llm # ... and the engines' /metrics while they are up
@@ -934,6 +944,18 @@ quietly becoming a fallback chain. A busy
slot's `prompt` is the user's in-flight request text, so the capture script blanks it and
the page never reads it.
`tests/rate_window_contract.py` is the same device for one number rather than a field table: the Δ
window each source measures its rates over — `RATE_WINDOW_SECONDS` for the host agents,
`LLM_INTERVAL_SECONDS` for the engines, one estimator per measuring cadence (phase 23). It holds
the two spans, the `Δ<window>` rounding the pages print, and a range assertion — the estimator's
own floor up to its `2 × span` ceiling, plus the printed label — shared by four files: the
`build_hub` wiring spy in
`test_api.py`, both throughput cells of both engines in `test_llm_source.py`, the host payload's
`window_seconds` in `test_host_source.py`, and the estimator's floor / `2 × span` ceiling / 8×
jump gate run at both spans in `test_rates.py`. The two payloads are the same shape and only the
number differs, so a pin naming one source's window is exactly how the other could quietly keep
advertising a window no poll ever measured.
## Battery-test buttons
Instant commands are the only endpoint in the application that changes anything **outside this