phase: 23_llm_rate_responsiveness
Phase 23 final verification complete — all green.
```
VERIFIED (criteria)
1 /api/llm/snapshot windows ≈2 s live (1.98–2.00), /api/system window_seconds ≈10 s (9.996–10.005);
/api/config llm.rate_window_seconds=2=LLM_INTERVAL_SECONDS; browser re-check ALL CHECKS PASSED
(data/logs/23-window-{llm,system}.png — /llm prints Δ2.0s, /system footnote Δ10.0s)
2 Phase-13 pins unchanged: floor, 2×span ceiling, 8× jump gate + independent anchor — all pass
(only import/span-line diffs in tests/; git diff confirms no assertion rewritten)
3 Live proof re-derived from artifacts: 780 rows, 0 unlabelled, windows 0.61–3.38 s, 5 starts +
5 ends settled by poll 2 (0 unsettled), 1 m rollups max 42.3 vs raw 201.6, 0 over 300 tok/s;
counterfactual 10 s window: median settle poll 3, |ln| 0.924 vs 0.037
4 uv run pytest: 1494 passed / 15 skipped · coverage 98.94% (floor 94) · ruff clean · pyright 0
errors · node --check clean · bash .agents/validate.sh exit 0 · live 11 passed · store_perf 4 passed
Dev server up, all nine sources ok. No git add/commit run; everything left in the working tree.
DOCS PASS — 4 inconsistencies fixed in AGENTS.md (last two runs above include them)
- UI conventions hero example still said the `.sub` footnote reads `Δ10.0s`; the only hero that
prints a Δ is /llm's → now `Δ2.0s` on /llm, "at the current load" on /ups
- gap 9: "found 10 busy-edges" contradicted its own artifact (17 edges, 10 judged, 7 reported-not-
judged) → 17; and the poll-2 ratios were a 3-of-5 cherry-pick → full five values both windows
- gap 12: the responsive-table fix is now scheduled — names 27_mobile_tables_and_audit
README, config table, .env.example, rates/prom docstrings, app/api bullets and the testing-policy
pins all re-checked against code and found consistent (store pragmas, budgets, throttle, fleet).
NOTES
- Advisory only, not phase 23: fastapi 0.153.0 → 0.156.8 available; dependencies untouched here,
so rule 8's "update on purpose" is left to the operator.
- NOTES.md is empty in the working tree (its items became phases 24–27); left as-is, untracked-file
rules forbid me removing it.
- Next phase: 24_dev_mtls_rollout (25–27 follow).
```
This commit is contained in:
@@ -0,0 +1,34 @@
|
||||
# Phase 23 — LLM rate responsiveness
|
||||
|
||||
**Source:** `TODO.md` L8 — "The LLM monitor has a strange delay and a 'ramp up/ramp down' effect that makes it look innacurate. Data reporting should be accurate to as small a delta as possible."
|
||||
**Story:** n/a
|
||||
**Context:** `src/dashboards/app.py` line 122 (`build_hub` builds **one** `RateEstimator(span=settings.rate_window)` and hands it to both `host_jobs` (line 124) and `llm_jobs` (line 130)), `src/dashboards/config.py` lines 197–199 (`llm_interval` 2 s, `llm_timeout` 3 s, `rate_window` 10 s), `src/dashboards/collectors.py` line 477 (`llm_jobs(settings, client, rates)` — the estimator is a parameter, so a second one is a wiring change, not a signature redesign), `src/dashboards/rates.py` (line 55 `MIN_WINDOW_SECONDS = 0.5`, line 61 `MAX_WINDOW_RATIO = 2.0`, the phase-13 jump gate — all untouched by this phase), `src/dashboards/web/static/page-llm.js` (line 39 `pollMs: 2000`, the `Δ<window>` labels printed from the payload's `window`), `src/dashboards/sources/llm.py` (the rate cells carry `computed:Δ<series>/Δt` + the honest `Δ<window>`; the window the estimator *used* travels with the value — rule 2), the mechanism the TODO describes (gap 9 / phase 13 in AGENTS.md: a 10 s window over a 2 s poll ramped the hero up over ~5 polls when a slot started and decayed over ~5 polls when it ended — the "ramp up/ramp down" the operator sees; the sub-8× skew family and the stall are **already** closed by phases 13/14 and stay closed: this phase changes the window size, nothing else)
|
||||
|
||||
## Objective
|
||||
|
||||
The LLM throughput rates are measured over a window as small as the app's own cadence allows — the LLM source gets its **own** `RateEstimator` whose span is the LLM poll interval (2 s), so a slot start reads steady within two polls instead of ramping over ten, while the `host` source keeps the 10 s `RATE_WINDOW_SECONDS` window its 10 s cadence needs (a 2 s window over a 10 s poll would refuse every window — the estimator's floor and ceiling would answer `n/a` forever). The phase-13 protections (the 0.5 s floor, the `2 × span` ceiling, the 8× jump gate with its independent-anchor confirmation) apply to the smaller window unchanged; the `Δ<window>` label prints the smaller window it actually used, so the number always says what it is.
|
||||
|
||||
## Dependencies
|
||||
- — (none) — the phase touches `app.py`/`collectors.py` wiring and the LLM page; it is scheduled after `22_disks_hide_unconfigured` only to keep the fleet-changing phase 24 last.
|
||||
|
||||
## Tasks
|
||||
1. `01_llm_window_per_poll.md` — the second estimator: `build_hub` hands `llm_jobs` a `RateEstimator(span=settings.llm_interval)`, the host keeps the 10 s one
|
||||
2. `02_window_pins.md` — the pins: LLM rate cells label the 2 s window, host cells keep 10 s, the phase-13 gates still hold over the smaller window
|
||||
3. `03_live_verification.md` — the live proof: a slot start reads steady within two polls (no 10 s ramp), the `Δ2.0s` label travels with the value, screenshots
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: `tests/test_llm.py` (or wherever the `parse_gufo_metrics`/`parse_vllm_metrics` tests live — grep `rate_window`/`RateEstimator` in `tests/` first) — the window-value pins for both sources, the phase-13 tests (floor refusal, ceiling, jump gate, the stale-anchor tests) run **unchanged** and green; `tests/test_api.py`'s `/api/llm/snapshot` shape pins stay green (the shape is unchanged — only the `window` values move)
|
||||
- Coverage: **>90%** on new/modified code
|
||||
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
|
||||
|
||||
## Documentation & Consistency
|
||||
|
||||
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**. AGENTS.md's `app.py`/`poller.py` bullets (the "one `RateEstimator`" sentence) and `rates.py`'s docstring context gain the two-estimator fact; gap 9's "the estimator no longer turns a slow fetch into 0.2 s arithmetic" history stays — this phase is the window-size decision on top of it, and the `RATE_WINDOW_SECONDS=10` line in `.env.example` stays (it is the host's knob now, and gains a comment saying so if it has none).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `/api/llm/snapshot` rate cells carry a `window` ≈ 2 s (the label prints `Δ2.0s` on the page) and `/api/system/snapshot` rate cells keep ≈ 10 s (pinned both ways)
|
||||
- [ ] the phase-13 tests (floor, ceiling, jump gate, stale anchor) pass unchanged
|
||||
- [ ] live: a slot start reads steady within two polls and a slot end decays within two — the 10 s ramp is gone (screenshots + the stored 1 m rollup in `data/logs/`)
|
||||
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
|
||||
- [ ] `bash .agents/validate.sh` green
|
||||
- [ ] no behavior change in completed work: the host rates, the estimator's gates, the rollups and the `Δ<window>` labelling are all unchanged in kind
|
||||
@@ -0,0 +1,28 @@
|
||||
# Task 01 — The LLM source gets its own 2 s-window estimator
|
||||
|
||||
**Phase:** `23_llm_rate_responsiveness` · **Source:** `TODO.md:8` — "The LLM monitor has a strange delay and a 'ramp up/ramp down' effect that makes it look innacurate. Data reporting should be accurate to as small a delta as possible."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
`build_hub` builds **two** estimators: the host's keeps `RateEstimator(span=settings.rate_window)` (10 s — its 10 s cadence needs the wide window; a 2 s span over a 10 s poll would sit the window-empty fallback against the `2 × span` ceiling and answer `n/a` every poll), and the LLM's is `RateEstimator(span=settings.llm_interval)` (2 s — one poll interval: the smallest window the 2 s cadence can honestly fill). `llm_jobs` (which already takes the estimator as a parameter, `collectors.py` line 477) receives the LLM one; `HostCollector` keeps the 10 s one. No new env var: the LLM window **tracks** `LLM_INTERVAL_SECONDS`, so the one existing knob stays the single source of truth for the LLM cadence-and-window pair.
|
||||
|
||||
- ASSUMPTION: **no new env var** — `RATE_WINDOW_SECONDS` (10, `.env.example` line 85) becomes the host's knob and the LLM window is `llm_interval` by construction. A 2 s window is 4× above the 0.5 s `MIN_WINDOW_SECONDS` floor, so the floor never engages on a healthy cadence; the 8× jump gate stays engaged (a real burst from idle still holds one poll for confirmation), and the poll cadence itself is **unchanged** — gufo and vLLM see exactly the same request rate as before (the TODO asks for a smaller *delta*, not more load on the engines).
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/app.py` — `build_hub` (lines 122–130): `rates` becomes `host_rates = RateEstimator(span=settings.rate_window)`; new `llm_rates = RateEstimator(span=settings.llm_interval)`; `host_jobs(..., HostCollector(host_rates), ...)` unchanged in shape; `llm_jobs(settings, client, llm_rates)` takes the new one. The two estimators are keyed on disjoint series names (rule 2's key is the resolved name + label set), so one shared object was never load-bearing — but the docstring/comment at line 122 (if it says "one estimator for all sources") is rewritten to the two-estimator shape.
|
||||
2. `src/dashboards/collectors.py` — line 477: no signature change expected (the parameter is already there); verify the LLM jobs thread `rates` through to `parse_gufo_metrics`/`parse_vllm_metrics` (they already take it, `sources/llm.py` lines 189/243) — if any LLM path constructs its own estimator, it takes the injected one (one estimator per source, the phase's invariant).
|
||||
3. `src/dashboards/config.py` — no change expected (the knobs exist); if a comment on `rate_window` (line 199) implies it is the one global knob, it gains the "host's knob; the LLM window is `llm_interval`" clause.
|
||||
4. `tests/` — the `build_hub`-level test (grep `build_hub` in `tests/` — the `tests/test_api.py` wiring pins) gains one assertion pair: the estimator the LLM jobs receive has `span == settings.llm_interval` and the host collector's has `span == settings.rate_window` (spy the two objects, the way the `agent_tls_map` pin spies its entry).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: the wiring pin above; the existing LLM parse tests (fixture-driven, `tests/fixtures/`) run with the injected estimator — their assertions on the `window` value, if any hard-code 10.0, are updated to the new 2.0 (grep `10.0`/`window` in `tests/test_llm.py` first); the host parse tests keep 10.0
|
||||
- Coverage: **>90%** on this task's new/modified code
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `build_hub` hands the LLM jobs a 2 s-span estimator and the host a 10 s-span one (pinned by the spy)
|
||||
- [ ] the fixture-driven LLM tests pass with the 2 s window; the host tests keep 10 s
|
||||
- [ ] `uv run pytest` green, `ruff` + `pyright` clean
|
||||
- [ ] full test suite green
|
||||
- [ ] no behavior change in completed work: the host's measured rates, the estimator's gates and the engines' poll cadence are untouched
|
||||
@@ -0,0 +1,27 @@
|
||||
# Task 02 — The pins: 2 s LLM window, 10 s host window, gates intact
|
||||
|
||||
**Phase:** `23_llm_rate_responsiveness` · **Source:** `TODO.md:8` — "…Data reporting should be accurate to as small a delta as possible."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The window decision is pinned on the surface that shows it, in both directions (the repo's testing-policy rule): a LLM rate cell answers its `window` ≈ the LLM interval with the `Δ<window>` label traveling with the value, a host rate cell keeps ≈ 10 s, and the phase-13 gate tests — the 0.5 s floor refusal, the `2 × span` ceiling, the 8× jump hold with its independent-anchor confirmation, and the stale-scrape-body regression — run **unchanged** and green, because they are estimator properties, not window-size properties.
|
||||
|
||||
## Work
|
||||
|
||||
1. `tests/test_llm.py` (or the file the LLM parse tests live in — grep first) — the two window pins: (a) a gufo metrics scrape driven through `parse_gufo_metrics` with a `RateEstimator(span=2.0)` (the `build_hub` shape, task 01's) answers the rate cell with a `window` in `(0.5, 2.0]` on a healthy 2 s cadence and the cell's `source` string still reads `computed:Δ<series>/Δt`; (b) the same test shape with `span=10.0` on the host's series (or the existing host fixture tests) keeps its `window` ≈ 10 s — the two sources' windows are **different numbers in the same payload shape**, and a pin that asserts only one of them is the gap this task closes.
|
||||
2. `tests/test_rates.py` (or wherever the phase-13 tests live) — run them unchanged: the floor test (a sub-0.5 s window answers `(None, dt)`), the ceiling test (`2 × span`), the jump-gate test (≥8× holds one poll, publishes on a *different-anchor* confirmation) and the two stale-body regression tests. If a test constructs its estimator with an explicit `span` that now disagrees with the app's shape, it is **not** edited to match — the estimator's constants are app-independent and the tests' own spans are theirs (the app's shape is task 01's pin's job).
|
||||
3. `tests/test_api.py` — the `/api/llm/snapshot` shape pins stay green (the shape is unchanged; only `window` values move). If a pin hard-codes a `window` number in the LLM payload, it is the one place it is updated — to a range assertion (`0.5 < window <= 2.0 × llm_interval`), not a literal, so a future cadence change does not rot it.
|
||||
4. `tests/test_config.py` — confirm `rate_window`'s pin (the 10 s default) stays; no new setting was added, so no new pin is needed (the "no new env var" ASSUMPTION is what the absence of a new setting is).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: the four pin groups above
|
||||
- Coverage: **>90%** on this task's new/modified code
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a LLM rate cell's `window` is pinned to the 2 s shape and its label prints `Δ2.0s` (or the honest sub-window when the floor engages)
|
||||
- [ ] a host rate cell's `window` is pinned to the 10 s shape
|
||||
- [ ] the phase-13 gate tests pass unchanged
|
||||
- [ ] `uv run pytest` green, `ruff` + `pyright` clean
|
||||
- [ ] full test suite green
|
||||
- [ ] no behavior change in completed work: the payload shapes, the `source` strings and the estimator constants are untouched
|
||||
@@ -0,0 +1,27 @@
|
||||
# Task 03 — Live verification: the ramp is gone
|
||||
|
||||
**Phase:** `23_llm_rate_responsiveness` · **Source:** `TODO.md:8` — "The LLM monitor has a strange delay and a 'ramp up/ramp down' effect that makes it look innacurate."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The smaller window is proven on the live engines, read-only: with the dev server up (invariant 10) and a decode workload running on the engines, the `/llm` hero reaches steady state within **two** polls of the slot starting (pre-phase it ramped over ~five) and decays within two of it ending, every value carrying its `Δ2.0s` label, and the stored 1 m rollups staying in the fleet's measured band (gap 9's pass line: anything above ~300 tok/s is arithmetic, not work). No engine is written to — the workload is whatever the operator's engines are doing, or a deliberately short local decode the operator starts; a live test that loads a shared engine is not a test.
|
||||
|
||||
## Work
|
||||
|
||||
1. `./scripts/dev-server.sh restart`, sign in with `testpass`; `./scripts/dev-server.sh status` — the nine-source table `ok` (a restart also proves the two-estimator wiring survives a real boot, not just the in-memory test store).
|
||||
2. Signed-in polling of `/api/llm/snapshot` at 2 s (a short script, the `scripts/watch-store-perf.py` pattern): capture the decode tok/s cell's `value` + `window` for ~60 s while a slot is busy, then again for ~30 s after it frees. The pass shape: within two polls of the transition the value is within 8× of steady (the jump gate's own ratio — a held value is the designed behaviour, and a *confirmation* within the next poll is the honest publish; a value still ramping at poll 5 is the pre-phase defect returning).
|
||||
3. The `Δ` label check (gap 9's rule, restated): **zero** values printed without their `Δ<window>` label; the label reads the window actually used (`Δ2.0s` on a healthy cadence, the honest sub-window — never a bare number — when the floor or the stall engages).
|
||||
4. The stored-rollup check (the phase-14 rule): the 1 m rollups of `llm.decode_tokens_per_second` for the window under test top out in the fleet's measured band (gap 9: 119.9 gufo / 171.7 vLLM were the real maxima; the 300 tok/s pass line stands) — a 1 m average set by a ramp would be the arithmetic this phase exists to remove.
|
||||
5. Screenshots of `/llm` mid-burst and post-free in `data/logs/` (invariant 10), plus the captured series as a TSV beside them (the `data/logs/llm-decode-*.tsv` precedent).
|
||||
|
||||
## Testing & Quality
|
||||
- Coverage: n/a (verification task; the pins are task 02's)
|
||||
- The read-only rule: no `POST` to the fleet, no `upsd` command, no engine write — observing a workload the engines are already doing is the test
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a slot start reads steady (within 8× of steady) within two polls, and a slot end decays within two — the 10 s ramp is gone (captured series in `data/logs/`)
|
||||
- [ ] zero values printed without their `Δ<window>` label, the label reading the window actually used
|
||||
- [ ] the 1 m rollups stay in the measured band (nothing above the 300 tok/s pass line)
|
||||
- [ ] screenshots in `data/logs/`
|
||||
- [ ] the full suite + `bash .agents/validate.sh` green
|
||||
Reference in New Issue
Block a user