phase: 13_llm_page_and_decode_rate
All green. Final verification summary:
**Verified / implemented (phase 13 final pass)**
- All 4 tasks already in `complete/`; this was the verification pass. Confirmed the CSS rhythm rules (`#strip`, `#banners:empty ~ #fleet`) are served and pinned, `data/logs/NOTES-decode-spike.md` names the mechanism, and the phase-07/04 surfaces still pass their pins.
- **Found and fixed a residual defect in this phase's own code**: a stalled fetch stamps its body with the poll's *finish* time, so that sample anchors every window for the rest of the span; the jump gate "confirmed" a jump measured from that same anchor → live gufo rows 232.28 tok/s (one request at ~50) + a decaying 146/114/95 tail, repeated at 200.6. `rates.py::_publish` now requires an **independent anchor** (`_pending` stores the base sample's ts; the hold continues until it ages out). Pinned by `test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor` + `test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine` (both fail on task 03's gate).
- Re-verified live after a dev-server restart: 10 min / 540 signed-in `/api/llm/snapshot` polls + read-only SQLite pass; **0 values > 300, 0 without a `Δ<window>` label**; 0 post-fix stored rows over the ceiling except the two pre-fix-shape rows above.
**Tests / lint (exact, all 100 % green)**
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1195 passed, 11 skipped, coverage 98.70 %**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; repo hygiene clean.
- `uv run pytest -m live --live` → **11 passed / 0 failed**. `/api/health` → **all nine sources ok**.
**Completion criteria**
- Fleet/strip + head/fleet gap with screenshots (`llm-fleet-spacing{,-top,-banner-case}.png`) — met (CSS unchanged this pass, pinned on served bytes).
- Task-02 findings file names the confirmed mechanism — met.
- `test_rates.py`/`test_llm_source.py` pin label-set, both window bounds and the always-honest `Δ` — met (+2 new pins).
- Sustained watch under the recorded ceiling (300 tok/s) — met (worst 203.05, all labelled).
- Full suite + validate green; no phase-07/04 behaviour change — met.
**Documentation**
- README: `RATE_WINDOW_SECONDS` row now states the hold/confirm rule (only change needed; rest verified consistent).
- AGENTS.md: `rates.py` bullet + gap 9 rewritten (residual closed *and* the remaining sub-8× skew measured, root-caused to gap 4, explicitly not fixed by inventing a new threshold).
- **Fleet facts changed under this pass and were corrected**: operator added driveripper's bearer token → `host:driveripper`/`disks:driveripper` `ok`, live suite 9/2→11/0; but driveripper reads **identity only, zero SMART numbers** (root `smartctl` gives 55 °C/14 %; `/dev/nvme0`, `/dev/ng0n1`, `/dev/bsg/*` are root-only; its unit is `DevicePolicy=auto`, not the installer's `strict`). Evidence appended to `data/logs/NOTES-agent-install.md` + `NOTES-decode-stable.md`.
**Notable decisions / deviations**
- Did **not** add a span-relative window floor: that would re-decide task 03's pinned `MIN_WINDOW_SECONDS` and its "in-bounds diluted row publishes" test. Recorded as the next phase's call, with the root cause left at gap 4 / `14_store_performance` (the ~10 s stall).
- No live fleet change made (read-only probes + one dev-server restart). Fixtures deliberately not re-captured (no parse change, task 02 proved no shape drift). Dependency-currency advisory left untouched (no dependency moved in this phase): certifi/fastapi/starlette/websockets have newer releases.
**Next pending phase:** `.agents/phases/todo/14_store_performance` (then `15_agent_mtls`).
This commit is contained in:
@@ -0,0 +1,60 @@
|
||||
# Phase 13 — The LLM page: fleet spacing and an honest decode tok/s
|
||||
|
||||
**Source:** `TODO.md` L4–L5 — "In the LLM engines page the 'fleet' section has no spacing between it and the engine summaries" / "Decode tok/s sometimes jumps unrealistically high, could be a calculation error"
|
||||
**Story:** n/a
|
||||
**Context:** `src/dashboards/web/pages/llm.html` (the `#banners`/`#fleet`/`#strip`/engines section stack), `src/dashboards/web/static/dash.css` (`.strip`, the `.fleet` block, the shared `.85rem` rhythm), `src/dashboards/rates.py` (`RateEstimator` — the one piece of mutable state in the app), `src/dashboards/sources/llm.py` (`parse_gufo_metrics`/`parse_vllm_metrics`, the `measured()` Δcounter/Δt cells), `src/dashboards/prom.py` (`scalar(agg="sum")` over label sets), `tests/test_rates.py` / `tests/test_llm_source.py`, `tests/fixtures/metrics_gufo.txt` / `metrics_vllm.txt`, `data/logs/` (findings + screenshots)
|
||||
|
||||
## Objective
|
||||
|
||||
Two defects on the `/llm` page, one phase. (1) The `#fleet` aggregate block and the
|
||||
`#strip` engine summaries sit flush — the page's shared `.85rem` vertical rhythm skips
|
||||
the two sections that have no inline margin, so the page reads as one unbroken wall
|
||||
above the engine panels. (2) The `decode tok/s` hero/chart occasionally prints values
|
||||
far above anything the engines can produce: the measured Δcounter/Δt path is keyed
|
||||
correctly by *name* (rule 2), but the *value* it diffs is a **sum over the label
|
||||
set**, and the estimator's window-empty fallback divides by a `dt` it never bounds —
|
||||
so a counter's carried value can leak into one interval's delta, or a stall can
|
||||
rescale the window. The fix keeps every number measured and labelled (`Δ<window>`),
|
||||
just makes the measurement honest.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `07_llm_monitor_style_pages` (complete) — the fleet block, the strip, and the
|
||||
one-`/api/llm/history`-request-per-refresh contract this phase must not break.
|
||||
- `04_llm_chart_readability` (complete) — the hero row and the `provenance()` tags.
|
||||
- `11_admin_login` (todo) — no shared files (this phase touches `dash.css`, `llm.html`
|
||||
optionally, `rates.py`, `sources/llm.py`, `prom.py` and their tests; 11 touches
|
||||
`app.py`/`config.py`/`auth.py`). Either order runs.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_fleet_strip_spacing.md` — restore the shared `.85rem` rhythm between `#fleet` and `#strip` (and above `#fleet` when no banners are shown) in `dash.css`; screenshot.
|
||||
2. `02_pin_decode_spike_mechanism.md` — diagnose the spike against the live engines and the stored history (AGENTS.md rule 7 — never guess the trigger); written findings in `data/logs/`.
|
||||
3. `03_bounded_estimator_and_per_series_rates.md` — the `rates.py`/`sources/llm.py` fix: per-label-series diffing plus a bounded effective window, so a label-set change contributes zero and a stall contributes an `n/a`, never a spike.
|
||||
4. `04_live_stable_verification.md` — the dev-server proof: sustained watch of the live `decode tok/s` with its `Δ<window>` label, screenshots, fixtures re-captured if the live shape changed.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: `tests/test_rates.py` — the new window/label-set behaviours (see task 03); the
|
||||
existing pinned behaviour (`test_window_is_labelled_with_the_span_actually_used`)
|
||||
keeps passing — the labelled-window contract is load-bearing.
|
||||
- Unit/integration: `tests/test_llm_source.py` — a metrics text whose label set gains a
|
||||
series between two polls produces no spike in `decode_tokens_per_second`; the
|
||||
fixture-driven parse tests pass unmodified in intent.
|
||||
- Page: `tests/test_api.py` — the served `llm.html` + `dash.css` pin the spacing rule
|
||||
(the same device that pins the other dash.css invariants).
|
||||
- Coverage: **>90%** on new/modified code; the 94% gate must hold.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `/llm` shows the shared gap between the fleet block and the engine summaries, and
|
||||
between the page head and the fleet block when no banners render — screenshots in
|
||||
`data/logs/` (`llm-fleet-spacing.png`)
|
||||
- [ ] the task-02 findings file exists and names the confirmed mechanism
|
||||
- [ ] `tests/test_rates.py` + `tests/test_llm_source.py` pin: no spike on a label-set
|
||||
change, no unbounded `dt`, the `Δ<window>` label always the window actually used
|
||||
- [ ] a sustained live watch shows no `decode tok/s` value above the confirmed
|
||||
physical ceiling for the fleet's cards (the task-02 note records the ceiling)
|
||||
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||
- [ ] no behavior change in completed phases: the strip, the fleet block, the slot grid
|
||||
and the trend charts render exactly as phase 07/04 shipped them, apart from the gap
|
||||
@@ -0,0 +1,59 @@
|
||||
# Task 01 — The gap between the fleet block and the engine summaries
|
||||
|
||||
**Phase:** `13_llm_page_and_decode_rate` · **Source:** `TODO.md:4` — "In the LLM engines page the 'fleet' section has no spacing between it and the engine summaries"
|
||||
|
||||
## Objective
|
||||
|
||||
`#llm`'s top-level sections break the page's shared `.85rem` vertical rhythm:
|
||||
`#banners` ends with `margin-bottom:.85rem`, the engine section below `#strip` carries
|
||||
an inline `margin-top:.85rem`, but `#fleet` and `#strip` carry **no** margin at all —
|
||||
so the fleet block and the engine summaries sit flush (and, with no banners, the fleet
|
||||
block also sits flush against the page head). Restore the rhythm with two CSS rules;
|
||||
no markup or JS change.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/web/static/dash.css` — in the fleet/strip section:
|
||||
- `#strip { margin-top: .85rem; }` — the gap the TODO names (fleet → engine
|
||||
summaries). Unconditional on purpose: when `#fleet` is `hidden` (the
|
||||
below-two-engines case) the strip is the first visible section and still needs
|
||||
the same top gap.
|
||||
- `#banners:empty ~ #fleet { margin-top: .85rem; }` — the gap above the fleet block
|
||||
*only* when no banner occupies the slot above it; with a banner present, the
|
||||
banner's own `margin-bottom:.85rem` already is the gap (never two stacked gaps).
|
||||
- The engine section's existing inline `margin-top:.85rem` stays (it already is the
|
||||
strip → panels gap).
|
||||
2. `src/dashboards/web/pages/llm.html` — no change expected; if the executor finds the
|
||||
sections are not siblings the way the markup suggests, fix the CSS selectors to the
|
||||
actual structure rather than re-flowing the markup (the section order is a phase-07
|
||||
contract).
|
||||
3. `tests/test_api.py` — pin the two rules on the served bytes (the existing dash.css
|
||||
pin device: a test that fails if the rules are missing from the served stylesheet),
|
||||
so a later "cleanup" that drops them fails a test.
|
||||
4. Screenshot (invariant 10): dev server up, `/llm` with **≥2 engines answering** (the
|
||||
fleet block visible — below two engines it stays hidden and the fix is invisible),
|
||||
full page top-to-bottom: `data/logs/llm-fleet-spacing.png`. If the live fleet cannot
|
||||
show two engines (the transport gap in AGENTS.md's fleet facts), pin the spacing on
|
||||
the simulated path is **not** an option for a layout claim — instead screenshot the
|
||||
one-engine page (strip visible, `#fleet` hidden, strip's `margin-top` doing its job)
|
||||
and record in the completion note that the two-engine gap was verified by the CSS
|
||||
rule + the served-bytes test, with the live two-engine capture deferred to the
|
||||
operator's engine restore.
|
||||
- ASSUMPTION: the fix is CSS-only in `dash.css`, not an HTML wrapper change — the
|
||||
page's section stack is a phase-07 layout contract and the page already uses
|
||||
inline `margin-top:.85rem` for the same rhythm, so a rule (not a restructure) is
|
||||
the change in the page's own vocabulary.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Page: the served-bytes pin in `tests/test_api.py`.
|
||||
- Live: the screenshot per step 4.
|
||||
- Coverage: **>90%** on new/modified code (CSS + test — the 94% gate must hold).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `#strip` has a `.85rem` top gap in all states (fleet visible, fleet hidden);
|
||||
`#fleet` has the gap above it exactly when no banner is shown
|
||||
- [ ] no double gap (banner + rule) and no flush pair anywhere in the section stack
|
||||
- [ ] `tests/test_api.py` pin green; screenshot in `data/logs/`
|
||||
- [ ] `bash .agents/validate.sh` green
|
||||
@@ -0,0 +1,72 @@
|
||||
# Task 02 — Pin the decode-tok/s spike against the live upstream
|
||||
|
||||
**Phase:** `13_llm_page_and_decode_rate` · **Source:** `TODO.md:5` — "Decode tok/s sometimes jumps unrealistically high, could be a calculation error"
|
||||
|
||||
## Objective
|
||||
|
||||
Rule 7: never guess the trigger. Before touching `rates.py`, confirm **which** mechanism
|
||||
produces the spike, from the stored history and the live engines, and record the
|
||||
findings in a file the fix task (03) and the verification task (04) both read.
|
||||
|
||||
## Work
|
||||
|
||||
1. Find the spikes in the stored history (the dev DB is the source of truth — AGENTS.md
|
||||
known gap 4 records it at ~610 MB with a year of raw/1m/1h rollups):
|
||||
`sqlite3 data/… "<db>" "SELECT ts, value FROM sample WHERE metric LIKE
|
||||
'decode_tokens_per_second%' ORDER BY ts DESC LIMIT 40000"` — list the top-N values
|
||||
and their timestamps. Note for each: which engine (`source`/`host`), and what the
|
||||
neighbouring values look like (a one-poll spike vs a sustained step).
|
||||
2. For each spike, read the **provenance** of the cell that produced it — the payload
|
||||
carries it (rule 2): `computed:Δ<series>/Δt` + `Δ<window>` (our estimator) vs the
|
||||
engine's own gauge name (the native `decode_tokens_per_second_native` source string).
|
||||
A spike from the *engine's* gauge is the engine's rolling average, not our math —
|
||||
the fix there is a label, not a code change. `sqlite3` the engine's native gauge
|
||||
series alongside to separate the two paths.
|
||||
3. For the estimator path, inspect the live exposure (from a host that can reach the
|
||||
engines — the laptop can reach both, per the fleet facts):
|
||||
- `curl -s <gufo>/metrics | grep -E "tokens_(predicted|generated)_total"` and the
|
||||
vLLM equivalent — are the counters **plain** (as in `tests/fixtures/`), or
|
||||
**labelled per slot/engine** in the live build? A label set that changes between
|
||||
polls is the leading suspect: `prom.Prom.scalar(agg="sum")` sums whatever label
|
||||
sets exist, so a series that appears with a carried cumulative value adds that
|
||||
whole value to one interval's delta (a `value < samples[-1]` decrease would reset
|
||||
the window — an *increase* from a new series does not).
|
||||
- Cross-check the spike timestamps against `GET /slots` history (which slots were
|
||||
starting/turning over) and the event feed (`/api/events`) for engine restarts in
|
||||
the same minutes.
|
||||
4. Record the findings in `data/logs/NOTES-decode-spike.md`: the confirmed mechanism
|
||||
(one of the candidates below, or a new one), the evidence (values, timestamps,
|
||||
captures), and the fleet's physical decode ceiling for the task-04 comparison —
|
||||
the observed max of the native gauge over 90d **and** the cards' spec ceiling
|
||||
(R9700 / Strix Halo 8060S), so "unrealistically high" has a number attached.
|
||||
|
||||
Candidates, in suspected order (the findings file must name the one that fired):
|
||||
- (a) **label-set sum**: `scalar(agg="sum")` over a changing set of labelled
|
||||
series — a new series' carried counter enters the delta as a one-interval spike;
|
||||
- (b) **window-empty fallback**: `rates.py`'s `base = samples[-2]` when fewer than
|
||||
two samples sit inside the span — the effective window is the gap to the previous
|
||||
sample, unbounded above by 2×span (the trim keeps the last two however old);
|
||||
- (c) **a lineage jump that is not a decrease**: a counter that restarts *above*
|
||||
zero (a renamed/relabelled series resolving under the same key, a pre-loaded
|
||||
counter) — `value < samples[-1]` never fires, so the carried value is diffed;
|
||||
- (d) **the engine's own native gauge** (vLLM's `avg_*_throughput` rolling average) —
|
||||
not our math; label-level fix only.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- No code changes in this task — diagnostics only, with the findings file as the
|
||||
artifact. If a candidate is confirmed hermetically reproducible, add the red test
|
||||
here (a `tests/test_llm_source.py` / `tests/test_rates.py` test that fails on the
|
||||
current code) so task 03 turns it green.
|
||||
- Coverage: **>90%** on any test added (the 94% gate must hold).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `data/logs/NOTES-decode-spike.md` exists: confirmed mechanism, evidence, and the
|
||||
physical ceiling with its two sources (observed native-gauge max + card spec)
|
||||
- [ ] every spike in the stored top-N is classified (our estimator / the engine's
|
||||
gauge / a real burst) — a spike that cannot be classified is a finding, not a
|
||||
skipped row
|
||||
- [ ] a red test exists for the confirmed mechanism if it is hermetically reproducible
|
||||
- [ ] the fixtures were compared against the live exposure and any shape drift is
|
||||
recorded (drift → `./scripts/capture-fixtures.sh` re-capture noted for task 04)
|
||||
+85
@@ -0,0 +1,85 @@
|
||||
# Task 03 — A rate that cannot be a carried counter
|
||||
|
||||
**Phase:** `13_llm_page_and_decode_rate` · **Source:** `TODO.md:5` — "Decode tok/s sometimes jumps unrealistically high, could be a calculation error"
|
||||
|
||||
## Objective
|
||||
|
||||
Fix the confirmed mechanism from task 02, plus the two generic weaknesses that make
|
||||
any mechanism possible: a summed counter whose *label set* changes between polls, and
|
||||
an effective window the estimator never bounds. After this task, a delta that includes
|
||||
a carried counter value is impossible by construction — a new series contributes
|
||||
nothing until it has a baseline, a stalled window answers `n/a` for one poll instead of
|
||||
rescaling, and a jump that looks like a lineage break is confirmed over two polls
|
||||
before it is published. Every published number still carries its `Δ<window>` label and
|
||||
`source` string (rule 2).
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/prom.py` — expose the **per-label-set** view the fix needs (the
|
||||
`series` map already holds them; `by_label` shows the shape): a `values_by_series`
|
||||
(or reuse `values` with labels) accessor that returns `[(labels, value)]` per
|
||||
resolved name, so the source can diff each labelled series individually. Keep
|
||||
`scalar` untouched (the gauges still sum/avg/max the same way — a *gauge's* sum is
|
||||
a real aggregate; only *counter* deltas may not mix series).
|
||||
2. `src/dashboards/sources/llm.py` — the `measured()` path (gufo's
|
||||
`tokens_generated_total`/`tokens_prompt_total` and vLLM's
|
||||
`generation_tokens_total`/`prompt_tokens_total`): resolve the per-label-set values
|
||||
and run **each series through the estimator under its own key** (the key is the
|
||||
resolved name *plus* its label set — `llamacpp:tokens_predicted_total{slot="2"}`),
|
||||
then `rate = Σ per-series rates`, with the cell's `window` the max of the
|
||||
contributing series' windows (the window label says the widest window actually
|
||||
used). A series with no previous sample contributes 0 tokens this interval — never
|
||||
its carried value. The `Δ<window>` label and the `computed:Δ…/Δt` source string stay
|
||||
in the `provenance()` format the page already prints (the source string names the
|
||||
resolved series, as today).
|
||||
- ASSUMPTION: per-series diffing is the fix for candidates (a) and (c) — a
|
||||
label-set change then changes *nothing* in the delta until the new series has two
|
||||
samples, which is exactly the "a renamed counter starts a fresh window" doctrine
|
||||
the estimator docstring already states, extended from names to label sets.
|
||||
3. `src/dashboards/rates.py` — the two generic bounds:
|
||||
- **stalled-window bound**: in the `len(inside) < 2` fallback, accept `samples[-2]`
|
||||
as the base only while its age is ≤ 1.5 × `span`; older than that, return
|
||||
`(None, None)` for this poll — one `n/a` on the poll after a stall is the honest
|
||||
answer (a 20-second window on a 10-second panel is a different claim), and the
|
||||
next poll recovers normally. The existing `test_window_is_labelled_with_the_span_actually_used`
|
||||
(a 9s fallback) keeps passing: 9s ≤ 15s.
|
||||
- **jump confirmation**: a published rate ≥ 8× the previous poll's published rate
|
||||
(previous > 0) is held — the previous value is repeated for one more poll, and the
|
||||
jump publishes only if *confirmed* by the next interval; an unconfirmed jump is
|
||||
dropped (the cell prints the held value, its `Δ<window>` label unchanged). This
|
||||
suppresses a single-interval artifact without flattening a real burst: a genuine
|
||||
step-up is confirmed one interval later and then publishes, and the confirmation
|
||||
state is per-key in the estimator.
|
||||
- ASSUMPTION: the 8×-over-previous threshold and the one-poll hold are the right
|
||||
confirmation rule — a real LLM decode step (a batch of long outputs starting)
|
||||
persists across a 10s interval, while a carried-counter artifact does not; the
|
||||
thresholds are named constants at the top of the file so they can move without a
|
||||
logic change.
|
||||
4. `tests/test_rates.py` — the pins: a stalled window (previous sample > 1.5×span old)
|
||||
answers `(None, None)` for one poll and recovers; a jump ≥ 8× holds for one poll
|
||||
and publishes when confirmed / drops when not; the pinned pre-existing tests all
|
||||
pass unmodified.
|
||||
5. `tests/test_llm_source.py` — the label-set pin: two parses of a metrics text whose
|
||||
counter series set differs between polls (a `{slot="2"}` series appears carrying
|
||||
50 000) — the delta of the second poll contains **no** 50 000; the flat-counter and
|
||||
counter-reset tests keep passing.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: `tests/test_rates.py` + `tests/test_llm_source.py` per the work list.
|
||||
- Regression: every pre-existing `test_llm_source.py` / `test_rates.py` test passes
|
||||
unmodified in intent; the fixture-driven parse tests (`metrics_gufo.txt`,
|
||||
`metrics_vllm.txt`) are the shape pins.
|
||||
- Coverage: **>90%** on this task's new/modified code; the 94% gate must hold.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] no delta can contain a value that was not generated inside the window: the
|
||||
label-set test, the reset test and the stall test all green
|
||||
- [ ] a real step-up (sustained over two polls) still publishes — the confirmation
|
||||
path is tested, not just the suppression path
|
||||
- [ ] every published rate still carries `Δ<window>` (the window actually used, possibly
|
||||
a sum-of-windows label) and its `source` string — `tests/test_llm_source.py`
|
||||
asserts the provenance shape
|
||||
- [ ] the task-02 red test is green
|
||||
- [ ] `bash .agents/validate.sh` green
|
||||
@@ -0,0 +1,50 @@
|
||||
# Task 04 — Live proof: a decode rate that stays under its ceiling
|
||||
|
||||
**Phase:** `13_llm_page_and_decode_rate` · **Source:** `TODO.md:5` — "Decode tok/s sometimes jumps unrealistically high, could be a calculation error"
|
||||
|
||||
## Objective
|
||||
|
||||
The dev-server proof that the fix holds on the real engines: a sustained watch of the
|
||||
live `decode tok/s` (payload cell + stored history) with its `Δ<window>` label, every
|
||||
observed value under the physical ceiling task 02 recorded, and the page itself
|
||||
screenshotted with the task-01 spacing in view.
|
||||
|
||||
## Work
|
||||
|
||||
1. Dev server up and the LLM sources `ok` (`./scripts/dev-server.sh status`); record
|
||||
the start time. Watch for at least one full poll-cycle cluster with traffic if the
|
||||
engines are idle (the operator's own requests are the traffic — do **not**
|
||||
generate load: a live test that puts real work on the engines' batteries is not a
|
||||
test, the AGENTS.md live-suite rule):
|
||||
- `curl -s http://127.0.0.1:8090/api/llm/snapshot` every poll for ≥ 10 minutes:
|
||||
log `decode_tokens_per_second` (value, `source`, `window`) per engine;
|
||||
- then `sqlite3` the `decode_tokens_per_second` raw series for the same period and
|
||||
confirm no stored value exceeds the task-02 ceiling.
|
||||
2. `data/logs/llm-decode-stable.png` — the `/llm` page showing the hero row with the
|
||||
`Δ<window>` provenance tag, the strip card, and (when ≥ 2 engines answer) the fleet
|
||||
block with the task-01 gap; a second screenshot if a real burst was observed
|
||||
mid-watch (the confirmed-and-published path — the burst's `Δ` label visible).
|
||||
3. `./scripts/capture-fixtures.sh` — re-capture **only if** task 02 recorded a live
|
||||
shape drift (a label set the fixtures lack); otherwise leave the fixtures
|
||||
untouched and say so.
|
||||
4. `uv run pytest -m live --live` — the opt-in live suite (read-only per its rule;
|
||||
the LLM engine tests skip when their source is unset). Record the tally in the
|
||||
completion note (the fleet's transport gap means the host/disks live tests may
|
||||
still fail — the LLM and UPS ones are this task's bar).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Live: the watch + `sqlite3` comparison per step 1, screenshots per step 2.
|
||||
- No code changes expected in this task — if the watch finds a residual spike the
|
||||
fix missed, the findings go back into `data/logs/NOTES-decode-spike.md` and task 03
|
||||
is re-opened (record the re-open in the completion note rather than fixing inline).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] ≥ 10 minutes of live polls: no `decode_tokens_per_second` value (payload or
|
||||
stored) above the task-02 ceiling; the `Δ<window>` label present on every
|
||||
non-`n/a` value
|
||||
- [ ] `data/logs/llm-decode-stable.png` (and the burst screenshot if one occurred)
|
||||
- [ ] fixture re-capture decision recorded; `bash .agents/validate.sh` green
|
||||
- [ ] the live tally recorded, with the known transport gap acknowledged as the reason
|
||||
any host/disks live failures are out of scope here
|
||||
Reference in New Issue
Block a user