services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 13_llm_page_and_decode_rate

Browse Source 
All green. Final verification summary:

**Verified / implemented (phase 13 final pass)**
- All 4 tasks already in `complete/`; this was the verification pass. Confirmed the CSS rhythm rules (`#strip`, `#banners:empty ~ #fleet`) are served and pinned, `data/logs/NOTES-decode-spike.md` names the mechanism, and the phase-07/04 surfaces still pass their pins.
- **Found and fixed a residual defect in this phase's own code**: a stalled fetch stamps its body with the poll's *finish* time, so that sample anchors every window for the rest of the span; the jump gate "confirmed" a jump measured from that same anchor → live gufo rows 232.28 tok/s (one request at ~50) + a decaying 146/114/95 tail, repeated at 200.6. `rates.py::_publish` now requires an **independent anchor** (`_pending` stores the base sample's ts; the hold continues until it ages out). Pinned by `test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor` + `test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine` (both fail on task 03's gate).
- Re-verified live after a dev-server restart: 10 min / 540 signed-in `/api/llm/snapshot` polls + read-only SQLite pass; **0 values > 300, 0 without a `Δ<window>` label**; 0 post-fix stored rows over the ceiling except the two pre-fix-shape rows above.

**Tests / lint (exact, all 100 % green)**
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1195 passed, 11 skipped, coverage 98.70 %**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; repo hygiene clean.
- `uv run pytest -m live --live` → **11 passed / 0 failed**. `/api/health` → **all nine sources ok**.

**Completion criteria**
- Fleet/strip + head/fleet gap with screenshots (`llm-fleet-spacing{,-top,-banner-case}.png`) — met (CSS unchanged this pass, pinned on served bytes).
- Task-02 findings file names the confirmed mechanism — met.
- `test_rates.py`/`test_llm_source.py` pin label-set, both window bounds and the always-honest `Δ` — met (+2 new pins).
- Sustained watch under the recorded ceiling (300 tok/s) — met (worst 203.05, all labelled).
- Full suite + validate green; no phase-07/04 behaviour change — met.

**Documentation**
- README: `RATE_WINDOW_SECONDS` row now states the hold/confirm rule (only change needed; rest verified consistent).
- AGENTS.md: `rates.py` bullet + gap 9 rewritten (residual closed *and* the remaining sub-8× skew measured, root-caused to gap 4, explicitly not fixed by inventing a new threshold).
- **Fleet facts changed under this pass and were corrected**: operator added driveripper's bearer token → `host:driveripper`/`disks:driveripper` `ok`, live suite 9/2→11/0; but driveripper reads **identity only, zero SMART numbers** (root `smartctl` gives 55 °C/14 %; `/dev/nvme0`, `/dev/ng0n1`, `/dev/bsg/*` are root-only; its unit is `DevicePolicy=auto`, not the installer's `strict`). Evidence appended to `data/logs/NOTES-agent-install.md` + `NOTES-decode-stable.md`.

**Notable decisions / deviations**
- Did **not** add a span-relative window floor: that would re-decide task 03's pinned `MIN_WINDOW_SECONDS` and its "in-bounds diluted row publishes" test. Recorded as the next phase's call, with the root cause left at gap 4 / `14_store_performance` (the ~10 s stall).
- No live fleet change made (read-only probes + one dev-server restart). Fixtures deliberately not re-captured (no parse change, task 02 proved no shape drift). Dependency-currency advisory left untouched (no dependency moved in this phase): certifi/fastapi/starlette/websockets have newer releases.

**Next pending phase:** `.agents/phases/todo/14_store_performance` (then `15_agent_mtls`).
This commit is contained in:
ducoterra
2026-10-19 14:53:58 -04:00
parent 04d934dd94
commit 1bbd868401
36 changed files with 1231 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__00_phase.a1.err
View File
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__00_phase.a1.md
+28
View File
@@ -0,0 +1,28 @@
All green. Final verification summary:
**Verified / implemented (phase 13 final pass)**
- All 4 tasks already in `complete/`; this was the verification pass. Confirmed the CSS rhythm rules (`#strip`, `#banners:empty ~ #fleet`) are served and pinned, `data/logs/NOTES-decode-spike.md` names the mechanism, and the phase-07/04 surfaces still pass their pins.
- **Found and fixed a residual defect in this phase's own code**: a stalled fetch stamps its body with the poll's *finish* time, so that sample anchors every window for the rest of the span; the jump gate "confirmed" a jump measured from that same anchor → live gufo rows 232.28 tok/s (one request at ~50) + a decaying 146/114/95 tail, repeated at 200.6. `rates.py::_publish` now requires an **independent anchor** (`_pending` stores the base sample's ts; the hold continues until it ages out). Pinned by `test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor` + `test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine` (both fail on task 03's gate).
- Re-verified live after a dev-server restart: 10 min / 540 signed-in `/api/llm/snapshot` polls + read-only SQLite pass; **0 values > 300, 0 without a `Δ<window>` label**; 0 post-fix stored rows over the ceiling except the two pre-fix-shape rows above.
**Tests / lint (exact, all 100 % green)**
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1195 passed, 11 skipped, coverage 98.70 %**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; repo hygiene clean.
- `uv run pytest -m live --live` → **11 passed / 0 failed**. `/api/health` → **all nine sources ok**.
**Completion criteria**
- Fleet/strip + head/fleet gap with screenshots (`llm-fleet-spacing{,-top,-banner-case}.png`) — met (CSS unchanged this pass, pinned on served bytes).
- Task-02 findings file names the confirmed mechanism — met.
- `test_rates.py`/`test_llm_source.py` pin label-set, both window bounds and the always-honest `Δ` — met (+2 new pins).
- Sustained watch under the recorded ceiling (300 tok/s) — met (worst 203.05, all labelled).
- Full suite + validate green; no phase-07/04 behaviour change — met.
**Documentation**
- README: `RATE_WINDOW_SECONDS` row now states the hold/confirm rule (only change needed; rest verified consistent).
- AGENTS.md: `rates.py` bullet + gap 9 rewritten (residual closed *and* the remaining sub-8× skew measured, root-caused to gap 4, explicitly not fixed by inventing a new threshold).
- **Fleet facts changed under this pass and were corrected**: operator added driveripper's bearer token → `host:driveripper`/`disks:driveripper` `ok`, live suite 9/2→11/0; but driveripper reads **identity only, zero SMART numbers** (root `smartctl` gives 55 °C/14 %; `/dev/nvme0`, `/dev/ng0n1`, `/dev/bsg/*` are root-only; its unit is `DevicePolicy=auto`, not the installer's `strict`). Evidence appended to `data/logs/NOTES-agent-install.md` + `NOTES-decode-stable.md`.
**Notable decisions / deviations**
- Did **not** add a span-relative window floor: that would re-decide task 03's pinned `MIN_WINDOW_SECONDS` and its "in-bounds diluted row publishes" test. Recorded as the next phase's call, with the root cause left at gap 4 / `14_store_performance` (the ~10 s stall).
- No live fleet change made (read-only probes + one dev-server restart). Fixtures deliberately not re-captured (no parse change, task 02 proved no shape drift). Dependency-currency advisory left untouched (no dependency moved in this phase): certifi/fastapi/starlette/websockets have newer releases.
**Next pending phase:** `.agents/phases/todo/14_store_performance` (then `15_agent_mtls`).
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__00_phase.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__00_phase.a1.validate
+63
View File
@@ -0,0 +1,63 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 53%]
........................................................................ [ 59%]
........................................................................ [ 65%]
....................sssssssssss......................................... [ 71%]
........................................................................ [ 77%]
........................................................................ [ 83%]
........................................................................ [ 89%]
........................................................................ [ 95%]
...................................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 32
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 348 0 100%
src/dashboards/app.py 178 0 100%
src/dashboards/auth.py 265 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 120 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
src/dashboards/rates.py 73 0 100%
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 178 1 99% 285
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 188 2 99% 255-256
------------------------------------------------------------------
TOTAL 2914 38 99%
Required test coverage of 94% reached. Total coverage: 98.70%
1195 passed, 11 skipped, 1 warning in 92.75s (0:01:32)
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
certifi 2026.7.22 2026.10.18 wheel
fastapi 0.143.5 0.149.0 wheel
starlette 1.8.0 1.9.0 wheel
websockets 17.2 17.3 wheel
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__01_fleet_strip_spacing.a1.err
View File
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__01_fleet_strip_spacing.a1.md
+20
View File
@@ -0,0 +1,20 @@
Everything is green. Final state:
**Implemented**
- `src/dashboards/web/static/dash.css` — added the two rhythm rules in the fleet-totals section: `#strip { margin-top: .85rem; }` (unconditional, so the strip keeps the gap when `#fleet` is `[hidden]`) and `#banners:empty ~ #fleet { margin-top: .85rem; }` (gap above the block only when the banner slot renders nothing), with a comment naming why each half is shaped that way. No markup/JS change; the sections were already siblings of `<main>`.
- `tests/test_api.py::test_llm_section_stack_keeps_the_shared_vertical_rhythm` — pins both rules on the served stylesheet, pins that **exactly one** rule styles each of `#strip`/`#fleet` (an unguarded second one = the double gap), pins the `~`-sibling structure and the inline `.85rem` margins the rules are written around. Verified to fail both ways (rules deleted; extra `#fleet` rule added).
- Live capture with **two engines answering**: `data/logs/llm-fleet-spacing.png` (full page), `-top.png`, `-banner-case.png`, and `data/logs/NOTES-llm-fleet-spacing.md` with the before/after table.
**Tests / lint (exact)**
- `bash .agents/validate.sh` → **EXIT=0**: `uv run pytest --cov --cov-fail-under=94` → **1173 passed, 11 skipped, coverage 98.63 %** (baseline 1172; +1 new test, coverage unchanged); `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → 0 errors; `node --check web/static/*.js` → clean.
- `uv run pytest -m live --live` → 8 passed / 3 failed — the three `driveripper` agent tests, i.e. the transport state AGENTS.md already records; unchanged by a CSS rule.
- Chromium measurements (`.85rem` = 13.6px): `#fleet → #strip` **0 → 13.59px**; `#strip → panels` 13.59px; banner slot → fleet 13.59px with a banner up (`#fleet` computes `margin-top: 0px`, i.e. no double gap) and with `#fleet` hidden.
**Docs**: AGENTS.md (`/llm` UI-conventions paragraph: the shared `.85rem` rule, both rule shapes, the test pins) and README.md (`/llm` walkthrough clause).
**Decisions/deviations**
- Measured honesty: only fleet→strip actually moves. The `:empty`-guarded rule changes no rendered distance today because the *empty* `#banners` slot carries its own inline `margin-bottom:.85rem` (the same 1rem + .85rem head spacing `/` and `/ups` already have); I kept the prescribed rule — it moves the gap from markup-that-renders-nothing onto the block and prevents the double gap — and recorded that measurement in the test docstring and the notes instead of claiming a second fix.
- Left the `/`–`/ups` page-head rhythm and `/disks`'s analogous fleet→"Per host" adjacency alone (out of this task's scope); noted `/disks` here as the same pattern for a future pass.
- `validate.sh` advisory (not a gate): certifi/fastapi/starlette have newer releases; this task moved no dependency.
**Next pending task**: `.agents/phases/todo/13_llm_page_and_decode_rate/02_pin_decode_spike_mechanism.md`.
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__01_fleet_strip_spacing.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__01_fleet_strip_spacing.a1.validate
+62
View File
@@ -0,0 +1,62 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 6%]
........................................................................ [ 12%]
........................................................................ [ 18%]
........................................................................ [ 24%]
........................................................................ [ 30%]
........................................................................ [ 36%]
........................................................................ [ 42%]
........................................................................ [ 48%]
........................................................................ [ 54%]
........................................................................ [ 60%]
........................................................................ [ 66%]
....................sssssssssss......................................... [ 72%]
........................................................................ [ 79%]
........................................................................ [ 85%]
........................................................................ [ 91%]
........................................................................ [ 97%]
................................ [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 32
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 348 0 100%
src/dashboards/app.py 178 0 100%
src/dashboards/auth.py 265 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 120 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 228
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 188 2 99% 255-256
------------------------------------------------------------------
TOTAL 2843 39 99%
Required test coverage of 94% reached. Total coverage: 98.63%
1173 passed, 11 skipped, 1 warning in 93.51s (0:01:33)
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
certifi 2026.7.22 2026.10.18 wheel
fastapi 0.143.5 0.148.0 wheel
starlette 1.8.0 1.9.0 wheel
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__02_pin_decode_spike_mechanism.a1.err
View File
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__02_pin_decode_spike_mechanism.a1.md
+15
View File
@@ -0,0 +1,15 @@
**Implemented**
- `data/logs/NOTES-decode-spike.md` — mechanism **confirmed, not guessed**: `RateEstimator` has no lower bound on its window (`base = inside[0] if len(inside)>1 else samples[-2]`) **and** `poller.Job._loop`'s `max(0.2, interval-elapsed)` catch-up makes a 0.2 s window after any slow fetch, so a stalled fetch's carried tokens are diffed over 0.2 s. Candidates (a) label-set sum, (c) lineage jump and (d) engine gauge are all excluded by measurement; (a) stays a latent defect with its own red test.
- Evidence: 277/277 stored gufo values >1000 tok/s are catch-up rows (every one with a >3 s fetch gap in the prior 15 s); no spike's numerator exceeds 1.12× what its elapsed span can produce; the 15.47 s incident totals 827 tokens = 53.5 tok/s, the engine's real rate; all sources gap at the same instant (process-wide stall, gap-4/phase-14 territory).
- Live read-only probes (`data/logs/probe-decode-spike{,2}.py`, `/metrics`+`/slots` only): gufo counters unlabelled + monotonic (max +76 tokens/read); vLLM has one label set and **no** throughput gauge; probe 1 reproduced a 0.297 s window, probe 2 reproduced the spike live (2043.66 tok/s, `Δ0.4s`) with the same tail as the stored incident.
- Fixture↔live shape: **no drift** (gufo 15/15, vLLM 128/128) → no re-capture needed for task 04.
- 4 red tests: 2 in `tests/test_rates.py` (spike + the inflated tail), 2 in `tests/test_llm_source.py` (gufo incident through `parse_gufo_metrics`; vLLM joining label set).
- Physical ceiling for task 04: spike-free max **119.9** (gufo) / **171.7** (vLLM) tok/s, engine lifetime means 30.0 / 2.8, native gauge 27.3–49.5, bandwidth bounds 256/640 GB/s → **>300 tok/s is arithmetic**, ≥1000 impossible.
**Tests/lint/coverage** — `uv run pytest --cov --cov-fail-under=94` → **1173 passed, 11 skipped, 4 xfailed, coverage 98.66%**; `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → 0 errors; `bash .agents/validate.sh` → exit 0 (incl. `node --check`, hygiene). No deps moved.
**Docs** — AGENTS.md: new known gap 9 (confirmed mechanism, evidence file, the four xfails that XPASS when fixed) + the `rates.py` architecture line now says the window has no floor yet. README: no doc impact (nothing user-visible changed yet).
**Decisions/deviations** — (1) "Red tests" ship as `xfail(strict=True)` so the suite is 100% green now and fails loudly (XPASS) when task 03 fixes the behaviour — markers must be removed, assertions kept. (2) No live change of any kind was needed or made; the ~10–15 s stall's root cause is left to gap 4 / `14_store_performance`, with a pointer in the notes.
**Next pending task** — `.agents/phases/todo/13_llm_page_and_decode_rate/03_bounded_estimator_and_per_series_rates.md`.
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__02_pin_decode_spike_mechanism.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__02_pin_decode_spike_mechanism.a1.validate
+62
View File
@@ -0,0 +1,62 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 6%]
........................................................................ [ 12%]
........................................................................ [ 18%]
........................................................................ [ 24%]
........................................................................ [ 30%]
........................................................................ [ 36%]
........................................................................ [ 42%]
........................................................................ [ 48%]
........................................................................ [ 54%]
........................................................................ [ 60%]
........................................................................ [ 66%]
....................sssssssssss....................................x.... [ 72%]
...........x............................................................ [ 78%]
........................................................................ [ 84%]
.................xx..................................................... [ 90%]
........................................................................ [ 96%]
.................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 32
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 348 0 100%
src/dashboards/app.py 178 0 100%
src/dashboards/auth.py 265 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 120 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 0 100%
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 228
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 188 2 99% 255-256
------------------------------------------------------------------
TOTAL 2843 38 99%
Required test coverage of 94% reached. Total coverage: 98.66%
1173 passed, 11 skipped, 4 xfailed, 1 warning in 95.47s (0:01:35)
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
certifi 2026.7.22 2026.10.18 wheel
fastapi 0.143.5 0.148.0 wheel
starlette 1.8.0 1.9.0 wheel
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__03_bounded_estimator_and_per_series_rates.a1.err
View File
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__03_bounded_estimator_and_per_series_rates.a1.md
+20
View File
@@ -0,0 +1,20 @@
All green. Final state verified against the recorded fleet baseline (the 3 live failures are the documented `driveripper`-no-agent ConnectErrors, unchanged by this task).
## Report — task 03 `bounded_estimator_and_per_series_rates`
**Implemented**
- `src/dashboards/rates.py`: effective window now bounded both sides — `MIN_WINDOW_SECONDS=0.5` floor (the poll loop's ~0.2 s catch-up answers `(None, dt)` — one `n/a` with its honest label — **and drops the stale base** so the inflated tail re-bases) and the documented `2 × span` ceiling enforced via `MAX_WINDOW_RATIO` on the window-empty fallback; `JUMP_CONFIRM_RATIO=8.0` one-poll hold (previous value + label repeat; jump publishes only when the next interval confirms, else dropped); reset and `forget()` clear the judgement state; `min_window=0.0` constructor escape for sub-cadence tests.
- `src/dashboards/prom.py`: `values_by_series()` per-label-set accessor; `scalar` untouched.
- `src/dashboards/sources/llm.py`: gufo + vLLM `measured()` now diff **each label set under its own key** (`rate_key()` = resolved name + sorted, quoted labels); cell = Σ per-series rates over the **max** contributing window; a baseline-less series contributes 0 (never its carried value); a refused window keeps `computed:Δ<series>/Δt` + `Δ<window>` beside `n/a`.
- Removed all four task-02 `xfail(strict=True)` markers, assertions kept; added pins for floor, ceiling, hold/confirm/drop, per-key state, per-series joins, provenance shape, `values_by_series`.
**Tests / lint / coverage** — `uv run pytest`: **1193 passed, 11 skipped**; `bash .agents/validate.sh` exit 0 (coverage **98.69%** ≥ 94; rates.py 100%, llm.py 99%, prom.py 96% — misses pre-existing in `_quantile`); `ruff` + `pyright` clean; dev server restarted, `/api/health` as recorded (driveripper DOWN = missing agent), live `/api/llm/snapshot` reads e.g. `42.5 tok/s @ Δ9.9s computed:Δllamacpp:tokens_predicted_total/Δt`; `pytest -m live --live` = 8 passed / 3 failed, all three the documented `driveripper:9120` ConnectError.
**Docs** — `AGENTS.md` (invariant 2 keyed-by-name+label-set, `rates.py`/`prom.py`/`sources/llm.py` bullets, gap 9 rewritten as fixed) and `README.md` (`RATE_WINDOW_SECONDS` row: a stall answers one `n/a`).
**Decisions/deviations**
- The task file's "accept fallback base ≤ 1.5 × span" contradicts its own pinned red test (asserts the diluted 7.47 row at a 15.25 s window, span 10). The test wins: ceiling enforced at the estimator's documented `2 × span`; the confirmed mechanism (NOTES candidate b, *lower*-bound form) is fixed by the floor + stale-base drop, as the findings file prescribes.
- A refused window is treated as an answer (skips vLLM's weaker `running/tpot` estimate for that poll) so the stall is visible as one self-labelled `n/a`.
- Two `test_collectors.py` real-time tests (fetches µs apart) now pass `min_window=0.0` — they pin window semantics, not the floor.
**Next pending task:** `13_llm_page_and_decode_rate/04_live_stable_verification.md` (sustained live watch ≤ 300 tok/s, screenshots).
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__03_bounded_estimator_and_per_series_rates.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__03_bounded_estimator_and_per_series_rates.a1.validate
+62
View File
@@ -0,0 +1,62 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 53%]
........................................................................ [ 59%]
........................................................................ [ 65%]
....................sssssssssss......................................... [ 71%]
........................................................................ [ 77%]
........................................................................ [ 83%]
........................................................................ [ 89%]
........................................................................ [ 95%]
.................................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 32
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 348 0 100%
src/dashboards/app.py 178 0 100%
src/dashboards/auth.py 265 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 120 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
src/dashboards/rates.py 69 0 100%
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 178 1 99% 285
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 188 2 99% 255-256
------------------------------------------------------------------
TOTAL 2910 38 99%
Required test coverage of 94% reached. Total coverage: 98.69%
1193 passed, 11 skipped, 1 warning in 97.58s (0:01:37)
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
certifi 2026.7.22 2026.10.18 wheel
fastapi 0.143.5 0.148.0 wheel
starlette 1.8.0 1.9.0 wheel
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__04_live_stable_verification.a1.err
View File
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__04_live_stable_verification.a1.md
+18
View File
@@ -0,0 +1,18 @@
Everything is green. Final report:
**Implemented (task 04 — live proof; no production code needed changing)**
- `data/logs/watch-decode-stable.py` — signed-in watcher: 11.0 min / 285 `/api/llm/snapshot` polls, logging `decode_tokens_per_second` value + `Δ<window>` + `source` per engine (no load generated).
- `data/logs/analyze-watch.py` — judge: TSV + `mode=ro` SQLite over the identical window; ceilings, label check, jump/hold check, 1 m rollups → `VERDICT: PASS`.
- `data/logs/NOTES-decode-stable.md` — the completion evidence (row-by-row holds/confirmations/refusals, fixtures decision, live tallies, screenshots, gap measurements).
- Screenshots: `llm-decode-stable.png` (full page), `llm-decode-stable-top.png` (fleet block + **13.59 px** `.85rem` gaps + strip + `DECODE` heroes with `Δ8.2s`/`Δ8.0s`), `llm-decode-burst.png` (real two-slot burst: 62.1 tok/s `Δ8.5s`, fleet sum 102.4).
**Results** — gufo max **117.57**, vLLM max **98.00** tok/s (ceiling 300): 0 payload and 0 stored raw rows over it, 0 values printed without a `Δ` label; 3 floor refusals (`Δ0.387/0.399/0.363 s`) and 2 ≥8× jumps held-then-confirmed ran live; 1 m rollups top out at 51.9/34.1 (task 02's bad minute was 222.7). `bash .agents/validate.sh` → **rc 0** (1193 passed, 11 skipped; coverage 98.69 % ≥ 94; ruff "All checks passed!"; pyright 0 errors; `node --check` clean). `pytest -m live --live` → **9 passed / 2 failed**, both `driveripper` agent tests.
**Docs** — `AGENTS.md`: gap 9 rewritten as *fixed and verified live* (numbers, screenshots, fixtures-not-recaptured), stale `1172 passed`/`98.6 %` counts corrected, and the fleet facts updated for a change that happened mid-task (see below). README: no doc impact (task 03 already carried the operator-facing line).
**Decisions / deviations**
- **Fixtures not re-captured**: task 02 proved no shape drift (gufo 15/15, vLLM 128/128); task 13 changed no parsing, so `capture-fixtures.sh` was deliberately not run.
- **Fleet fact moved under the task**: the operator installed `driveripper`'s `dashboards-agent` at 13:27 (`active`, `/health` 200, `/snapshot` 401) — the two `driveripper` jobs are still DOWN but now on a **missing bearer token** (`HOST_TARGETS` has none for driveripper), which is why the live tally moved 8/3 → 9/2. I made **no live change**: reading/rotating that root-owned token is an operator action; AGENTS.md now records it and the one step that turns those rows green.
- The ~10 s process-wide stall persists (38 catch-up polls in 11 min) — gap 4 / phase 14's problem; the estimator absorbs it, so no re-open of task 03.
**Next pending task:** none in phase 13 — `13_llm_page_and_decode_rate/04` was the last; next is `14_store_performance` (and `15_agent_mtls` now in `todo/`).
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__04_live_stable_verification.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/13_llm_page_and_decode_rate/13_llm_page_and_decode_rate__04_live_stable_verification.a1.validate
+63
View File
@@ -0,0 +1,63 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 53%]
........................................................................ [ 59%]
........................................................................ [ 65%]
....................sssssssssss......................................... [ 71%]
........................................................................ [ 77%]
........................................................................ [ 83%]
........................................................................ [ 89%]
........................................................................ [ 95%]
.................................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 32
src/dashboards/alerts.py 176 0 100%
src/dashboards/api.py 348 0 100%
src/dashboards/app.py 178 0 100%
src/dashboards/auth.py 265 0 100%
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
src/dashboards/config.py 120 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 165 3 98% 192-194, 279
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
src/dashboards/rates.py 69 0 100%
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/disks.py 130 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 178 1 99% 285
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
src/dashboards/sources/nut_sim.py 134 0 100%
src/dashboards/store.py 188 2 99% 255-256
------------------------------------------------------------------
TOTAL 2910 38 99%
Required test coverage of 94% reached. Total coverage: 98.69%
1193 passed, 11 skipped, 1 warning in 95.31s (0:01:35)
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
certifi 2026.7.22 2026.10.18 wheel
fastapi 0.143.5 0.149.0 wheel
starlette 1.8.0 1.9.0 wheel
websockets 17.2 17.3 wheel