phase: 13_llm_page_and_decode_rate

All green. Final verification summary:

**Verified / implemented (phase 13 final pass)**
- All 4 tasks already in `complete/`; this was the verification pass. Confirmed the CSS rhythm rules (`#strip`, `#banners:empty ~ #fleet`) are served and pinned, `data/logs/NOTES-decode-spike.md` names the mechanism, and the phase-07/04 surfaces still pass their pins.
- **Found and fixed a residual defect in this phase's own code**: a stalled fetch stamps its body with the poll's *finish* time, so that sample anchors every window for the rest of the span; the jump gate "confirmed" a jump measured from that same anchor → live gufo rows 232.28 tok/s (one request at ~50) + a decaying 146/114/95 tail, repeated at 200.6. `rates.py::_publish` now requires an **independent anchor** (`_pending` stores the base sample's ts; the hold continues until it ages out). Pinned by `test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor` + `test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine` (both fail on task 03's gate).
- Re-verified live after a dev-server restart: 10 min / 540 signed-in `/api/llm/snapshot` polls + read-only SQLite pass; **0 values > 300, 0 without a `Δ<window>` label**; 0 post-fix stored rows over the ceiling except the two pre-fix-shape rows above.

**Tests / lint (exact, all 100 % green)**
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1195 passed, 11 skipped, coverage 98.70 %**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; repo hygiene clean.
- `uv run pytest -m live --live` → **11 passed / 0 failed**. `/api/health` → **all nine sources ok**.

**Completion criteria**
- Fleet/strip + head/fleet gap with screenshots (`llm-fleet-spacing{,-top,-banner-case}.png`) — met (CSS unchanged this pass, pinned on served bytes).
- Task-02 findings file names the confirmed mechanism — met.
- `test_rates.py`/`test_llm_source.py` pin label-set, both window bounds and the always-honest `Δ` — met (+2 new pins).
- Sustained watch under the recorded ceiling (300 tok/s) — met (worst 203.05, all labelled).
- Full suite + validate green; no phase-07/04 behaviour change — met.

**Documentation**
- README: `RATE_WINDOW_SECONDS` row now states the hold/confirm rule (only change needed; rest verified consistent).
- AGENTS.md: `rates.py` bullet + gap 9 rewritten (residual closed *and* the remaining sub-8× skew measured, root-caused to gap 4, explicitly not fixed by inventing a new threshold).
- **Fleet facts changed under this pass and were corrected**: operator added driveripper's bearer token → `host:driveripper`/`disks:driveripper` `ok`, live suite 9/2→11/0; but driveripper reads **identity only, zero SMART numbers** (root `smartctl` gives 55 °C/14 %; `/dev/nvme0`, `/dev/ng0n1`, `/dev/bsg/*` are root-only; its unit is `DevicePolicy=auto`, not the installer's `strict`). Evidence appended to `data/logs/NOTES-agent-install.md` + `NOTES-decode-stable.md`.

**Notable decisions / deviations**
- Did **not** add a span-relative window floor: that would re-decide task 03's pinned `MIN_WINDOW_SECONDS` and its "in-bounds diluted row publishes" test. Recorded as the next phase's call, with the root cause left at gap 4 / `14_store_performance` (the ~10 s stall).
- No live fleet change made (read-only probes + one dev-server restart). Fixtures deliberately not re-captured (no parse change, task 02 proved no shape drift). Dependency-currency advisory left untouched (no dependency moved in this phase): certifi/fastapi/starlette/websockets have newer releases.

**Next pending phase:** `.agents/phases/todo/14_store_performance` (then `15_agent_mtls`).

tests/test_llm_source.py

@@ -15,6 +15,7 @@ from dashboards.sources.llm import (
     fleet_totals,
     parse_gufo_metrics,
     parse_vllm_metrics,
+    rate_key,
     series_names,
 )
 from slots_contract import load_slots_capture
@@ -94,6 +95,58 @@ def test_gufo_counter_reset_produces_n_a_not_a_spike(metrics_text) -> None:
     assert metrics.get("decode_tokens_per_second", {}).get("value") is None
 
 
+def test_gufo_a_stalled_poll_then_the_catch_up_is_not_a_spike(metrics_text) -> None:
+    """The exact stored incident, through the production parse path.
+
+    2026-10-18 09:16:31 on deskwork: a 15 s gap in every source, a first result carrying 114
+    tokens (its body was generated ~2 s into the stall), then the poll loop's 0.2 s catch-up
+    printing **3284.59 tok/s** for an engine doing 53. `test_rates.py` holds the estimator-level
+    pair; this is the same defect where the operator sees it.
+    """
+    text = metrics_text("gufo")
+    rates = RateEstimator(span=10.0)
+    parse_gufo_metrics(text, rates, 1_000.0)
+    text = bump_metric(text, "llamacpp:tokens_predicted_total", 114.0)
+    parse_gufo_metrics(text, rates, 1_015.25)
+    text = bump_metric(text, "llamacpp:tokens_predicted_total", 713.0)
+    metrics = parse_gufo_metrics(text, rates, 1_015.47)
+    cell = metrics["decode_tokens_per_second"]
+    assert cell["window"] == pytest.approx(0.22)
+    assert cell["value"] is None or cell["value"] < 100.0, (
+        f"53 tok/s of real work printed as {cell['value']:.0f} tok/s")
+
+
+def test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine(metrics_text) -> None:
+    """The other half of the stall: the reply is stamped when the poll *finished*, not when the
+    engine rendered it, so its counter is the stall's length older than its own timestamp.
+
+    That sample then anchors every window for the rest of the span and each poll re-diffs the
+    same skewed delta over a longer one — the live 2026-10-19 12:42 incident printed 232 tok/s
+    for one request running at ~50, then a decaying 146 / 114 / 95 tail. The estimator's jump
+    gate holds the first of those and now refuses to call a window anchored on the same sample a
+    confirmation, so nothing above the engine's real rate reaches the payload.
+    """
+    text = metrics_text("gufo")
+    rates = RateEstimator(span=10.0)
+    parse_gufo_metrics(text, rates, 1_000.0)
+    # (stamp, tokens the engine had really produced by then): the 1011 reply is the stalled one —
+    # its body was rendered ~9 s before the poll stamped it — and from there the counter advances
+    # at a steady 45 tok/s, so 45 is the only rate that is true to print.
+    timeline = [(1_011.0, 90.0), (1_012.0, 540.0), (1_014.0, 630.0), (1_016.0, 720.0),
+                (1_018.0, 810.0), (1_022.0, 990.0), (1_026.0, 1_170.0)]
+    produced = 0.0
+    cells = []
+    for stamp, total in timeline:
+        text = bump_metric(text, "llamacpp:tokens_predicted_total", total - produced)
+        produced = total
+        cells.append(parse_gufo_metrics(text, rates, stamp)["decode_tokens_per_second"])
+    for cell in cells:
+        assert cell["value"] is None or cell["value"] <= 100.0, \
+            f"a 45 tok/s engine printed {cell['value']:.0f} tok/s ({cells})"
+    assert cells[-1]["value"] == pytest.approx(45.0), "the stall ends in the truth, not a tail"
+    assert cells[-1]["window"] == pytest.approx(10.0)
+
+
 def test_absent_gufo_metrics_are_none_with_no_source() -> None:
     metrics = parse_gufo_metrics("# nothing here\n")
     assert metrics["requests_processing"]["value"] is None
@@ -191,6 +244,79 @@ def test_vllm_measured_rate_when_the_build_has_no_throughput_gauge(metrics_text)
     assert metrics["decode_tokens_per_second"]["value"] == pytest.approx(25.0)
 
 
+def test_vllm_a_joining_label_set_does_not_lend_its_carried_counter(metrics_text) -> None:
+    """A second `engine=`/`model_name=` series appearing mid-window must not be a rate.
+
+    vLLM's counters are labelled, so `scalar(agg="sum")` added whatever label sets exist right
+    now. A rank joining (or a reloaded model whose old series is still exported) carries its own
+    cumulative counter, and `value < samples[-1]` never fires for an *increase*, so the carried
+    value was diffed. Confirmed as a defect, not as this fleet's trigger: `generation_tokens_total`
+    has exactly one label set live (`{engine="0",model_name="juggernaut"}`). Fixed in task 03 by
+    diffing each label set under its own estimator key — the joining series has no baseline and
+    contributes 0 tokens until it has one.
+    """
+    text = metrics_text("vllm")
+    rates = RateEstimator(span=10.0)
+    parse_vllm_metrics(text, rates, 2_000.0)
+    gained = text + 'vllm:generation_tokens_total{engine="1",model_name="juggernaut"} 500000.0\n'
+    metrics = parse_vllm_metrics(gained, rates, 2_010.0)
+    cell = metrics["decode_tokens_per_second"]
+    assert cell["value"] is None or cell["value"] < 100.0, (
+        f"a joining label set printed {cell['value']:.0f} tok/s")
+
+
+def test_vllm_a_label_set_that_survives_starts_contributing_its_own_delta(metrics_text) -> None:
+    """Fresh series contribute nothing on their first poll and their real Δ on the second."""
+    text = metrics_text("vllm")
+    rates = RateEstimator(span=10.0)
+    parse_vllm_metrics(text, rates, 2_000.0)
+    gained = text + 'vllm:generation_tokens_total{engine="1",model_name="juggernaut"} 500000.0\n'
+    metrics = parse_vllm_metrics(gained, rates, 2_010.0)
+    assert metrics["decode_tokens_per_second"]["value"] == pytest.approx(0.0), \
+        "the joining series is baseline-less: 500 000 carried, 0 tokens diffed"
+    grown = bump_metric(gained, "vllm:generation_tokens_total", 100.0)  # bumps every label set
+    metrics = parse_vllm_metrics(grown, rates, 2_020.0)
+    assert metrics["decode_tokens_per_second"]["value"] == pytest.approx(20.0), \
+        "two series x 100 tokens / 10 s: the rate is the sum of the per-series rates"
+    assert metrics["decode_tokens_per_second"]["window"] == pytest.approx(10.0)
+
+
+def test_gufo_measured_cell_keeps_the_provenance_shape(metrics_text) -> None:
+    """Rule 2 after the per-series rewrite: value, `computed:Δ<series>/Δt`, `Δ<window>`."""
+    text = metrics_text("gufo")
+    rates = RateEstimator(span=10)
+    parse_gufo_metrics(text, rates, 1_000.0)
+    text = bump_metric(text, "llamacpp:tokens_predicted_total", 400.0)
+    decode = parse_gufo_metrics(text, rates, 1_010.0)["decode_tokens_per_second"]
+    assert decode["value"] == pytest.approx(40.0)
+    assert decode["derived"] is True
+    assert decode["source"] == "computed:Δllamacpp:tokens_predicted_total/Δt"
+    assert decode["window"] == pytest.approx(10.0), "the widest window a contributing used"
+
+
+def test_gufo_a_refused_window_is_n_a_and_names_the_window_it_refused(metrics_text) -> None:
+    """The stall refusal is an answer: value None, but the series and Δ still travel."""
+    text = metrics_text("gufo")
+    rates = RateEstimator(span=10)
+    parse_gufo_metrics(text, rates, 1_000.0)
+    text = bump_metric(text, "llamacpp:tokens_predicted_total", 50.0)
+    parse_gufo_metrics(text, rates, 1_010.0)
+    text = bump_metric(text, "llamacpp:tokens_predicted_total", 60.0)
+    decode = parse_gufo_metrics(text, rates, 1_010.3)["decode_tokens_per_second"]
+    assert decode["value"] is None, "0.3 s is the loop's catch-up gap, not a rate"
+    assert decode["window"] == pytest.approx(0.3)
+    assert decode["source"] == "computed:Δllamacpp:tokens_predicted_total/Δt"
+
+
+def test_rate_key_puts_the_label_set_in_the_estimator_key() -> None:
+    """The key is the resolved name plus its labels, label order included-stable."""
+    assert rate_key("llamacpp:tokens_predicted_total", {}) == "llamacpp:tokens_predicted_total"
+    assert rate_key("vllm:generation_tokens_total", {"model_name": "m", "engine": "0"}) == \
+        'vllm:generation_tokens_total{engine="0",model_name="m"}'
+    assert rate_key("m", {"weird": 'a"b\\c'}) == 'm{weird="a\\"b\\\\c"}', \
+        "a quote in a label value must not splice two keys into one"
+
+
 def test_vllm_waiting_requests_are_broken_down_by_reason(metrics_text) -> None:
     metrics = parse_vllm_metrics(metrics_text("vllm"))
     reasons = metrics["waiting_by_reason"]["value"]