phase: 13_llm_page_and_decode_rate
All green. Final verification summary:
**Verified / implemented (phase 13 final pass)**
- All 4 tasks already in `complete/`; this was the verification pass. Confirmed the CSS rhythm rules (`#strip`, `#banners:empty ~ #fleet`) are served and pinned, `data/logs/NOTES-decode-spike.md` names the mechanism, and the phase-07/04 surfaces still pass their pins.
- **Found and fixed a residual defect in this phase's own code**: a stalled fetch stamps its body with the poll's *finish* time, so that sample anchors every window for the rest of the span; the jump gate "confirmed" a jump measured from that same anchor → live gufo rows 232.28 tok/s (one request at ~50) + a decaying 146/114/95 tail, repeated at 200.6. `rates.py::_publish` now requires an **independent anchor** (`_pending` stores the base sample's ts; the hold continues until it ages out). Pinned by `test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor` + `test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine` (both fail on task 03's gate).
- Re-verified live after a dev-server restart: 10 min / 540 signed-in `/api/llm/snapshot` polls + read-only SQLite pass; **0 values > 300, 0 without a `Δ<window>` label**; 0 post-fix stored rows over the ceiling except the two pre-fix-shape rows above.
**Tests / lint (exact, all 100 % green)**
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1195 passed, 11 skipped, coverage 98.70 %**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; repo hygiene clean.
- `uv run pytest -m live --live` → **11 passed / 0 failed**. `/api/health` → **all nine sources ok**.
**Completion criteria**
- Fleet/strip + head/fleet gap with screenshots (`llm-fleet-spacing{,-top,-banner-case}.png`) — met (CSS unchanged this pass, pinned on served bytes).
- Task-02 findings file names the confirmed mechanism — met.
- `test_rates.py`/`test_llm_source.py` pin label-set, both window bounds and the always-honest `Δ` — met (+2 new pins).
- Sustained watch under the recorded ceiling (300 tok/s) — met (worst 203.05, all labelled).
- Full suite + validate green; no phase-07/04 behaviour change — met.
**Documentation**
- README: `RATE_WINDOW_SECONDS` row now states the hold/confirm rule (only change needed; rest verified consistent).
- AGENTS.md: `rates.py` bullet + gap 9 rewritten (residual closed *and* the remaining sub-8× skew measured, root-caused to gap 4, explicitly not fixed by inventing a new threshold).
- **Fleet facts changed under this pass and were corrected**: operator added driveripper's bearer token → `host:driveripper`/`disks:driveripper` `ok`, live suite 9/2→11/0; but driveripper reads **identity only, zero SMART numbers** (root `smartctl` gives 55 °C/14 %; `/dev/nvme0`, `/dev/ng0n1`, `/dev/bsg/*` are root-only; its unit is `DevicePolicy=auto`, not the installer's `strict`). Evidence appended to `data/logs/NOTES-agent-install.md` + `NOTES-decode-stable.md`.
**Notable decisions / deviations**
- Did **not** add a span-relative window floor: that would re-decide task 03's pinned `MIN_WINDOW_SECONDS` and its "in-bounds diluted row publishes" test. Recorded as the next phase's call, with the root cause left at gap 4 / `14_store_performance` (the ~10 s stall).
- No live fleet change made (read-only probes + one dev-server restart). Fixtures deliberately not re-captured (no parse change, task 02 proved no shape drift). Dependency-currency advisory left untouched (no dependency moved in this phase): certifi/fastapi/starlette/websockets have newer releases.
**Next pending phase:** `.agents/phases/todo/14_store_performance` (then `15_agent_mtls`).
This commit is contained in:
@@ -2421,6 +2421,96 @@ def test_llm_fleet_block_colours_no_number(client) -> None:
|
||||
"the fleet block's head row lost its styling"
|
||||
|
||||
|
||||
#: Elements the browser treats as empty, so they never open a nesting level.
|
||||
_VOID_TAGS = {"meta", "link", "br", "img", "input", "hr", "source", "track", "wbr"}
|
||||
|
||||
|
||||
def _depth_at(html: str, needle: str) -> int:
|
||||
"""How deeply the `needle` sits inside `<main>` (1 = a direct child of it).
|
||||
|
||||
The `~` in `#banners:empty ~ #fleet` only resolves between **siblings**, so the gap rule
|
||||
depends on the markup as much as on the stylesheet. Counting tags is enough here and keeps
|
||||
the check on the served bytes: no HTML parser, no dependency, and comments are already
|
||||
gone by the time `_code` hands the page over.
|
||||
"""
|
||||
body = html[html.index("<main>"):html.index(needle, html.index("<main>"))]
|
||||
depth = 0
|
||||
for closing, tag, _attrs, self_closed in re.findall(
|
||||
r"<(/?)([a-zA-Z][\w-]*)([^>]*?)(/?)>", body):
|
||||
if self_closed or tag.lower() in _VOID_TAGS:
|
||||
continue
|
||||
depth += -1 if closing else 1
|
||||
return depth
|
||||
|
||||
|
||||
def test_llm_section_stack_keeps_the_shared_vertical_rhythm(client) -> None:
|
||||
"""Fleet block → engine summaries → engine panels sit the shared `.85rem` apart.
|
||||
|
||||
`TODO.md:4`: the fleet section had no spacing between it and the engine summaries. Every
|
||||
boundary below it was already handled inline — the section under `#strip` carries
|
||||
`margin-top:.85rem` — so `#fleet` and `#strip` skipped the rhythm and the top of the page
|
||||
read as one unbroken wall. Measured in Chromium at the 16px root (`data/logs/`
|
||||
`llm-fleet-spacing*.png`), before → after:
|
||||
|
||||
| gap | before | after |
|
||||
|-----|--------|-------|
|
||||
| `#fleet` → `#strip` | **0 (flush)** | 13.59px |
|
||||
| `#strip` → engine panels | 13.59px | 13.59px |
|
||||
| banner slot → `#fleet`, no banner | 13.59px | 13.59px |
|
||||
| banner slot → `#fleet`, one banner up | 13.59px | 13.59px (`margin-top: 0px`) |
|
||||
|
||||
One rendered distance moves, and it is the one the operator named. What the *shape* of the
|
||||
two rules buys is that no other state can put it back:
|
||||
|
||||
* `#strip`'s gap is **unconditional**, because below two engines `#fleet` is `[hidden]`
|
||||
(`display:none !important`, so it contributes nothing) and the strip is then the first
|
||||
visible section: the 13.59px below the banner slot in that state is now the strip's own
|
||||
margin rather than one borrowed from a slot above it, which is what `#fleet ~ #strip`
|
||||
would have been;
|
||||
* `#fleet`'s gap is guarded by `#banners:empty`, so the block owns its top gap exactly when
|
||||
nothing occupies the slot above it, and a banner's own `margin-bottom:.85rem` stays the
|
||||
single gap under it. An unguarded `#fleet { margin-top }` is a second rule for one gap:
|
||||
margin collapsing hides it at `.85rem`/`.85rem` today, and it doubles the moment either
|
||||
margin is changed. The empty slot keeps its inline bottom margin, so head → fleet stays
|
||||
the 1rem + .85rem the other two banner pages (`/`, `/ups`) already render.
|
||||
|
||||
Pinned on the served bytes because a stylesheet "cleanup" that drops either rule is
|
||||
invisible in a diff and only shows up as a page that reads worse.
|
||||
"""
|
||||
raw = _css(client)
|
||||
section = _css_section(raw, "fleet totals")
|
||||
assert "#strip { margin-top: .85rem; }" in section, \
|
||||
"the fleet block and the engine summaries render flush again (TODO.md:4)"
|
||||
assert "#banners:empty ~ #fleet { margin-top: .85rem; }" in section, (
|
||||
"the gap above the fleet block stopped belonging to the block: a banner-less page "
|
||||
"would then have it only as long as the empty banner slot keeps its inline margin")
|
||||
|
||||
# one rule per element: a second one (or an unguarded `#fleet`) is the double gap
|
||||
css = _code(raw)
|
||||
for element, expected in (("#strip", {"#strip"}),
|
||||
("#fleet", {"#banners:empty ~ #fleet"})):
|
||||
selectors = {s.strip() for s in re.findall(rf"([^{{}}]*{element}[^{{}}]*)\{{", css)}
|
||||
assert selectors == expected, (
|
||||
f"dash.css styles {element} with {sorted(selectors)} — exactly one rule, the "
|
||||
"guarded one, or the page grows a double gap or loses the single one")
|
||||
|
||||
html = _code(client.get("/llm").text)
|
||||
for element in ("banners", "fleet", "strip"):
|
||||
assert _depth_at(html, f'id="{element}"') == 1, (
|
||||
f"`#{element}` is no longer a direct child of <main>, so the `~` gap rule cannot "
|
||||
"match: rewrite the selectors for the new structure instead of dropping the gap")
|
||||
assert html.index('id="banners"') < html.index('id="fleet"') < html.index('id="strip"') \
|
||||
< html.index('id="engines"'), "the section stack was re-ordered"
|
||||
|
||||
# the inline half of the rhythm, which the two rules are written around
|
||||
assert 'style="margin-bottom:.85rem"' in html[:html.index('id="fleet"')], (
|
||||
"#banners lost the inline bottom margin that is the banner → fleet gap, and today the "
|
||||
"only thing spacing the page head from the block")
|
||||
after_strip = html[html.index('id="strip"'):]
|
||||
assert after_strip.count('style="margin-top:.85rem"') == 3, \
|
||||
"a section below the strip lost the inline gap the strip's rule hands off to"
|
||||
|
||||
|
||||
# ---- the `/llm` gufo slot grid (phase 07 task 04) -------------------------
|
||||
#
|
||||
# `llm_monitor`'s slot cards (`index.html` ~L1179), rebuilt against the capture instead of a
|
||||
|
||||
@@ -62,7 +62,10 @@ async def test_a_real_agent_snapshot_parses() -> None:
|
||||
return httpx.Response(200, json=body)
|
||||
|
||||
target = Target("driveripper", "http://agent.test")
|
||||
collector = HostCollector(RateEstimator())
|
||||
# min_window=0: these fetches land microseconds apart, far under any real poll
|
||||
# cadence; this test pins the Δjiffies window semantics, not the phase-13 floor
|
||||
# (tests/test_rates.py pins the floor at a real cadence).
|
||||
collector = HostCollector(RateEstimator(min_window=0.0))
|
||||
async with client_for(handler) as client:
|
||||
first = (await fetch_host_snapshot(client, target, collector)).payload
|
||||
# CPU usage is Δjiffies/Δt: one scrape has no window, so it is n/a and not 0 %
|
||||
@@ -392,7 +395,9 @@ async def test_second_gufo_poll_measures_throughput_over_a_window() -> None:
|
||||
return httpx.Response(200, json=[{"index": 0, "n_ctx": 8192, "predicted": 12}])
|
||||
return httpx.Response(404)
|
||||
|
||||
rates = RateEstimator(span=10.0)
|
||||
# min_window=0: the two fetches are microseconds apart (no real loop polls this
|
||||
# fast); this test pins that the measured cell divides by the window it labels.
|
||||
rates = RateEstimator(span=10.0, min_window=0.0)
|
||||
async with client_for(handler) as client:
|
||||
await fetch_engine(client, "gufo", "http://engine.test", "", rates)
|
||||
state["text"] = bump(GUFO, "llamacpp:tokens_predicted_total", 40.0)
|
||||
|
||||
@@ -15,6 +15,7 @@ from dashboards.sources.llm import (
|
||||
fleet_totals,
|
||||
parse_gufo_metrics,
|
||||
parse_vllm_metrics,
|
||||
rate_key,
|
||||
series_names,
|
||||
)
|
||||
from slots_contract import load_slots_capture
|
||||
@@ -94,6 +95,58 @@ def test_gufo_counter_reset_produces_n_a_not_a_spike(metrics_text) -> None:
|
||||
assert metrics.get("decode_tokens_per_second", {}).get("value") is None
|
||||
|
||||
|
||||
def test_gufo_a_stalled_poll_then_the_catch_up_is_not_a_spike(metrics_text) -> None:
|
||||
"""The exact stored incident, through the production parse path.
|
||||
|
||||
2026-10-18 09:16:31 on deskwork: a 15 s gap in every source, a first result carrying 114
|
||||
tokens (its body was generated ~2 s into the stall), then the poll loop's 0.2 s catch-up
|
||||
printing **3284.59 tok/s** for an engine doing 53. `test_rates.py` holds the estimator-level
|
||||
pair; this is the same defect where the operator sees it.
|
||||
"""
|
||||
text = metrics_text("gufo")
|
||||
rates = RateEstimator(span=10.0)
|
||||
parse_gufo_metrics(text, rates, 1_000.0)
|
||||
text = bump_metric(text, "llamacpp:tokens_predicted_total", 114.0)
|
||||
parse_gufo_metrics(text, rates, 1_015.25)
|
||||
text = bump_metric(text, "llamacpp:tokens_predicted_total", 713.0)
|
||||
metrics = parse_gufo_metrics(text, rates, 1_015.47)
|
||||
cell = metrics["decode_tokens_per_second"]
|
||||
assert cell["window"] == pytest.approx(0.22)
|
||||
assert cell["value"] is None or cell["value"] < 100.0, (
|
||||
f"53 tok/s of real work printed as {cell['value']:.0f} tok/s")
|
||||
|
||||
|
||||
def test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine(metrics_text) -> None:
|
||||
"""The other half of the stall: the reply is stamped when the poll *finished*, not when the
|
||||
engine rendered it, so its counter is the stall's length older than its own timestamp.
|
||||
|
||||
That sample then anchors every window for the rest of the span and each poll re-diffs the
|
||||
same skewed delta over a longer one — the live 2026-10-19 12:42 incident printed 232 tok/s
|
||||
for one request running at ~50, then a decaying 146 / 114 / 95 tail. The estimator's jump
|
||||
gate holds the first of those and now refuses to call a window anchored on the same sample a
|
||||
confirmation, so nothing above the engine's real rate reaches the payload.
|
||||
"""
|
||||
text = metrics_text("gufo")
|
||||
rates = RateEstimator(span=10.0)
|
||||
parse_gufo_metrics(text, rates, 1_000.0)
|
||||
# (stamp, tokens the engine had really produced by then): the 1011 reply is the stalled one —
|
||||
# its body was rendered ~9 s before the poll stamped it — and from there the counter advances
|
||||
# at a steady 45 tok/s, so 45 is the only rate that is true to print.
|
||||
timeline = [(1_011.0, 90.0), (1_012.0, 540.0), (1_014.0, 630.0), (1_016.0, 720.0),
|
||||
(1_018.0, 810.0), (1_022.0, 990.0), (1_026.0, 1_170.0)]
|
||||
produced = 0.0
|
||||
cells = []
|
||||
for stamp, total in timeline:
|
||||
text = bump_metric(text, "llamacpp:tokens_predicted_total", total - produced)
|
||||
produced = total
|
||||
cells.append(parse_gufo_metrics(text, rates, stamp)["decode_tokens_per_second"])
|
||||
for cell in cells:
|
||||
assert cell["value"] is None or cell["value"] <= 100.0, \
|
||||
f"a 45 tok/s engine printed {cell['value']:.0f} tok/s ({cells})"
|
||||
assert cells[-1]["value"] == pytest.approx(45.0), "the stall ends in the truth, not a tail"
|
||||
assert cells[-1]["window"] == pytest.approx(10.0)
|
||||
|
||||
|
||||
def test_absent_gufo_metrics_are_none_with_no_source() -> None:
|
||||
metrics = parse_gufo_metrics("# nothing here\n")
|
||||
assert metrics["requests_processing"]["value"] is None
|
||||
@@ -191,6 +244,79 @@ def test_vllm_measured_rate_when_the_build_has_no_throughput_gauge(metrics_text)
|
||||
assert metrics["decode_tokens_per_second"]["value"] == pytest.approx(25.0)
|
||||
|
||||
|
||||
def test_vllm_a_joining_label_set_does_not_lend_its_carried_counter(metrics_text) -> None:
|
||||
"""A second `engine=`/`model_name=` series appearing mid-window must not be a rate.
|
||||
|
||||
vLLM's counters are labelled, so `scalar(agg="sum")` added whatever label sets exist right
|
||||
now. A rank joining (or a reloaded model whose old series is still exported) carries its own
|
||||
cumulative counter, and `value < samples[-1]` never fires for an *increase*, so the carried
|
||||
value was diffed. Confirmed as a defect, not as this fleet's trigger: `generation_tokens_total`
|
||||
has exactly one label set live (`{engine="0",model_name="juggernaut"}`). Fixed in task 03 by
|
||||
diffing each label set under its own estimator key — the joining series has no baseline and
|
||||
contributes 0 tokens until it has one.
|
||||
"""
|
||||
text = metrics_text("vllm")
|
||||
rates = RateEstimator(span=10.0)
|
||||
parse_vllm_metrics(text, rates, 2_000.0)
|
||||
gained = text + 'vllm:generation_tokens_total{engine="1",model_name="juggernaut"} 500000.0\n'
|
||||
metrics = parse_vllm_metrics(gained, rates, 2_010.0)
|
||||
cell = metrics["decode_tokens_per_second"]
|
||||
assert cell["value"] is None or cell["value"] < 100.0, (
|
||||
f"a joining label set printed {cell['value']:.0f} tok/s")
|
||||
|
||||
|
||||
def test_vllm_a_label_set_that_survives_starts_contributing_its_own_delta(metrics_text) -> None:
|
||||
"""Fresh series contribute nothing on their first poll and their real Δ on the second."""
|
||||
text = metrics_text("vllm")
|
||||
rates = RateEstimator(span=10.0)
|
||||
parse_vllm_metrics(text, rates, 2_000.0)
|
||||
gained = text + 'vllm:generation_tokens_total{engine="1",model_name="juggernaut"} 500000.0\n'
|
||||
metrics = parse_vllm_metrics(gained, rates, 2_010.0)
|
||||
assert metrics["decode_tokens_per_second"]["value"] == pytest.approx(0.0), \
|
||||
"the joining series is baseline-less: 500 000 carried, 0 tokens diffed"
|
||||
grown = bump_metric(gained, "vllm:generation_tokens_total", 100.0) # bumps every label set
|
||||
metrics = parse_vllm_metrics(grown, rates, 2_020.0)
|
||||
assert metrics["decode_tokens_per_second"]["value"] == pytest.approx(20.0), \
|
||||
"two series x 100 tokens / 10 s: the rate is the sum of the per-series rates"
|
||||
assert metrics["decode_tokens_per_second"]["window"] == pytest.approx(10.0)
|
||||
|
||||
|
||||
def test_gufo_measured_cell_keeps_the_provenance_shape(metrics_text) -> None:
|
||||
"""Rule 2 after the per-series rewrite: value, `computed:Δ<series>/Δt`, `Δ<window>`."""
|
||||
text = metrics_text("gufo")
|
||||
rates = RateEstimator(span=10)
|
||||
parse_gufo_metrics(text, rates, 1_000.0)
|
||||
text = bump_metric(text, "llamacpp:tokens_predicted_total", 400.0)
|
||||
decode = parse_gufo_metrics(text, rates, 1_010.0)["decode_tokens_per_second"]
|
||||
assert decode["value"] == pytest.approx(40.0)
|
||||
assert decode["derived"] is True
|
||||
assert decode["source"] == "computed:Δllamacpp:tokens_predicted_total/Δt"
|
||||
assert decode["window"] == pytest.approx(10.0), "the widest window a contributing used"
|
||||
|
||||
|
||||
def test_gufo_a_refused_window_is_n_a_and_names_the_window_it_refused(metrics_text) -> None:
|
||||
"""The stall refusal is an answer: value None, but the series and Δ still travel."""
|
||||
text = metrics_text("gufo")
|
||||
rates = RateEstimator(span=10)
|
||||
parse_gufo_metrics(text, rates, 1_000.0)
|
||||
text = bump_metric(text, "llamacpp:tokens_predicted_total", 50.0)
|
||||
parse_gufo_metrics(text, rates, 1_010.0)
|
||||
text = bump_metric(text, "llamacpp:tokens_predicted_total", 60.0)
|
||||
decode = parse_gufo_metrics(text, rates, 1_010.3)["decode_tokens_per_second"]
|
||||
assert decode["value"] is None, "0.3 s is the loop's catch-up gap, not a rate"
|
||||
assert decode["window"] == pytest.approx(0.3)
|
||||
assert decode["source"] == "computed:Δllamacpp:tokens_predicted_total/Δt"
|
||||
|
||||
|
||||
def test_rate_key_puts_the_label_set_in_the_estimator_key() -> None:
|
||||
"""The key is the resolved name plus its labels, label order included-stable."""
|
||||
assert rate_key("llamacpp:tokens_predicted_total", {}) == "llamacpp:tokens_predicted_total"
|
||||
assert rate_key("vllm:generation_tokens_total", {"model_name": "m", "engine": "0"}) == \
|
||||
'vllm:generation_tokens_total{engine="0",model_name="m"}'
|
||||
assert rate_key("m", {"weird": 'a"b\\c'}) == 'm{weird="a\\"b\\\\c"}', \
|
||||
"a quote in a label value must not splice two keys into one"
|
||||
|
||||
|
||||
def test_vllm_waiting_requests_are_broken_down_by_reason(metrics_text) -> None:
|
||||
metrics = parse_vllm_metrics(metrics_text("vllm"))
|
||||
reasons = metrics["waiting_by_reason"]["value"]
|
||||
|
||||
@@ -33,6 +33,23 @@ def test_labels_are_parsed_and_filterable(metrics_text) -> None:
|
||||
assert by_model and all(model for model in by_model)
|
||||
|
||||
|
||||
def test_values_by_series_keeps_the_label_sets_apart() -> None:
|
||||
"""Counter deltas diff per label set; the summed view would let a joining series
|
||||
lend its carried value to one interval (phase 13). This is the accessor that view."""
|
||||
prom = Prom('x{a="1"} 5.0\nx{a="2"} 7.0\ny 1.0\n')
|
||||
assert prom.values_by_series(["x"]) == [({"a": "1"}, 5.0), ({"a": "2"}, 7.0)]
|
||||
assert prom.values_by_series(["y"]) == [({}, 1.0)]
|
||||
assert prom.values_by_series(["x"], {"a": "2"}) == [({"a": "2"}, 7.0)]
|
||||
assert prom.values_by_series(["nope"]) == []
|
||||
|
||||
|
||||
def test_values_by_series_on_a_real_labelled_capture(metrics_text) -> None:
|
||||
prom = Prom(metrics_text("vllm"))
|
||||
series = prom.values_by_series(["vllm:generation_tokens_total"])
|
||||
assert series, "vLLM exposes generation_tokens_total"
|
||||
assert all(labels.get("model_name") for labels, _v in series), "the capture lost its labels"
|
||||
|
||||
|
||||
def test_aggregations(metrics_text) -> None:
|
||||
prom = Prom(metrics_text("vllm"))
|
||||
total = prom.scalar(["vllm:prompt_tokens_total"], agg="sum")
|
||||
|
||||
@@ -52,6 +52,203 @@ def test_nan_and_inf_are_treated_as_missing() -> None:
|
||||
assert r.rate("x", bad, now=1.0) == (None, None)
|
||||
|
||||
|
||||
# --- phase 13: the live decode-tok/s spike ---------------------------------- #
|
||||
# Confirmed against the stored history and reproduced on live hardware; the findings are in
|
||||
# data/logs/NOTES-decode-spike.md. These two tests were written RED (as strict xfails) by
|
||||
# task 02 and turned GREEN by task 03's bounded window; the markers were removed and every
|
||||
# assertion kept verbatim, as the phase mandated. They are permanent pins now.
|
||||
#
|
||||
# The incident, from `sample` (source=llm, host=gufo, metric=llm.decode_tokens_per_second,
|
||||
# res=raw) on 2026-10-18, with the counter value reconstructed from the rows' own windows:
|
||||
#
|
||||
# 09:16:15.839 53.64 tok/s window ~10 s (steady work on deskwork: 53 tok/s)
|
||||
# ... 15.25 s with no sample at all: every source in the process gaps at the same instant ...
|
||||
# 09:16:31.091 7.47 tok/s window 15.25 s (114 tokens — a body generated ~2 s into it)
|
||||
# 09:16:31.308 3284.59 tok/s window 0.217 s (713 tokens — the rest of the stall)
|
||||
# 09:16:33.311 371.23 tok/s window 2.22 s (the stale sample is still base[0])
|
||||
# 09:16:35.315 222.32, 09:16:37.318 166.23, 09:16:39.316 138.37, then back to ~50
|
||||
#
|
||||
# 827 tokens across the whole 15.47 s is 53.5 tok/s — the engine's true rate. The tokens are
|
||||
# real; the window is the lie. `probe-decode-spike2.py` produced the same row pair (and nearly
|
||||
# the same tail: 379 / 232 / 174 / 140) against live gufo with the production parser.
|
||||
|
||||
GUFO_TOKENS = "llamacpp:tokens_predicted_total"
|
||||
|
||||
|
||||
def stalled_estimator(span: float = 10.0) -> RateEstimator:
|
||||
"""A window fed the steady 2 s cadence at exactly 50 tok/s."""
|
||||
r = RateEstimator(span=span)
|
||||
for i in range(6):
|
||||
r.rate(GUFO_TOKENS, 100.0 * i, now=1_000.0 + 2.0 * i)
|
||||
return r
|
||||
|
||||
|
||||
def test_a_catch_up_poll_after_a_stall_answers_na_not_a_spike() -> None:
|
||||
"""A fetch that takes 15 s and the loop's 0.2 s catch-up floor must not make a rate.
|
||||
|
||||
`poller.Job._loop` sleeps `max(0.2, interval - elapsed)`, so any poll slower than the
|
||||
interval is followed by a second one ~0.2 s later. Whatever the stalled fetch carried is
|
||||
then diffed over that 0.2 s.
|
||||
"""
|
||||
r = stalled_estimator() # steady 50 tok/s, last sample (1010.0, 500.0)
|
||||
# the stalled fetch: 15.25 s after the last sample, but its body was generated ~2 s in,
|
||||
# so it carries 114 tokens and gets stamped 13 s later than the engine measured them
|
||||
diluted, _window = r.rate(GUFO_TOKENS, 614.0, now=1_025.25)
|
||||
assert diluted == pytest.approx(7.47, abs=0.01) # the diluted row the DB actually holds
|
||||
# the catch-up: 0.22 s later, the current counter (713 tokens of stall still unaccounted)
|
||||
rate, window = r.rate(GUFO_TOKENS, 1_327.0, now=1_025.47)
|
||||
assert window == pytest.approx(0.22)
|
||||
assert rate is None, "a 0.22 s window is not a rate over the span; it must be n/a"
|
||||
|
||||
|
||||
def test_a_stalled_sample_does_not_inflate_the_polls_after_it() -> None:
|
||||
"""The tail is as wrong as the spike: base[0] is the stale sample for the next ~span."""
|
||||
r = stalled_estimator()
|
||||
r.rate(GUFO_TOKENS, 614.0, now=1_025.25)
|
||||
r.rate(GUFO_TOKENS, 1_327.0, now=1_025.47)
|
||||
# back to a normal 2 s cadence and the same honest 50 tok/s
|
||||
tail = [r.rate(GUFO_TOKENS, 1_327.0 + 100.0 * n, now=1_029.47 + 2.0 * (n - 1))[0]
|
||||
for n in range(1, 5)]
|
||||
assert all(v is None or v <= 60.0 for v in tail), f"inflated tail, real rate is 50: {tail}"
|
||||
|
||||
|
||||
# --- phase 13 task 03: both window bounds and the confirmed jump ------------- #
|
||||
|
||||
def test_window_below_the_floor_answers_na_with_its_honest_label() -> None:
|
||||
"""A 0.4 s catch-up gap is not a measurement over a 10 s span; it labels itself."""
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("x", 100.0, now=0.0)
|
||||
rate, window = r.rate("x", 103.0, now=0.4)
|
||||
assert rate is None
|
||||
assert window == pytest.approx(0.4), "the n/a still names the window it refused"
|
||||
# the refused base was dropped: the next normal poll measures from the catch-up sample
|
||||
rate, window = r.rate("x", 203.0, now=4.4)
|
||||
assert rate == pytest.approx(25.0)
|
||||
assert window == pytest.approx(4.0)
|
||||
|
||||
|
||||
def test_fallback_base_older_than_two_spans_answers_na_and_recovers() -> None:
|
||||
"""A 20-second window on a 10-second panel is a different claim — one n/a, then normal."""
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("x", 0.0, now=0.0)
|
||||
assert r.rate("x", 50.0, now=30.0) == (None, None)
|
||||
rate, window = r.rate("x", 100.0, now=32.0)
|
||||
assert rate == pytest.approx(25.0)
|
||||
assert window == pytest.approx(2.0), "the next poll re-bases on the fresh sample"
|
||||
|
||||
|
||||
def test_a_fallback_inside_the_documented_ceiling_still_publishes() -> None:
|
||||
"""The stall row the DB actually holds (15.25 s ≤ 2 x 10 s) keeps its diluted rate."""
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("x", 0.0, now=0.0)
|
||||
rate, window = r.rate("x", 100.0, now=15.25)
|
||||
assert rate == pytest.approx(100.0 / 15.25)
|
||||
assert window == pytest.approx(15.25)
|
||||
|
||||
|
||||
def test_two_samples_at_the_same_instant_are_not_a_rate() -> None:
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("x", 100.0, now=5.0)
|
||||
assert r.rate("x", 200.0, now=5.0) == (None, None)
|
||||
|
||||
|
||||
def test_a_jump_is_held_for_one_poll_and_publishes_when_confirmed() -> None:
|
||||
"""A real step-up survives one more interval and then prints — one poll late.
|
||||
|
||||
The confirming window is anchored on the sample 10 s back, which has moved on from the one
|
||||
the held jump was measured from — that is what makes it a second measurement. The other
|
||||
half is `test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor`.
|
||||
"""
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("x", 0.0, now=0.0)
|
||||
r.rate("x", 100.0, now=10.0) # 10 tok/s published
|
||||
rate, window = r.rate("x", 900.0, now=20.0) # 80 tok/s = 8x: held
|
||||
assert rate == pytest.approx(10.0)
|
||||
assert window == pytest.approx(10.0), "the held value keeps the Δ label it was published with"
|
||||
rate, window = r.rate("x", 1_720.0, now=30.0) # 82 tok/s: the step is real — publish
|
||||
assert rate == pytest.approx(82.0)
|
||||
assert window == pytest.approx(10.0)
|
||||
|
||||
|
||||
def test_an_unconfirmed_jump_is_dropped_and_the_held_value_keeps_printing() -> None:
|
||||
"""A one-interval artifact never publishes; the cell kept printing through it anyway."""
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("x", 0.0, now=0.0)
|
||||
r.rate("x", 100.0, now=10.0) # 10 tok/s
|
||||
assert r.rate("x", 900.0, now=20.0)[0] == pytest.approx(10.0) # 80: held
|
||||
assert r.rate("x", 1_005.0, now=30.0)[0] == pytest.approx(10.0) # 10.5: artifact dropped
|
||||
rate, _ = r.rate("x", 1_105.0, now=40.0) # ordinary publishing resumes
|
||||
assert rate == pytest.approx(10.0)
|
||||
|
||||
|
||||
def test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor() -> None:
|
||||
"""A stalled fetch stamps its body late, and that body then anchors the whole span.
|
||||
|
||||
Measured on live gufo at 2026-10-19 12:42 (`data/logs/NOTES-decode-stable.md`'s window,
|
||||
one request running at ~50 tok/s): the process stalled 11 s, the first reply back carried
|
||||
the counter gufo had rendered ~2 s into it, and the estimator diffed the rest of the stall
|
||||
over a 3 s window — 232 tok/s — then re-diffed **the same delta** over 5, 7 and 9 s
|
||||
(146, 114, 95) because the stale sample stayed `inside[0]`. Measured twice from one sample
|
||||
is one measurement, so the confirmation gate requires a window anchored on a different
|
||||
sample; the hold simply continues until the stale anchor ages out of the span.
|
||||
"""
|
||||
r = RateEstimator(span=10.0)
|
||||
tokens = lambda t: 45.0 * t # noqa: E731 the engine: a steady 45 tok/s
|
||||
polls = [(0.0, tokens(0.0)),
|
||||
(11.0, tokens(2.2)), # the stalled reply: honest body, 9 s-old stamp
|
||||
(12.0, tokens(12.0)), # the catch-up: ≥8x the diluted row, so it is held
|
||||
(14.0, tokens(14.0)), # …and every poll still anchored on that stale sample
|
||||
(16.0, tokens(16.0)),
|
||||
(18.0, tokens(18.0)),
|
||||
(20.0, tokens(20.0)),
|
||||
(22.0, tokens(22.0)), # the anchor has aged out; one poll still denies the jump
|
||||
(24.0, tokens(24.0)),
|
||||
(26.0, tokens(26.0))]
|
||||
published = [r.rate("g", value, now=1_000.0 + t)[0] for t, value in polls]
|
||||
assert all(v is None or v <= 60.0 for v in published), \
|
||||
f"a 45 tok/s engine printed {max(v for v in published if v is not None):.0f}: {published}"
|
||||
assert published[-1] == pytest.approx(45.0), "the stall ends in the truth, not a tail"
|
||||
|
||||
|
||||
def test_a_rise_from_zero_is_never_held() -> None:
|
||||
"""The first work an idle engine does must print on its own poll, not one late."""
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("x", 0.0, now=0.0)
|
||||
r.rate("x", 0.0, now=10.0) # 0 tok/s published
|
||||
assert r.rate("x", 500.0, now=20.0)[0] == pytest.approx(50.0)
|
||||
|
||||
|
||||
def test_confirmation_state_is_per_key() -> None:
|
||||
"""One key's pending jump must not hold another key's numbers."""
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("busy", 0.0, now=0.0)
|
||||
r.rate("busy", 100.0, now=10.0) # 10 tok/s — a jump base exists
|
||||
r.rate("idle", 5.0, now=0.0)
|
||||
r.rate("idle", 5.0, now=10.0) # 0 tok/s — previous is 0, nothing to confirm against
|
||||
assert r.rate("busy", 900.0, now=20.0)[0] == pytest.approx(10.0)
|
||||
assert r.rate("idle", 505.0, now=20.0)[0] == pytest.approx(50.0)
|
||||
|
||||
|
||||
def test_a_counter_restart_clears_the_confirmation_state() -> None:
|
||||
"""A restarted counter has no previous lineage to be judged against."""
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("x", 0.0, now=0.0)
|
||||
r.rate("x", 100.0, now=10.0) # 10 tok/s
|
||||
assert r.rate("x", 10.0, now=20.0) == (None, None) # restart drops the window
|
||||
rate, _ = r.rate("x", 910.0, now=30.0) # 90 tok/s on the new counter
|
||||
assert rate == pytest.approx(90.0)
|
||||
|
||||
|
||||
def test_forget_drops_the_judgement_state_with_the_window() -> None:
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("host:x", 0.0, now=0.0)
|
||||
r.rate("host:x", 100.0, now=10.0)
|
||||
r.rate("host:x", 900.0, now=20.0) # held: one pending jump, one published rate
|
||||
assert r._published and r._pending
|
||||
assert r.forget("host:") == 1
|
||||
assert r._published == {} and r._pending == {}
|
||||
|
||||
|
||||
def test_counter_reset_yields_none_once_then_recovers() -> None:
|
||||
"""A restart must not report a huge negative — or, worse, a huge positive — rate."""
|
||||
r = RateEstimator(span=60)
|
||||
|
||||
Reference in New Issue
Block a user