phase: 14_store_performance
**Phase 14 `store_performance` — final verification pass (all 4 tasks already in `complete/`)** - Re-verified every task's work in the tree: read caches (`series_names`/`stats`), capped compound `pick_res_many`, `history_many` + `api._history_batch`, windowed `maintain()` + per-resolution prune + `PRAGMA optimize`/`automatic_analyze=0`/WAL pragmas, `store_perf.BUDGETS` (14 ceilings), probe, live watch. - **Completion criteria:** baseline+after side by side in `data/logs/NOTES-store-perf.md` ✅ · budget tests green (`uv run pytest -m store_perf --store-perf` → **4 passed**, 14/14 inside) ✅ · 25-min dev-server watch re-run **independently** (74 passes, `scripts/watch-store-perf.py`): **0 `store-error`** (total still 190, newest 2026‑10‑18 06:33:30), WAL 7.77 MB ≤ 67.1 MB, medians 1.7–3.2 ms, p95 720/613 ms = the documented cold-cache window; a 3rd 27-pass watch on a freshly restarted server reproduced it (3.1/708, 1.8/572) ✅ · `maintain()` 5,368 ms → 2.7–6.8 ms and WAL bounded across two cycles (test + notes) ✅ · no behaviour change (all four history routes pinned against `query()`) ✅ - **Full gates:** `uv run pytest` → **1267 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0**, coverage **98.75%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` ok, repo hygiene ok; `uv run pytest -m live --live` → **11 passed**; dev server up, 9/9 sources `ok`, pages 200 in 1.5–3.3 ms. - **Docs fixed:** README/AGENTS still claimed a flat "20 s per pass" while `host_agent.py` (in this same uncommitted change) scales the budget to 5 s × disk count — corrected in both plus the two illustrative "20 s of wedged disk" sentences; broken bold (`` `driveripper** ``) repaired; stale gap-4 title + "tuning is its own phase" wording updated to say phase 14 ran; README's absolute "nothing … is answered by a whole-table scan" qualified to *per call* with the cold-window cost named; "third opt-in marker" → second; conftest `--store-perf` help named the real marker; **`data/logs/watch-store-perf.py` moved to `scripts/watch-store-perf.py`** (both docs told a clone to run a gitignored path) with root-anchored paths, `DASH_WATCH_OUT`, and dead code removed. - **New finding recorded:** gap 9's "~10 s process-wide stall is still gap 4's / `14_store_performance`'s problem" is now closed — coincident >8 s `llm` + >12 s `ups` raw-stream gaps went 393 (00:00–12:00) and 198 (12:00–21:36) to **0** across 110+ min of the phase-14 store; written into AGENTS.md gaps 4 and 9 and the notes table. - **Deviations:** did *not* pull either deferred cost knob (`STATS_CACHE_TTL_SECONDS`, warming caches from the maintenance tick) — AGENTS.md records that decision and its reasoning; `uv pip list --outdated` still reports certifi/fastapi/starlette/websockets (pre-existing advisory, no dependency touched this phase, gate treats it as advisory). Overwrote nothing: task 04's watch preserved as `store-perf-watch.task04.*`, my two runs as `.verify25.*` / `.current-code.*`. - **Next pending phase:** `15_agent_mtls`, then `16_agent_smart_pass_budget` (owns the scaled-budget tests + its doc criterion, now partly pre-satisfied by this pass), `17_installer_device_filter`.
This commit is contained in:
+170
-1
@@ -19,6 +19,7 @@ import pytest
|
||||
from fastapi import HTTPException
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
import store_perf
|
||||
from auth_client import sign_in
|
||||
from auth_contract import ADMIN_PASSWORD, FAST_PBKDF2
|
||||
from dashboards import alerts, auth
|
||||
@@ -34,7 +35,7 @@ from dashboards.sources.llm import (
|
||||
parse_gufo_metrics,
|
||||
parse_vllm_metrics,
|
||||
)
|
||||
from dashboards.store import Event, Sample, SampleStore
|
||||
from dashboards.store import RES_RAW, Event, Sample, SampleStore
|
||||
from slots_contract import (
|
||||
CANONICAL_TASK_ID_FIELD,
|
||||
OPTIONAL_NEXT_TOKEN_FIELDS,
|
||||
@@ -3153,6 +3154,84 @@ def test_one_request_per_host_answers_every_per_host_chart(
|
||||
assert gap["series"][0]["points"] == []
|
||||
|
||||
|
||||
def test_every_history_endpoint_answers_exactly_what_the_per_series_path_did(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Phase 14 task 02: the batched store door moved the cost of an N x M request, not its answer.
|
||||
|
||||
All four history routes now ask `store.history_many` for the whole batch instead of looping
|
||||
`store.history` per pair. That is only safe if the JSON is the same document the loop built,
|
||||
so this compares every route against points read straight out of the store with `query()` —
|
||||
a third path, in neither the loop nor the batch — and pins the series order the pages parse
|
||||
(host-major, metric-minor) plus the exact key set of one series entry.
|
||||
"""
|
||||
with build_client(tmp_path, monkeypatch, simulate=True, nut_host="127.0.0.1",
|
||||
host_targets=(Target("driveripper", DEAD_URL),
|
||||
Target("deskwork", DEAD_URL)),
|
||||
gufo_url=DEAD_URL) as test_client:
|
||||
store = test_client.app.state.ctx.store # type: ignore[attr-defined]
|
||||
now = time.time()
|
||||
writes = [
|
||||
("host", "driveripper", "cpu.usage_pct", [1.0, 2.0, 3.0]),
|
||||
("host", "driveripper", "mem.used_pct", [10.0, 20.0]),
|
||||
("host", "deskwork", "cpu.usage_pct", [4.0, 5.0, 6.0]),
|
||||
("host", "deskwork", "mem.used_pct", [30.0]),
|
||||
("disks", "driveripper", "disk.temp_c@nvme0n1", [38.0, 39.0]),
|
||||
("llm", "gufo", "llm.decode_tokens_per_second", [40.0, 45.0]),
|
||||
# a UPS name the simulator does not export, so its 5-second poll cannot add rows to
|
||||
# the series this assertion is about
|
||||
("ups", "notaserial", "ups.charge_pct", [99.0, 100.0]),
|
||||
]
|
||||
for source, host, metric, values in writes:
|
||||
store.add_many([Sample(source, host, metric, value, now - (len(values) - index) * 60)
|
||||
for index, value in enumerate(values)])
|
||||
|
||||
def expected(source: str, host: str, metric: str) -> list[dict[str, float]]:
|
||||
rows = store.query(source, host, metric, res=RES_RAW, start=now - 3_600.0, tail=True)
|
||||
return [{"ts": ts, "value": value} for ts, value in rows]
|
||||
|
||||
routes = [
|
||||
("/api/system/history", {"hosts": "driveripper,deskwork",
|
||||
"metrics": "cpu.usage_pct,mem.used_pct", "range": "1h"},
|
||||
[("host", host, metric) for host in ("driveripper", "deskwork")
|
||||
for metric in ("cpu.usage_pct", "mem.used_pct")]),
|
||||
("/api/disks/history", {"hosts": "driveripper",
|
||||
"metrics": "disk.temp_c@nvme0n1", "range": "1h"},
|
||||
[("disks", "driveripper", "disk.temp_c@nvme0n1")]),
|
||||
("/api/llm/history", {"engines": "gufo", "metrics": "decode_tokens_per_second",
|
||||
"range": "1h"},
|
||||
[("llm", "gufo", "llm.decode_tokens_per_second")]),
|
||||
("/api/ups/history", {"ups": "notaserial", "metrics": "ups.charge_pct",
|
||||
"range": "1h"},
|
||||
[("ups", "notaserial", "ups.charge_pct")]),
|
||||
]
|
||||
for path, params, pairs in routes:
|
||||
body = test_client.get(path, params=params).json()
|
||||
assert len(body["series"]) == len(pairs), path
|
||||
for series, (source, host, metric) in zip(body["series"], pairs, strict=True):
|
||||
assert set(series) == {"source", "host", "metric", "range_seconds", "points"}
|
||||
assert (series["source"], series["host"], series["metric"]) == (source, host, metric)
|
||||
assert series["range_seconds"] == 3_600.0
|
||||
assert series["points"] == expected(source, host, metric), f"{path} {metric}"
|
||||
|
||||
# and an empty answer is still an answer: a name with no rows, in a batch with names that
|
||||
# do, comes back as an empty series rather than a missing entry or a 404
|
||||
body = test_client.get("/api/system/history", params={
|
||||
"hosts": "driveripper", "metrics": "cpu.usage_pct,gpu.busy_pct@7",
|
||||
"range": "1h"}).json()
|
||||
assert [series["points"] for series in body["series"]] == [expected(
|
||||
"host", "driveripper", "cpu.usage_pct"), []]
|
||||
|
||||
# no metric named is no series asked for: the batch door is never opened with an empty
|
||||
# list (SQL `IN ()` is a syntax error waiting for that, and 500 is not the answer anyway)
|
||||
empty = test_client.get("/api/system/history",
|
||||
params={"hosts": "driveripper", "metrics": ""}).json()
|
||||
assert empty["hosts"] == ["driveripper"] and empty["series"] == []
|
||||
no_hosts = test_client.get("/api/system/history",
|
||||
params={"hosts": "", "metrics": "cpu.usage_pct"}).json()
|
||||
assert no_hosts["hosts"] == ["driveripper", "deskwork"], "empty means every configured host"
|
||||
assert len(no_hosts["series"]) == 2
|
||||
|
||||
|
||||
def test_llm_snapshot_keeps_an_engine_that_never_answered_in_the_list(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A configured-but-dead engine is listed as unreachable, not dropped (invariant 4)."""
|
||||
@@ -4079,3 +4158,93 @@ def test_samples_written_by_the_poller_are_queryable(tmp_path: Path,
|
||||
"range": "15m"}).json()["points"]
|
||||
assert [p["value"] for p in points] == [40.0, 45.0]
|
||||
store.close()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# phase 14 task 04: the endpoint budgets
|
||||
#
|
||||
# `store_perf.BUDGETS` is the one table of ceilings (it is what the probe's report and the notes
|
||||
# render, so a ceiling relaxed in one place is relaxed everywhere). The ceilings are 2-3x the
|
||||
# measured "after" on the seeded production-shaped store, which makes them a regression gate
|
||||
# rather than a stopwatch: the `/system` page's own chart request costs ~2 ms and still gets
|
||||
# 100 ms, while putting a full-table scan back on a per-request path crosses by three orders of
|
||||
# magnitude. Wall time lives behind `--store-perf` for that reason; what runs in every `uv run
|
||||
# pytest` is the key-name pin below, because a budget that silently stops being priced is as
|
||||
# useless as a budget that is crossed.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_every_http_budget_names_a_route_the_probe_prices(tmp_path: Path) -> None:
|
||||
"""The stale-pointer check for the HTTP half, on a throwaway store.
|
||||
|
||||
`measure_http` is what the priced test runs, and it owns the request shapes; if a key there is
|
||||
renamed, or a route stops being measured, the priced test would report the gap rather than the
|
||||
cost — which is why this asserts the key set here, cheaply, on every run.
|
||||
"""
|
||||
report = store_perf.measure_budget_http(tmp_path / "budget-keys.sqlite3",
|
||||
hosts=("driveripper",), metrics=("cpu.usage_pct",))
|
||||
assert set(report["http"]) == set(store_perf.HTTP_MEASURED_KEYS), (
|
||||
"the HTTP pass and the budget table no longer describe the same requests")
|
||||
assert store_perf.missing_budget_keys(report, ("http",)) == []
|
||||
# and the request the budgets describe is the one the task names: 3 hosts x 9 series
|
||||
nine = store_perf.HISTORY_METRICS
|
||||
assert len(nine) == 9, "the 90 d budget is priced against a 3x9 request, not some other shape"
|
||||
|
||||
|
||||
def test_metrics_stops_scanning_for_series_names_once_the_hub_is_polling(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The live shape behind one budgeted endpoint, guarded by statements rather than a clock.
|
||||
|
||||
Task 04's watch caught `/api/metrics` answering with its full `SELECT DISTINCT` scan on
|
||||
practically every call of the running server (535 ms, measured): task 02's cache was invalidated
|
||||
by every sample write, and the hub writes every 2-300 s, so the cache never lived long enough to
|
||||
serve. The store-level half of this lives in `tests/test_store.py`; this is the half that says
|
||||
the *endpoint* is what stops scanning, on the app the operator runs.
|
||||
"""
|
||||
db = str(tmp_path / "metrics-busy.sqlite3")
|
||||
store = SampleStore(db)
|
||||
store.add_many([Sample("host", "driveripper", "cpu.usage_pct", 1.0, time.time()),
|
||||
Sample("host", "deskwork", "cpu.usage_pct", 1.0, time.time())])
|
||||
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
|
||||
pbkdf2_iterations=FAST_PBKDF2), store)
|
||||
try:
|
||||
with TestClient(app) as http:
|
||||
sign_in(http)
|
||||
assert "cpu.usage_pct" in http.get("/api/metrics", params={"source": "host"}).json()["metrics"]
|
||||
|
||||
# a poll of series the store already has, then the same request with the log open
|
||||
store.add_many([Sample("host", "driveripper", "cpu.usage_pct", 2.0, time.time()),
|
||||
Sample("host", "deskwork", "cpu.usage_pct", 2.0, time.time())])
|
||||
seen: list[str] = []
|
||||
# the statement log is the test: `SampleStore._db` is the handle the request runs on
|
||||
store._db.set_trace_callback(seen.append)
|
||||
try:
|
||||
assert "cpu.usage_pct" in http.get("/api/metrics",
|
||||
params={"source": "host"}).json()["metrics"]
|
||||
finally:
|
||||
store._db.set_trace_callback(None)
|
||||
assert [s for s in seen if "DISTINCT metric" in s] == [], (
|
||||
f"`/api/metrics` went back to the table while nothing changed: {seen}")
|
||||
|
||||
# and a source that starts publishing something new is in the very next answer
|
||||
store.add("host", "driveripper", "gpu.busy_pct@0", 1.0)
|
||||
assert "gpu.busy_pct@0" in http.get("/api/metrics",
|
||||
params={"source": "host"}).json()["metrics"]
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.mark.store_perf
|
||||
def test_the_budgeted_http_paths_fit_their_pinned_margins(perf_store_path: Path) -> None:
|
||||
"""`/api/health`, `/api/metrics`, `/api/system/history` (1 h and 90 d), `/api/summary`.
|
||||
|
||||
Priced through the real ASGI gate with a signed-in session, on the seeded 6.4 M-row store —
|
||||
the same call the operator's browser makes, in-process so no network is measured. This is the
|
||||
line `TODO.md:7` is about: `/api/health` used to answer in 1.27 s because it counted the whole
|
||||
`sample` table on every poll, on every tab.
|
||||
"""
|
||||
report = store_perf.measure_budget_http(perf_store_path)
|
||||
print()
|
||||
print(store_perf.render_budgets_markdown(store_perf.budget_rows(report, ("http",))))
|
||||
assert store_perf.budget_violations(report, ("http",)) == []
|
||||
|
||||
Reference in New Issue
Block a user