phase: 14_store_performance
**Phase 14 `store_performance` — final verification pass (all 4 tasks already in `complete/`)** - Re-verified every task's work in the tree: read caches (`series_names`/`stats`), capped compound `pick_res_many`, `history_many` + `api._history_batch`, windowed `maintain()` + per-resolution prune + `PRAGMA optimize`/`automatic_analyze=0`/WAL pragmas, `store_perf.BUDGETS` (14 ceilings), probe, live watch. - **Completion criteria:** baseline+after side by side in `data/logs/NOTES-store-perf.md` ✅ · budget tests green (`uv run pytest -m store_perf --store-perf` → **4 passed**, 14/14 inside) ✅ · 25-min dev-server watch re-run **independently** (74 passes, `scripts/watch-store-perf.py`): **0 `store-error`** (total still 190, newest 2026‑10‑18 06:33:30), WAL 7.77 MB ≤ 67.1 MB, medians 1.7–3.2 ms, p95 720/613 ms = the documented cold-cache window; a 3rd 27-pass watch on a freshly restarted server reproduced it (3.1/708, 1.8/572) ✅ · `maintain()` 5,368 ms → 2.7–6.8 ms and WAL bounded across two cycles (test + notes) ✅ · no behaviour change (all four history routes pinned against `query()`) ✅ - **Full gates:** `uv run pytest` → **1267 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0**, coverage **98.75%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` ok, repo hygiene ok; `uv run pytest -m live --live` → **11 passed**; dev server up, 9/9 sources `ok`, pages 200 in 1.5–3.3 ms. - **Docs fixed:** README/AGENTS still claimed a flat "20 s per pass" while `host_agent.py` (in this same uncommitted change) scales the budget to 5 s × disk count — corrected in both plus the two illustrative "20 s of wedged disk" sentences; broken bold (`` `driveripper** ``) repaired; stale gap-4 title + "tuning is its own phase" wording updated to say phase 14 ran; README's absolute "nothing … is answered by a whole-table scan" qualified to *per call* with the cold-window cost named; "third opt-in marker" → second; conftest `--store-perf` help named the real marker; **`data/logs/watch-store-perf.py` moved to `scripts/watch-store-perf.py`** (both docs told a clone to run a gitignored path) with root-anchored paths, `DASH_WATCH_OUT`, and dead code removed. - **New finding recorded:** gap 9's "~10 s process-wide stall is still gap 4's / `14_store_performance`'s problem" is now closed — coincident >8 s `llm` + >12 s `ups` raw-stream gaps went 393 (00:00–12:00) and 198 (12:00–21:36) to **0** across 110+ min of the phase-14 store; written into AGENTS.md gaps 4 and 9 and the notes table. - **Deviations:** did *not* pull either deferred cost knob (`STATS_CACHE_TTL_SECONDS`, warming caches from the maintenance tick) — AGENTS.md records that decision and its reasoning; `uv pip list --outdated` still reports certifi/fastapi/starlette/websockets (pre-existing advisory, no dependency touched this phase, gate treats it as advisory). Overwrote nothing: task 04's watch preserved as `store-perf-watch.task04.*`, my two runs as `.verify25.*` / `.current-code.*`. - **Next pending phase:** `15_agent_mtls`, then `16_agent_smart_pass_budget` (owns the scaled-budget tests + its doc criterion, now partly pre-satisfied by this pass), `17_installer_device_filter`.
This commit is contained in:
@@ -0,0 +1,9 @@
|
||||
**Phase 14 `store_performance` — final verification pass (all 4 tasks already in `complete/`)**
|
||||
|
||||
- Re-verified every task's work in the tree: read caches (`series_names`/`stats`), capped compound `pick_res_many`, `history_many` + `api._history_batch`, windowed `maintain()` + per-resolution prune + `PRAGMA optimize`/`automatic_analyze=0`/WAL pragmas, `store_perf.BUDGETS` (14 ceilings), probe, live watch.
|
||||
- **Completion criteria:** baseline+after side by side in `data/logs/NOTES-store-perf.md` ✅ · budget tests green (`uv run pytest -m store_perf --store-perf` → **4 passed**, 14/14 inside) ✅ · 25-min dev-server watch re-run **independently** (74 passes, `scripts/watch-store-perf.py`): **0 `store-error`** (total still 190, newest 2026‑10‑18 06:33:30), WAL 7.77 MB ≤ 67.1 MB, medians 1.7–3.2 ms, p95 720/613 ms = the documented cold-cache window; a 3rd 27-pass watch on a freshly restarted server reproduced it (3.1/708, 1.8/572) ✅ · `maintain()` 5,368 ms → 2.7–6.8 ms and WAL bounded across two cycles (test + notes) ✅ · no behaviour change (all four history routes pinned against `query()`) ✅
|
||||
- **Full gates:** `uv run pytest` → **1267 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0**, coverage **98.75%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` ok, repo hygiene ok; `uv run pytest -m live --live` → **11 passed**; dev server up, 9/9 sources `ok`, pages 200 in 1.5–3.3 ms.
|
||||
- **Docs fixed:** README/AGENTS still claimed a flat "20 s per pass" while `host_agent.py` (in this same uncommitted change) scales the budget to 5 s × disk count — corrected in both plus the two illustrative "20 s of wedged disk" sentences; broken bold (`` `driveripper** ``) repaired; stale gap-4 title + "tuning is its own phase" wording updated to say phase 14 ran; README's absolute "nothing … is answered by a whole-table scan" qualified to *per call* with the cold-window cost named; "third opt-in marker" → second; conftest `--store-perf` help named the real marker; **`data/logs/watch-store-perf.py` moved to `scripts/watch-store-perf.py`** (both docs told a clone to run a gitignored path) with root-anchored paths, `DASH_WATCH_OUT`, and dead code removed.
|
||||
- **New finding recorded:** gap 9's "~10 s process-wide stall is still gap 4's / `14_store_performance`'s problem" is now closed — coincident >8 s `llm` + >12 s `ups` raw-stream gaps went 393 (00:00–12:00) and 198 (12:00–21:36) to **0** across 110+ min of the phase-14 store; written into AGENTS.md gaps 4 and 9 and the notes table.
|
||||
- **Deviations:** did *not* pull either deferred cost knob (`STATS_CACHE_TTL_SECONDS`, warming caches from the maintenance tick) — AGENTS.md records that decision and its reasoning; `uv pip list --outdated` still reports certifi/fastapi/starlette/websockets (pre-existing advisory, no dependency touched this phase, gate treats it as advisory). Overwrote nothing: task 04's watch preserved as `store-perf-watch.task04.*`, my two runs as `.verify25.*` / `.current-code.*`.
|
||||
- **Next pending phase:** `15_agent_mtls`, then `16_agent_smart_pass_budget` (owns the scaled-budget tests + its doc criterion, now partly pre-satisfied by this pass), `17_installer_device_filter`.
|
||||
@@ -0,0 +1 @@
|
||||
ok
|
||||
@@ -0,0 +1,64 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 22%]
|
||||
.............................................s.......................... [ 28%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 61%]
|
||||
........................sssssssssss..................................... [ 67%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 95%]
|
||||
sss....................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 356 0 100%
|
||||
src/dashboards/app.py 178 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 130 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 3047 38 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.75%
|
||||
1267 passed, 15 skipped, 1 warning in 110.62s (0:01:50)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
certifi 2026.7.22 2026.10.19 wheel
|
||||
fastapi 0.143.5 0.150.0 wheel
|
||||
starlette 1.8.0 1.9.0 wheel
|
||||
websockets 17.2 17.3 wheel
|
||||
+1
@@ -0,0 +1 @@
|
||||
Now let me write the baseline NOTES document.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 65%]
|
||||
....................sssssssssss......................................... [ 71%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 95%]
|
||||
...s....................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 348 0 100%
|
||||
src/dashboards/app.py 178 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 130 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 188 2 99% 255-256
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2914 38 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.70%
|
||||
1199 passed, 12 skipped, 1 warning in 117.25s (0:01:57)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
certifi 2026.7.22 2026.10.18 wheel
|
||||
fastapi 0.143.5 0.149.0 wheel
|
||||
starlette 1.8.0 1.9.0 wheel
|
||||
websockets 17.2 17.3 wheel
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
**Task 02 — no full table scan on a per-request path: done, all green.**
|
||||
|
||||
**Implemented**
|
||||
- `store.py`: `series_names()` cached 60 s (invalidated by `add`/`add_many`/`delete_series`/`maintain` via a generation counter; bounded to 256 `(source,host)` keys since `/api/metrics` takes both as free text); `stats()` counts cached 30 s with `last_maintain`/`size_bytes` deliberately never cached; `invalidate_read_caches()` for cold probes.
|
||||
- `pick_res()` now one compound statement of per-`res` `COUNT`s capped by `LIMIT max_points+1` (exact: the decision only asks `0` / `≤max_points` / neither), plus new `pick_res_many()` and `history_many()` (chunked at `COMPOUND_CHUNK`, index-range seek per branch).
|
||||
- `api.py`: all four history routes go through `_history_batch()` → `store.history_many()`; response dicts unchanged.
|
||||
- `tests/store_perf.py`: stats/`series_names` measured **cold and cached**, batch-vs-loop measured at 27 series × 24 h/90 d, EXPLAIN of the statement that actually runs.
|
||||
- Fixed a task-01 defect found in the full suite: the seeded determinism test pinned `SUM(value)`, whose float accumulation order changes when `INSERT OR REPLACE` moves rowids — now counts/windows exact, sum `rel=1e-9`, plus an exact PK-ordered row slice.
|
||||
- Docs: `AGENTS.md` (store.py/api.py bullets, gap 4 split into read-half-closed, commands), `README.md` (store.py, api.py, store_perf, new test pins), `data/logs/NOTES-store-perf.md` task-02 section (before/after, rejected shapes, EXPLAIN, live curl proof).
|
||||
|
||||
**Measured** (`uv run pytest -m store_perf --store-perf`): HTTP `/api/health` median 1 272 ms → **1.2 ms**; `/api/metrics?source=host` 1 415 → **1.6 ms**; `/api/system/history` 3×9 at 90 d 100.3 → **49.8 ms**; `pick_res()`×3 3.0 → **0.3 ms**. Live dev server on the real 636 MB store: `/api/health` 0.700 s → **1.5 ms**, all nine sources `ok`.
|
||||
|
||||
**Tests/lint** (exact): `bash .agents/validate.sh` → pytest `--cov --cov-fail-under=94` **1214 passed, 12 skipped, coverage 98.73%**; `ruff` All checks passed; `pyright` 0 errors; `node --check` ok. Final confirmation run `uv run pytest -q --cov --cov-fail-under=94` → **EXIT=0**, 0 FAILED (log `data/logs/pytest-final2.log`). No dependencies touched (validate's outdated-dep line is pre-existing advisory).
|
||||
|
||||
**Deviations**: batched the resolution *decision*, not the row fetch — `ROW_NUMBER()` window (73 ms) and a compound of the 27 tail selects (38 ms) both measured slower than 27 index seeks (27 ms) on the seeded store, and task 01 requires the numbers not to regress; recorded with its table in the notes. The literal `SELECT res, COUNT(*) … GROUP BY res` shape was also measured slower (4.3 vs 2.7 ms) than the capped compound that shipped. No live-infra changes beyond `./scripts/dev-server.sh restart` (local dev, invariant 10).
|
||||
|
||||
**Next pending task**: `.agents/phases/todo/14_store_performance/03_windowed_rollups_and_wal_tuning.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+64
@@ -0,0 +1,64 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 64%]
|
||||
.....................sssssssssss........................................ [ 70%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 93%]
|
||||
..................s..................................................... [ 99%]
|
||||
.. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 356 0 100%
|
||||
src/dashboards/app.py 178 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 130 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 266 2 99% 445-446
|
||||
------------------------------------------------------------------
|
||||
TOTAL 3000 38 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.73%
|
||||
1214 passed, 12 skipped, 1 warning in 122.10s (0:02:02)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
certifi 2026.7.22 2026.10.18 wheel
|
||||
fastapi 0.143.5 0.149.0 wheel
|
||||
starlette 1.8.0 1.9.0 wheel
|
||||
websockets 17.2 17.3 wheel
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
**Phase 14 / Task 03 — windowed rollups + WAL tuning: done.**
|
||||
|
||||
Implementation (`src/dashboards/store.py`)
|
||||
- `maintain()` now windows both rollup steps off `last_maintain` (read via `get_meta()` *outside* the lock), overlapped by one bucket + `MAINTAIN_SLACK_SECONDS=60` and **snapped down to a bucket boundary** — a floor that cut into a bucket re-rolled a partial average from its tail rows (2.5 → 4.5); missing/non-numeric row falls back to full retention with a fixed-line warning.
|
||||
- Prune keeps its **full retention bound** but is now one `DELETE` per resolution: the old single `OR` of the three was unplannable (`SCAN sample`, 554 ms of a 562 ms pass while deleting nothing).
|
||||
- Statistics are the other half of the cost bound: `maintain()` ends with `PRAGMA optimize`, `__init__` sets `automatic_analyze=0` (SQLite's own refresh lands inside a poller's commit). WAL: `wal_autocheckpoint=1000`, `journal_size_limit=64 MB`; `busy_timeout=5000` deliberately unchanged.
|
||||
- Measured (seeded 6.4 M-row store): `maintain()` **5,368 ms → 2.7 ms** steady, 48 ms ten-minute catch-up; race at the app's cadence **100/100 inserts, 0 `database is locked`, max wait 21 ms** where the pre-fix control still waits 4.76 s; WAL 38 MB-forever → 0–4 MB. Live store: WAL 52.3 → 5.9 MB, `sqlite_stat1` created, real-store passes ~40 ms, `/api/health` 3.5 ms.
|
||||
|
||||
Tests / gate
|
||||
- `tests/test_store.py`: +9 tests (window bounds from a trace callback, catch-up-not-a-gap, windowed-vs-full-window row-for-row equivalence, re-roll only gains rows, corrupt-row fallback, `maintain` leaves statistics, optimize failure is a log line not a failed tick, WAL pragmas, two-cycle WAL cap).
|
||||
- `tests/store_perf.py`: write passes and races moved to a `scratch_copy()` (baseline never moves), added `RACE_MAINTAIN_GAP_SECONDS` cadence races + the whole-window **control**, per-window plans derived from `ROLLUP_INSERT`.
|
||||
- `bash .agents/validate.sh`: **1240 passed, 12 skipped**, coverage **98.75%** (floor 94), ruff + pyright + `node --check` clean; `-m store_perf --store-perf` 1 passed; dev server restarted, all 9 sources `ok`.
|
||||
|
||||
Docs
|
||||
- `AGENTS.md` (store.py bullet incl. the bucket-alignment rule, known gap 4 → write half closed, Commands, new testing-policy rule "a cost claim is pinned as a shape and a control"), `README.md` (store.py + `store_perf` paragraphs), `data/logs/NOTES-store-perf.md` (full task-03 tables, race-shape key, no-read-regression table, live smoke check).
|
||||
|
||||
Deviations
|
||||
- The task's "windowing + WAL tuning" needed two extra fixes to actually reach the budget (prune `OR`, missing statistics); both are measured and recorded rather than assumed, and the prune's retention bound is unchanged.
|
||||
- The saturated task-01 race is kept for comparability but the pass line moved to the cadence-shaped race (a maintainer looping with no gap starves a writer at *any* pass length); the control beside it proves the race can still see the failure.
|
||||
- Left alone: `uv pip list --outdated` advisory (certifi/fastapi/starlette/websockets) — pre-existing, no dependencies touched.
|
||||
|
||||
Next pending task: `.agents/phases/todo/14_store_performance/04_budget_gates_and_live_proof.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+111
@@ -0,0 +1,111 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 63%]
|
||||
.....................sssssssssss........................................ [ 69%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 92%]
|
||||
..............F.............................s........................... [ 97%]
|
||||
............................ [100%]
|
||||
=================================== FAILURES ===================================
|
||||
_______ test_a_second_maintain_reads_only_the_window_since_the_last_run ________
|
||||
|
||||
db = <dashboards.store.SampleStore object at 0x7fe8d2126c50>
|
||||
|
||||
def test_a_second_maintain_reads_only_the_window_since_the_last_run(db) -> None:
|
||||
"""The cost change, pinned as a bound and as behaviour.
|
||||
|
||||
A bucket that filled after the previous pass is inside the next window and gets rolled. A raw
|
||||
row that arrives for a bucket ten minutes behind it is not, and that bucket keeps the average
|
||||
the earlier pass computed — the documented cost of a window, stated here as a passing test
|
||||
rather than as a surprise. Reaching it in production costs a poller stall measured in minutes
|
||||
(the poller stamps samples with its own wall clock); the alternative it replaced was
|
||||
re-averaging 3.97 M rows every minute.
|
||||
"""
|
||||
now = time.time()
|
||||
bucket_a = bucket_start(now - 10 * MINUTE)
|
||||
add_series(db, "host", "h", "m", [10, 20], start=bucket_a, step=1)
|
||||
db.maintain(now=now)
|
||||
assert db.query("host", "h", "m", res=RES_MIN, start=bucket_a - 1,
|
||||
end=bucket_a + MINUTE) == [(bucket_a, 15.0)]
|
||||
|
||||
second = now + 60
|
||||
bucket_b = bucket_start(second - 2 * MINUTE)
|
||||
add_series(db, "host", "h", "m", [40, 50], start=bucket_b, step=1)
|
||||
add_series(db, "host", "h", "m", [1000], start=bucket_a + 30, step=1)
|
||||
with query_log(db) as seen:
|
||||
counts = db.maintain(now=second)
|
||||
|
||||
windows = rollup_windows(seen)
|
||||
# the bound: one minute (one hour) of rows plus the overlap, not 48 h of raw (not 30 d of 1m)
|
||||
assert windows[RES_RAW] == pytest.approx(
|
||||
(second - MINUTE - MAINTAIN_SLACK_SECONDS, bucket_start(now - MINUTE - MAINTAIN_SLACK_SECONDS)))
|
||||
> assert windows[RES_MIN] == pytest.approx(
|
||||
(second - HOUR - MAINTAIN_SLACK_SECONDS, bucket_start(now - HOUR - MAINTAIN_SLACK_SECONDS)))
|
||||
E assert (1792449272.5... 1792447200.0) == approx((17924....0 ± 1.8e+03))
|
||||
E
|
||||
E comparison failed. Mismatched elements: 1 / 2:
|
||||
E Max absolute difference: 1980.0
|
||||
E Max relative difference: 1.104635048664195e-06
|
||||
E Index | Obtained | Expected
|
||||
E 1 | 1792447200.0 | 1792449180.0 ± 1.8e+03
|
||||
|
||||
tests/test_store.py:802: AssertionError
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 356 0 100%
|
||||
src/dashboards/app.py 178 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 130 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 304 2 99% 510-511
|
||||
------------------------------------------------------------------
|
||||
TOTAL 3038 38 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.75%
|
||||
=========================== short test summary info ============================
|
||||
FAILED tests/test_store.py::test_a_second_maintain_reads_only_the_window_since_the_last_run
|
||||
1 failed, 1239 passed, 12 skipped, 1 warning in 111.30s (0:01:51)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
certifi 2026.7.22 2026.10.18 wheel
|
||||
fastapi 0.143.5 0.150.0 wheel
|
||||
starlette 1.8.0 1.9.0 wheel
|
||||
websockets 17.2 17.3 wheel
|
||||
validation FAILED (see output above)
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
**Phase 14 / Task 03 — re-run after the harness failure: green.**
|
||||
|
||||
What was fixed
|
||||
- The failing test was mine, and it was wall-clock luck, not the store: `test_a_second_maintain_reads_only_the_window_since_the_last_run` asserted the 1m→1h step's floor as a **minute**-aligned number while `_rollup_floor` snaps to an **hour**. `pytest.approx`'s relative tolerance (≈±1 800 s at epoch scale) covered the half of the hour where the two are within that, so it passed twice and failed at 19:4x.
|
||||
- Fix: `WINDOW_CLOCK_OFFSETS = (0, 1, 30, 59, 1 800, 3 539, 3 599)` over a base epoch that sits exactly on both boundaries — every edge is now a case instead of a coin flip — and each bound is snapped to *its own step's* bucket; the same parametrization went on the first-pass and catch-up tests so no window assertion in this task reads `time.time()`.
|
||||
- Found and fixed a second latent flake in the same test: the "due" bucket sat one step behind the ceiling, so a clock within a second of a boundary put the pair's second row outside the window and rolled 40.0 instead of 45.0 (now two steps back). `db.query()` defaults `end` to the wall clock — an explicit `end=` is now passed where the test runs on its own clock.
|
||||
- Added `test_the_window_floor_is_always_aligned_to_its_own_bucket` (aligned, never above the bound, never a whole window back, and the two steps really differ). Mutation-checked: removing the `floor(…/bucket)*bucket` snap fails 6 of 7 clock cases plus that test; restored code passes.
|
||||
|
||||
Results
|
||||
- `bash .agents/validate.sh`: **1259 passed, 12 skipped, 0 failed** · coverage **98.75%** (floor 94, unchanged) · `ruff check src tests agent` all passed · `pyright` 0 errors · `node --check` ok · hygiene ok · no `validation FAILED`.
|
||||
- `uv run pytest -m store_perf --store-perf`: **1 passed** (window plans seek, catch-up ≤ 2× steady + slack, the whole-window control still contends, the cadence race is 0 `database is locked`).
|
||||
- Dev server healthy: all 9 sources `ok`, real-store WAL holding at ~7.6 MB under the 64 MB cap.
|
||||
|
||||
Notable decisions
|
||||
- No production-code change this pass — the defect was in the test's expectation, so the implementation (windowed rollups, per-resolution prune, `PRAGMA optimize` + `automatic_analyze=0`, WAL pragmas, `busy_timeout` untouched) stands as measured: `maintain()` 5 368 ms → 2.7 ms steady / 48 ms catch-up.
|
||||
- Preferred explicit clocks and a property test over widening a tolerance, and recorded the flake plus both root causes in `data/logs/NOTES-store-perf.md` ("One window assertion was wall-clock luck").
|
||||
- Left alone: the pre-existing `uv pip list --outdated` advisory (certifi/fastapi/starlette/websockets) — no dependencies touched in this task.
|
||||
|
||||
Next pending task: `.agents/phases/todo/14_store_performance/04_budget_gates_and_live_proof.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+64
@@ -0,0 +1,64 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 62%]
|
||||
.....................sssssssssss........................................ [ 67%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 90%]
|
||||
...............................................................s........ [ 96%]
|
||||
............................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 356 0 100%
|
||||
src/dashboards/app.py 178 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 130 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 304 2 99% 510-511
|
||||
------------------------------------------------------------------
|
||||
TOTAL 3038 38 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.75%
|
||||
1259 passed, 12 skipped, 1 warning in 112.04s (0:01:52)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
certifi 2026.7.22 2026.10.18 wheel
|
||||
fastapi 0.143.5 0.150.0 wheel
|
||||
starlette 1.8.0 1.9.0 wheel
|
||||
websockets 17.2 17.3 wheel
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
Task 04 is complete and everything is green.
|
||||
|
||||
**What shipped**
|
||||
- `tests/store_perf.py` — `BUDGETS`: 14 pinned ceilings (2‑3× the measured "after"), `budget_rows`/``budget_violations`/`missing_budget_keys`, `measure_budget_reads/_http`, and the budget table rendered into the probe report.
|
||||
- Budget tests: 2 default-run pins (a budget whose measurement was renamed fails; the table is internally coherent) + 4 `store_perf` passes (reads, HTTP, `maintain`, race). `uv run pytest -m store_perf --store-perf` → 4 passed, **14/14 budgets inside**.
|
||||
- **The watch found a real defect in task 02's fix** and I fixed it: the series cache was invalidated by *every* sample write, so on a store nine pollers write to it never served and `/api/metrics` paid the full `SELECT DISTINCT` scan (measured 535 ms/call; the scan alone is 572 ms on the live store). `store.py` now invalidates only the write that can change the answer (`_note_series_write` + a capped `_known_series` existence set; `DELETE` paths still invalidate unconditionally). Pinned by statement counts, not clocks, in `tests/test_store.py` and through the served endpoint in `tests/test_api.py`.
|
||||
- Live proof (`data/logs/watch-store-perf.py` → `store-perf-watch.{log,tsv,json}`, `store-perf-live.png`): 25 min / 74 passes / 296 signed-in requests on the operator's 636 MB store — **0 new `store-error` rows** (190 total, newest 2026‑10‑18 06:33), **WAL 6.5 → 7.77 MB** (cap 67 MB), 0 non‑200, medians 1.8–3.3 ms for `/api/health`, `/api/summary`, `/api/metrics` and a 3×9 90‑day `/api/system/history`.
|
||||
- Docs: `data/logs/NOTES-store-perf.md` gained the task‑04 section (baseline vs after side by side, EXPLAIN shapes, the watch table, the defect write-up); `README.md` and `AGENTS.md` updated (gap 4 now reads: closed, with the honest remainder).
|
||||
- Gates: `bash .agents/validate.sh` exit 0 — 1267 passed, 15 skipped, coverage 98.75%, ruff + pyright + `node --check` clean. Dev server restarted, all 9 sources `ok`.
|
||||
|
||||
**Honest leftovers**
|
||||
1. The p95 is the cache window, not the request: on the live store a cold `series_names('host')` costs 599 ms and `stats()`' counts 729 ms, once per 60 s / 30 s (37 of 74 passes had one request >250 ms). Closing that needs a counts table or a bounded series index — task 03's note already names it, and it is a phase, not this one.
|
||||
2. `uv pip list --outdated` is non-empty (fastapi 0.143.5→0.150.0, starlette, certifi, websockets) — pre-existing drift, untouched here; rule 8 wants a deliberate decision.
|
||||
3. Per AGENTS rule 4 I did **not** modify `.agents/phases/todo/14_store_performance/04_...md` — say the word and I'll tick the completion criteria and move it to `complete/`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+64
@@ -0,0 +1,64 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 22%]
|
||||
.............................................s.......................... [ 28%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 61%]
|
||||
........................sssssssssss..................................... [ 67%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 95%]
|
||||
sss....................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 356 0 100%
|
||||
src/dashboards/app.py 178 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 130 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 3047 38 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.75%
|
||||
1267 passed, 15 skipped, 1 warning in 113.83s (0:01:53)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
certifi 2026.7.22 2026.10.19 wheel
|
||||
fastapi 0.143.5 0.150.0 wheel
|
||||
starlette 1.8.0 1.9.0 wheel
|
||||
websockets 17.2 17.3 wheel
|
||||
Reference in New Issue
Block a user