Compare commits
33
Commits
a08e693698
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
883e36c727 | ||
|
|
219d7a7c3b | ||
|
|
86c8bf4992 | ||
|
|
0271387053
|
||
|
|
51772ffef2 | ||
|
|
25fd74b7c5 | ||
|
|
e366f599f3 | ||
|
|
fa4f38c694 | ||
|
|
44d61d3ab9 | ||
|
|
5d37444fb0 | ||
|
|
9ff3c7889b | ||
|
|
2bf236fd92 | ||
|
|
558a83d0be | ||
|
|
89d9fb9f89 | ||
|
|
5f3ed9bcd4 | ||
|
|
895f309f8e | ||
|
|
5e267737dc | ||
|
|
973b7299fe | ||
|
|
6444a158f2 | ||
|
|
95df8b695b | ||
|
|
f9b1565abb | ||
|
|
10f86772ee | ||
|
|
b02c13f7d3 | ||
|
|
7e367a9014 | ||
|
|
4b0f53c499 | ||
|
|
cffbac5771 | ||
|
|
569ca25fdb | ||
|
|
1bbd868401 | ||
|
|
04d934dd94 | ||
|
|
29524b3ebc
|
||
|
|
28139df6d6 | ||
|
|
1734d81e48 | ||
|
|
eb4788cba1 |
@@ -0,0 +1,55 @@
|
|||||||
|
# Phase 12 — A passed battery self-test is news, not a warning
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — ""Last self-test: Done and passed" still shows up as a warning in the overview and UPS section"
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** `src/dashboards/sources/nut.py` (`assess()` — the explicit `health.checks[]` list, the "Last self-test" check at the `test_result` branch), `src/dashboards/alerts.py` (the overview alert rule that reads `health.checks[]`), `tests/test_nut.py` (the two existing "Last self-test" assertions, the `SimUps` test-result tests), `tests/fixtures/nut_apc.vars` / `nut_cyberpower.vars` (verbatim `LIST VAR` captures), `src/dashboards/sources/nut_sim.py` (the simulator's `PASSED`/`FAILED` window), `src/dashboards/web/pages/ups.html` + `src/dashboards/web/static/page-ups.js` (the check list the `/ups` card renders), `data/logs/` (screenshots)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The `Last self-test` check grades the driver's own word about the last battery test, and a
|
||||||
|
passed test — the real drivers answer **`Done and passed`** — must read `ok`, not `warn`.
|
||||||
|
The current word lists (`"ok", "passed", "pass", "completed"` / `"failed", "fail", "error",
|
||||||
|
"aborted", "test incomplete"`) let the one phrasing this fleet's hardware actually uses fall
|
||||||
|
through to `warn`, so a *healthy* UPS carries a warning on the overview and the UPS page.
|
||||||
|
After this phase the verdict follows the test's outcome: pass → `ok`, fail → `fail`,
|
||||||
|
never-tested / unset → `unknown`, and the overview's alert feed stops naming it.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
- `05_ups_hero_battery_glyph` (complete) — the `health.checks[]` contract and the `/ups`
|
||||||
|
check list this phase changes.
|
||||||
|
- `11_admin_login` (todo) — **no shared files**: 11 touches `app.py`/`config.py`/`store.py`/
|
||||||
|
the auth surface; this phase touches `sources/nut.py` + its tests + fixtures. Either order
|
||||||
|
runs; the only coupling is the shared `tests/test_nut.py` — apply whichever runs second on
|
||||||
|
top of the file state at that time.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
|
||||||
|
1. `01_pin_self_test_wording.md` — capture the real drivers' `ups.test.result` values and pin the current (wrong) verdict in a failing test, so the fix has a before/after.
|
||||||
|
2. `02_verdict_map_for_test_results.md` — the `assess()` word map: pass phrasings → `ok`, fail phrasings → `fail`, "No test initiated"/unset → `unknown` (never `warn`); fixtures + `tests/test_nut.py` + the simulator's wording stay consistent.
|
||||||
|
3. `03_no_false_warning_overview_and_ups.md` — end-to-end: a passed test produces no alert on `/api/summary` and an `ok` chip on `/ups`; pinned by API test and a live screenshot.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Unit: `tests/test_nut.py` — the full word map (each pass phrasing `ok`, each fail phrasing
|
||||||
|
`fail`, `No test initiated` and unset `unknown`, case/whitespace-insensitive), driven from
|
||||||
|
the verbatim fixture wording where it exists and synthetic `UpsReading`s elsewhere.
|
||||||
|
- Integration: `tests/test_api.py` — a `health.checks[]` entry with a passed self-test
|
||||||
|
produces no alert in the `/api/summary` payload; a failed one does.
|
||||||
|
- The `SimUps` test-result tests (`tests/test_nut.py`) keep passing: the simulator's
|
||||||
|
`PASSED`/`FAILED` window words were already in the right tiers.
|
||||||
|
- Coverage: **>90%** on new/modified code; the overall gate (94%) must hold.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `printf 'LIST VAR apc ups.test.result\n' | nc <driveripper> 3051` (run from a host
|
||||||
|
that can reach upsd — the laptop still cannot, per the fleet facts) captured and
|
||||||
|
recorded in `tests/test_nut.py` / the fixture; the captured word asserts `ok`
|
||||||
|
- [ ] the "Last self-test" check renders `ok` (dimmed, no warning class) on `/ups` and does
|
||||||
|
not appear in the overview's Warnings panel; screenshots in `data/logs/`
|
||||||
|
(`self-test-passed-overview.png`, `self-test-passed-ups.png`)
|
||||||
|
- [ ] `uv run pytest` green, `uv run pytest --cov --cov-fail-under=94` green,
|
||||||
|
`uv run ruff check src tests agent && uv run pyright` clean, `bash .agents/validate.sh`
|
||||||
|
green
|
||||||
|
- [ ] no behavior change in completed phases: every other `health.checks[]` check (power
|
||||||
|
path, charge, runtime, battery age/temperature, load, alarms) grades exactly as before
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
# Task 01 — Pin the self-test word the real drivers use
|
||||||
|
|
||||||
|
**Phase:** `12_nut_self_test_verdict` · **Source:** `TODO.md:3` — ""Last self-test: Done and passed" still shows up as a warning in the overview and UPS section"
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Prove the bug in a test before touching the fix: the `Last self-test` check in
|
||||||
|
`assess()` must be shown to grade the drivers' real wording (`Done and passed`) as
|
||||||
|
`warn`, and the verbatim `ups.test.result` value must be captured from the running
|
||||||
|
upsd so the fixture/test asserts the hardware's actual sentence, not a memory of it
|
||||||
|
(AGENTS.md rule 7 — verify against the deployed upstream; the factory state both UPSes
|
||||||
|
answer is `No test initiated`, pinned in `AGENTS.md`'s fleet facts).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `tests/test_nut.py` — add the pinning test **first**: build the `UpsReading` the way
|
||||||
|
the two existing "Last self-test" tests do (the `SimUps`/`vars` helpers in the file),
|
||||||
|
with `test_result="Done and passed"`, and assert the current behaviour
|
||||||
|
(`check["status"] == "warn"`) with a comment that the *fix* flips it to `ok`. This is
|
||||||
|
the red test the whole phase turns green.
|
||||||
|
2. Capture the real value — from a host that can reach upsd (the laptop cannot yet,
|
||||||
|
per the fleet facts; `driveripper` itself, or the deployed dashboard's network):
|
||||||
|
`printf 'LIST VAR apc ups.test.result\nLIST VAR cyberpower ups.test.result\n' | nc
|
||||||
|
<upsd-host> 3051` — record the exact words in `tests/test_nut.py` (an assertion per
|
||||||
|
UPS where the value is a stable driver constant) and note the capture date. If a test
|
||||||
|
has been run on the hardware, the value is the driver's pass word; if not, it is
|
||||||
|
`No test initiated` — either way it goes into the test, because the verdict map must
|
||||||
|
cover both.
|
||||||
|
3. `tests/fixtures/nut_apc.vars` / `nut_cyberpower.vars` — if the live `ups.test.result`
|
||||||
|
line differs from the captured fixture (a test has since run), re-capture with
|
||||||
|
`./scripts/capture-fixtures.sh` (the script scrubs serials; the test-result line is
|
||||||
|
not a secret) so the fixtures stay verbatim ground truth.
|
||||||
|
4. Confirm the symptom end-to-end while it is live: `curl -s
|
||||||
|
http://127.0.0.1:8090/api/summary` (signed in, per phase 11) shows the alert row for
|
||||||
|
the `Last self-test` check, and `/api/ups/snapshot`'s `health.checks[]` carries the
|
||||||
|
`warn` entry — the two surfaces the TODO names.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Unit: the pinning test (red now, green after task 02).
|
||||||
|
- Coverage: **>90%** on this task's new/modified code.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `uv run pytest tests/test_nut.py -k self_test` shows the new test **failing** on
|
||||||
|
the current code (or passing if the wording already graded `ok` — in which case
|
||||||
|
record that and task 02 narrows to the `No test initiated` branch only)
|
||||||
|
- [ ] the real `ups.test.result` word is captured from the running upsd and recorded in
|
||||||
|
the test suite with its capture date
|
||||||
|
- [ ] fixtures re-captured if the live value differs from the recorded one
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
# Task 02 — Grade the test outcome, not the phrasing
|
||||||
|
|
||||||
|
**Phase:** `12_nut_self_test_verdict` · **Source:** `TODO.md:3` — ""Last self-test: Done and passed" still shows up as a warning in the overview and UPS section"
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The `Last self-test` branch in `assess()` maps the driver's word to a verdict with two
|
||||||
|
hardcoded tuples, and any phrasing outside both falls through to `warn` — which is how
|
||||||
|
`Done and passed` (the real drivers' pass word) and `No test initiated` (the factory
|
||||||
|
state) both read as warnings. Rewrite the branch as an explicit outcome map: pass
|
||||||
|
phrasings → `ok`, fail phrasings → `fail`, "no test has run" phrasings and unset →
|
||||||
|
`unknown`. A warning for a *passed* test is the bug; a warning for a machine that never
|
||||||
|
ran a test is noise this phase removes too.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/sources/nut.py` — the `test_result` branch in `assess()`:
|
||||||
|
- pass tier: the existing `("ok", "passed", "pass", "completed")` **plus**
|
||||||
|
`done and passed` (the live word captured in task 01, kept lowercase/trimmed —
|
||||||
|
keep the existing `strip().lower()` normalisation);
|
||||||
|
- fail tier: unchanged (`"failed", "fail", "error", "aborted", "test incomplete"`);
|
||||||
|
- new `unknown` tier: `no test initiated` (and the empty-string case) — reuse the
|
||||||
|
existing `None`-branch sentence ("no self-test result reported — run a battery
|
||||||
|
test") so the two "never tested" states say the same thing;
|
||||||
|
- the fall-through for any *other* phrasing stays `warn` (an unrecognised word is
|
||||||
|
the one thing worth a warning — a driver inventing a new word is drift worth
|
||||||
|
noticing), and the check keeps printing the raw word plus the `test_date` suffix.
|
||||||
|
- ASSUMPTION: `No test initiated` (the factory state on both UPSes, per AGENTS.md
|
||||||
|
fleet facts) maps to `unknown`, not `warn` — a pack that has never been
|
||||||
|
self-tested is a suggestion to run one, not a fault; the `None` branch already
|
||||||
|
says exactly that, so the two states read identically.
|
||||||
|
2. `tests/test_nut.py` — the task-01 red test now asserts `ok`; add the full map:
|
||||||
|
every pass phrasing (including the captured live word) → `ok`, every fail phrasing
|
||||||
|
→ `fail`, `No test initiated` / empty / unset → `unknown` with the run-a-test
|
||||||
|
sentence, an unknown phrasing (e.g. `mystery`) → `warn`; case and whitespace
|
||||||
|
variations of the live word.
|
||||||
|
3. `tests/fixtures/nut_apc.vars` / `nut_cyberpower.vars` — if the captured live value
|
||||||
|
(task 01) is the factory `No test initiated`, the fixtures already carry it; if a
|
||||||
|
test has run, the fixture carries `Done and passed` — either way the fixture-driven
|
||||||
|
tests in `tests/test_ups_contract.py` / `test_nut.py` must pass against the new map
|
||||||
|
(the fixture's word is part of the hero contract's check list).
|
||||||
|
4. `src/dashboards/sources/nut_sim.py` — no code change expected: the simulator's
|
||||||
|
`PASSED` / `FAILED` window words are already in the pass/fail tiers; run the
|
||||||
|
`SimUps` test-result tests to prove it. If the simulator ever answers a *third*
|
||||||
|
word, add it to the map — do not special-case the simulator.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Unit: the full word-map table in `tests/test_nut.py` (the tiers above, plus
|
||||||
|
`test_date` suffix still printed on the pass/fail detail line).
|
||||||
|
- Regression: every pre-existing `test_nut.py` / `test_ups_contract.py` test passes
|
||||||
|
unmodified in intent (the fixtures' words land in the right tier).
|
||||||
|
- Coverage: **>90%** on this task's new/modified code.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `uv run pytest tests/test_nut.py tests/test_ups_contract.py` green, including the
|
||||||
|
task-01 test now asserting `ok` for `Done and passed`
|
||||||
|
- [ ] no phrasing the two real drivers answer (verified against the task-01 capture)
|
||||||
|
falls into `warn`
|
||||||
|
- [ ] the `unknown` tier's detail sentence is the existing "run a battery test" one, so
|
||||||
|
unset and never-initiated read the same
|
||||||
+60
@@ -0,0 +1,60 @@
|
|||||||
|
# Task 03 — No false warning on the overview and the UPS page
|
||||||
|
|
||||||
|
**Phase:** `12_nut_self_test_verdict` · **Source:** `TODO.md:3` — ""Last self-test: Done and passed" still shows up as a warning in the overview and UPS section"
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Prove the fix end-to-end on the two surfaces the TODO names: a passed self-test must
|
||||||
|
produce **no alert** in `/api/summary` (the overview's Warnings panel) and an `ok`
|
||||||
|
entry in `/api/ups/snapshot`'s `health.checks[]` (the `/ups` check list), while a
|
||||||
|
*failed* test still alerts — the check keeps its reason for existing.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `tests/test_api.py` — the alert-absence pin: an app (the shared `create_app`
|
||||||
|
test harness, `auth_contract` seeding) whose UPS payload carries a `Last self-test`
|
||||||
|
check with a passed word answers `/api/summary` with **no** alert row naming it
|
||||||
|
(the `alerts` list is the thing the overview renders — a row there is the warning
|
||||||
|
the TODO sees); the same harness with a `fail`-tier word **does** produce the
|
||||||
|
alert, so the test pins both directions.
|
||||||
|
2. `src/dashboards/web/static/page-ups.js` / `src/dashboards/web/pages/ups.html` — no
|
||||||
|
code change expected (the check list renders `health.checks[]` verdicts verbatim);
|
||||||
|
verify the `ok` chip styling for the "Last self-test" row is the existing dim path
|
||||||
|
(`ok` stays quiet ink, per the shared verdict vocabulary). If the row needs no
|
||||||
|
markup, say so in the completion note — do not restyle.
|
||||||
|
3. Live verification (AGENTS.md invariant 10 — UI changes are screenshotted): the dev
|
||||||
|
server (`./scripts/dev-server.sh status` first; the source must be `ok`), signed in
|
||||||
|
with the dev password:
|
||||||
|
- `curl -s http://127.0.0.1:8090/api/summary` — no `Last self-test` alert while the
|
||||||
|
live UPSes report a passed (or never-initiated) result;
|
||||||
|
- the `/` overview: Warnings & errors panel shows no self-test row;
|
||||||
|
- the `/ups` page: the "Last self-test" row renders the driver's word with the `ok`
|
||||||
|
(or `unknown`) class.
|
||||||
|
4. `data/logs/` — `self-test-passed-overview.png` and `self-test-passed-ups.png`
|
||||||
|
(the overview with the Warnings panel visible, and the `/ups` card with the check
|
||||||
|
list visible). If the live UPSes are at the factory `No test initiated`, the
|
||||||
|
screenshots show the `unknown` dim state — the *absence of the warning* is the
|
||||||
|
point, in either state.
|
||||||
|
- ASSUMPTION: no battery test is run on the live hardware to force a `Done and
|
||||||
|
passed` state — a self-test puts real load on a battery things are plugged into,
|
||||||
|
which stays the operator's call (the AGENTS.md known-gaps rule). The hermetic
|
||||||
|
`fail`/`ok` pins in `tests/test_api.py` carry the behaviour; the live screenshots
|
||||||
|
carry whichever state the hardware currently reports.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Integration: the two-direction alert pin in `tests/test_api.py`.
|
||||||
|
- Live: the two surfaces checked per step 3, screenshots recorded.
|
||||||
|
- Coverage: **>90%** on this task's new/modified code (mostly test code — the overall
|
||||||
|
94% gate must hold).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `curl -s http://127.0.0.1:8090/api/summary | grep -i "self-test"` returns nothing
|
||||||
|
while the live `health.checks[]` entry is `ok`/`unknown`, and returns the alert
|
||||||
|
row when a `fail`-tier word is injected (the `tests/test_api.py` pin)
|
||||||
|
- [ ] `/` renders no self-test row in Warnings & errors; `/ups` renders the check with
|
||||||
|
its verdict class — both screenshotted in `data/logs/`
|
||||||
|
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||||
|
- [ ] the `Dev server stays up` rule held: `./scripts/dev-server.sh status` shows the
|
||||||
|
`ups` source `ok` before and after the screenshots
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
# Phase 13 — The LLM page: fleet spacing and an honest decode tok/s
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L4–L5 — "In the LLM engines page the 'fleet' section has no spacing between it and the engine summaries" / "Decode tok/s sometimes jumps unrealistically high, could be a calculation error"
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** `src/dashboards/web/pages/llm.html` (the `#banners`/`#fleet`/`#strip`/engines section stack), `src/dashboards/web/static/dash.css` (`.strip`, the `.fleet` block, the shared `.85rem` rhythm), `src/dashboards/rates.py` (`RateEstimator` — the one piece of mutable state in the app), `src/dashboards/sources/llm.py` (`parse_gufo_metrics`/`parse_vllm_metrics`, the `measured()` Δcounter/Δt cells), `src/dashboards/prom.py` (`scalar(agg="sum")` over label sets), `tests/test_rates.py` / `tests/test_llm_source.py`, `tests/fixtures/metrics_gufo.txt` / `metrics_vllm.txt`, `data/logs/` (findings + screenshots)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Two defects on the `/llm` page, one phase. (1) The `#fleet` aggregate block and the
|
||||||
|
`#strip` engine summaries sit flush — the page's shared `.85rem` vertical rhythm skips
|
||||||
|
the two sections that have no inline margin, so the page reads as one unbroken wall
|
||||||
|
above the engine panels. (2) The `decode tok/s` hero/chart occasionally prints values
|
||||||
|
far above anything the engines can produce: the measured Δcounter/Δt path is keyed
|
||||||
|
correctly by *name* (rule 2), but the *value* it diffs is a **sum over the label
|
||||||
|
set**, and the estimator's window-empty fallback divides by a `dt` it never bounds —
|
||||||
|
so a counter's carried value can leak into one interval's delta, or a stall can
|
||||||
|
rescale the window. The fix keeps every number measured and labelled (`Δ<window>`),
|
||||||
|
just makes the measurement honest.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
- `07_llm_monitor_style_pages` (complete) — the fleet block, the strip, and the
|
||||||
|
one-`/api/llm/history`-request-per-refresh contract this phase must not break.
|
||||||
|
- `04_llm_chart_readability` (complete) — the hero row and the `provenance()` tags.
|
||||||
|
- `11_admin_login` (todo) — no shared files (this phase touches `dash.css`, `llm.html`
|
||||||
|
optionally, `rates.py`, `sources/llm.py`, `prom.py` and their tests; 11 touches
|
||||||
|
`app.py`/`config.py`/`auth.py`). Either order runs.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
|
||||||
|
1. `01_fleet_strip_spacing.md` — restore the shared `.85rem` rhythm between `#fleet` and `#strip` (and above `#fleet` when no banners are shown) in `dash.css`; screenshot.
|
||||||
|
2. `02_pin_decode_spike_mechanism.md` — diagnose the spike against the live engines and the stored history (AGENTS.md rule 7 — never guess the trigger); written findings in `data/logs/`.
|
||||||
|
3. `03_bounded_estimator_and_per_series_rates.md` — the `rates.py`/`sources/llm.py` fix: per-label-series diffing plus a bounded effective window, so a label-set change contributes zero and a stall contributes an `n/a`, never a spike.
|
||||||
|
4. `04_live_stable_verification.md` — the dev-server proof: sustained watch of the live `decode tok/s` with its `Δ<window>` label, screenshots, fixtures re-captured if the live shape changed.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Unit: `tests/test_rates.py` — the new window/label-set behaviours (see task 03); the
|
||||||
|
existing pinned behaviour (`test_window_is_labelled_with_the_span_actually_used`)
|
||||||
|
keeps passing — the labelled-window contract is load-bearing.
|
||||||
|
- Unit/integration: `tests/test_llm_source.py` — a metrics text whose label set gains a
|
||||||
|
series between two polls produces no spike in `decode_tokens_per_second`; the
|
||||||
|
fixture-driven parse tests pass unmodified in intent.
|
||||||
|
- Page: `tests/test_api.py` — the served `llm.html` + `dash.css` pin the spacing rule
|
||||||
|
(the same device that pins the other dash.css invariants).
|
||||||
|
- Coverage: **>90%** on new/modified code; the 94% gate must hold.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `/llm` shows the shared gap between the fleet block and the engine summaries, and
|
||||||
|
between the page head and the fleet block when no banners render — screenshots in
|
||||||
|
`data/logs/` (`llm-fleet-spacing.png`)
|
||||||
|
- [ ] the task-02 findings file exists and names the confirmed mechanism
|
||||||
|
- [ ] `tests/test_rates.py` + `tests/test_llm_source.py` pin: no spike on a label-set
|
||||||
|
change, no unbounded `dt`, the `Δ<window>` label always the window actually used
|
||||||
|
- [ ] a sustained live watch shows no `decode tok/s` value above the confirmed
|
||||||
|
physical ceiling for the fleet's cards (the task-02 note records the ceiling)
|
||||||
|
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||||
|
- [ ] no behavior change in completed phases: the strip, the fleet block, the slot grid
|
||||||
|
and the trend charts render exactly as phase 07/04 shipped them, apart from the gap
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
# Task 01 — The gap between the fleet block and the engine summaries
|
||||||
|
|
||||||
|
**Phase:** `13_llm_page_and_decode_rate` · **Source:** `TODO.md:4` — "In the LLM engines page the 'fleet' section has no spacing between it and the engine summaries"
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`#llm`'s top-level sections break the page's shared `.85rem` vertical rhythm:
|
||||||
|
`#banners` ends with `margin-bottom:.85rem`, the engine section below `#strip` carries
|
||||||
|
an inline `margin-top:.85rem`, but `#fleet` and `#strip` carry **no** margin at all —
|
||||||
|
so the fleet block and the engine summaries sit flush (and, with no banners, the fleet
|
||||||
|
block also sits flush against the page head). Restore the rhythm with two CSS rules;
|
||||||
|
no markup or JS change.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/static/dash.css` — in the fleet/strip section:
|
||||||
|
- `#strip { margin-top: .85rem; }` — the gap the TODO names (fleet → engine
|
||||||
|
summaries). Unconditional on purpose: when `#fleet` is `hidden` (the
|
||||||
|
below-two-engines case) the strip is the first visible section and still needs
|
||||||
|
the same top gap.
|
||||||
|
- `#banners:empty ~ #fleet { margin-top: .85rem; }` — the gap above the fleet block
|
||||||
|
*only* when no banner occupies the slot above it; with a banner present, the
|
||||||
|
banner's own `margin-bottom:.85rem` already is the gap (never two stacked gaps).
|
||||||
|
- The engine section's existing inline `margin-top:.85rem` stays (it already is the
|
||||||
|
strip → panels gap).
|
||||||
|
2. `src/dashboards/web/pages/llm.html` — no change expected; if the executor finds the
|
||||||
|
sections are not siblings the way the markup suggests, fix the CSS selectors to the
|
||||||
|
actual structure rather than re-flowing the markup (the section order is a phase-07
|
||||||
|
contract).
|
||||||
|
3. `tests/test_api.py` — pin the two rules on the served bytes (the existing dash.css
|
||||||
|
pin device: a test that fails if the rules are missing from the served stylesheet),
|
||||||
|
so a later "cleanup" that drops them fails a test.
|
||||||
|
4. Screenshot (invariant 10): dev server up, `/llm` with **≥2 engines answering** (the
|
||||||
|
fleet block visible — below two engines it stays hidden and the fix is invisible),
|
||||||
|
full page top-to-bottom: `data/logs/llm-fleet-spacing.png`. If the live fleet cannot
|
||||||
|
show two engines (the transport gap in AGENTS.md's fleet facts), pin the spacing on
|
||||||
|
the simulated path is **not** an option for a layout claim — instead screenshot the
|
||||||
|
one-engine page (strip visible, `#fleet` hidden, strip's `margin-top` doing its job)
|
||||||
|
and record in the completion note that the two-engine gap was verified by the CSS
|
||||||
|
rule + the served-bytes test, with the live two-engine capture deferred to the
|
||||||
|
operator's engine restore.
|
||||||
|
- ASSUMPTION: the fix is CSS-only in `dash.css`, not an HTML wrapper change — the
|
||||||
|
page's section stack is a phase-07 layout contract and the page already uses
|
||||||
|
inline `margin-top:.85rem` for the same rhythm, so a rule (not a restructure) is
|
||||||
|
the change in the page's own vocabulary.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Page: the served-bytes pin in `tests/test_api.py`.
|
||||||
|
- Live: the screenshot per step 4.
|
||||||
|
- Coverage: **>90%** on new/modified code (CSS + test — the 94% gate must hold).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `#strip` has a `.85rem` top gap in all states (fleet visible, fleet hidden);
|
||||||
|
`#fleet` has the gap above it exactly when no banner is shown
|
||||||
|
- [ ] no double gap (banner + rule) and no flush pair anywhere in the section stack
|
||||||
|
- [ ] `tests/test_api.py` pin green; screenshot in `data/logs/`
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
# Task 02 — Pin the decode-tok/s spike against the live upstream
|
||||||
|
|
||||||
|
**Phase:** `13_llm_page_and_decode_rate` · **Source:** `TODO.md:5` — "Decode tok/s sometimes jumps unrealistically high, could be a calculation error"
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Rule 7: never guess the trigger. Before touching `rates.py`, confirm **which** mechanism
|
||||||
|
produces the spike, from the stored history and the live engines, and record the
|
||||||
|
findings in a file the fix task (03) and the verification task (04) both read.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. Find the spikes in the stored history (the dev DB is the source of truth — AGENTS.md
|
||||||
|
known gap 4 records it at ~610 MB with a year of raw/1m/1h rollups):
|
||||||
|
`sqlite3 data/… "<db>" "SELECT ts, value FROM sample WHERE metric LIKE
|
||||||
|
'decode_tokens_per_second%' ORDER BY ts DESC LIMIT 40000"` — list the top-N values
|
||||||
|
and their timestamps. Note for each: which engine (`source`/`host`), and what the
|
||||||
|
neighbouring values look like (a one-poll spike vs a sustained step).
|
||||||
|
2. For each spike, read the **provenance** of the cell that produced it — the payload
|
||||||
|
carries it (rule 2): `computed:Δ<series>/Δt` + `Δ<window>` (our estimator) vs the
|
||||||
|
engine's own gauge name (the native `decode_tokens_per_second_native` source string).
|
||||||
|
A spike from the *engine's* gauge is the engine's rolling average, not our math —
|
||||||
|
the fix there is a label, not a code change. `sqlite3` the engine's native gauge
|
||||||
|
series alongside to separate the two paths.
|
||||||
|
3. For the estimator path, inspect the live exposure (from a host that can reach the
|
||||||
|
engines — the laptop can reach both, per the fleet facts):
|
||||||
|
- `curl -s <gufo>/metrics | grep -E "tokens_(predicted|generated)_total"` and the
|
||||||
|
vLLM equivalent — are the counters **plain** (as in `tests/fixtures/`), or
|
||||||
|
**labelled per slot/engine** in the live build? A label set that changes between
|
||||||
|
polls is the leading suspect: `prom.Prom.scalar(agg="sum")` sums whatever label
|
||||||
|
sets exist, so a series that appears with a carried cumulative value adds that
|
||||||
|
whole value to one interval's delta (a `value < samples[-1]` decrease would reset
|
||||||
|
the window — an *increase* from a new series does not).
|
||||||
|
- Cross-check the spike timestamps against `GET /slots` history (which slots were
|
||||||
|
starting/turning over) and the event feed (`/api/events`) for engine restarts in
|
||||||
|
the same minutes.
|
||||||
|
4. Record the findings in `data/logs/NOTES-decode-spike.md`: the confirmed mechanism
|
||||||
|
(one of the candidates below, or a new one), the evidence (values, timestamps,
|
||||||
|
captures), and the fleet's physical decode ceiling for the task-04 comparison —
|
||||||
|
the observed max of the native gauge over 90d **and** the cards' spec ceiling
|
||||||
|
(R9700 / Strix Halo 8060S), so "unrealistically high" has a number attached.
|
||||||
|
|
||||||
|
Candidates, in suspected order (the findings file must name the one that fired):
|
||||||
|
- (a) **label-set sum**: `scalar(agg="sum")` over a changing set of labelled
|
||||||
|
series — a new series' carried counter enters the delta as a one-interval spike;
|
||||||
|
- (b) **window-empty fallback**: `rates.py`'s `base = samples[-2]` when fewer than
|
||||||
|
two samples sit inside the span — the effective window is the gap to the previous
|
||||||
|
sample, unbounded above by 2×span (the trim keeps the last two however old);
|
||||||
|
- (c) **a lineage jump that is not a decrease**: a counter that restarts *above*
|
||||||
|
zero (a renamed/relabelled series resolving under the same key, a pre-loaded
|
||||||
|
counter) — `value < samples[-1]` never fires, so the carried value is diffed;
|
||||||
|
- (d) **the engine's own native gauge** (vLLM's `avg_*_throughput` rolling average) —
|
||||||
|
not our math; label-level fix only.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- No code changes in this task — diagnostics only, with the findings file as the
|
||||||
|
artifact. If a candidate is confirmed hermetically reproducible, add the red test
|
||||||
|
here (a `tests/test_llm_source.py` / `tests/test_rates.py` test that fails on the
|
||||||
|
current code) so task 03 turns it green.
|
||||||
|
- Coverage: **>90%** on any test added (the 94% gate must hold).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `data/logs/NOTES-decode-spike.md` exists: confirmed mechanism, evidence, and the
|
||||||
|
physical ceiling with its two sources (observed native-gauge max + card spec)
|
||||||
|
- [ ] every spike in the stored top-N is classified (our estimator / the engine's
|
||||||
|
gauge / a real burst) — a spike that cannot be classified is a finding, not a
|
||||||
|
skipped row
|
||||||
|
- [ ] a red test exists for the confirmed mechanism if it is hermetically reproducible
|
||||||
|
- [ ] the fixtures were compared against the live exposure and any shape drift is
|
||||||
|
recorded (drift → `./scripts/capture-fixtures.sh` re-capture noted for task 04)
|
||||||
+85
@@ -0,0 +1,85 @@
|
|||||||
|
# Task 03 — A rate that cannot be a carried counter
|
||||||
|
|
||||||
|
**Phase:** `13_llm_page_and_decode_rate` · **Source:** `TODO.md:5` — "Decode tok/s sometimes jumps unrealistically high, could be a calculation error"
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Fix the confirmed mechanism from task 02, plus the two generic weaknesses that make
|
||||||
|
any mechanism possible: a summed counter whose *label set* changes between polls, and
|
||||||
|
an effective window the estimator never bounds. After this task, a delta that includes
|
||||||
|
a carried counter value is impossible by construction — a new series contributes
|
||||||
|
nothing until it has a baseline, a stalled window answers `n/a` for one poll instead of
|
||||||
|
rescaling, and a jump that looks like a lineage break is confirmed over two polls
|
||||||
|
before it is published. Every published number still carries its `Δ<window>` label and
|
||||||
|
`source` string (rule 2).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/prom.py` — expose the **per-label-set** view the fix needs (the
|
||||||
|
`series` map already holds them; `by_label` shows the shape): a `values_by_series`
|
||||||
|
(or reuse `values` with labels) accessor that returns `[(labels, value)]` per
|
||||||
|
resolved name, so the source can diff each labelled series individually. Keep
|
||||||
|
`scalar` untouched (the gauges still sum/avg/max the same way — a *gauge's* sum is
|
||||||
|
a real aggregate; only *counter* deltas may not mix series).
|
||||||
|
2. `src/dashboards/sources/llm.py` — the `measured()` path (gufo's
|
||||||
|
`tokens_generated_total`/`tokens_prompt_total` and vLLM's
|
||||||
|
`generation_tokens_total`/`prompt_tokens_total`): resolve the per-label-set values
|
||||||
|
and run **each series through the estimator under its own key** (the key is the
|
||||||
|
resolved name *plus* its label set — `llamacpp:tokens_predicted_total{slot="2"}`),
|
||||||
|
then `rate = Σ per-series rates`, with the cell's `window` the max of the
|
||||||
|
contributing series' windows (the window label says the widest window actually
|
||||||
|
used). A series with no previous sample contributes 0 tokens this interval — never
|
||||||
|
its carried value. The `Δ<window>` label and the `computed:Δ…/Δt` source string stay
|
||||||
|
in the `provenance()` format the page already prints (the source string names the
|
||||||
|
resolved series, as today).
|
||||||
|
- ASSUMPTION: per-series diffing is the fix for candidates (a) and (c) — a
|
||||||
|
label-set change then changes *nothing* in the delta until the new series has two
|
||||||
|
samples, which is exactly the "a renamed counter starts a fresh window" doctrine
|
||||||
|
the estimator docstring already states, extended from names to label sets.
|
||||||
|
3. `src/dashboards/rates.py` — the two generic bounds:
|
||||||
|
- **stalled-window bound**: in the `len(inside) < 2` fallback, accept `samples[-2]`
|
||||||
|
as the base only while its age is ≤ 1.5 × `span`; older than that, return
|
||||||
|
`(None, None)` for this poll — one `n/a` on the poll after a stall is the honest
|
||||||
|
answer (a 20-second window on a 10-second panel is a different claim), and the
|
||||||
|
next poll recovers normally. The existing `test_window_is_labelled_with_the_span_actually_used`
|
||||||
|
(a 9s fallback) keeps passing: 9s ≤ 15s.
|
||||||
|
- **jump confirmation**: a published rate ≥ 8× the previous poll's published rate
|
||||||
|
(previous > 0) is held — the previous value is repeated for one more poll, and the
|
||||||
|
jump publishes only if *confirmed* by the next interval; an unconfirmed jump is
|
||||||
|
dropped (the cell prints the held value, its `Δ<window>` label unchanged). This
|
||||||
|
suppresses a single-interval artifact without flattening a real burst: a genuine
|
||||||
|
step-up is confirmed one interval later and then publishes, and the confirmation
|
||||||
|
state is per-key in the estimator.
|
||||||
|
- ASSUMPTION: the 8×-over-previous threshold and the one-poll hold are the right
|
||||||
|
confirmation rule — a real LLM decode step (a batch of long outputs starting)
|
||||||
|
persists across a 10s interval, while a carried-counter artifact does not; the
|
||||||
|
thresholds are named constants at the top of the file so they can move without a
|
||||||
|
logic change.
|
||||||
|
4. `tests/test_rates.py` — the pins: a stalled window (previous sample > 1.5×span old)
|
||||||
|
answers `(None, None)` for one poll and recovers; a jump ≥ 8× holds for one poll
|
||||||
|
and publishes when confirmed / drops when not; the pinned pre-existing tests all
|
||||||
|
pass unmodified.
|
||||||
|
5. `tests/test_llm_source.py` — the label-set pin: two parses of a metrics text whose
|
||||||
|
counter series set differs between polls (a `{slot="2"}` series appears carrying
|
||||||
|
50 000) — the delta of the second poll contains **no** 50 000; the flat-counter and
|
||||||
|
counter-reset tests keep passing.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Unit: `tests/test_rates.py` + `tests/test_llm_source.py` per the work list.
|
||||||
|
- Regression: every pre-existing `test_llm_source.py` / `test_rates.py` test passes
|
||||||
|
unmodified in intent; the fixture-driven parse tests (`metrics_gufo.txt`,
|
||||||
|
`metrics_vllm.txt`) are the shape pins.
|
||||||
|
- Coverage: **>90%** on this task's new/modified code; the 94% gate must hold.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] no delta can contain a value that was not generated inside the window: the
|
||||||
|
label-set test, the reset test and the stall test all green
|
||||||
|
- [ ] a real step-up (sustained over two polls) still publishes — the confirmation
|
||||||
|
path is tested, not just the suppression path
|
||||||
|
- [ ] every published rate still carries `Δ<window>` (the window actually used, possibly
|
||||||
|
a sum-of-windows label) and its `source` string — `tests/test_llm_source.py`
|
||||||
|
asserts the provenance shape
|
||||||
|
- [ ] the task-02 red test is green
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
# Task 04 — Live proof: a decode rate that stays under its ceiling
|
||||||
|
|
||||||
|
**Phase:** `13_llm_page_and_decode_rate` · **Source:** `TODO.md:5` — "Decode tok/s sometimes jumps unrealistically high, could be a calculation error"
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The dev-server proof that the fix holds on the real engines: a sustained watch of the
|
||||||
|
live `decode tok/s` (payload cell + stored history) with its `Δ<window>` label, every
|
||||||
|
observed value under the physical ceiling task 02 recorded, and the page itself
|
||||||
|
screenshotted with the task-01 spacing in view.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. Dev server up and the LLM sources `ok` (`./scripts/dev-server.sh status`); record
|
||||||
|
the start time. Watch for at least one full poll-cycle cluster with traffic if the
|
||||||
|
engines are idle (the operator's own requests are the traffic — do **not**
|
||||||
|
generate load: a live test that puts real work on the engines' batteries is not a
|
||||||
|
test, the AGENTS.md live-suite rule):
|
||||||
|
- `curl -s http://127.0.0.1:8090/api/llm/snapshot` every poll for ≥ 10 minutes:
|
||||||
|
log `decode_tokens_per_second` (value, `source`, `window`) per engine;
|
||||||
|
- then `sqlite3` the `decode_tokens_per_second` raw series for the same period and
|
||||||
|
confirm no stored value exceeds the task-02 ceiling.
|
||||||
|
2. `data/logs/llm-decode-stable.png` — the `/llm` page showing the hero row with the
|
||||||
|
`Δ<window>` provenance tag, the strip card, and (when ≥ 2 engines answer) the fleet
|
||||||
|
block with the task-01 gap; a second screenshot if a real burst was observed
|
||||||
|
mid-watch (the confirmed-and-published path — the burst's `Δ` label visible).
|
||||||
|
3. `./scripts/capture-fixtures.sh` — re-capture **only if** task 02 recorded a live
|
||||||
|
shape drift (a label set the fixtures lack); otherwise leave the fixtures
|
||||||
|
untouched and say so.
|
||||||
|
4. `uv run pytest -m live --live` — the opt-in live suite (read-only per its rule;
|
||||||
|
the LLM engine tests skip when their source is unset). Record the tally in the
|
||||||
|
completion note (the fleet's transport gap means the host/disks live tests may
|
||||||
|
still fail — the LLM and UPS ones are this task's bar).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Live: the watch + `sqlite3` comparison per step 1, screenshots per step 2.
|
||||||
|
- No code changes expected in this task — if the watch finds a residual spike the
|
||||||
|
fix missed, the findings go back into `data/logs/NOTES-decode-spike.md` and task 03
|
||||||
|
is re-opened (record the re-open in the completion note rather than fixing inline).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] ≥ 10 minutes of live polls: no `decode_tokens_per_second` value (payload or
|
||||||
|
stored) above the task-02 ceiling; the `Δ<window>` label present on every
|
||||||
|
non-`n/a` value
|
||||||
|
- [ ] `data/logs/llm-decode-stable.png` (and the burst screenshot if one occurred)
|
||||||
|
- [ ] fixture re-capture decision recorded; `bash .agents/validate.sh` green
|
||||||
|
- [ ] the live tally recorded, with the known transport gap acknowledged as the reason
|
||||||
|
any host/disks live failures are out of scope here
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
# Phase 14 — The store: fast reads, short writes, no "database is locked"
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L7 — "Pages take forever to load and seem to lock up the database. Fix that."
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** `src/dashboards/store.py` (the `sample`/`event`/`meta` schema, `pick_res`, `history`, `series_names`, `stats`, `maintain`, the single `self._lock`, `busy_timeout=5000`), `src/dashboards/api.py` (the N-hosts × M-metrics history loops, `/api/metrics`, the polled `/api/health`), `src/dashboards/poller.py` (`add_many` on every successful poll — the write side that loses the `busy_timeout` race), `tests/test_store.py`, `tests/test_api.py`, `data/logs/` (the ~610 MB dev DB per AGENTS.md known gap 4 — the `store-error: OperationalError: database is locked` line)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The page load time and the `database is locked` lines are one disease: the hot read
|
||||||
|
paths of a 610 MB / 140 MB-WAL database are **full-table operations on every request**
|
||||||
|
(`series_names()` scans for `DISTINCT metric`, the browser-polled `/api/health` runs
|
||||||
|
`stats()` which counts the whole `sample` table, `pick_res()` pays three COUNT
|
||||||
|
round-trips per series), while `maintain()`'s rollups are long whole-table write
|
||||||
|
transactions that hold the WAL write lock long enough for a poller's `add_many` to
|
||||||
|
outlive `busy_timeout=5000` and surface as the reported `store-error` (AGENTS.md
|
||||||
|
known gap 4). This phase makes every hot path bounded (indexes, one grouped query,
|
||||||
|
short-TTL caches) and every write short (windowed rollups, WAL checkpoint tuning),
|
||||||
|
with timing budgets pinned as tests so a regression is a red test, not a slow page.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
- the foundation (complete) — `store.py`/`api.py`/`poller.py` as shipped; the rollup
|
||||||
|
and retention contract in AGENTS.md invariant 9.
|
||||||
|
- `11_admin_login` (todo) — **shared files, run-order note**: 11 rewrites `api.py`
|
||||||
|
(the session gate) and adds `get_meta`/`set_meta`/`delete_meta` to `store.py`. Apply
|
||||||
|
this phase's `store.py`/`api.py` changes on top of the file state at the time it
|
||||||
|
runs; never restore a line the other phase owns. The two phases touch disjoint
|
||||||
|
*regions* (11: the meta helpers + the auth routes; 14: the query paths + the
|
||||||
|
maintain rollups) — the coupling is the file, not the logic.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
|
||||||
|
1. `01_baseline_store_costs.md` — a production-shaped seeded store + the timing/EXPLAIN baseline for every hot path, recorded in `data/logs/`.
|
||||||
|
2. `02_no_full_scan_on_hot_paths.md` — `series_names`/`stats` off the per-request full scan (bounded caches), `pick_res` to one grouped query, the N×M history endpoints to one grouped round trip.
|
||||||
|
3. `03_windowed_rollups_and_wal_tuning.md` — `maintain()` rolls up only the window since `last_maintain`; WAL checkpoint + journal-size tuning so the 140 MB WAL stops growing between checkpoints.
|
||||||
|
4. `04_budget_gates_and_live_proof.md` — the timing budgets pinned as tests, the live dev-server proof (page load measured, zero `store-error` over a watch window), docs updated.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Unit/integration: `tests/test_store.py` — the budget tests (a seeded
|
||||||
|
production-shaped store: the task-01 recipe), the cache-invalidation behaviour, the
|
||||||
|
windowed-rollup correctness (a rollup run twice rolls the new window only, and the
|
||||||
|
1m/1h values match a whole-table rollup on the same data — the windowing must not
|
||||||
|
change a single averaged value).
|
||||||
|
- Integration: `tests/test_api.py` — the N×M history endpoints' response shape is
|
||||||
|
byte-identical to today (only the cost changes), under the new budget.
|
||||||
|
- Live: the dev-server watch per task 04 (AGENTS.md invariant 10 — the dev server stays
|
||||||
|
up while work happens; invariant 9 — the DB stays bounded, on a volume in
|
||||||
|
deployment).
|
||||||
|
- Coverage: **>90%** on new/modified code; the 94% gate must hold.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] the task-01 baseline and the task-04 "after" numbers are both in
|
||||||
|
`data/logs/NOTES-store-perf.md`, side by side
|
||||||
|
- [ ] the budget tests are green: the full `/system` history request, `/api/metrics`,
|
||||||
|
`/api/health` and a 90d `history()` call all inside their pinned budgets on the
|
||||||
|
seeded store
|
||||||
|
- [ ] a 25-minute dev-server watch (the task-04 recipe) shows **zero** `store-error`
|
||||||
|
events and page loads inside the budget — the known-gap-4 failure mode is gone,
|
||||||
|
not hidden
|
||||||
|
- [ ] `maintain()`'s write transaction is measurably shorter (the task-03 note) and
|
||||||
|
the WAL stays bounded across two maintain cycles
|
||||||
|
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||||
|
- [ ] no behavior change in completed phases: every history payload, rollup value and
|
||||||
|
event answer is identical to today's shape (the `tests/test_api.py` pins)
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
# Task 01 — What the hot paths actually cost
|
||||||
|
|
||||||
|
**Phase:** `14_store_performance` · **Source:** `TODO.md:7` — "Pages take forever to load and seem to lock up the database. Fix that."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
A before/after exists only if the "before" is measured, not remembered. Seed a
|
||||||
|
production-shaped store, time every hot path of `store.py`/`api.py` against it,
|
||||||
|
`EXPLAIN QUERY PLAN` the queries that will change, and record the baseline in
|
||||||
|
`data/logs/NOTES-store-perf.md` — the numbers task 04 compares against.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `tests/test_store.py` — the seeded-store recipe (a fixture the budget tests of this
|
||||||
|
phase all reuse): an on-disk SQLite file (WAL, like production — the in-memory DB
|
||||||
|
hides the lock behaviour) seeded to the dev DB's *shape and retention semantics*
|
||||||
|
(`raw_keep_hours=48`, `minute_keep_days=30`, `hour_keep_days=400` — the
|
||||||
|
`SampleStore` defaults, i.e. what a 90 d chart actually reads: 1m/1h, never raw
|
||||||
|
beyond 48 h):
|
||||||
|
- ~3 hosts × ~30 series per host, 10 s cadence for the 48 h raw window;
|
||||||
|
- their 1m rollups for the 30 d window and 1h rollups for the 400 d window
|
||||||
|
(generated by calling `maintain()` — a few hundred MB, the dev DB's order of
|
||||||
|
magnitude, a temp file, never the dev DB itself: tests stay hermetic per the
|
||||||
|
AGENTS.md testing policy);
|
||||||
|
- ~2 000 event rows and the `meta` rows the app writes (`last_maintain` included).
|
||||||
|
The test asserts the row counts, not the wall time of seeding.
|
||||||
|
- ASSUMPTION: "production-shaped" means the dev DB's *shape and magnitude* (~610 MB,
|
||||||
|
per AGENTS.md known gap 4), reproduced in a temp file — the dev DB itself is
|
||||||
|
never opened by a test (tests stay hermetic, the AGENTS.md testing policy), and
|
||||||
|
a temp DB of the same shape is the honest substitute.
|
||||||
|
2. Measure (a script or the test's `--store-perf` marker; record in
|
||||||
|
`data/logs/NOTES-store-perf.md` with the machine's specs):
|
||||||
|
- `store.series_names()` (what `/api/metrics` runs per call) — wall time +
|
||||||
|
`EXPLAIN QUERY PLAN` (expected: a full scan of `sample`);
|
||||||
|
- `store.stats()` (what the browser-polled `/api/health` runs) — wall time +
|
||||||
|
EXPLAIN (expected: a full COUNT scan of `sample` + `event`);
|
||||||
|
- `store.pick_res()` × 3 + `store.history()` for one 90 d window (what one chart
|
||||||
|
series costs) — wall time + EXPLAIN;
|
||||||
|
- the full `/api/system/history` request shape (3 hosts × 9 series — the
|
||||||
|
`tests/test_system_contract.py` table, 27 `_history` calls) through the test app
|
||||||
|
— total wall time;
|
||||||
|
- a `maintain()` run on the seeded store — the write transaction's wall time
|
||||||
|
(the lock-holding window; note the `to_hour` step already scans the *entire*
|
||||||
|
30 d 1m window on every run — the suspected long write) and the WAL file's
|
||||||
|
size before/after;
|
||||||
|
3. Record the baseline table in `data/logs/NOTES-store-perf.md`: path, wall time,
|
||||||
|
EXPLAIN shape, and the wall time of the poller's `add_many` (a 30-sample insert)
|
||||||
|
issued **concurrently** with a `stats()`/`series_names()` call — the race known gap
|
||||||
|
4 describes, made visible as a number (how often does the insert wait, and does it
|
||||||
|
ever outlive `busy_timeout=5000`? run the race loop ~100× and count the waits).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Unit: the seeded-store fixture in `tests/test_store.py` (asserts row counts and the
|
||||||
|
rollup values match a second `maintain()` pass — determinism check).
|
||||||
|
- No behaviour changes in this task — measurement only.
|
||||||
|
- Coverage: **>90%** on the fixture code; the 94% gate must hold.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `data/logs/NOTES-store-perf.md` exists: the full baseline table (wall times,
|
||||||
|
EXPLAIN shapes, the concurrent-insert wait count), the machine's specs, and the
|
||||||
|
seeded DB's row counts
|
||||||
|
- [ ] the baseline confirms (or refutes — record the refutation) the three suspect
|
||||||
|
mechanisms: `series_names` full scan, `stats` full scan, `maintain`'s long write
|
||||||
|
window
|
||||||
|
- [ ] the fixture is green and deterministic (two runs, same rollup values)
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
# Task 02 — No full table scan on a per-request path
|
||||||
|
|
||||||
|
**Phase:** `14_store_performance` · **Source:** `TODO.md:7` — "Pages take forever to load and seem to lock up the database. Fix that."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Three per-request paths pay a whole-table price on a 610 MB database:
|
||||||
|
`series_names()` (a `SELECT DISTINCT metric` scan, on every `/api/metrics` call),
|
||||||
|
`stats()` (a `COUNT(*) GROUP BY res` over all of `sample` plus `event`, on every
|
||||||
|
browser-polled `/api/health`), and the N-hosts × M-metrics history endpoints
|
||||||
|
(`pick_res()` is three COUNT round-trips per series, so one `/system` chart request
|
||||||
|
is ~108 serialized round-trips through the single `self._lock`). Make every hot path
|
||||||
|
bounded: short-TTL caches for the two scanned answers, one grouped query where three
|
||||||
|
stand, and one grouped round trip where N×M stand. The response shapes are byte-
|
||||||
|
identical — only the cost moves.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/store.py` — `series_names()`: an in-memory cache of the distinct
|
||||||
|
metric set (per `source`/`host` argument tuple, a 60 s TTL, invalidated on
|
||||||
|
`add`/`add_many`/`delete_series` via a dirty flag — a cache that outlives a write
|
||||||
|
answers "why is this chart empty" with yesterday's truth). `stats()`: cache the
|
||||||
|
`sample`/`event` counts the same way (30 s TTL) — the `/api/health` poll (the
|
||||||
|
browser's `app.js` masthead loop) must never run a full COUNT scan; `last_maintain`
|
||||||
|
stays read through `get_meta()` as today (the meta table is tiny; do not cache
|
||||||
|
what a three-row table costs nothing to read).
|
||||||
|
2. `src/dashboards/store.py` — `pick_res()`: replace the three per-`res` COUNT queries
|
||||||
|
with **one** `SELECT res, COUNT(*) … WHERE … ts >= ? GROUP BY res` (the three
|
||||||
|
resolutions in a single index-range pass), same decision logic, same return value.
|
||||||
|
3. `src/dashboards/api.py` — the four history endpoints' loops (system/disks/llm/ups —
|
||||||
|
the `[host, metric]` nesting in each) call `store.history()` per pair, which is
|
||||||
|
`pick_res()` (now 1 query) + the thinning query. Add the batched door in
|
||||||
|
`store.py`: `pick_res_many(hosts, metrics, span, max_points)` and
|
||||||
|
`history_many(hosts, metrics, span, max_points)` — one grouped COUNT over the whole
|
||||||
|
`host IN (…) AND metric IN (…) AND res IN (…) AND ts >= ?` range, and one
|
||||||
|
per-resolution thinning query (the per-series `LIMIT ?` tail-thinning kept by a
|
||||||
|
window function or a per-series subquery — the shape of each series' answer must
|
||||||
|
stay identical: ascending `(ts, value)`, `tail=True` semantics preserved). The
|
||||||
|
endpoints' response dicts are unchanged (the same series, the same order); the
|
||||||
|
`tests/test_api.py` pins are the contract.
|
||||||
|
- ASSUMPTION: one grouped round trip (plus one thinning query per resolution
|
||||||
|
present) is the target for the N×M endpoints — the alternative (N×M separate
|
||||||
|
queries, each now cheaper) leaves ~100 serialized round-trips on a 3-host request,
|
||||||
|
which on a 610 MB file is still the page-load tail; the grouped shape is the fix
|
||||||
|
the TODO asks for. The per-series thinning keeps `tail=True` (the "the chart must
|
||||||
|
have 'now'" docstring) exactly as today.
|
||||||
|
4. `tests/test_store.py` — the pins: `series_names()`/`stats()` answer from cache
|
||||||
|
within the TTL (a write after the read invalidates `series_names`; `stats` expires
|
||||||
|
on the clock); `pick_res_many`/`history_many` equal the per-series answers on the
|
||||||
|
seeded store (the byte-identical-shape test); a cache hit makes a second
|
||||||
|
`series_names()` call measurably not scan (assert via a query-counter hook or the
|
||||||
|
wall-time budget of task 04).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Unit/integration: per the work list, on the task-01 seeded store.
|
||||||
|
- Regression: every pre-existing `tests/test_store.py` / `tests/test_api.py` test
|
||||||
|
passes unmodified in intent — the response shapes are the contract.
|
||||||
|
- Coverage: **>90%** on this task's new/modified code; the 94% gate must hold.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] no per-request path in `api.py` triggers a full-table scan of `sample` or
|
||||||
|
`event` (the task-04 budgets are the proof; `EXPLAIN QUERY PLAN` of the grouped
|
||||||
|
queries shows index-range access, recorded in the notes file)
|
||||||
|
- [ ] the N×M endpoints' answers are byte-identical to the per-series path on the
|
||||||
|
seeded store (the equality test)
|
||||||
|
- [ ] a fresh `series_names()`/`stats()` call costs the full scan (documented — the
|
||||||
|
first poll after a process start pays it once), a repeated call costs the cache
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
# Task 03 — A rollup that holds the write lock for seconds, not minutes
|
||||||
|
|
||||||
|
**Phase:** `14_store_performance` · **Source:** `TODO.md:7` — "Pages take forever to load and seem to lock up the database. Fix that."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`maintain()` is "safe to call every minute" and is idempotent — but on a real-sized
|
||||||
|
store the `to_hour` step's `SELECT … GROUP BY` scans the **entire 30-day 1m window**
|
||||||
|
on every run (and `to_minute` the entire 48 h raw window), and that GROUP BY is
|
||||||
|
inside the one `with self._lock, self._db:` transaction: a multi-second write
|
||||||
|
transaction that holds the WAL write lock long enough for a poller's `add_many` to
|
||||||
|
outlive `busy_timeout=5000` and surface as the `store-error: OperationalError:
|
||||||
|
database is locked` line (AGENTS.md known gap 4). Window each step to *what became
|
||||||
|
due since the last run* — the idempotent `INSERT OR REPLACE` makes a slightly wider
|
||||||
|
re-scan safe, so the window is a cost bound, not a correctness device — and tune the
|
||||||
|
WAL so its 140 MB growth stops being a per-run surprise.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/store.py` — `maintain()`, both rollup steps:
|
||||||
|
- read `last_maintain` through `get_meta()` **before** the transaction (the
|
||||||
|
existing `stats()` note applies: the meta read is outside the non-reentrant
|
||||||
|
lock);
|
||||||
|
- `to_minute`: the raw scan bound moves from `ts >= now - raw_keep_seconds` to
|
||||||
|
`ts >= max(now - raw_keep_seconds, last_maintain - MINUTE - 60)` — steady state
|
||||||
|
(a run per minute) scans ~one minute of raw rows; a missed interval scans the
|
||||||
|
catch-up window; a first run on a fresh DB (no `last_maintain` row) scans the
|
||||||
|
full retention window exactly as today;
|
||||||
|
- `to_hour`: the same pattern against the 1m rows (`ts >= max(now -
|
||||||
|
minute_keep_seconds, last_maintain - HOUR - 60)`). The `ts < now - {MINUTE|HOUR}
|
||||||
|
- 60` incompleteness guard stays — a bucket is still only rolled once its whole
|
||||||
|
window is in the past.
|
||||||
|
- the `DELETE … pruned` step is unchanged (the retention window is the correct
|
||||||
|
bound for a prune; it is `DELETE`s on indexed ranges, not a GROUP BY).
|
||||||
|
2. `src/dashboards/store.py` — the WAL tuning in `__init__` (after the existing
|
||||||
|
pragmas): `PRAGMA wal_autocheckpoint = 1000` (the default 1000 pages ≈ 4 MB
|
||||||
|
auto-checkpoint is already the default — the 140 MB WAL means checkpoints lose to
|
||||||
|
the write rate; the real lever is the journal-size cap) and
|
||||||
|
`PRAGMA journal_size_limit = 67108864` (64 MB — the WAL stops growing past it;
|
||||||
|
checkpoints then truncate it). Keep `journal_mode=WAL` and
|
||||||
|
`synchronous=NORMAL` (the "a lost sample is fine, corruption is not" comment
|
||||||
|
stands); keep `busy_timeout=5000` — the fix is a shorter write, not a longer wait.
|
||||||
|
- ASSUMPTION: 64 MB is the journal-size cap — the dev DB's 140 MB WAL is the
|
||||||
|
measured "too big" (AGENTS.md known gap 4), 64 MB is the nearest power of two
|
||||||
|
below it, and the cap is a pragma, not a migration; if the live watch (task 04)
|
||||||
|
shows checkpoint truncation starving writers, the cap moves (named constant in
|
||||||
|
`__init__`, one-line change).
|
||||||
|
3. `tests/test_store.py` — the pins:
|
||||||
|
- **windowing correctness**: seed a store, run `maintain()` (full window, no
|
||||||
|
`last_maintain` yet), advance time and add new 1m buckets, run `maintain()`
|
||||||
|
again — the second run's `to_hour`/`to_minute` scan (assert via the EXPLAIN/row
|
||||||
|
count hook, or a `sqlite3` trace) touches only the since-`last_maintain` window,
|
||||||
|
**and** the rolled 1m/1h values are byte-identical to a reference whole-window
|
||||||
|
rollup on the same data (the windowing must not change a single averaged value —
|
||||||
|
`INSERT OR REPLACE` idempotency is what makes this safe, prove it);
|
||||||
|
- **catch-up**: no `maintain` for 10 minutes, then one run — the 10 minutes of
|
||||||
|
buckets are all rolled (the wider window, not a gap);
|
||||||
|
- **the lock race**: the task-01 concurrent loop (a poller-style `add_many` during
|
||||||
|
a `maintain()` run) now completes with **zero** `OperationalError`s over the
|
||||||
|
100× loop (the baseline's wait count is the before, this is the after).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Unit: per the work list, on the task-01 seeded store.
|
||||||
|
- Regression: every pre-existing `tests/test_store.py` test passes unmodified in
|
||||||
|
intent (the idempotency/retention tests are the load-bearing ones).
|
||||||
|
- Coverage: **>90%** on this task's new/modified code; the 94% gate must hold.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] the second `maintain()` run's write transaction is measured shorter than the
|
||||||
|
task-01 baseline (both numbers in `data/logs/NOTES-store-perf.md`), and the
|
||||||
|
rollup values are unchanged (the equality test)
|
||||||
|
- [ ] the 100× concurrent `add_many`-during-`maintain` loop: zero `OperationalError`
|
||||||
|
(baseline wait count vs after, in the notes file)
|
||||||
|
- [ ] two `maintain()` cycles on the seeded store: the WAL file size stays bounded by
|
||||||
|
the 64 MB cap (before/after sizes recorded)
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
# Task 04 — The budgets, and the dev server that proves them
|
||||||
|
|
||||||
|
**Phase:** `14_store_performance` · **Source:** `TODO.md:7` — "Pages take forever to load and seem to lock up the database. Fix that."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The "after" numbers, pinned as tests so a regression is red instead of slow: the
|
||||||
|
task-01 baseline vs the fixed store, side by side in `data/logs/NOTES-store-perf.md`,
|
||||||
|
plus the live dev-server proof on the real 610 MB DB — page loads inside budget, zero
|
||||||
|
`store-error` events over a watch window (the known-gap-4 failure mode gone, not
|
||||||
|
hidden), and the docs updated.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `tests/test_store.py` / `tests/test_api.py` — the budget tests (the task-01 recipe
|
||||||
|
on the seeded store; budgets are the task-01 baseline's *target* — record the
|
||||||
|
chosen numbers in the notes file with the reasoning):
|
||||||
|
- the full `/api/system/history` request (3 hosts × 9 series, 1 h and 90 d):
|
||||||
|
wall time budget (the 90 d case reads the 1m/1h rollups — the one that used to
|
||||||
|
be the page-load tail);
|
||||||
|
- `store.series_names()` / `store.stats()` second call (cache hit): wall time
|
||||||
|
budget, asserted to be a fraction of the first call's (the scan is gone);
|
||||||
|
- one 90 d `store.history()` call: wall time budget;
|
||||||
|
- the budgets are generous (2–3× the measured "after"), so a slower CI machine does
|
||||||
|
not flake red — the notes file records both the machine's specs and the margin.
|
||||||
|
2. The live proof (AGENTS.md invariant 10 — the dev server stays up while work
|
||||||
|
happens, and a fix is screenshotted, not claimed; invariant 9 — the DB is the
|
||||||
|
volume's business in deployment, the dev DB is the test bench):
|
||||||
|
- `./scripts/dev-server.sh restart` (its `stop` waits for the old pid to die —
|
||||||
|
known gap 4's "two servers on one file" cause stays fixed), `status` shows the
|
||||||
|
sources;
|
||||||
|
- page loads: `curl -o /dev/null -s -w '%{time_total}'` on `/` and on the
|
||||||
|
history-heavy pages' API calls (signed in, per phase 11), five runs each,
|
||||||
|
recorded; the browser's `/llm` or `/system` page with a 90 d window selected is
|
||||||
|
the worst case — screenshot `data/logs/store-perf-live.png` (the page loaded,
|
||||||
|
the charts drawn, the masthead dots `ok`);
|
||||||
|
- the watch: ≥ 25 minutes of `./scripts/dev-server.sh logs` (longer than the
|
||||||
|
gap-4 baseline's "a few times an hour" rate) with **zero**
|
||||||
|
`store-error: OperationalError: database is locked` lines; the `store.stats()`
|
||||||
|
`event` count for `kind='store-error'` over the window is 0 (the `sqlite3`
|
||||||
|
check on the dev DB — read-only);
|
||||||
|
- the WAL file's size after the watch (the cap from task 03, in the real DB).
|
||||||
|
- ASSUMPTION: the watch window is 25 minutes against a "few times an hour"
|
||||||
|
baseline rate — enough to cover > 0.4 of the baseline's expected occurrences at
|
||||||
|
the mean rate; if the baseline was at the mean, a zero-count 25-minute window
|
||||||
|
already moves the needle, and the `sqlite3` event-table check over *the whole
|
||||||
|
dev DB's* `store-error` rows (not just the window) shows when the last
|
||||||
|
occurrence predates this phase.
|
||||||
|
3. Docs: `README.md` — the *Configuration*/*Development* notes gain the two new
|
||||||
|
`PRAGMA`s' one-line "why" (the journal cap, and that it is a cap not a target);
|
||||||
|
`AGENTS.md` — known gap 4's entry is updated to record the fix (the windowed
|
||||||
|
rollups, the journal cap, the budget tests) and the gap's "what remains" line
|
||||||
|
(checkpoint starvation was never shown to be a cause — the controlled 25 s read
|
||||||
|
transaction produced none — so the entry's honest wording stays, with the
|
||||||
|
`database is locked` line now attributed to the rollup's write window). (The
|
||||||
|
AGENTS.md/README update is the phase's *normal work* per the repo's definition of
|
||||||
|
done — this instruction is for the phase's executor, not for the conversion.)
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Unit/integration: the budget tests per step 1.
|
||||||
|
- Live: the step-2 watch, screenshots and log evidence in `data/logs/`.
|
||||||
|
- Coverage: **>90%** on new/modified code; the 94% gate must hold.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `data/logs/NOTES-store-perf.md` has the baseline and the after table side by
|
||||||
|
side (every task-01 path, wall time before/after, EXPLAIN shapes)
|
||||||
|
- [ ] the budget tests are green with the recorded margins; `bash .agents/validate.sh`
|
||||||
|
green
|
||||||
|
- [ ] the 25-minute watch: zero `store-error` lines in the dev-server logs and zero
|
||||||
|
new `store-error` rows in the dev DB's `event` table; `data/logs/` holds the
|
||||||
|
page-load numbers and `store-perf-live.png`
|
||||||
|
- [ ] the WAL's post-watch size is inside the 64 MB cap on the real DB
|
||||||
|
- [ ] `README.md`/`AGENTS.md` carry the updated, accurate wording (gap 4: fixed, with
|
||||||
|
its "what remains" honest)
|
||||||
|
- [ ] no behavior change in completed phases: the `tests/test_api.py` payload pins all
|
||||||
|
pass — the answers are the same, only faster
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# Phase 15 — Protected exporter web servers, auth by certificate
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** `agent/host_agent.py` (the stdlib-only exporter web server; today: a bearer token via `--token`/`--token-file`, refuses an off-loopback bind without one, `ThreadingHTTPServer`, the `resolve_token` idiom), `src/dashboards/config.py` (`Target`/`parse_targets` — token in URL userinfo, `https://` URLs already accepted; **no change needed**), `src/dashboards/collectors.py` (`fetch_snapshot`, the one `UpstreamError`, redacted display URLs, the one shared `httpx.AsyncClient` from the lifespan), `src/dashboards/app.py` (the `PAGES` dict with the settings-page comment, `build_hub`/`build_context`, the lifespan that builds `ctx.auth`), `src/dashboards/api.py` (`AppContext`, the gated-write + event pattern), `src/dashboards/store.py` (`get_meta`/`set_meta`/`delete_meta` — the only durable key/value state, home of `auth.*`), `deploy/install-agent.sh` (the token-file idiom `0640 root:agent-user`, the `ProtectSystem=strict` unit with no `RuntimeDirectory`, the `--status` probe), `tests/test_agent.py` (the in-process agent harness), `tests/test_collectors.py`, `tests/test_deploy_scripts.py`, `tests/test_api.py` (the route walk, the settings-page pin, the orphan-id audit), `src/dashboards/web/pages/settings.html` + `src/dashboards/web/static/page-settings.js` (the "a page, not a dashboard" pattern the new `/certs` admin page follows)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Every exporter (host agent) web server is protected end-to-end by certificates the dashboard itself generates and manages: the dashboard holds a root CA in the `meta` table, signs its own client cert and each agent's server cert, the agent serves TLS with a per-path client-cert check (only the dashboard can read it), the dashboard dials `https://` with its client cert, and a new `/certs` admin page lists every cert with its expiry and status and triggers rotation, revocation, and bundle re-delivery. The operator's stated follow-up — firewalling upsd 3051 and talking to the agent only — is the motivation, not part of this phase. The agent's bearer-token mode stays a compatibility mode; the dashboard's own login is a separate auth domain and is untouched.
|
||||||
|
|
||||||
|
**Scope (operator-confirmed):** "exporters" = the host agents this repo installs. gufo / vLLM / upsd are third-party services this repo does not deploy; their TLS (e.g. upsd `STARTTLS`) is a separate operator concern.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
- `14_store_performance` (todo) — **shared-file, run-order note**: 14 rewrites the hot paths in `store.py`/`api.py`; 15 only *adds* meta keys and new routes/pages. Apply on top of the file state at the time it runs; never restore a line the other phase owns.
|
||||||
|
- `11_admin_login` (complete) — the session gate (invariant 12) for the new routes, the gated-write + CSRF + event pattern for the new `POST /api/certs/*`, the `meta` helpers, and the settings-page pattern `/certs` follows.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
|
||||||
|
1. `01_cert_management.md` — `src/dashboards/certs.py`: the dashboard-managed root CA, client cert, per-agent server certs, revocation and reissue, all in `meta`; the `cryptography` dependency (dashboard-side only).
|
||||||
|
2. `02_agent_tls.md` — `host_agent.py` TLS mode: one installed `--cert-bundle`, the per-path client-cert check, the bundle revocation list, `bundle_rev` in the snapshot; token mode unchanged.
|
||||||
|
3. `03_collector_tls.md` — the dashboard dials `https://` agent URLs with its client cert, verifies against its root, refuses a revoked agent without dialing; error sentences stay redacted.
|
||||||
|
4. `04_cert_api.md` — the gated `/api/certs` surface: list, per-host bundle, rotate, revoke, root-rotate; fixed-text events, no keys out of `meta`.
|
||||||
|
5. `05_certs_admin_page.md` — the `/certs` admin page (the settings pattern): every cert with its expiry and status, issued vs deployed bundle rev, rotate/revoke/root-rotate actions, bundle download.
|
||||||
|
6. `06_deploy_cert_bundle.md` — `install-agent.sh --cert-bundle` installs `/etc/dashboards-agent.certs` (`0600 root:agent-user`), the unit gains the flag + `RuntimeDirectory`, `--status` probes over TLS; the script pins move with it.
|
||||||
|
7. `07_live_rollout.md` — the two live agents flip to TLS, `HOST_TARGETS` goes `https://`, the live suite + health table + screenshot prove it; the `driveripper` install procedure is documented.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Unit: `tests/test_certs.py` (new) — the whole lifecycle against a dict `MetaStore`, exactly how `tests/test_auth.py` does it.
|
||||||
|
- Integration: `tests/test_agent.py` (the TLS handshake matrix against the in-process agent), `tests/test_collectors.py` (the `https` dial, the revoked refusal with a no-dial spy, the error sentences), `tests/test_api.py` (the `/api/certs` shapes, the gate/CSRF on the new writes, the served `/certs` page pins), `tests/test_deploy_scripts.py` (the new flags/files/permissions).
|
||||||
|
- Live: task 07 (AGENTS.md invariant 10 — the dev server stays up; screenshots in `data/logs/`).
|
||||||
|
- Coverage: **>90%** on new/modified code; the 94% gate holds. `uv pip list --outdated` empty (the `cryptography` dependency — invariant 8).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] a fresh database boots with a `meta` root CA, generated once and idempotent across restarts (pinned in `tests/test_certs.py`)
|
||||||
|
- [ ] an agent run with a bundle answers `/snapshot` only to a client cert signed by its root; a wrong-CA or certless client is refused; a revoked serial gets the 403; `/health` stays probeable with the CA only
|
||||||
|
- [ ] with `https://` `HOST_TARGETS`, the dev server's `host:`/`disks:` jobs are `ok`, and a revoked host says "certificate revoked" without any dial (spy-pinned)
|
||||||
|
- [ ] signed-in `GET /certs` lists every cert with expiry and status; rotate/revoke/root-rotate work through the gated API; the bundle download is byte-identical to the stored bundle
|
||||||
|
- [ ] the installer places the bundle `0600 root:agent-user`, the unit comes up in TLS mode, `--status` is green over TLS
|
||||||
|
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green, `uv pip list --outdated` empty
|
||||||
|
- [ ] live: deskwork + gateway in TLS mode, `https://` dials, live suite green, screenshots in `data/logs/`
|
||||||
|
- [ ] no behaviour change in completed phases: the token-mode agent is byte-identical, the engine/UPS polling is untouched, the public surface is unchanged (the route walk)
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
# Task 01 — The dashboard owns a root CA and every cert it signs
|
||||||
|
|
||||||
|
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The certificate lifecycle becomes a pure, testable module: the dashboard generates a root CA once (idempotent across restarts), signs its own client cert and each agent's server cert, and supports reissue and revocation — all state in the `meta` table, exactly how `auth.py` keeps the verifier and the session secret.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `pyproject.toml` — add `cryptography` to `dependencies` (dashboard-side only; the agent stays stdlib-only) and run `uv lock`. Invariant 8: `uv pip list --outdated` empty afterwards.
|
||||||
|
- ASSUMPTION: `cryptography` is the only new dependency and it stays on the dashboard side; the agent (task 02) gains no third-party code — it loads the PEMs with stdlib `ssl`.
|
||||||
|
2. `src/dashboards/certs.py` (new) — the whole lifecycle as pure functions over the three-method `MetaStore` protocol (the same protocol `auth.load_auth_state` uses, so this module imports nothing else from the package, and its tests use a dict):
|
||||||
|
- `ensure_certs(store, *, now: float | None = None)` → a frozen `CertsState` carrying the PEM text (`root_key`, `root_cert`, `client_key`, `client_cert`) plus the root/client `serial` + `not_after`. Reads the `meta` rows first; only generates and persists when missing (first boot); an existing row wins and is **never regenerated on restart** — regenerating on every start would rotate the CA and invalidate every installed agent bundle. `now` is injected (the module never reads a clock itself — the `auth.py` rule).
|
||||||
|
- meta keys (one concern per key, the `auth.*` precedent): `certs.root.key`, `certs.root.cert`, `certs.client.key`, `certs.client.cert`, `certs.agents` (one JSON object mapping `host name` → `{serial, not_after, status, bundle_rev, bundle, san}` — the bundle text is stored, so what the agent holds is byte-identical to what the API serves).
|
||||||
|
- Shapes: root CA self-signed, CN `dashboards-root`, **10 years**; client cert signed by the root, CN `dashboards`, **1 year**; agent server certs signed by the root, CN = the host name, **1 year**, `SAN` = the address(es) the dashboard dials for that host.
|
||||||
|
- ASSUMPTION: 10 y root / 1 y leaf lifetimes. No env seed for the root CA (unlike the admin password, the root key is a secret only the app needs to know, so it is auto-generated on first boot and merely persisted).
|
||||||
|
- `issue_agent_cert(store, state, name, san: list[str])` / `reissue_agent_cert(store, state, name, san)` — a fresh serial, `bundle_rev` +1 (1 on first issue), `status: "active"`, a fresh `bundle` (below). Idempotent per call — never issued on a read.
|
||||||
|
- `rotate_root(store, state)` — a fresh root CA + fresh client cert + every existing agent entry reissued (each `bundle_rev` +1): one action re-arms the whole fleet.
|
||||||
|
- `revoke(store, state, name)` — `status: "revoked"` (the serial stays on the denylist until a reissue gives the host a fresh identity; there is no "unrevoke" — reissue is the fix).
|
||||||
|
- the **bundle** (the install artifact, the TODO's "root certificate installed with each exporter"): one PEM text = the agent's server cert + the agent's key + the root CA *cert* (never the root key) + a `# BUNDLE-REV <n>` line + a `# REVOKED <serial>` line (one per revoked serial of the *current* root generation — a root rotation resets the list, since every cert is reissued anyway).
|
||||||
|
- ASSUMPTION: one client cert for the whole dashboard (the agent verifies "signed by our root", not a per-process identity); one agent entry per `HOST_TARGETS` name.
|
||||||
|
- invariant 6, the `auth.py` rule: no `logging` and no raise with anything interpolated — a malformed stored PEM is a fixed-text `RuntimeError` telling the operator to delete the `certs.*` meta keys (the shape of the auth reset), and nothing else.
|
||||||
|
3. `src/dashboards/app.py` (+ the `AppContext` in `src/dashboards/api.py`) — the startup path builds `ctx.certs` (an `ensure_certs` read, the one-time bootstrap included) **before** `build_hub`, in both the owned-store (`build_context`) and the injected-store (lifespan) branches, through **one** helper so the two wiring sites cannot drift (the `build_hub` rule in AGENTS.md); the PEMs are written once to a private temp dir (mode `0600`, lives as long as the process) and the *paths* are what `AppContext.certs` carries for task 03's dialing.
|
||||||
|
- A `certs` whose stored PEM is malformed (a truncated `meta` row) is a fail-closed startup — a dashboard that cannot present its own credential must not start the loops (the `ctx.auth` rule, the one deliberate raise).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- `tests/test_certs.py` (new) — a dict `MetaStore` (the `tests/test_auth.py` idiom):
|
||||||
|
- the first `ensure_certs` call persists exactly the five `certs.*` keys; the second is byte-identical (no regeneration — pin the `certs.root.cert` value);
|
||||||
|
- the parsed chain (`cryptography.x509` loaders) verifies: client → root, agent → root; the SAN list appears in the agent cert; the expiries match the assumed lifetimes (injected `now`);
|
||||||
|
- `reissue_agent_cert` bumps `serial` and `bundle_rev` and sets `status` active; `revoke` flips the status and puts the serial on the bundle's `# REVOKED` line; a reissue after a revoke gives a fresh serial that is *not* on the list;
|
||||||
|
- `rotate_root` reissues every agent entry (each `bundle_rev` bumped), the bundle carries the new root cert, and no stored value still contains the old root key;
|
||||||
|
- the bundle contains the agent cert + key + root cert + `# BUNDLE-REV` + the `# REVOKED` lines and **never** the root key (assert via the PEM header and a parsed-key mismatch);
|
||||||
|
- a malformed stored PEM → the fixed-text `RuntimeError` naming the meta keys, nothing interpolated.
|
||||||
|
- Coverage: **>90%** on this task's new code.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `uv run pytest tests/test_certs.py` green; the full suite green
|
||||||
|
- [ ] `uv pip list --outdated` empty; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||||
|
- [ ] a fresh store boots with the `certs.*` rows present; a second boot changes no byte of `certs.root.cert`
|
||||||
|
- [ ] no behaviour change in completed work: `auth.*` keys, the public surface, and the poller wiring are untouched
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
# Task 02 — The agent serves a protected TLS web server from one installed bundle
|
||||||
|
|
||||||
|
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`host_agent.py` gains a TLS mode that makes the web server protected by certificate: one installed bundle holds the server cert + key, the root CA, a bundle revision, and a revocation list; the agent requires a client cert signed by the root on every path except `/health` (mirroring the token-mode `/health` exemption, so an installer probe can reach `/health` with the CA only), and it reports the bundle revision it loaded so the dashboard can tell a stale bundle from a fresh one. The token mode is byte-identical to today.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `agent/host_agent.py` — stdlib only (an `ssl` import; no new third-party dependency):
|
||||||
|
- a new flag `--cert-bundle <file>` (env `AGENT_CERT_BUNDLE`): one PEM text containing the server cert, the server key, the root CA cert, a `# BUNDLE-REV <n>` line, and `# REVOKED <serial>` lines. `resolve_bundle(path) -> (context, bundle_rev, revoked: frozenset[str], error)` is a pure function (the `resolve_token` idiom): parse the PEM blocks by text, extract the three blocks, write them to a temp dir for `load_cert_chain` (into `/run/dashboards-agent` when present and writable — the unit's `RuntimeDirectory`, task 06 — else `tempfile.mkdtemp`), and build the context.
|
||||||
|
- The TLS context: `ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)`, `load_cert_chain`, `load_verify_locations(ca)`, `verify_mode = ssl.CERT_OPTIONAL` — the handshake is optional and the **handler** enforces it: on every path except `/health`, `self.connection.getpeercert()` must carry a verified certificate (subject present), else `403 {"ok": false, "detail": "client certificate required"}`; that cert's serial must not be in the bundle's `# REVOKED` set, else `403 {"ok": false, "detail": "client certificate revoked"}`. The denylist file is re-read on mtime change per request (a small file; a redeployed bundle takes effect without a restart — the reason the `# REVOKED` lines live in the bundle at all).
|
||||||
|
- The startup gate: off-loopback with neither a token nor a bundle → the existing exit-2 refusal (its message now names the bundle too); a bundle present → the off-loopback bind is allowed (the client cert is the protection); a token plus a bundle → the token check is skipped on the TLS path (the handshake is the auth).
|
||||||
|
- The snapshot payload gains two **additive** top-level fields in TLS mode: `tls: true` and `bundle_rev: <n>` (absent in token mode — an additive field never changes an existing shape); the `serve()` start line says `tls (client cert required, rev <n>)`.
|
||||||
|
- `AGENT_VERSION` 1.0.0 → 1.1.0; the docstring's deployment paragraph gains one line for the TLS mode (no flavour text — the AGENTS.md audit rule).
|
||||||
|
- ASSUMPTION: `CERT_OPTIONAL` + the per-path check (not `CERT_REQUIRED`) is what keeps `/health` probeable with the CA only — the installer's `--status` and the systemd probe (task 06) cannot present a client cert the installer does not hold, and a probe that cannot health-check is a dead install.
|
||||||
|
- ASSUMPTION: the agent never generates or renews certificates (stdlib cannot mint x509); the bundle is the artifact the dashboard signs and the installer places.
|
||||||
|
2. `tests/test_agent.py` — the existing in-process harness (`serve()` in a thread on `127.0.0.1`, `urllib`/`ssl` clients), with a test CA + leafs generated by `src/dashboards/certs.py` over a dict store (no second generator):
|
||||||
|
- a client with a matching client cert → `/snapshot` 200 with `tls: true` and `bundle_rev` equal to the bundle's;
|
||||||
|
- a client with no cert → the handshake fails or a 403 on a non-`/health` path; `/health` still 200 with a CA-only context (the probe path);
|
||||||
|
- a client signed by a *different* CA → refused;
|
||||||
|
- a client whose serial is in the bundle's `# REVOKED` list → 403 "revoked"; rewrite the bundle file without the line (mtime change) → the same client is accepted on the next request (the denylist re-read);
|
||||||
|
- a token-mode agent (no bundle): the existing tests unchanged and green (byte-identical behaviour — no `tls`/`bundle_rev` fields);
|
||||||
|
- `resolve_bundle` error paths: a missing file, an empty file, a bundle missing the key block, a bundle missing the `# BUNDLE-REV` line (an error — a revless bundle is not one this app signed; ASSUMPTION: the `# BUNDLE-REV` line is mandatory).
|
||||||
|
3. `deploy/` is untouched here (task 06); the non-systemd fallback (`tempfile.mkdtemp`) keeps dev and tests working.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- the `tests/test_agent.py` matrix above; the existing token-mode tests stay green unchanged.
|
||||||
|
- Coverage: **>90%** on this task's new/modified code.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] a TLS-mode agent on a loopback port answers the full handshake matrix (tests) and reports `bundle_rev`
|
||||||
|
- [ ] a token-mode agent is byte-identical to today (existing tests green, no new fields present)
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Task 03 — The dashboard dials agents over https with its own client cert
|
||||||
|
|
||||||
|
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
An `https://` `HOST_TARGETS` entry is now a certificate-authenticated dial: the dashboard presents the client cert task 01 gives it, verifies the agent against its own root CA, refuses a *revoked* agent before dialing (invariant 3's shape — a source it knows is dead says so without a network), and passes the agent's `tls`/`bundle_rev` through the `system` and `disks` payloads so task 05's page can compare the issued rev against the deployed one. Engines and token-mode agents are untouched.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/config.py` — **no change** (an `https://` URL already parses; a TLS host carries no userinfo, and the userinfo token stays available for token-mode hosts).
|
||||||
|
2. `src/dashboards/collectors.py` —
|
||||||
|
- a small frozen `AgentTls` dataclass: `verify: str` (the CA path), `cert: tuple[str, str]` (the client cert + key paths), `status: Callable[[], str]` — the `status` callable returns `"active"`/`"revoked"` from a **fresh** `get_meta("certs.agents")` read (a small-table `SELECT`, cheap at poll cadence, and phase 14's work keeps it off the hot paths; reading it live is what makes a runtime revocation take effect on the next poll without a restart).
|
||||||
|
- `fetch_snapshot(client, target, *, timeout=None, tls: AgentTls | None = None)`: when `tls` is given **and** `target.url` is `https://`, the request passes `verify=<ca path>` and `cert=(cert path, key path)` to `client.get` (httpx per-request overrides — the shared client dials both plain and TLS upstreams, so a per-client `verify` would poison the engine dials).
|
||||||
|
- Revocation before any network: `tls is not None` and `tls.status() == "revoked"` → `UpstreamError(f"{target.name}: agent certificate revoked")` **before** `client.get` — the poller's stale/last-good semantics then do the rest (invariant 4).
|
||||||
|
- A `403` from an agent gets its own honest sentence (the existing `401` branch keeps the token-mode wording): `f"{target.name}: agent refused the client certificate ({detail})"` with the `detail` from the agent's JSON (`client certificate required` / `client certificate revoked`), the redacted `display_url` (invariant 6) like every message in this function.
|
||||||
|
- `fetch_host_snapshot`/`fetch_disk_snapshot` pass the `tls` through; the per-host `AgentTls` is plumbed from `AppContext.certs` into the `host_jobs`/`disks_jobs` lambdas (a new optional `tls` parameter on both `*_jobs`, `build_hub` takes it — one call site per branch so the production and injected wiring cannot drift, the `build_hub` rule in AGENTS.md). `status` reads from the store the jobs already hold.
|
||||||
|
3. `src/dashboards/app.py` (+ the `AppContext` in `api.py`) — `AppContext.certs` (task 01) exposes the temp-dir paths (`ca`, `client_cert`, `client_key`); `build_context` and the lifespan's injected branch build the per-host `AgentTls` objects **before** `build_hub`.
|
||||||
|
4. `src/dashboards/sources/host.py` + `src/dashboards/sources/disks.py` — pass the agent's `tls`/`bundle_rev` snapshot fields through into the per-host payload (additive, the `degraded` idiom) so `/api/system/snapshot` and `/api/disks/snapshot` carry the deployed rev. **No** `severity` for a rev mismatch — a count is not a graded number (the fleet-header rule), the page colours nothing for it.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- `tests/test_collectors.py` — an in-process TLS agent (task 02's harness, in a thread) + `AgentTls` material generated by `certs.ensure_certs` over a dict store:
|
||||||
|
- an `https` target with `tls` → the 200 body; a missing client-cert file and a wrong-CA failure still surface as `UpstreamError` with the redacted URL;
|
||||||
|
- a revoked status (dict store with `status: "revoked"`) → the error is raised and a spy on `client.get` (the `tests/test_api.py` spy idiom) proves **no dial happened**;
|
||||||
|
- an `http` token-mode target: the existing tests unchanged;
|
||||||
|
- the 403 sentence: a scripted agent 403 (a `tests/conftest.py` fake) → the `detail` appears in the message, the URL redacted;
|
||||||
|
- the payload passthrough: a snapshot with `tls: true, bundle_rev: 3` → both the `system` and the `disks` payloads carry it.
|
||||||
|
- Coverage: **>90%** on this task's new/modified code.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `uv run pytest` green; the engine/UPS/token-host paths are byte-identical (existing pins hold)
|
||||||
|
- [ ] a revoked host fails its `host:` and `disks:` jobs with "certificate revoked" and no network dial (spy-pinned)
|
||||||
|
- [ ] `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# Task 04 — The gated /api/certs surface: list, bundle, rotate, revoke, root-rotate
|
||||||
|
|
||||||
|
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The API half of "the collector manages the certs": a signed-in operator can list every cert with its expiry and status, download the exact bundle an agent should hold, and trigger rotation / revocation / root rotation. The routes dial nothing (invariant 1's shape), are gated like every other write (invariant 12), and write fixed-text events — a key never leaves `meta` (invariant 6).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/api.py` — the new routes (the middleware gates them by default; the route-walk test covers them for free):
|
||||||
|
- `GET /api/certs` — `{root: {serial, not_after}, client: {serial, not_after}, agents: [{name, serial, not_after, status, bundle_rev, deployed_bundle_rev, san}]}`. `deployed_bundle_rev` is the `bundle_rev` from the host's last `system` job payload (the same source `alerts.py` reads job states from); a host that never answered → `null`. Reads never issue (no side effects on a read — a `GET` that mints a cert is a surprise).
|
||||||
|
- `GET /api/certs/<name>/bundle` — the stored bundle text, `Content-Type: application/x-pem-file`, `Content-Disposition: attachment; filename="agent-<name>.bundle"` — byte-identical to what `certs.py` stored (the installer's input, task 06/07).
|
||||||
|
- `POST /api/certs/<name>/rotate` — a reissue (`certs.reissue_agent_cert`, the SANs taken from the configured `HOST_TARGETS` URL host(s) + the name); a `cert-rotated` event (a fixed message naming the host — no PEM, no key).
|
||||||
|
- `POST /api/certs/<name>/revoke` — `certs.revoke`; a `cert-revoked` event. The next poll refuses without dialing (task 03) — that is the enforcement; the event is the audit line.
|
||||||
|
- `POST /api/certs/root/rotate` — `certs.rotate_root`; a `cert-root-rotated` event; the response carries `{"ok": true, "restart_required": true}` — the client cert and CA are startup material, so a new root is fully in force at the next process start (the page, task 05, prints that; the honest constraint, not a hidden one).
|
||||||
|
- ASSUMPTION: a root rotation requires a dashboard restart to be fully in force (the client cert + CA load at startup); the API says so instead of pretending a hot-swap that does not exist.
|
||||||
|
- A `<name>` that is not a configured `HOST_TARGETS` name → `404 {"configured": false}` **without** issuing, dialing, or touching the store (invariant 3, the `/api/disks` gate shape, spy-pinned in `tests/test_api.py`).
|
||||||
|
- Every `POST` is a gated write: session + `Origin`/`Sec-Fetch-Site` + the cookie's `X-CSRF-Token` (the `POST /api/ups/command` test idiom — a tokenless write records nothing on the wire, with the token it does).
|
||||||
|
- invariant 6: no `meta` value (a key PEM) may appear in an `event` row, a `/api/health` field, or an error `detail`; the events carry the host name + the action in fixed text.
|
||||||
|
2. `tests/test_api.py` — the signed-in harness (`tests/auth_client.py::sign_in`):
|
||||||
|
- `GET /api/certs` with zero / one / three configured hosts: the key paths pinned (the `/api/disks` pin idiom — a panel that retires can never shrink the payload);
|
||||||
|
- an unknown `<name>`: the `404` and the spies on `httpx.AsyncClient.get` and `collectors.fetch_snapshot` (the `tests/test_api.py` spy idiom) prove no network and no store write (a `meta` row count before/after);
|
||||||
|
- `POST rotate` → `bundle_rev` +1, a `cert-rotated` event row, the `meta` value changed, and the old serial is not on the new bundle's denylist;
|
||||||
|
- `POST revoke` → the status, the event, and (through task 03's path) the next `fetch_snapshot` for that host refuses without a dial (the spy);
|
||||||
|
- `POST root/rotate` → a fresh root serial, every agent reissued, `restart_required: true` in the body;
|
||||||
|
- `GET bundle` → the body byte-identical to the stored `meta` bundle, and a reissue changes the served bytes;
|
||||||
|
- the unsigned gate: the route-walk test (existing, automatic) plus one explicit unsigned `303`/`401` on `GET /api/certs` and a `POST` refusal (the `tests/test_auth_api.py` idiom).
|
||||||
|
- Coverage: **>90%** on this task's new/modified code.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] the routes are signed-in + CSRF-gated (the wire test); the unsigned shape pinned
|
||||||
|
- [ ] an unknown name 404s with no dial and no store write (spy-pinned)
|
||||||
|
- [ ] rotate / revoke / root-rotate each leave a fixed-text event row and no key in the `event`/`health` tables (a test greps the stored events for a PEM header and finds none)
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Task 05 — The /certs admin page: every cert, its expiry, rotate, revoke, re-deliver
|
||||||
|
|
||||||
|
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter." (operator addition: an admin page to see generated certs and expirations, trigger rotation, and revoke certs)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
A signed-in operator sees every cert the dashboard has generated — the root CA, its own client, and one per host — with its expiry and status, can rotate / revoke / root-rotate through task 04's API, and can download the exact bundle an agent should hold. The page follows `settings.html`'s precedent exactly: a page and not a dashboard — in `PAGES` (so it is a route), not in `.tabs`, linked from `.masthead-right`, with no `aria-current` on the tabs and no status dots.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/pages/certs.html` + `src/dashboards/web/static/page-certs.js` (the `page-settings.js` idiom: the same `<head>` + masthead, `Dash.fetchJSON`, errors printed through `textContent`, a `confirm()` for destructive actions — the typed `DANGER` word is reserved for power cuts):
|
||||||
|
- the **Root CA** block: the CN, the serial, the expiry (a local timestamp **and** the time left, the settings session-display idiom), and the **Rotate root CA** button → `POST /api/certs/root/rotate` with a confirm dialog; the response's `restart_required` prints "restart the dashboard to activate the new root" (the API's own sentence, not an invented one).
|
||||||
|
- the **Agents** table: one row per configured `HOST_TARGETS` host (the `GET /api/certs` `agents[]`): the name, the serial, the `status` chip (`active` → `ok`, `revoked` → `fail` — a string-to-class map, the page never compares a number), the expiry (local timestamp + time left), the **issued** `bundle_rev` against the **deployed** `bundle_rev` (`n/a` when the agent never reported — a `null` verdict, uncoloured; issued > deployed → a neutral `stale` note, uncoloured — a count is not a graded number, the fleet-header rule), and the `san` list in mono.
|
||||||
|
- per-row actions: **Rotate** (a confirm → `POST …/rotate`), **Revoke** (a confirm that says it stops the dashboard trusting that host until a reissue and a re-delivered bundle — a fixed sentence from the page, not the API), and **bundle** (a plain `<a href="/api/certs/<name>/bundle">` — a signed-in `GET` needs no CSRF).
|
||||||
|
- a client-cert line (the serial + the expiry) — the one cert the agents check. It is **display-only**: no revoke button (a revoked client cert is a dashboard that reaches no agent — a footgun the page should not make easy); its rotation happens with the root.
|
||||||
|
- ASSUMPTION: the client cert is display-only on the page (no revoke control); revoking it is deliberately not offered from the UI.
|
||||||
|
2. `src/dashboards/app.py` — `PAGES` gains `"/certs": "certs.html"` (the settings-page comment at the `PAGES` dict extends by one line of the same shape); a `Certs` link (a `⚷` glyph is acceptable) in `.masthead-right` of every page including `/settings`, next to `⚙ Account` — **outside** `.tabs` (the settings precedent, `tests/test_api.py::test_every_dashboard_page_links_to_settings` pins that structure; extend the pin, do not break it).
|
||||||
|
3. `src/dashboards/web/static/dash.css` — reuse existing vocabulary only (`.chip`, the table/`.hero`-family styles present in `settings.html`); **no** new `:root` tokens (the `tests/test_api.py` pin fails a CSS rule that reaches for an undeclared token).
|
||||||
|
4. `tests/test_api.py` — the served-page pins (the settings-pin idiom):
|
||||||
|
- the `Certs` masthead link on every dashboard page and on `/settings` (extend `test_every_dashboard_page_links_to_settings` into a two-link pin, or a sibling test);
|
||||||
|
- the served `/certs` HTML: the root block, the agents table container, the three action labels; the page's own tab structure carries **no** `aria-current` on any of the five (the settings pin, copied);
|
||||||
|
- the automatic audits cover the rest: every `id` the page keeps is referenced by `page-certs.js` (the orphan-id audit), no `Dash.severityFor` named (the existing pin), the deleted-flavour-text audit, the `no-store`/gate shape from the route walk.
|
||||||
|
5. the live verification screenshot is task 07 (invariant 10).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- the served-HTML pins above; the JS is verified by the live screenshot (the repo's UI test policy — no JS test runner).
|
||||||
|
- Coverage: **>90%** on the modified Python (`app.py`'s `PAGES`/route code).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] signed-in `GET /certs` serves the page; unsigned is a `303` to `/login?next=/certs` (the query included — `auth.safe_next`)
|
||||||
|
- [ ] the masthead pin green (every page links to both settings and certs)
|
||||||
|
- [ ] the orphan-id + flavour-text + `severityFor` audits green on the new page
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 06 — The installer places the bundle; the unit runs TLS; --status probes over TLS
|
||||||
|
|
||||||
|
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`deploy/install-agent.sh` can install the certificate bundle the same way it already installs the token — one file, the right owner and mode, the unit flag, a `--status` that proves the TLS path end-to-end — and `tests/test_deploy_scripts.py` moves its pins so a regression is a red test, not a dead agent.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `deploy/install-agent.sh` —
|
||||||
|
- a new flag `--cert-bundle <file>` (the operator already fetched it from `GET /api/certs/<name>/bundle` — task 04's surface; the script does not dial the dashboard, per its existing no-dial rule): validate the bundle before touching anything (a `python3` one-liner: three PEM blocks — server cert, key, CA cert — and a `# BUNDLE-REV` line; a bad bundle refuses with the reason, the `resolve_bundle` idiom).
|
||||||
|
- install to `/etc/dashboards-agent.certs` as `0600 root:$AGENT_USER` (the token-file idiom, with one line of the same comment explaining why the mode matters — the bundle holds the agent's key, so only root and the agent's own service user may read it); on re-run, replace it only if the flag is given (an existing bundle survives an unrelated re-run — the token-reuse idiom).
|
||||||
|
- the unit: when a bundle is installed, `ExecStart` gains `--cert-bundle /etc/dashboards-agent.certs` (the `--token-file` line stays — task 02's token check is skipped in TLS mode, so the harmless dual mode eases a rollout) and `RuntimeDirectory=dashboards-agent` (the unit's `ProtectSystem=strict` leaves the agent no writable file but this — the extracted PEM blocks live there; `PrivateTmp` already exists, so a non-systemd dev run falls back to `tempfile`).
|
||||||
|
- the `--status` probe: when the unit runs with a bundle, probe `https://127.0.0.1:$PORT/health` with a `python3 -c` `ssl` context that `load_verify_locations`-es the bundle's CA block and presents **no** client cert (task 02's `CERT_OPTIONAL` + the `/health` exemption is what makes this possible — the installer holds no client cert); a token-mode install keeps the plain `/health` probe. The probe is `python3` (the unit's own interpreter), never a new binary dependency (no `openssl` — one less thing to install on a host).
|
||||||
|
2. `deploy/agent-sudoers` — **no** new line (the bundle travels through the same root installer invocation as the token file).
|
||||||
|
3. `tests/test_deploy_scripts.py` — the new pins (the file's existing idiom: read the script, run it unprivileged so it exits before touching anything, and assert on the lines):
|
||||||
|
- `--cert-bundle` in the usage + the `/etc/dashboards-agent.certs` path with the `0600` and `root:$AGENT_USER` `chown` (assert the `install -m`/`chown` line);
|
||||||
|
- a `RuntimeDirectory=dashboards-agent` line in the unit when a bundle is installed (assert the conditional that emits it);
|
||||||
|
- the `--status` probe branch: the `python3` `ssl` one-liner for TLS, the plain probe for token mode;
|
||||||
|
- the existing pins unchanged (the sudoers-line identity, phase 10's no-tunnel name/port).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- the `tests/test_deploy_scripts.py` pins above; the real-host smoke is task 07 (invariant 10 keeps the dev server up).
|
||||||
|
- Coverage: the script is shell — the pins are its tests (the file's stated policy: no shell test framework; a subprocess buys nothing more).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] an unprivileged `--cert-bundle /nonexistent` run refuses with the reason and touches nothing
|
||||||
|
- [ ] the pins green: the bundle path/mode, the unit flag + `RuntimeDirectory`, the TLS `--status` probe
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# Task 07 — The fleet runs TLS: install, flip, verify, document
|
||||||
|
|
||||||
|
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The two live agents (deskwork, gateway) present the dashboard-issued server cert and require the dashboard's client cert; the dashboard dials `https://`; the health table, the live suite, and a `/certs` screenshot prove it — and the still-missing `driveripper` install is documented to carry a bundle from day one.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. **Fleet (operator root actions, the `deploy/agent-sudoers` grant):** for `deskwork` and `gateway`: `GET /api/certs/<name>/bundle` (a signed-in `curl`), `scp` the bundle to the host, `sudo ./deploy/install-agent.sh --cert-bundle <file>`, `systemctl restart dashboards-agent`, `./deploy/install-agent.sh --status` green over TLS. Record the exact commands + output in `data/logs/NOTES-agent-install.md` (an append to the file's existing per-host record; local evidence, not tracked, the AGENTS.md rule).
|
||||||
|
2. **The flip:** `.env.dev` — the `HOST_TARGETS` entries become `https://<same address>:9120` (no userinfo — the cert is the auth; the token goes away for a TLS host), `./scripts/dev-server.sh restart`, `./scripts/dev-server.sh status` shows `host:deskwork ok`, `host:gateway ok`, `disks:deskwork ok`, `disks:gateway ok` (the two `driveripper` jobs stay `DOWN` — unchanged, the designed degradation).
|
||||||
|
3. **Proof (invariant 10):** `uv run pytest -m live --live` — the host/disks tests over the `https` path; a `/certs` screenshot (the table showing both rows `active`, issued rev == deployed rev) **and** a `/system` screenshot (the host rows not dimmed) in `data/logs/`; a `data/logs/NOTES-verify.md`-style note: the `curl` codes (unsigned `/api/certs` refused, signed-in 200), one line for a wrong-CA client's handshake failure (the redacted URL only).
|
||||||
|
4. **The `driveripper` procedure:** an append to `data/logs/NOTES-agent-install.md` — when the agent is finally installed (the outstanding operator action, the AGENTS.md fleet section), install the bundle in the same invocation (`sudo ./deploy/install-agent.sh --bind … --cert-bundle …`) and point `HOST_TARGETS` at `https://` from day one; a token-only install after this phase is a host the `https`-dialing dashboard cannot reach.
|
||||||
|
5. **Docs (the AGENTS.md definition of done):** the AGENTS.md fleet section (the transport fact: the agent is now mTLS, the bundle file, the `/certs` page) + the commands section (the bundle-fetch line); a short "Protecting the agent" section in `README.md` (the operator's how-to: fetch the bundle, install, flip the URL — three sentences, no flavour).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- the live suite is read-only (the repo's rule) — no bundle mutations on the hardware; the revocation/rotation proof happens on a dev-box agent, not the fleet's.
|
||||||
|
- invariant 10: the dev server stays up throughout; screenshots in `data/logs/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `./scripts/dev-server.sh status`: `host:deskwork ok`, `host:gateway ok`, `disks:deskwork ok`, `disks:gateway ok` over the `https://` dials
|
||||||
|
- [ ] `uv run pytest -m live --live` green for the host/disks paths
|
||||||
|
- [ ] the `/certs` + `/system` screenshots in `data/logs/` show `active`, issued rev == deployed rev, the host rows not dimmed
|
||||||
|
- [ ] the `data/logs/` note records the exact commands + the `curl` codes + the wrong-CA failure line
|
||||||
|
- [ ] AGENTS.md + `README.md` updated (the definition of done)
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
# Phase 16 — Pin the agent's scaled smart-pass budget
|
||||||
|
|
||||||
|
**Source:** chat 2026-10-19 — the `driveripper` 16-drive host: the flat 20 s smart-pass
|
||||||
|
budget truncates a 16-device pass (a real read takes ~30–80 s), and because the deadline
|
||||||
|
resets each pass, the tail drives would have stayed `smart: null` every pass. The fix —
|
||||||
|
budget = 20 s floor scaled to 5 s × disk count, an explicit override still winning — was
|
||||||
|
applied to the working tree on 2026-10-19 (operator-confirmed, verified on 16/16 live
|
||||||
|
drives). This phase pins that already-applied change with tests and documentation.
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** `agent/host_agent.py` (`_smart_pass` — the scaled `budget = max(SMARTCTL_BUDGET_SECONDS, self.smart_timeout_seconds * len(device_names)) when the default is in force, else the explicit override; the `pass budget of {budget:.0f}s exhausted` error line; the module docstring "5 s per device and a pass budget of 20 s scaled to the disk count"), `tests/test_agent.py::test_the_pass_budget_skips_the_rest_with_one_error` (sets `collector.smart_budget_seconds = 0.0` — the override-wins branch), `AGENTS.md` line 185 ("5 s per device / 20 s per pass / 240 s cache") and the `deploy/install-agent.sh` header comment (line 25 area, "bounded 5 s per device and 20 s per pass" if still present there), `README.md` line 94 ("20 s per pass, results cached 4 minutes")
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The smart pass budget scale-up that is already in the working tree is pinned: a test proves the *default* scales (16 devices → 80 s deadline, one device → 20 s floor, the exhaustion line names the scaled figure), the override-wins branch stays covered, and every doc that still says "20 s per pass" says the floor-scaled truth. No new behaviour — the code change is already deployed on `driveripper` and verified live.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
- `09_disks_smartctl` (complete) — the smart pass, its bounds, and `tests/test_agent.py`'s in-process agent harness
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
|
||||||
|
1. `01_pin_the_scaled_budget.md` — the tests that the default budget scales with the disk count and that the error line names the scaled figure
|
||||||
|
2. `02_docs_and_gate.md` — every "20 s per pass" mention in `AGENTS.md`/`README.md`/`agent/host_agent.py` says the floor-scaled truth; the gate is green
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Unit: `tests/test_agent.py` — the scaled default (a default-attribute collector over 16 device names exhausts only when the 80 s budget runs out; over one device the floor is 20 s; the exhaustion error line carries the scaled figure, not the floor)
|
||||||
|
- Coverage: **>90%** on the modified code (the `_smart_pass` branch)
|
||||||
|
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
|
||||||
|
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the smart-pass bounds are a documented contract of `host_agent.py`).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `uv run pytest tests/test_agent.py` green, including the new scaled-budget tests and the untouched override test
|
||||||
|
- [ ] `grep -rn "20 s per pass" AGENTS.md README.md agent/ deploy/` returns nothing (the wording is floor-scaled everywhere)
|
||||||
|
- [ ] `bash .agents/validate.sh` green (pytest --cov --cov-fail-under=94, ruff, pyright, node --check)
|
||||||
|
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
|
||||||
|
- [ ] no behavior change in completed work: the deployed agent on `driveripper` is byte-identical to the repo's `agent/host_agent.py` (md5), the live suite is green
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# Task 01 — Pin the scaled smart-pass budget
|
||||||
|
|
||||||
|
**Phase:** `16_agent_smart_pass_budget` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The default smart-pass budget (20 s floor, scaled to 5 s × disk count) is proven by tests: the deadline actually moves with the disk count, the exhaustion error line names the *scaled* figure, and the existing override-wins branch (a test sets `smart_budget_seconds = 0.0`) keeps passing untouched. The code change is already in `agent/host_agent.py::_smart_pass` — this task adds the tests, it does not rewrite the code.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `tests/test_agent.py` — add a helper that builds the in-process collector with **default** smart attributes (do not touch `smart_budget_seconds`/`smart_timeout_seconds`) and a `smartctl` fake that records each device's deadline window, in the same style as `test_the_pass_budget_skips_the_rest_with_one_error` (which stays untouched):
|
||||||
|
- `test_the_pass_budget_scales_with_the_disk_count`: 16 device names → the last device's `smartctl` run happens with `left` (its deadline margin) consistent with a ~80 s budget (assert the deadline the code computes: `max(20, 5 × 16)`; the fake binary is instant, so assert via the recorded `min(left, 5.0)` timeout being 5.0 for a late device and the pass *not* truncating — every device got a `smartctl` run); 1 device name → budget 20 s (assert the recorded timeout is 5.0 and, with the fake made slow enough to exhaust a 20 s but not an 80 s budget in a *separate* case if feasible — if the harness cannot fake wall time, assert the computed `budget` expression via the error-line route instead: force exhaustion with a `time.monotonic` monkeypatch offset).
|
||||||
|
- `test_the_pass_budget_error_line_names_the_scaled_figure`: force exhaustion on 16 devices (monkeypatch the clock forward past the 80 s deadline between two device runs, as the existing test's `0.0` override does for its 0 s budget) → the error line reads `smart: pass budget of 80s exhausted, N disk(s) skipped` — the scaled figure, not the 20 s floor.
|
||||||
|
2. `agent/host_agent.py` — **no code change expected.** If a test exposes a real bug in the already-applied change (e.g. the `== SMARTCTL_BUDGET_SECONDS` default check), fix it minimally and record the diff in the task's completion note; do not redesign.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Unit: the two new tests above + the untouched `test_the_pass_budget_skips_the_rest_with_one_error` (override wins: `0.0` still exhausts immediately on 3 devices with one error line)
|
||||||
|
- Coverage: **>90%** on `agent/host_agent.py::_smart_pass`'s budget branch (the default-scaled path must now be exercised, not just the override)
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `uv run pytest tests/test_agent.py -q` green: the two new tests, the untouched override test, and every pre-existing agent test
|
||||||
|
- [ ] the 80 s case asserts the *scaled* figure in the error line (a test that passed with a 20 s budget would fail on the scaled code — no tautological assertions)
|
||||||
|
- [ ] `uv run ruff check agent tests` and `uv run pyright` clean
|
||||||
|
- [ ] no behavior change in completed work: `agent/host_agent.py` unchanged unless a real bug was found (then the fix is minimal and recorded)
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# Task 02 — Docs and gate for the scaled budget
|
||||||
|
|
||||||
|
**Phase:** `16_agent_smart_pass_budget` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Every place that still says the smart pass is "20 s per pass" says the floor-scaled truth (20 s floor, 5 s × disk count), and the phase gate is green. The agent's module docstring was already updated on 2026-10-19 to the scaled wording — this task verifies it and fixes the two remaining doc lines.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `AGENTS.md` line ~185 (the `host_agent.py` bullet): `5 s per device / 20 s per pass / 240 s cache` → `5 s per device / a pass budget of 20 s scaled to 5 s × disk count / 240 s cache` (keep the bullet's voice; one line).
|
||||||
|
2. `README.md` line ~94: `20 s per pass, results cached 4 minutes` → `a pass budget of 20 s scaled to the disk count (5 s each), results cached 4 minutes` (one line).
|
||||||
|
3. `agent/host_agent.py` module docstring — verify it reads `bounded 5 s per device and a pass budget of 20 s scaled to the disk count, cached 240 s`; fix only if it drifted.
|
||||||
|
4. `bash .agents/validate.sh` — the four checks (pytest --cov --cov-fail-under=94, ruff, pyright, node --check) + repo hygiene, all `uv run`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- No new logic in this task; the gate run is the check.
|
||||||
|
- Coverage: **>90%** gate holds (task 01's tests carry the new branch).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `grep -rn "20 s per pass" AGENTS.md README.md agent/ deploy/` returns nothing
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
|
- [ ] the two edited doc lines read the floor-scaled truth and nothing else changed in either file
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
# Phase 17 — The installer's broken device filter and the kernel-gate variant
|
||||||
|
|
||||||
|
**Source:** chat 2026-10-19 — the `driveripper` install: the template's
|
||||||
|
`DevicePolicy=strict` + `DeviceAllow` lines produced a **deny-all device filter** on
|
||||||
|
kernel 7.2 (systemd resolves `DeviceAllow` globs by name against `/proc/devices`, where
|
||||||
|
this kernel lists NVMe as `blkext` (major 259), so `nvme*` matched nothing and the whole
|
||||||
|
device namespace was locked). The operator removed the lines by hand; the unit now also
|
||||||
|
runs `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` (kernel 7.2's
|
||||||
|
unprivileged-passthrough gate — `data/logs/NOTES-driveripper-agent.md`). The template
|
||||||
|
still ships the broken lines, `User=$AGENT_USER`, and an empty bounding set, and re-running
|
||||||
|
the installer on `driveripper` would **clobber the working unit** (unconditional
|
||||||
|
`cat > "$UNIT"`). This phase makes the template the single source of the working unit.
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** `deploy/install-agent.sh` (`do_install` — the unconditional unit heredoc with `DevicePolicy=strict` + nine `DeviceAllow` lines, the `PrivateDevices=false` rationale comment block, `User=$AGENT_USER`, `CapabilityBoundingSet=`, the `--bind/--port/--token/--rotate-token/--status/--uninstall` flag set), `tests/test_deploy_scripts.py` (the fake-root install sandbox `test_install_path_installs_into_its_own_paths_and_prints_the_token_once`, the token-file permission test, the `bash -n` + function-naming pins), `AGENTS.md` (the fleet bullet "The deployed unit has since diverged from the template in the three ways the SMART block below records", gap 10's "the template change is its own phase"), `README.md` (the install section, line ~46 `sudo deploy/install-agent.sh --bind <addr> --port 9120`)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Re-running `deploy/install-agent.sh` on any host — including `driveripper`, with
|
||||||
|
`--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` — produces the working unit: no
|
||||||
|
`DevicePolicy`/`DeviceAllow` lines (device access falls back to DAC via the `disk` group
|
||||||
|
+ SELinux, the standard SMART-monitoring arrangement), the kernel-gate variant expressible
|
||||||
|
by flags, and a re-run that would change an existing unit prints the diff and refuses
|
||||||
|
without `--force`. The default install (dedicated user, empty bounding set) is otherwise
|
||||||
|
byte-identical to what the template wrote before this phase.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
- `09_disks_smartctl` (complete) — the smart pass the unit exists to serve
|
||||||
|
- `15_agent_mtls` (todo) — **shared-file, run-order note**: 15's task 06 adds `--cert-bundle`/`RuntimeDirectory` to the same `do_install`; apply on top of the file state at the time it runs, never restore a line the other phase owns. If 15 lands first, its new lines survive this phase's filter-line removal untouched.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
|
||||||
|
1. `01_drop_the_broken_device_filter.md` — remove `DevicePolicy=strict` + all nine `DeviceAllow` lines and rewrite the rationale comment block
|
||||||
|
2. `02_root_and_caps_flags.md` — `--as-root` and `--caps CAP1,CAP2` write the kernel-gate unit lines
|
||||||
|
3. `03_no_silent_clobber.md` — a re-run that would change an existing unit prints the diff and refuses without `--force`
|
||||||
|
4. `04_tests_docs_gate.md` — the unit pins in `tests/test_deploy_scripts.py`, `AGENTS.md`/`README.md` updated, gate green
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Integration: `tests/test_deploy_scripts.py` — the template unit carries **no** `DevicePolicy`/`DeviceAllow` line; a default-install sandbox unit is byte-identical to the pre-phase template unit minus the filter lines; `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` writes `User=root` and `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` exactly; the re-run guard refuses (exit 1, diff on stdout) and overwrites with `--force`; `bash -n` and the existing function-naming pin still pass
|
||||||
|
- Live (task 04, operator-verified): `driveripper`'s deployed unit is reproduced by `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` — `systemctl diff` empty, `CapEff` still `0000000000220000`, 16/16 drives still live; `deskwork`/`gateway` untouched (their units are not re-run)
|
||||||
|
- Coverage: the script itself is pinned by text assertions + `bash -n` (house pattern; >90% applies to the new Python test code)
|
||||||
|
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
|
||||||
|
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the new flags are operator-facing; gap 10's "the template change is its own phase" becomes "done in phase 17" with the `driveripper` flag line).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing (the `PrivateDevices=false` line stays)
|
||||||
|
- [ ] `sudo deploy/install-agent.sh --help` names `--as-root` and `--caps` with the kernel-gate sentence
|
||||||
|
- [ ] a fresh sandbox install (fake root, no flags) produces a unit byte-identical to the pre-phase template unit minus the filter lines (pinned in `tests/test_deploy_scripts.py`)
|
||||||
|
- [ ] a re-run that would change an existing unit exits 1 with the diff on stdout; with `--force` it overwrites
|
||||||
|
- [ ] operator-verified on `driveripper`: the flags reproduce the working unit (`systemctl diff` empty, 16/16 drives live after restart)
|
||||||
|
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||||
|
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
|
||||||
|
- [ ] no behavior change in completed work: the default install's unit (minus the filter lines) and every existing flag behave as before; the live suite is green
|
||||||
+25
@@ -0,0 +1,25 @@
|
|||||||
|
# Task 01 — Drop the broken device filter
|
||||||
|
|
||||||
|
**Phase:** `17_installer_device_filter` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The template unit no longer carries `DevicePolicy=strict` or any `DeviceAllow` line, and the comment block that justified them is rewritten to say why the *absence* is the fix. The `PrivateDevices=false` line stays (explicit, default behaviour, and the rest of the lockdown is unchanged).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `deploy/install-agent.sh` — in the `do_install` unit heredoc: delete the `DevicePolicy=strict` line and all nine `DeviceAllow=…` lines (`/dev/null`, `/dev/zero`, `/dev/full`, `/dev/random`, `/dev/urandom`, `/dev/nvme*`, `/dev/sd*`, `/dev/vd*`, `/dev/sg*`).
|
||||||
|
2. `deploy/install-agent.sh` — rewrite the comment block above `PrivateDevices=false` (the "The pseudo nodes are not decoration…" paragraph) to the new truth: **no device filter on purpose** — systemd resolves `DeviceAllow` globs by name against `/proc/devices` (fnmatch in `bpf_devices_allow_list_major`), where kernel 7.2 lists NVMe as `blkext` (major 259), so `nvme*` matched nothing and `DevicePolicy=strict` denied the *whole* device namespace (the `driveripper` install went deny-all, root included; `data/logs/NOTES-driveripper-agent.md`). Without the policy, device access falls back to DAC (the `disk` group the script already grants) + SELinux — the standard SMART-monitoring arrangement. Keep the one operational fact that survives: `PrivateDevices=false` is the default (a `true` sandbox with no `DeviceAllow` list is what dies at spawn with `status=208/STDIN` — keep that warning sentence, it is still why the line is written explicitly).
|
||||||
|
3. `deploy/install-agent.sh` — the script's top-of-file header comment (line ~25: "the unit device-allows the disk nodes") → "the unit applies no device filter (kernel 7.2's `/proc/devices` name matching made the old `nvme*` glob deny-all); device access is DAC via the `disk` group + SELinux".
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Integration: `bash -n deploy/install-agent.sh` clean (the existing pin covers it); a quick sandbox run (the `test_install_path_installs_into_its_own_paths_and_prints_the_token_once` pattern, run manually) shows no `DevicePolicy`/`DeviceAllow` in the written unit — the formal pins are task 04, this task keeps the change small and reviewable.
|
||||||
|
- Coverage: **>90%** n/a (shell script; the pins are text assertions per the house pattern).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing
|
||||||
|
- [ ] `bash -n deploy/install-agent.sh` clean
|
||||||
|
- [ ] a manual sandbox install (fake root, `--bind 127.0.0.1 --port 9199`) writes a unit with the filter lines gone, `PrivateDevices=false` present, and every other line byte-identical to the pre-phase template unit
|
||||||
|
- [ ] the rewritten comment names the `/proc/devices`/`blkext` mechanism in one sentence (an executor reading only the script can understand *why* there is no filter)
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 02 — `--as-root` and `--caps` flags
|
||||||
|
|
||||||
|
**Phase:** `17_installer_device_filter` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The kernel-gate variant of the unit (what `driveripper` runs) is expressible from the template: `--as-root` writes `User=root` instead of `User=$AGENT_USER`, and `--caps CAP1,CAP2` (comma-separated, e.g. `--caps CAP_SYS_RAWIO,CAP_SYS_ADMIN`) writes `CapabilityBoundingSet=CAP1 CAP2` instead of the empty `CapabilityBoundingSet=` line. Both default off; a flag without its friend is refused with a message.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `deploy/install-agent.sh` — flag parsing: add `--as-root` (sets `AS_ROOT=1`) and `--caps <list>` (sets `CAPS="<list>"`, validates non-empty, rejects an empty list with exit 1) in the existing `case`/flag block, and `--help` text:
|
||||||
|
- `--as-root run the service as root (for a kernel-gated host: kernel 7.2 gates unprivileged smartctl passthrough behind capabilities, not uid — a root process under the default empty bounding set is capability-less, so this flag needs --caps to mean anything)`
|
||||||
|
- `--caps LIST CapabilityBoundingSet line for the unit (comma-separated, e.g. CAP_SYS_RAWIO,CAP_SYS_ADMIN); default: empty (capability-less service)`
|
||||||
|
2. `deploy/install-agent.sh` — validation: `--as-root` without `--caps` → print "note: --as-root with the default empty CapabilityBoundingSet leaves the process capability-less (the kernel gates check capabilities, not uid); pair it with --caps for a kernel-gated host" and **still proceed** (a capability-less root is a legitimate, if unusual, choice — do not hard-refuse; the note is the operator's reminder). `--caps` without `--as-root` → refuse, exit 1 ("--caps needs --as-root: a capability grant on the service user is not the kernel-gate fix").
|
||||||
|
3. `deploy/install-agent.sh` — the unit heredoc: `User=$AGENT_USER` → `User=$( [ "$AS_ROOT" -eq 1 ] && echo root || echo "$AGENT_USER" )`; the empty `CapabilityBoundingSet=` line → the `CAPS` list converted comma→space when set. Add one comment line above: `# --caps: kernel 7.2 gates unprivileged smartctl passthrough (NVMe admin log: CAP_SYS_ADMIN; SG_IO allow-list: CAP_SYS_RAWIO); a root process needs them named, because an empty bounding set bounds even root to CapEff 0`.
|
||||||
|
4. `deploy/install-agent.sh` — the `--status` probe is unchanged (it probes the agent over HTTP; the user the unit runs as is invisible to it). The token file stays `0640 root:$AGENT_USER` — with `--as-root` the group read is irrelevant (root reads it), so do **not** change the token-file logic.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Integration: manual sandbox runs (fake root): default → `User=dashboards-agent` + empty `CapabilityBoundingSet=`; `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` → `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`; `--caps` alone → exit 1 with the message; `--as-root` alone → proceeds with the note on stdout. (Formal pins: task 04.)
|
||||||
|
- Coverage: **>90%** n/a (shell script; text pins per the house pattern).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `sudo deploy/install-agent.sh --help` names both flags with the kernel-gate sentence (a fresh clone's executor can learn the `driveripper` recipe from `--help` alone)
|
||||||
|
- [ ] the two sandbox units match the exact lines above (byte-for-byte on the `User=` and `CapabilityBoundingSet=` lines)
|
||||||
|
- [ ] `bash -n deploy/install-agent.sh` clean; every pre-existing flag (`--bind/--port/--token/--rotate-token/--status/--uninstall`) behaves as before in a sandbox run
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 03 — No silent clobber on re-run
|
||||||
|
|
||||||
|
**Phase:** `17_installer_device_filter` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
A re-run that would change an already-installed unit stops: it prints the diff between the installed unit and the unit the script would write, and exits 1 without touching the file. `--force` proceeds with the overwrite. A re-run whose generated unit is identical to the installed one proceeds silently (idempotent re-runs stay the zero-noise path the script has always had).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `deploy/install-agent.sh` — add a `--force` flag to the flag block (`FORCE=1`) and `--help` text: `--force overwrite an existing unit even if the generated unit differs (prints the diff first)`.
|
||||||
|
2. `deploy/install-agent.sh` — in `do_install`, immediately before the `cat > "$UNIT"` heredoc write: build the would-be unit into a temp file (the heredoc already exists — render it once to `$(mktemp)`), and if `"$UNIT"` exists and `! cmp -s "$UNIT" "$tmp_unit"`:
|
||||||
|
- if `FORCE=1`: print the diff (`diff -u "$UNIT" "$tmp_unit"` on stdout) and proceed with the overwrite;
|
||||||
|
- else: print `refusing to overwrite $UNIT (generated unit differs); run with --force to apply:` + the same diff, and `exit 1` **before** the token-file write, the `systemctl daemon-reload`, and the `systemctl enable --now` — a refused run must leave the host exactly as it was (token file, unit, and running service all untouched; note the token-reuse logic runs earlier in `do_install` and only *reads* the existing token, so ordering the guard before the unit write is enough).
|
||||||
|
- identical unit → no output, normal proceed.
|
||||||
|
3. `deploy/install-agent.sh` — `--uninstall` is unchanged (it already removes the unit deliberately; the guard applies to `do_install` only).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Integration: manual sandbox sequence (fake root): install → modify the installed unit by one line → re-run with the same flags → exit 1, diff on stdout, installed unit byte-identical to the modified version, service still the old one; re-run with `--force` → diff printed, unit overwritten; re-run again unchanged → silent, exit 0. (Formal pins: task 04.)
|
||||||
|
- Coverage: **>90%** n/a (shell script; text/behaviour pins per the house pattern).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] the three-way sequence above (refuse → force → silent) holds in the sandbox, with the token file and the running service untouched by a refused run
|
||||||
|
- [ ] `--force` prints the diff *before* overwriting (an operator sees exactly what changed)
|
||||||
|
- [ ] `bash -n deploy/install-agent.sh` clean; a fresh install (no existing unit) is unaffected by the guard
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# Task 04 — Pins, docs, gate
|
||||||
|
|
||||||
|
**Phase:** `17_installer_device_filter` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The template's new shape is pinned in `tests/test_deploy_scripts.py` (no filter lines, the flags' exact lines, the clobber guard's refuse/force/silent sequence), `AGENTS.md` and `README.md` name the `driveripper` recipe, and the gate is green. The operator-verified `driveripper` reproduction (the completion criterion in `00_phase.md`) is the last check.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `tests/test_deploy_scripts.py` — extend the existing sandbox pattern:
|
||||||
|
- `test_the_template_unit_has_no_device_filter_lines` — read `deploy/install-agent.sh`'s unit heredoc region and assert no line starts with `DevicePolicy=` or `DeviceAllow=`, and `PrivateDevices=false` is still present (the filter's absence is the fix; a regression that re-adds `DeviceAllow=/dev/nvme*` is the deny-all returning).
|
||||||
|
- `test_a_default_install_unit_is_the_pre_phase_unit_minus_the_filter_lines` — the sandbox-installed unit equals the known pre-phase template unit text minus the ten filter lines (pin the whole unit, so a future template line drift fails here).
|
||||||
|
- `test_as_root_and_caps_write_the_kernel_gate_unit_lines` — `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` → `User=root` and `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` exactly; `--caps` alone → exit 1 with the `--as-root` message; `--as-root` alone → proceeds with the capability-less note on stdout.
|
||||||
|
- `test_a_re_run_that_differs_refuses_without_force_and_applies_with_it` — the refuse/force/silent sequence (exit 1 + diff on stdout + installed unit untouched; `--force` overwrites with the diff printed; an unchanged re-run is silent).
|
||||||
|
2. `AGENTS.md` — the fleet bullet's "re-running `deploy/install-agent.sh` on driveripper would clobber the working unit, because it rewrites the unit unconditionally (gap 10)" → the post-phase truth: the installer no longer writes the device-filter lines, and the `driveripper` unit is reproducible with `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` (a re-run that would change an existing unit refuses without `--force`). Update gap 10's "the template change is its own phase" → "done in phase 17" with the flag line.
|
||||||
|
3. `README.md` — the install section: one line naming the flags' purpose (the kernel-gated-host recipe), keeping the how-to voice.
|
||||||
|
4. `bash .agents/validate.sh` — all four checks green.
|
||||||
|
5. **Operator-verified (live, read-only + one restart):** on `driveripper`, `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` → `systemctl diff dashboards-agent` empty, `systemctl is-active` `active`, `CapEff` still `0000000000220000`, and 16/16 drives still live in the next `/snapshot` (the restart is the only fleet action; `deskwork`/`gateway` are not re-run). Screenshot `data/logs/disks-17-installer-repro.png` per invariant 10.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Integration: the four new pins above + the pre-existing pins (sudoers, `bash -n`, function naming, token-file permissions) all green.
|
||||||
|
- Coverage: **>90%** on the new test code; the 94% gate holds.
|
||||||
|
- Live: the operator-verified reproduction (item 5) — the phase's `driveripper` criterion.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] the four new pins pass and every pre-existing `test_deploy_scripts.py` pin passes
|
||||||
|
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing
|
||||||
|
- [ ] `AGENTS.md` fleet bullet + gap 10 and `README.md` name the `driveripper` flag recipe
|
||||||
|
- [ ] the operator-verified reproduction: `systemctl diff` empty, 16/16 drives live, screenshot in `data/logs/`
|
||||||
|
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||||
|
- [ ] the live suite (`set -a; . ./.env.dev; set +a; uv run pytest -m live --live`) is green
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# Phase 18 — Collapse the nav bar into a hamburger
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "The nav bar needs to collapse into a hamburger like brain of reese does"
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** the seven mastheads that carry the tab nav — `src/dashboards/web/pages/{index,system,disks,llm,ups,settings,certs}.html` (each has `nav class="tabs" aria-label="Dashboards"` with the five dashboard links; `login.html` deliberately has no tabs — it renders only the brand link and the theme toggle, and stays untouched), `src/dashboards/web/static/app.js` (`Dash.initChrome()`, called from the one `DOMContentLoaded` at line 786 — the `themeButton`/`clock`/`nav-status` blocks there are the per-element `if (element)` no-op pattern this phase's binding follows), `src/dashboards/web/static/dash.css` (`.masthead` at line 95 — `position: sticky; z-index: 40; height: var(--header-h)`, `nav.tabs` at line 131, the existing `@media (max-width: 640px)` block at line 286, the `prefers-reduced-motion` block at line 938), and the reference implementation the TODO names — `brain_of_reese` (sibling repo, read-only): `frontend/assets/header.js` lines 326–366 (the phase-46 binding: `#nav-toggle` shown ≤640px, `setNavMenu(open)` funnel, close on link click / Esc with focus return / `matchMedia` resize-back). Pinned test neighbours: `tests/test_api.py::test_the_disks_tab_is_in_the_masthead_of_every_page` (line 3837) and the served-page orphan-`id` audit in `tests/test_auth_api.py` (line 1654)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
On viewports ≤640px the masthead's five tab links collapse into a dropdown opened by a hamburger button, mirroring brain of reese's phase-46 behaviour: one shared binding in `app.js` (a no-op on any page without the elements — `/login`), the inline nav hidden and the dropdown absolutely positioned under the sticky masthead, and the desktop masthead byte-identical in behaviour above the breakpoint. The `masthead-right` cluster stays visible on phones; only the tab nav collapses.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- — (none; builds on the shipped masthead, `initChrome`, and the 640px media block)
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_hamburger_markup.md` — the `#nav-toggle` button and the `nav` id, identical across the seven tabbed pages, absent from `/login`
|
||||||
|
2. `02_dropdown_css.md` — the ≤640px block: toggle visible, inline tabs hidden, `.is-open` dropdown under the masthead, two-row masthead
|
||||||
|
3. `03_toggle_binding.md` — the `initChrome` binding: toggle + `aria-expanded`, close on link click / Esc / resize-back
|
||||||
|
4. `04_tests_and_verification.md` — the pins (present on all seven, absent on `/login`, CSS and binding), the orphan-`id` audit, and the narrow-viewport browser verification with screenshots
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: `tests/test_api.py` — the served-page pins (see task 04); the existing masthead/tab pins (`test_the_disks_tab_is_in_the_masthead_of_every_page`, `test_every_page_links_to_both_admin_pages`) and the `tests/test_auth_api.py` orphan-`id` audit pass untouched — the new `id`s are referenced by `aria-controls`, the stylesheet, and `app.js`, so the audit must still clear
|
||||||
|
- Coverage: **>90%** on new/modified code (the phase is markup + CSS + one small JS binding; the coverage gate applies to the changed Python test files)
|
||||||
|
- The gate (`bash .agents/validate.sh`) includes `node --check` of `web/static` — the modified `app.js` must pass it
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
|
||||||
|
- AGENTS.md's `UI conventions` section gains the masthead-collapsing convention (the 640px breakpoint, the toggle's identity, the close paths, and that `/login` carries no toggle) so a future masthead edit knows the contract. The README's page table is unchanged — the nav is the same five links at every width.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] at ≤640px: the tab links are hidden, the hamburger is visible, and clicking it opens a dropdown containing the five links; above 640px the desktop masthead is unchanged
|
||||||
|
- [ ] the dropdown closes on link click, on Esc (focus returns to the button), and on resize back above 640px; `aria-expanded` tracks the state
|
||||||
|
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
|
||||||
|
- [ ] `bash .agents/validate.sh` green (including `node --check` of the modified `app.js`)
|
||||||
|
- [ ] narrow-viewport screenshots (closed, open, and one desktop no-regression shot) in `data/logs/` against the running dev server (invariant 10)
|
||||||
|
- [ ] no behavior change in completed work: the desktop masthead and every page render identically at widths >640px
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 01 — The hamburger button markup
|
||||||
|
|
||||||
|
**Phase:** `18_nav_hamburger` · **Source:** `TODO.md:3` — "The nav bar needs to collapse into a hamburger like brain of reese does"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The seven pages that carry the tab nav each gain the hamburger button and the `nav` id, so the (later task's) CSS and binding have stable anchors to work on. The markup is identical on every page, and `/login` — which deliberately renders only the brand link and the theme toggle — stays untouched.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/pages/index.html` · `system.html` · `disks.html` · `llm.html` · `ups.html` · `settings.html` · `certs.html` — in each masthead, immediately before `nav class="tabs"`, add the toggle:
|
||||||
|
```html
|
||||||
|
<button class="ghost nav-toggle" id="nav-toggle" type="button"
|
||||||
|
aria-expanded="false" aria-controls="nav-tabs" aria-label="Toggle navigation">☰</button>
|
||||||
|
```
|
||||||
|
and give the nav its id: `<nav class="tabs" id="nav-tabs" aria-label="Dashboards">`. The button sits between `.brand` and `nav.tabs`, inside the same `<header class="masthead">` — one element per page, no script changes in this task.
|
||||||
|
2. `src/dashboards/web/static/dash.css` — add the one base rule beside the other `button.ghost, a.ghost` family (the masthead section): `.nav-toggle { display: none; }` — the button is invisible on every width until task 02's ≤640px block shows it, so the project stays exactly as it looks today after this task.
|
||||||
|
3. `src/dashboards/web/pages/login.html` — **no change.** Verify by `grep -c 'id="nav-toggle"'` returning 0 (the pin lands in task 04).
|
||||||
|
4. `ASSUMPTION:` the TODO named brain of reese's behaviour, not its breakpoint or glyph. This task takes the 640px breakpoint (brain's `(max-width: 640px)` and `dash.css`'s existing 640px block) and the `☰` glyph with `aria-label="Toggle navigation"`; the CSS task owns the visual.
|
||||||
|
5. `ASSUMPTION:` only the tab nav collapses on phones. The `masthead-right` cluster (`#nav-status` dots, pause, `⚷ Certs`, `⚙ Account`, theme toggle) stays visible; the masthead wraps to two rows at ≤640px (task 02).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the served pages still pass the untouched masthead pins (`test_the_disks_tab_is_in_the_masthead_of_every_page`, `test_every_page_links_to_both_admin_pages`) and the orphan-`id` audit — `#nav-toggle` is referenced by the (task 02) stylesheet rule and `aria-controls`, so the audit must not flag it
|
||||||
|
- Coverage: **>90%** on modified code (HTML only — the gate's `node --check` and pytest suite are the checks)
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `grep -l 'id="nav-toggle"' src/dashboards/web/pages/*.html` lists exactly the seven tabbed pages, never `login.html`
|
||||||
|
- [ ] every one of the seven carries the identical button block and the `nav` id (diff of the blocks across pages is empty)
|
||||||
|
- [ ] `uv run pytest -q` green; `bash .agents/validate.sh` green
|
||||||
|
- [ ] no behavior change: at every width the pages render exactly as before after this task (the button is `display:none` from the base rule shipped with it)
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# Task 02 — The ≤640px dropdown CSS
|
||||||
|
|
||||||
|
**Phase:** `18_nav_hamburger` · **Source:** `TODO.md:3` — "The nav bar needs to collapse into a hamburger like brain of reese does"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The CSS that makes the hamburger real at ≤640px: the toggle button visible, the inline tab nav hidden, the `.is-open` nav rendered as a dropdown under the sticky masthead, and the masthead wrapping to two rows so the `masthead-right` cluster does not overflow a phone. Above 640px nothing changes.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/static/dash.css` — the base `.nav-toggle { display: none; }` rule already landed with task 01's markup (the button is invisible on every width until this task's block). Extend the existing `@media (max-width: 640px)` block at line 286 (or add a sibling block — the file's convention is one block per concern; a `/* mobile nav */`-commented block beside the masthead section is the cleaner home):
|
||||||
|
- `.masthead { height: auto; min-height: var(--header-h); flex-wrap: wrap; }` — the single-row fixed-height masthead cannot hold brand + toggle + the whole `masthead-right` cluster at 360px; two rows, first row brand + toggle, second row the cluster.
|
||||||
|
- `.nav-toggle { display: inline-flex; align-items: center; }` (it is a `button.ghost`, so the outlined style is inherited — no second palette, the file's rule).
|
||||||
|
- `nav.tabs { display: none; }` and `nav.tabs.is-open { display: flex; flex-direction: column; gap: 0; position: absolute; top: 100%; left: 0; right: 0; background: var(--bg-raised); border-bottom: 1px solid var(--border); box-shadow: 0 8px 24px rgba(0, 0, 0, .35); padding: .4rem; z-index: 41; }` — the dropdown. The masthead is `position: sticky`, which is a non-static containing block for its absolute descendants, so `top: 100%` pins it directly under the (wrapped, two-row) masthead. `z-index: 41` is one above the masthead's 40 so the dropdown sits on the page content.
|
||||||
|
- `nav.tabs.is-open a { padding: .6rem .7rem; border-radius: var(--radius-sm); }` — touch-sized rows; the existing `:hover`/`[aria-current="page"]` rules carry over unchanged.
|
||||||
|
- Optional at most: a 120ms opacity/translateY transition on `nav.tabs.is-open`, **disabled inside the existing `@media (prefers-reduced-motion: reduce)` block at line 938** (add the dropdown selector to that block's list of motion-killers if used). If in doubt, ship without the transition.
|
||||||
|
2. `ASSUMPTION:` the dropdown is a single full-width column under the masthead (brain's shape, adapted: brain hides the nav in a `#app-nav` container; here the `nav` element itself is the container and gets `id="nav-tabs"` + `.is-open`). No separate menu panel element.
|
||||||
|
3. `ASSUMPTION:` no `body`-level marker class (brain's `nav-menu-open` exists for its sticky bottom chat cluster, which the dashboards do not have — the dropdown competes with nothing; do not copy a side effect nobody needs).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the served-stylesheet pins land in task 04 (base rule present, ≤640px block contains the toggle-visible + tabs-hidden + `.is-open` rules); the untouched `node --check` gate passes (CSS is not checked, but confirm no syntax error by loading the page in the browser — task 04's verification)
|
||||||
|
- Coverage: **>90%** on modified code (CSS only; the gate's pytest/ruff/pyright must stay green)
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] at a 360px-wide browser window: the tab links are gone from the row, the `☰` button is visible, and toggling `.is-open` (by hand in devtools) shows the full-width dropdown under the masthead with the five links stacked and the `aria-current` tab highlighted
|
||||||
|
- [ ] at 700px and 1200px the masthead is byte-identical in layout to before (one row, inline tabs, no button)
|
||||||
|
- [ ] the dropdown does not trap or cover the `masthead-right` cluster, and the page content scrolls under it
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
# Task 03 — The toggle binding in `initChrome`
|
||||||
|
|
||||||
|
**Phase:** `18_nav_hamburger` · **Source:** `TODO.md:3` — "The nav bar needs to collapse into a hamburger like brain of reese does"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
One shared binding in `app.js` (the file every page loads, including `/login`) that makes the button work on all seven tabbed pages and is a no-op where the elements are absent. The behaviour mirrors brain of reese's phase-46 binding: a `setNavMenu(open)` funnel so no close path can leave the state stuck, `aria-expanded` honest at all times, close on link click, on Esc (focus returns to the button), and on resize back above the breakpoint.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/static/app.js` — inside `initChrome()` (line 620), after the `nav-status` block and before the `return`, add a block in the file's per-element guard style (the `themeButton` pattern at line 626 is the shape: `if (element) { … }`):
|
||||||
|
```js
|
||||||
|
const navToggle = $('#nav-toggle');
|
||||||
|
const appNav = $('#nav-tabs');
|
||||||
|
if (navToggle && appNav) {
|
||||||
|
const setNavMenu = (open) => {
|
||||||
|
appNav.classList.toggle('is-open', open);
|
||||||
|
navToggle.setAttribute('aria-expanded', open ? 'true' : 'false');
|
||||||
|
};
|
||||||
|
navToggle.addEventListener('click', () =>
|
||||||
|
setNavMenu(!appNav.classList.contains('is-open')));
|
||||||
|
// A link click navigates — shut the menu (event-delegated: the five
|
||||||
|
// links are static markup, so one listener on the nav covers all of them).
|
||||||
|
appNav.addEventListener('click', (event) => {
|
||||||
|
if (event.target.closest('a')) setNavMenu(false);
|
||||||
|
});
|
||||||
|
// Esc closes while open; focus returns to the opener.
|
||||||
|
document.addEventListener('keydown', (event) => {
|
||||||
|
if (event.key === 'Escape' && appNav.classList.contains('is-open')) {
|
||||||
|
setNavMenu(false);
|
||||||
|
navToggle.focus();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
// Resize back to desktop: the inline nav reappears — drop the open state
|
||||||
|
// so aria-expanded stays honest (the CSS hides the dropdown above 640px
|
||||||
|
// regardless, but a stuck `is-open` would re-open on the next shrink).
|
||||||
|
const mq = window.matchMedia('(max-width: 640px)');
|
||||||
|
const onMqChange = () => { if (!mq.matches) setNavMenu(false); };
|
||||||
|
if (mq.addEventListener) mq.addEventListener('change', onMqChange);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Notes: the breakpoint literal must match task 02's CSS block — if one drifts, the menu opens into a display:none nav. `ASSUMPTION:` that match is pinned by task 04's served-asset test (assert the `640px` literal appears in both the served stylesheet's nav block and the served `app.js`), not by a shared constant — the file's convention is per-file literals (the 640px in CSS, 900px for the clock, and the page scripts' own numbers do not share a constant either).
|
||||||
|
2. No `body`-level marker class (task 02's assumption carries: nothing on this app competes with the dropdown for taps, so brain's `nav-menu-open` marker is not copied).
|
||||||
|
3. No change to the login page: `initChrome` already ships to it, and the `if (navToggle && appNav)` guard is the no-op — verify in task 04 that `/login`'s served script is the same shared file (it is: every page loads `/static/app.js`) and the guard makes the binding inert there.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the binding's behaviour is verified in the browser (task 04 — the repo has no JS test runner; the pins are structural on the served bytes: the block is present in the served `app.js`, the four close paths named in a comment, `node --check` passes); the untouched Python suite stays green
|
||||||
|
- Coverage: **>90%** on modified code (one JS function; the Python coverage gate applies to the test files task 04 adds)
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] at 360px: clicking `☰` opens the dropdown and `aria-expanded` reads `true`; clicking again closes it; clicking a tab link closes it as the page navigates
|
||||||
|
- [ ] with the menu open: Esc closes it and focus is on the `☰` button; resizing to 700px closes it and the next shrink back to 360px leaves it closed (no stale `is-open`)
|
||||||
|
- [ ] on `/login` the script loads and the guard no-ops (no console error, `node --check` clean)
|
||||||
|
- [ ] `uv run pytest -q` green; `bash .agents/validate.sh` green (includes `node --check` of the modified `app.js`)
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 04 — Pins and narrow-viewport verification
|
||||||
|
|
||||||
|
**Phase:** `18_nav_hamburger` · **Source:** `TODO.md:3` — "The nav bar needs to collapse into a hamburger like brain of reese does"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The phase is pinned on the served bytes the way this repo pins every page surface: the button is on all seven tabbed pages and **absent** from `/login`, the CSS and the binding carry the matching 640px literal, and the orphan-`id` audit still clears. Then the designed degradation is proven in a real browser at a phone width — the invariant-10 screenshot pass.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `tests/test_api.py` — new tests, in the style of `test_the_disks_tab_is_in_the_masthead_of_every_page` (line 3837; it iterates the served pages and checks the masthead):
|
||||||
|
- `test_the_nav_toggle_is_in_the_masthead_of_every_tabbed_page`: for each of the seven served pages (`/`, `/system`, `/disks`, `/llm`, `/ups`, `/settings`, `/certs`) — a `GET` (signed, like the other served-page pins) finds `button#nav-toggle` with `aria-expanded="false"`, `aria-controls="nav-tabs"`, `aria-label="Toggle navigation"`, and `type="button"`, and `nav#nav-tabs` with its five tab links intact; the button precedes the nav in the markup (the dropdown's `top: 100%` anchor needs the nav inside the masthead, which the pin asserts via the `header.masthead` containing both).
|
||||||
|
- `test_the_login_page_carries_no_nav_and_no_toggle`: `/login` (fetched the way the audit fetches it — unsigned, because a signed-in `GET /login` is a `303`) has no `nav.tabs`, no `#nav-toggle`, and no `#nav-tabs` — the page's contract is brand + theme toggle only, and a stray hamburger there is the bug this test exists to fail.
|
||||||
|
- `test_the_nav_toggle_css_and_binding_carry_the_same_breakpoint`: the served `dash.css` contains the base `.nav-toggle` `display: none` rule, a `@media (max-width: 640px)` (or the 640px block) with the toggle-visible, `nav.tabs`-hidden, and `.is-open` rules; the served `app.js` contains the binding block (the `is-open` toggle, the `Escape` close, the `matchMedia('(max-width: 640px)')` call) — and the `640px` literal appears in both, so a drift between CSS and JS fails here instead of opening a menu into a `display:none` nav.
|
||||||
|
2. `tests/test_auth_api.py` — **no change expected**: the served-page orphan-`id` audit (line 1654) must pass untouched, because `#nav-toggle` and `#nav-tabs` are referenced (the stylesheet rule, `aria-controls`, the `app.js` selectors). If the audit flags either id, the fix is in the CSS/JS references, not in the audit.
|
||||||
|
3. Browser verification (invariant 10): `./scripts/dev-server.sh restart`, sign in with `testpass`, and with a ≤640px viewport (Playwright or a devtools device width — the repo's screenshots land in `data/logs/`):
|
||||||
|
- `data/logs/nav-hamburger-closed-360.png` — the phone-width masthead: no tab links, the `☰` visible, the `masthead-right` cluster intact.
|
||||||
|
- `data/logs/nav-hamburger-open-360.png` — the dropdown open: five stacked links, the `aria-current` tab highlighted, the cluster not covered.
|
||||||
|
- `data/logs/nav-masthead-desktop-1280.png` — the no-regression shot: one-row masthead, inline tabs, no button.
|
||||||
|
4. `AGENTS.md` — in the `UI conventions` section (where the `/login` masthead contract and the `app.js` ownership are documented), add the collapsing contract in the section's voice: the 640px breakpoint, the toggle's identity and its seven-page scope, the three close paths (link click, Esc with focus return, resize-back), the `aria-expanded` rule, and that `/login` carries no toggle. `README.md` is checked in the phase's final pass (its page table — "power events" etc. — is unchanged by this phase).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the three new served-page tests above + the untouched orphan-`id` audit + `test_the_disks_tab_is_in_the_masthead_of_every_page` (the nav's id addition must not break it)
|
||||||
|
- Coverage: **>90%** on the new test code; the full gate green
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/test_api.py tests/test_auth_api.py -q` green, including the three new tests and the untouched audit
|
||||||
|
- [ ] `grep -c 'nav-toggle' data/logs/…` — the three screenshots exist and show the closed / open / desktop states as named
|
||||||
|
- [ ] `bash .agents/validate.sh` green (pytest --cov --cov-fail-under=94, ruff, pyright, node --check)
|
||||||
|
- [ ] AGENTS.md's UI conventions name the contract; README.md reviewed and consistent
|
||||||
|
- [ ] no behavior change in completed work: the desktop pages and the login page are pixel-identical in structure (the pins prove it)
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# Phase 19 — Expandable UPS power-events list
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L4 — "The UPS tab should have an expandable, clear list of last incidents with dates/times"
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** `src/dashboards/web/pages/ups.html` lines 72–76 (the `Power events` card: `<header><h3>` + a `ul.events#events` holding a `loading…` placeholder), `src/dashboards/web/static/page-ups.js` (`refreshEvents()` at line 430 — fetches `/api/events?limit=200&source=ups` and hands the rows to `Dash.eventsPager` with the date/time + severity-chip + host row shape; called unconditionally from `refresh()` at line 427, which the page's `poll(refresh, state.pollMs)` loop drives, `state.pollMs` from `config.ups.poll_ms || 5000`), `src/dashboards/web/static/app.js` (`Dash.eventsPager` — 50 rows a page, paging state in a `WeakMap` keyed by the `<ul>`, rows re-written only when their markup changed), `src/dashboards/web/static/dash.css` (`details.charts-box` at lines 714–727 — the summary marker `▸`/`▾`, hidden webkit marker, hover; the `.events` bounded 320px scroll and `.events-pager` strip), and the pattern this phase mirrors — `/system`'s per-host Charts disclosure (`system.html` line 81, `page-system.js` line 315: "a `<details>` renders nothing, so … nothing is built or drawn while the stack is shut") and its pin `test_system_history_charts_are_collapsed_behind_a_toggle`
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The `/ups` power-events list becomes what the TODO asks for — *expandable*: the card is collapsed by default behind a `<details>` disclosure in the repo's own `charts-box` style, nothing is fetched or drawn while it is shut (the `/system` rule), the first expand fetches and builds, a re-expand paints from the data in hand unless it is stale, and the page's poll loop refreshes the list only while the disclosure is open. The "clear" half already ships and stays: every row carries its date/time, the severity chip, and the host, and the 50-per-page pager bounds the list. No new API surface — the existing `/api/events?source=ups` is the whole backend.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `18_nav_hamburger` (todo) — both phases touch `ups.html` (the masthead, the events card); landing 18 first keeps this phase's diff confined to the events section. No shared code between them.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_events_disclosure.md` — the `ups.html` markup: the card body becomes a collapsed-by-default `<details class="events-box">`
|
||||||
|
2. `02_events_lazy_loading.md` — `page-ups.js`: no fetch while closed, first-expand fetch, re-expand stale rule, poll-while-open
|
||||||
|
3. `03_disclosure_css.md` — `dash.css`: the `details.events-box` summary styling in the `charts-box` shape, pager and bounded list inside
|
||||||
|
4. `04_tests_and_verification.md` — the served-page pins (disclosure present and unopened, the lazy rule, the unchanged row shape) and the closed/open browser verification
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: `tests/test_api.py` — the served-page pins (see task 04), alongside the untouched `Dash.eventsPager` behaviour (a re-paint with the same data keeps the reader's page — the `WeakMap` rule, pinned by the existing event-feed tests)
|
||||||
|
- Coverage: **>90%** on new/modified code (the phase is markup + JS + CSS; the gate applies to the changed Python test files)
|
||||||
|
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
|
||||||
|
- AGENTS.md's `UI conventions` section already documents the event pager ("An event feed is a window, not a printout") and the `/system` Charts disclosure; it gains the `/ups` disclosure's sentence (collapsed by default, lazy-built, poll-while-open, the stale rule) in the section's voice. The README's `/ups` table row ("power events") stays true — the list is on the page, now behind a summary.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `/ups` loads with the Power events card collapsed (a summary line, no list visible, no `/api/events` request in the page's first poll) and the expand shows the bounded, paged list with date/time rows
|
||||||
|
- [ ] closing and re-opening within one poll interval paints the cached rows without a refetch; a stale cache refetches on open; while open, the poll loop keeps the list current
|
||||||
|
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
|
||||||
|
- [ ] `bash .agents/validate.sh` green (including `node --check` of the modified `page-ups.js`)
|
||||||
|
- [ ] closed- and open-state screenshots in `data/logs/` against the running dev server (invariant 10)
|
||||||
|
- [ ] no behavior change in completed work: every other element on `/ups` (cards, charts, dialog, pager behaviour) is unchanged
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
# Task 01 — The disclosure markup
|
||||||
|
|
||||||
|
**Phase:** `19_ups_events_expandable` · **Source:** `TODO.md:4` — "The UPS tab should have an expandable, clear list of last incidents with dates/times"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The `Power events` card on `/ups` becomes a collapsed-by-default `<details>` disclosure, in the markup shape the repo already owns for `/system`'s Charts stack. The card stays a card; only its body moves inside the disclosure. The `loading…` placeholder stays in the `ul` — it is the honest state between the first expand and the first successful fetch (the pager's output replaces it on arrival; an empty feed then prints the pager's own `empty` sentence, which is a different fact).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/pages/ups.html` — restructure the section at lines 72–76 from
|
||||||
|
```html
|
||||||
|
<section class="card" style="margin-top:.85rem">
|
||||||
|
<header><h3>Power events</h3></header>
|
||||||
|
<div class="body"><ul class="events" id="events"><li class="muted">loading…</li></ul></div>
|
||||||
|
</section>
|
||||||
|
```
|
||||||
|
to
|
||||||
|
```html
|
||||||
|
<section class="card" style="margin-top:.85rem">
|
||||||
|
<details class="events-box">
|
||||||
|
<summary>Power events</summary>
|
||||||
|
<div class="body"><ul class="events" id="events"><li class="muted">loading…</li></ul></div>
|
||||||
|
</details>
|
||||||
|
</section>
|
||||||
|
```
|
||||||
|
— no `open` attribute (collapsed by default; `ASSUMPTION:` confirmed by the operator — "expandable" means the list is shut until the operator opens it, the `/system` Charts pattern). The `#events` id, the `ul.events` class (the bounded 320px scroll + `.events-pager` strip target it), the `loading…` placeholder, and the pager's contract are all the page-script's, so nothing else in the file moves.
|
||||||
|
2. `ASSUMPTION:` the summary is the plain word `Power events` (the old `<h3>` text) — no count chip on the summary. A count that has not been fetched would be an invented number (rule 2), and the pager under the list already says `1–50 of N` once data is in hand.
|
||||||
|
3. `ASSUMPTION:` the disclosure is a plain `details` (no `data-k` attribute) — `/system`'s `data-k="charts"` exists for the per-host JS that rebuilds the stack; here the single page-level `page-ups.js` queries it by class, and the pin in task 04 names the class, not an attribute.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the served-page pin lands in task 04 (`details.events-box` present on the served `/ups`, no `open` attribute, the `summary` label, the `#events` ul inside, no `loading…` placeholder); the untouched page renders (the browser verification is task 04)
|
||||||
|
- Coverage: **>90%** on modified code (HTML only; the gate's suite must stay green)
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the served `/ups` contains the restructured section exactly as named (a `GET` pin in task 04: `details.events-box`, no `open` attribute, the `summary` label, the `#events` ul with its `loading…` placeholder inside)
|
||||||
|
- [ ] the page loads in a browser with the card collapsed: a `Power events` summary line, no list visible (task 04's screenshot)
|
||||||
|
- [ ] `uv run pytest -q` green; `bash .agents/validate.sh` green
|
||||||
|
- [ ] no behavior change in completed work: the rest of the page (cards, charts, dialog) is byte-identical in markup
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
# Task 02 — The lazy loading in `page-ups.js`
|
||||||
|
|
||||||
|
**Phase:** `19_ups_events_expandable` · **Source:** `TODO.md:4` — "The UPS tab should have an expandable, clear list of last incidents with dates/times"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Nothing is fetched or drawn while the disclosure is shut (the `/system` rule: "nothing is built or drawn while the stack is shut"), the first expand fetches and builds, a re-expand paints from the data in hand unless it is stale, and the page's poll loop refreshes the list only while the disclosure is open. The row shape, the pager, and the `loading…`-then-rows sequence are unchanged.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/static/page-ups.js` — `state` (line 17) gains one key, in the object's existing style: `events: { open: false, painted: false, fetchedAt: 0 }`.
|
||||||
|
2. Same file, the `refresh()` function (the `await refreshEvents();` at line 427 is its last statement) — the unconditional call becomes the open-gate:
|
||||||
|
```js
|
||||||
|
if (state.events.open) await refreshEvents();
|
||||||
|
```
|
||||||
|
— the poll loop (the `state.poller = poll(refresh, state.pollMs)` at line 464) keeps driving the list **only while the disclosure is open**; closed, the page pays nothing for a feed nobody can see.
|
||||||
|
3. Same file, `refreshEvents()` (line 430) — on success, before/after the existing `Dash.eventsPager(…)` call, stamp the state: `state.events.fetchedAt = Date.now(); state.events.painted = true;`. The existing `catch (err) { /* next poll */ }` stays untouched — a failed fetch leaves `painted`/`fetchedAt` where they were, so the next open re-fetches, and a failed poll while open keeps the last good rows (the invariant-4 shape, already the page's behaviour for everything else).
|
||||||
|
4. Same file, inside the `DOMContentLoaded` handler, before `state.poller = poll(…)`:
|
||||||
|
```js
|
||||||
|
const eventsBox = $('.events-box');
|
||||||
|
if (eventsBox) {
|
||||||
|
state.events.open = eventsBox.open; // false — task 01 ships no `open` attribute
|
||||||
|
eventsBox.addEventListener('toggle', () => {
|
||||||
|
state.events.open = eventsBox.open;
|
||||||
|
if (!eventsBox.open) return;
|
||||||
|
// First expand: `painted` is false — fetch. Re-expand: paint from the data
|
||||||
|
// in hand only while it is fresh; a cache older than one poll interval is
|
||||||
|
// stale and re-fetches. A fresh, already-painted list needs nothing: the
|
||||||
|
// `<details>` keeps its children, so the rows (and the pager's page, in its
|
||||||
|
// WeakMap) are already where the operator left them.
|
||||||
|
if (!state.events.painted || Date.now() - state.events.fetchedAt > state.pollMs) {
|
||||||
|
refreshEvents();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
```
|
||||||
|
`ASSUMPTION:` the stale bound is **one poll interval** (`state.pollMs`, from `config.ups.poll_ms || 5000`) — the `/system` Charts rule re-fetches a cached response "from another window"; here the window is the page's own, so one interval is the honest bound. A re-open within the interval paints the cache; a re-open after it re-fetches.
|
||||||
|
5. `ASSUMPTION:` no `loading…` fiddling — the placeholder from task 01 stays in the `ul` until the first successful fetch's pager output replaces it (a `<details>` does not destroy its children, so the placeholder, the rows, and the pager strip all survive collapse/re-expand untouched).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the served-script pins land in task 04 (the `state.events` shape, the `refresh()` open-gate, the `toggle` listener with its stale rule — asserted on the served bytes, the repo's way for JS behaviour since it has no JS test runner); `node --check` of the modified file passes (the gate); the untouched `Dash.eventsPager` tests stay green
|
||||||
|
- Coverage: **>90%** on modified code (JS; the gate's Python checks apply to task 04's test files)
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] in a browser: the page's first poll makes **no** `/api/events` request (devtools network panel) while the card is collapsed; the first expand makes exactly one; the list then updates on the poll cadence while open and stops when closed
|
||||||
|
- [ ] close → re-open within one poll interval: no request, the rows are exactly as left (page, scroll); close → wait longer than the interval → open: one request, fresh rows
|
||||||
|
- [ ] a failed fetch (devtools offline) while open keeps the last good rows and the next poll retries; the next open after a failure re-fetches
|
||||||
|
- [ ] `uv run pytest -q` green; `bash .agents/validate.sh` green (includes `node --check`)
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
# Task 03 — The disclosure CSS
|
||||||
|
|
||||||
|
**Phase:** `19_ups_events_expandable` · **Source:** `TODO.md:4` — "The UPS tab should have an expandable, clear list of last incidents with dates/times"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The `details.events-box` summary renders in the shape the repo already owns for `/system`'s Charts disclosure (`details.charts-box`, `dash.css` lines 714–727): the `▸`/`▾` marker, the hidden webkit marker, the hover, and the open-state de-emphasis — so a second opinion about the drive's word is never the bug, and a page that ships the summary without the style is the drift this task's pin catches.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/static/dash.css` — beside the existing `details.charts-box` rules (lines 714–727), add the `events-box` rules in the same block's voice (one selector per property, the file's style):
|
||||||
|
```css
|
||||||
|
details.events-box { min-width: 0; }
|
||||||
|
details.events-box > summary {
|
||||||
|
cursor: pointer; list-style: none;
|
||||||
|
font-weight: 600; color: var(--text-muted);
|
||||||
|
padding: .4rem .8rem .4rem 1.4rem;
|
||||||
|
position: relative;
|
||||||
|
}
|
||||||
|
details.events-box > summary::-webkit-details-marker { display: none; }
|
||||||
|
details.events-box > summary::before {
|
||||||
|
content: "\25B8"; position: absolute; left: .6rem;
|
||||||
|
font-size: 9px; color: var(--text-faint);
|
||||||
|
}
|
||||||
|
details.events-box[open] > summary::before { content: "\25BE"; }
|
||||||
|
details.events-box > summary:hover { color: var(--text-muted); }
|
||||||
|
details.events-box[open] > summary { color: var(--text-muted); margin-bottom: .55rem; }
|
||||||
|
```
|
||||||
|
(match the `charts-box` block's actual property set line-for-line — the snippet is the shape, the neighbouring block is the ground truth; do not invent tokens `:root` does not declare — the file's rule, pinned by `tests/test_api.py`).
|
||||||
|
2. The card's existing `.body` wrapper (task 01's markup) carries the `ul.events` bounded scroll (the existing 320px rule) and the `.events-pager` strip — **no new rules** for those; verify they resolve inside the disclosure (the `.body` is a child of `details`, and the existing selectors are class-based, so they do — confirm in the browser, task 04).
|
||||||
|
3. `ASSUMPTION:` no open/close animation on the disclosure — `charts-box` has none, and the file's `prefers-reduced-motion` block exists for the things that do move; a summary that snaps is the house style.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the served-stylesheet pin lands in task 04 (the `details.events-box` rule family present, the `::before` markers, no token a `:root` does not declare — the existing `test` that fails if a CSS rule reaches for an undeclared token must pass untouched); the `node --check` gate is unaffected (CSS), and the page loads in the browser (task 04)
|
||||||
|
- Coverage: **>90%** on modified code (CSS only; the gate's suite must stay green)
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] collapsed: the `Power events` summary line shows the `▸` marker in the card header area, muted, with the card's existing padding rhythm
|
||||||
|
- [ ] open: the marker flips to `▾`, the summary de-emphasises, and the bounded scroll list + pager strip render exactly as the other feeds' (the 320px scroll, the `1–50 of N` strip with its arrows)
|
||||||
|
- [ ] the page's other cards (the battery charts' `details`-free section, the `#ups-cards` grid) are pixel-identical
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 04 — Pins and browser verification
|
||||||
|
|
||||||
|
**Phase:** `19_ups_events_expandable` · **Source:** `TODO.md:4` — "The UPS tab should have an expandable, clear list of last incidents with dates/times"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The phase is pinned on the served bytes the way this repo pins every page surface: the disclosure is on the served `/ups` and **unopened**, the lazy-load rule is in the served script, and the row shape — the "clear" half of the TODO (date/time, severity chip, host) — is unchanged. Then the designed degradation is proven in a real browser: the closed state, the open state, and the no-request-while-closed fact.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `tests/test_api.py` — new tests, in the file's served-page style (the `self_test_app` harness and the `test_system_history_charts_are_collapsed_behind_a_toggle` pin at the `/system` disclosure are the models):
|
||||||
|
- `test_the_ups_power_events_list_is_a_collapsed_disclosure`: a signed `GET /ups` finds `details.events-box` with **no** `open` attribute, a `summary` reading `Power events`, and the `ul.events#events` (with its `loading…` placeholder) inside it — and the old always-open shape is gone (no bare `header > h3` directly above the `ul` at the section level).
|
||||||
|
- `test_the_ups_events_fetch_is_gated_on_the_disclosure`: the served `page-ups.js` carries the lazy rule — the `state.events` shape (`open`, `painted`, `fetchedAt`), the `refresh()` open-gate (`state.events.open` guard on the `refreshEvents()` call), and the `toggle` listener with the stale bound (`state.pollMs`) — asserted on the served script text, the repo's established device for JS behaviour (no JS test runner; the `node --check` in the gate covers syntax).
|
||||||
|
- `test_the_ups_event_rows_keep_the_date_time_and_severity_shape`: the served `page-ups.js` row template still renders the `<time>` with the date/time `toLocaleString` options, the `sev ${event.severity}` chip, and the host — the "clear list with dates/times" the TODO asks for, pinned so a "cleanup" that drops the timestamp fails here instead of shipping.
|
||||||
|
2. The existing event-feed pins (`tests/test_api.py`'s `Dash.eventsPager`-behaviour tests, the `EVENT_FEED_LIMIT`/`EVENT_PAGE_SIZE` pairing) pass **untouched** — the phase adds no API surface and no pagination vocabulary.
|
||||||
|
3. Browser verification (invariant 10): `./scripts/dev-server.sh restart`, sign in with `testpass`, real upsd in the loop (the dev config dials the fleet's `upsd` on 3051 — two real UPSes, no `NUT_SIMULATE`):
|
||||||
|
- `data/logs/ups-events-closed.png` — the page with the card collapsed: the `Power events` summary line, no list, and the devtools network panel showing **no** `/api/events?source=ups` request on the first poll (screenshot the panel or note the request count in the filename's companion note).
|
||||||
|
- `data/logs/ups-events-open.png` — the expanded card: the bounded scroll list with date/time rows, the severity chips, and the `1–50 of N` pager strip (or the empty sentence, if the event table is genuinely empty — either is the correct state; the pin is the *shape*, not the data).
|
||||||
|
4. `AGENTS.md` — in the `UI conventions` section, beside the event-pager paragraph ("An event feed is a window, not a printout") and the `/system` Charts-disclosure paragraph, add the `/ups` sentence in the section's voice: the list is a collapsed-by-default `details` disclosure, nothing is fetched while it is shut, the first expand fetches and builds, a re-expand paints the cache while it is fresher than one poll interval (stale re-fetches), and the poll loop refreshes only while open. `README.md` is checked in the phase's final pass (its `/ups` row — "power events" — stays true; the list is on the page, behind a summary).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the three new served-page tests above + the untouched pager/limit pairing tests + `test_system_history_charts_are_collapsed_behind_a_toggle` (the pattern pin, untouched)
|
||||||
|
- Coverage: **>90%** on the new test code; the full gate green
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/test_api.py -q` green, including the three new tests and the untouched event-feed pins
|
||||||
|
- [ ] `data/logs/ups-events-closed.png` and `data/logs/ups-events-open.png` exist and show the states as named (closed: summary only; open: bounded list + pager; the closed shot's note records the zero `/api/events` requests on the first poll)
|
||||||
|
- [ ] `bash .agents/validate.sh` green (pytest --cov --cov-fail-under=94, ruff, pyright, node --check)
|
||||||
|
- [ ] AGENTS.md's UI conventions carry the disclosure sentence; README.md reviewed and consistent
|
||||||
|
- [ ] no behavior change in completed work: the rest of `/ups` (cards, charts, the confirm dialog, the banners) renders exactly as before
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# Phase 20 — Masthead & navigation polish
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L4, L5, L6, L10, L11 — "masthead row layout, un-sticky bar, hamburger animation, always-visible admin links"
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** `src/dashboards/web/static/dash.css` (`.masthead` at line 99 — `position: sticky; top: 0; z-index: 40; display: flex; height: var(--header-h)`; `.masthead-right` at line 148 — `margin-inline-start: auto`; `#nav-toggle` at line 190 — the `--nav-toggle-display` token reveal; the ≤640px block at lines 208–280 — masthead wrap, `masthead-right { justify-content: flex-end }`, the `nav.tabs` dropdown at `top: 100%` under the sticky bar, the `env(safe-area-inset-top, 0px)` padding line), `src/dashboards/web/static/app.js` (the `setNavMenu()` funnel at lines 695–722 — four close paths, `matchMedia('(max-width: 640px)')`), `src/dashboards/web/pages/*.html` (the masthead in every page: brand, `#nav-toggle`, `nav.tabs#nav-tabs`, `.masthead-right` holding `#nav-status` + clock + `⚷ Certs` + `⚙ Account` + theme toggle; `certs.html` lacks `⚷ Certs`, `settings.html` lacks `⚙ Account` — the phase-15 "no self-link" rule this phase inverts), `tests/test_api.py` (`test_every_page_links_to_both_admin_pages` at line 999, `test_the_disks_tab_is_in_the_masthead_of_every_page` at line 3913, `test_the_nav_toggle_is_in_the_masthead_of_every_tabbed_page` at line 3957)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The masthead becomes a three-row bar at **all widths** — row 1 the brand and the five dashboard tabs, row 2 the per-source status indicators left-justified in their own row, row 3 the `⚷ Certs` / `⚙ Account` / `☾ Dark` controls left-justified in their own row under the indicators — the bar stops being sticky (it scrolls away, maximising data screen space), the hamburger menu slides out and back in (180 ms, the `brain_of_reese` pattern, stilled under `prefers-reduced-motion`), and both admin links stay visible on every page including the two admin pages themselves. The served page markup is unchanged except the two added admin links; the tab structure (five links, one `aria-current`), the `#nav-toggle` token reveal, the safe-area padding line and the four close paths all stay exactly as phase 18 pinned them.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `19_ups_events_expandable` (todo) — both phases touch `ups.html` (the masthead, the events card); landing 19 first keeps this phase's diff confined to the masthead.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_two_row_masthead.md` — the three-row left-justified masthead at all widths (CSS-only reflow; the mobile block re-cut for the new rows)
|
||||||
|
2. `02_unsticky_masthead.md` — the bar scrolls away: `sticky` → `relative`, the dropdown anchor kept
|
||||||
|
3. `03_hamburger_slide.md` — the 180 ms slide+fade on the ≤640px dropdown, stilled under `prefers-reduced-motion`
|
||||||
|
4. `04_admin_links_visible.md` — `⚷ Certs` on `/certs`, `⚙ Account` on `/settings`; the "no self-link" pin inverted to "all eight pages link to both"
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: `tests/test_api.py` — the served-page pins (task 04's rewritten admin-link test over all eight non-login pages, the unchanged tab-structure pins at lines 3913/3957, the CSS pins for the row layout and the un-sticky change), the unchanged `setNavMenu` behaviour
|
||||||
|
- Coverage: **>90%** on new/modified code (the phase is markup + JS + CSS; the gate applies to the changed Python test files)
|
||||||
|
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**. AGENTS.md's masthead paragraph (the phase-18 "specificity ladder" and the `top: 100%`-under-the-sticky-bar wording) and the phase-15 "neither admin page links to itself" sentence (gap 12's "now questioned" item — this phase is the decision) both get rewritten to the new state in the section's voice.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] at 1280 px and at 360 px, every tabbed page and both admin pages render the three-row masthead: brand + tabs, left-justified indicators row, left-justified controls row under it (screenshots in `data/logs/`, invariant 10)
|
||||||
|
- [ ] scrolling any page at 1280 px and 360 px leaves the bar behind (no sticky offset), while the mobile dropdown still opens directly under the bar
|
||||||
|
- [ ] at 360 px the hamburger menu slides out and back in (180 ms); with `prefers-reduced-motion` it snaps
|
||||||
|
- [ ] `/certs` renders `⚷ Certs` and `/settings` renders `⚙ Account`, before the theme toggle and outside `.tabs`, exactly like the five dashboards
|
||||||
|
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
|
||||||
|
- [ ] `bash .agents/validate.sh` green (including `node --check` of the modified `app.js`)
|
||||||
|
- [ ] no behavior change in completed work: the tab structure, the token reveal, the safe-area padding line, the four close paths, the status dots and the theme toggle are all unchanged
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 01 — The three-row left-justified masthead, at all widths
|
||||||
|
|
||||||
|
**Phase:** `20_masthead_nav_polish` · **Source:** `TODO.md:5,6` — "The hosts disks engines and UPS indicators should be left justified in their own row" / "The ⚷ Certs ⚙ Account ☾ Dark buttons should be left justified in their own row under the hosts disks engines UPS indicators"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The single-row masthead becomes a three-row bar at **every** width: row 1 keeps the brand, `#nav-toggle` and the five dashboard tabs; row 2 is the `#nav-status` indicator row (the per-source dots), left-justified and on its own; row 3 is the `.masthead-right` cluster (clock, `⚷ Certs`, `⚙ Account`, `☾ Dark` — plus the `⏸ Pause` button until phase 21 removes it), left-justified and on its own row under the indicators. The change is CSS-only: no page markup changes in this task, so the served-HTML pins (tab structure, toggle placement) stay green untouched.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/static/dash.css` — `.masthead` (line 99): `display: flex; flex-direction: column; align-items: stretch; gap: .35rem` (replacing the row layout's `align-items: center; gap: 1.1rem`), `height: var(--header-h)` replaced by `min-height: var(--header-h)` (the ≤640px block already carries its own `min-height` — keep the mobile one, the two must not drift apart; see the comment it carries), `position: sticky; top: 0` **left in place in this task** (task 02 removes the stickiness; keep `z-index: 40`), and a positioned wrapper for row 1: the first flex child is the existing `brand` + `#nav-toggle` + `nav.tabs` trio — since the markup is not touched, use CSS `order`/`flex` only: `#nav-status` and `.masthead-right` each get `flex-basis: 100%` (forcing the row break), `nav.tabs` keeps its current inline position on row 1 (it already flows after the toggle). If `flex-basis: 100%` does not force a row break on the `nav` element in the pinned browser, the fallback is a `display: contents` wrapper-free `flex-wrap: wrap` on `.masthead` with the same `flex-basis: 100%` rules — measure in a browser, do not deduce (the phase-18 rule).
|
||||||
|
2. `src/dashboards/web/static/dash.css` — `#nav-status` (the `gap` container, line 148 neighbourhood): its own row styling — `flex-basis: 100%; display: flex; align-items: center; gap: .55rem; justify-content: flex-start` (left-justified, no `margin-inline-start: auto`).
|
||||||
|
3. `src/dashboards/web/static/dash.css` — `.masthead-right` (line 148): `margin-inline-start: auto` removed, `flex-basis: 100%; justify-content: flex-start` (left-justified, the TODO's words). The `⏸ Pause` button stays in this row (phase 21 deletes it; do not pre-empt it).
|
||||||
|
4. `src/dashboards/web/static/dash.css` — the ≤640px block (lines 208–280): re-cut for the new rows — the whole-bar `flex-wrap`/`min-height` machinery of the old wrap is replaced by the mobile half of the same three-row layout: `#nav-toggle` keeps its `--nav-toggle-display: inline-flex` reveal on row 1 (the token ladder stays exactly as pinned), `nav.tabs` stays hidden except `nav.tabs.is-open` (the dropdown), the `masthead-right > *` no-wrap rule and the `env(safe-area-inset-top, 0px)` top padding line (pinned by `tests/test_api.py`'s rule loop) survive verbatim, and the comment at line 203 (which describes the old 360 px wrap) is rewritten to the new three-row reality. The dropdown's `top: 100%` now lands under the **whole bar** (three rows) on mobile — that is the intended look (a menu under the bar, not wedged between rows); verify in a browser at 360 px.
|
||||||
|
5. `tests/test_api.py` — extend the served-stylesheet pins for the new layout: `.masthead` has `flex-direction: column` (or the `flex-wrap` fallback, whichever shipped) and no fixed `height: var(--header-h)`, `#nav-status` and `.masthead-right` are both `flex-basis: 100%` + `justify-content: flex-start`, and `.masthead-right` no longer declares `margin-inline-start: auto`. The existing pins (the media-query-vs-`matchMedia` comparison, the safe-area line, the token ladder) must stay green — run them before touching the file.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: `tests/test_api.py` — the new layout pins above run against the **served** bytes; the unchanged pins (tab structure line 3913, toggle line 3957, the safe-area rule loop) stay green
|
||||||
|
- Coverage: **>90%** on this task's new/modified code (the gate applies to the changed Python test files)
|
||||||
|
- Browser verification (invariant 10): screenshots of `/`, `/system`, `/certs`, `/settings` at 1280 px and 360 px in `data/logs/` — the three rows, indicators and controls left-justified, the mobile dropdown under the bar
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] at 1280 px and 360 px, the masthead renders three rows in the order brand+tabs / indicators / controls, the last two left-justified, on every tabbed page and both admin pages (screenshots in `data/logs/`)
|
||||||
|
- [ ] `tests/test_api.py` green including the new layout pins and every pre-existing masthead pin
|
||||||
|
- [ ] full test suite green
|
||||||
|
- [ ] no behavior change in completed work: the tabs, the toggle, the dots, the clock, the links and the theme toggle all behave exactly as before
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 02 — The masthead is not sticky
|
||||||
|
|
||||||
|
**Phase:** `20_masthead_nav_polish` · **Source:** `TODO.md:10` — "The navbar is only sticky part way down certain pages on mobile? It shouldn't be sticky at all. We want to maximize the amount of screen space for data viewing."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The masthead scrolls away with the page. The "sticky part way down" quirk is the fixed-height sticky bar fighting the wrapped rows on mobile; removing the stickiness removes the quirk and the bar's vertical footprint. The ≤640px dropdown (`nav.tabs.is-open` at `top: 100%`) still opens directly under the bar, because the bar stays a **positioned** ancestor — `position: relative` keeps the anchor without the stick.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/static/dash.css` — `.masthead` (line 99): `position: sticky; top: 0` → `position: relative` (keeps the positioned-ancestor role for the dropdown's `top: 100%` and the `left/right: 0` span; `z-index: 40` stays — the dropdown's `z-index: 41` is "one above the masthead" and that sentence stays true).
|
||||||
|
2. `src/dashboards/web/static/dash.css` — the dropdown comment (line 266: "`position: sticky`, which is a positioned ancestor") rewritten to `position: relative`; any other comment in the file that calls the bar sticky gets the same fix.
|
||||||
|
3. `tests/test_api.py` — any pin that asserts the `position: sticky` declaration (grep the test files for `sticky`) is updated to assert `position: relative` **and** the absence of `top: 0` on `.masthead`; if no pin named the declaration, add one to the layout pin loop from task 01 so the stickiness cannot quietly return.
|
||||||
|
4. `src/dashboards/web/static/dash.css` — check the `sticky; top: 0` at line 796 (a **different** element — the event-feed pager strip's own sticky header): it is not the masthead, do not touch it.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: `tests/test_api.py` — the `.masthead` pin above, the unchanged dropdown pin (the media query still names `top: 100%` under the bar)
|
||||||
|
- Coverage: **>90%** on this task's new/modified code
|
||||||
|
- Browser verification (invariant 10): at 1280 px and 360 px, scroll `/system` and `/` past one viewport — the bar is gone from the top of the viewport; at 360 px open the dropdown — it still renders directly under the bar (screenshot pair in `data/logs/`)
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] scrolling any page at any width leaves the masthead behind (no element pinned to the top edge but the pager strip's own header)
|
||||||
|
- [ ] the mobile dropdown at 360 px still opens directly under the bar (screenshot in `data/logs/`)
|
||||||
|
- [ ] `tests/test_api.py` green including the new `.masthead` pin
|
||||||
|
- [ ] full test suite green
|
||||||
|
- [ ] no behavior change in completed work: the pager strip's sticky header (line 796) and the dropdown's `z-index` stacking are untouched
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 03 — The hamburger menu slides out and back in
|
||||||
|
|
||||||
|
**Phase:** `20_masthead_nav_polish` · **Source:** `TODO.md:11` — "I want a nice animation for the hamburger menu - a slide out and slide back in would be perfect. See ~/Project/Personal/brain-of-reese for an example."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The ≤640px dropdown (`nav.tabs.is-open`) opens and closes with an 180 ms slide+fade instead of snapping — the exact pattern the operator pointed at in `brain_of_reese`: closed = `visibility: hidden; opacity: 0; transform: translateY(-8px); pointer-events: none` with a `visibility 0s linear 180ms` delayed transition, open = `visibility: visible; opacity: 1; transform: none; pointer-events: auto` with the `visibility` delay dropped, and the whole thing stilled under `prefers-reduced-motion` with **both states named** (the reference project's phase-46 comment: an open-state transition that is not stilled still animates). `app.js`'s `setNavMenu()` funnel and its four close paths are untouched — the animation is pure CSS over the existing class flip.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. Reference (read-only): `/var/home/ducoterra/Projects/Personal/brain_of_reese/frontend/assets/styles.css` lines 6960–6976 (the `.app-nav` closed/open transition pair — note the TODO's path `~/Project/Personal/brain-of-reese` is a typo; the project lives at `~/Projects/Personal/brain_of_reese`) and lines 7228–7235 (the `prefers-reduced-motion` block that names **both** `.app-nav` and `.app-nav.is-open`, with the specificity comment: an open rule that is not named keeps animating).
|
||||||
|
2. `src/dashboards/web/static/dash.css` — inside the ≤640px block (lines 208–280), the `nav.tabs` rule (the dropdown): add the closed-state properties + transition from the reference (adapted: this nav is `position: absolute; top: 100%`, so the slide is `transform: translateY(-8px)` → `none`, same 180 ms ease; keep the existing `visibility: hidden`/`pointer-events: none` closed-state behaviour if it is already there, otherwise add it — measure the current rule first, do not assume). The `nav.tabs.is-open` state gets the visible counterpart with the `visibility 0s` (undelayed) transition.
|
||||||
|
3. `src/dashboards/web/static/dash.css` — a `@media (prefers-reduced-motion: reduce)` block (the file's first, or merged with an existing one if there is one) that sets `transition: none` on **both** `nav.tabs` and `nav.tabs.is-open` — both states named, per the reference's phase-46 comment.
|
||||||
|
4. `tests/test_api.py` — extend the served-CSS pin for the mobile block: the `nav.tabs` rule carries an `opacity`/`transform` transition of 180 ms, the `is-open` state carries the visible counterpart, and the reduced-motion block names both states. The existing media-query-vs-`matchMedia` comparison test stays green untouched (the transition is additive; the `640px` literal still lives in exactly the two pinned places).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: `tests/test_api.py` — the pins above; the unchanged four-close-path behaviour (the `setNavMenu` funnel tests)
|
||||||
|
- Coverage: **>90%** on this task's new/modified code
|
||||||
|
- Browser verification (invariant 10): at 360 px, open and close the menu — a CDP check that the computed `transition` runs (a mid-transition screenshot is not required; assert the closed state's `opacity: 0` + `transform: translateY(-8px)` computed values and that a frame after `.is-open` is added shows `opacity > 0`), and with `prefers-reduced-motion: reduce` emulated the menu snaps (computed `transition: none`) — extend the phase-18 verification script pattern (`data/logs/nav-task04-verify.cjs`) rather than hand-asserting
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] at 360 px the menu slides out (fade + 8 px rise) and back in over 180 ms; all four close paths still work
|
||||||
|
- [ ] under `prefers-reduced-motion: reduce` the menu snaps in both directions
|
||||||
|
- [ ] `tests/test_api.py` green including the new transition pins and the pre-existing `640px` comparison pin
|
||||||
|
- [ ] full test suite green
|
||||||
|
- [ ] no behavior change in completed work: the token reveal, the safe-area line, the dropdown position under the bar and the close paths are untouched
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 04 — Both admin links stay visible on the admin pages
|
||||||
|
|
||||||
|
**Phase:** `20_masthead_nav_polish` · **Source:** `TODO.md:4` — "Clicking the certs tab causes the certs button to disappear, then clicking the account tab causes the accounts button to disappear. These should remain visible."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Phase 15's deliberate "neither admin page links to itself" rule is inverted by operator decision (it was already "now questioned" in AGENTS.md gap 12): `certs.html` gains the `⚷ Certs` link and `settings.html` gains the `⚙ Account` link, so the masthead's controls are identical on all eight non-login pages. The links sit before the theme toggle and outside `.tabs` exactly as on the five dashboards; the tab structure (five links, no `aria-current` on the admin pages) and the `/login` page (brand + theme toggle only) are untouched.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/pages/certs.html` — the masthead (lines 14–28): insert `<a class="ghost account" href="/certs" title="Agent certificates">⚷ Certs</a>` between the clock and the `⚙ Account` link (the dashboard order is clock → `⚷ Certs` → `⚙ Account` → theme toggle; `settings.html` already holds that order), and rewrite the HTML comment at lines 14–17, which currently justifies the *missing* link, to the new rule (both links everywhere; a self-link is a no-op the operator can click and not be surprised by — the TODO is the decision).
|
||||||
|
2. `src/dashboards/web/pages/settings.html` — the masthead (lines 14–30): insert `<a class="ghost account" href="/settings" title="Session and password">⚙ Account</a>` between the `⚷ Certs` link and the theme toggle; the comment at lines 12–14 ("It is reached from the `⚙ Account` link in the masthead of every dashboard") gains the "including this one" sentence.
|
||||||
|
3. `tests/test_api.py` — rewrite `test_every_page_links_to_both_admin_pages` (line 999): the two loops become **one** loop over all eight non-login pages (`*PAGES, SETTINGS_PAGE, CERTS_PAGE`) asserting both links are present, both before `id="theme-toggle"`, and `"/certs"` is not in the `.tabs` markup; the two `assert … not in … "links to itself"` lines are **deleted** (that was the old rule); add one explicit assertion that `login.html` carries **neither** link (the pre-auth surface renders brand + theme toggle only — that sentence in AGENTS.md stays true).
|
||||||
|
4. `AGENTS.md` — (Documentation & Consistency pass) the phase-15 "neither admin page links to itself, which is why the pinned two-link audit loops over two different tuples" sentence and gap 12's "now questioned" item are resolved to the new state.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: `tests/test_api.py` — the rewritten `test_every_page_links_to_both_admin_pages`, the unchanged `test_the_disks_tab_is_in_the_masthead_of_every_page` (line 3913 — the tab structure is untouched by this task)
|
||||||
|
- Coverage: **>90%** on this task's new/modified code
|
||||||
|
- Browser verification (invariant 10): signed in, click `⚷ Certs` on `/certs` (stays on `/certs`, no navigation loop) and `⚙ Account` on `/settings` (same) — screenshots of both pages' mastheads in `data/logs/`
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `/certs` renders `⚷ Certs` and `/settings` renders `⚙ Account`, both before the theme toggle and outside `.tabs`
|
||||||
|
- [ ] all eight non-login pages render both admin links; `/login` renders neither
|
||||||
|
- [ ] `tests/test_api.py` green with the rewritten test
|
||||||
|
- [ ] full test suite green
|
||||||
|
- [ ] no behavior change in completed work: the tab strip, the `aria-current` absence on the admin pages, the clock and the theme toggle are untouched
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# Phase 21 — Page cleanups: pause, GPU values, certs scroll
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L7, L9, L12 — "remove the pause button, GPU data as its own values, /certs table horizontal scroll"
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** `src/dashboards/web/pages/{disks,llm,system,ups}.html` line 25 (the `#pause` button, the only one in the served markup — `/`, `/certs` and `/settings` have none), `src/dashboards/web/static/app.js` (lines 587–615 the `poll()` `_paused` plumbing, lines 773–783 the `#pause` binding plus the `Space` shortcut), `src/dashboards/web/static/dash.css` line 203 (the comment naming the pause button), `src/dashboards/web/static/page-system.js` (lines 150–156 `gpuSub()` — the dense sub-line the TODO quotes, lines 164–188 `renderGpuSlot()`, line 72 the `data-k="gpus"` hero slot), `tests/test_api.py` (line 1425 — the six-heroes pin: "exactly six heroes: CPU, MEM, TEMP, DISK, GPU, NET", asserted over the three captures: discrete GPU, APU, no-GPU), `src/dashboards/web/pages/certs.html` line 81 (`<table class="data" id="agents-table">` — no `.scroll` modifier; `dash.css` gives `table.data` a scroll container only with the `.scroll` class), `src/dashboards/web/static/page-certs.js` (lines 55–56 — the table element references)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Three small page-level cleanups, one per workstream: the pause button is gone from every page (the hidden-tab skip stays — a background tab idling its polls is not the operator's pause), the `/system` GPU data is split out of the dense sub-line into its own labelled values and a host with no GPU **omits** the slot instead of printing `n/a`, and the `/certs` agents table scrolls horizontally so the right-hand columns are reachable on a phone. No API surface changes; the pins that name the old behaviour (six heroes incl. an `n/a` GPU slot, the unpinned pause button) are rewritten, not left rotting.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `20_masthead_nav_polish` (todo) — the pause button lives in the masthead row task 01.01 reflows, and the `/system` hero row is re-measured under the new bar; landing 20 first keeps 21's diff confined to the page bodies.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_remove_pause_button.md` — delete the button, the `Space` shortcut, the `setPaused`/`_paused` plumbing and the CSS comment; the hidden-tab skip survives
|
||||||
|
2. `02_gpu_own_values.md` — the dense `gpuSub()` line becomes four labelled values per card; a GPU-less host omits the slot (five heroes, no `n/a`)
|
||||||
|
3. `03_certs_table_scroll.md` — `#agents-table` gets the `.scroll` modifier so its right-hand columns are reachable at phone width
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: `tests/test_api.py` — the served-page pins: the pause button absent from the four pages (a return of `#pause` fails a test), the six-heroes pin rewritten (GPU-present host: six, GPU-absent host: five with no `data-k="gpus"` slot), the `#agents-table` `.scroll` pin; `tests/test_system_contract.py` + `tests/card_contract.py` stay green (the API fields are unchanged — only the page's reading of them moves)
|
||||||
|
- Coverage: **>90%** on new/modified code (the phase is markup + JS + CSS; the gate applies to the changed Python test files)
|
||||||
|
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**. AGENTS.md's `/system` paragraph ("a missing number or verdict — gateway's CPU temp, any host's absent GPU — prints `n/a` **uncoloured**") is the one rule this phase inverts for the GPU slot (the CPU-temp `n/a` stays — the TODO quotes the GPU line only); the sentence is split, the GPU half rewritten to the omission rule, and the pause button's mention in the masthead comment dropped.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] no page serves a `#pause` button or binds the `Space` key; a hidden tab still skips its polls (served-bytes pin + a CDP `visibilitychange` check)
|
||||||
|
- [ ] `/system`'s GPU slot prints model / VRAM / pool / temp as four labelled values; a GPU-less host (the no-GPU capture, gateway) renders five heroes with no GPU slot and no `n/a` in it (screenshots in `data/logs/`)
|
||||||
|
- [ ] at 360 px the `/certs` agents table scrolls horizontally to its last column (screenshot in `data/logs/`)
|
||||||
|
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
|
||||||
|
- [ ] `bash .agents/validate.sh` green (including `node --check` of the modified page scripts)
|
||||||
|
- [ ] no behavior change in completed work: the six-hero order on GPU hosts, the hidden-tab skip, the GPU history charts, the `/certs` row contents and the event pager are all unchanged
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 01 — Remove the pause button
|
||||||
|
|
||||||
|
**Phase:** `21_page_cleanups` · **Source:** `TODO.md:7` — "Completely remove the pause button"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The `⏸ Pause` control is gone from the served markup, the page scripts and the shared poller: no button, no `Space`-key shortcut, no `setPaused`/`_paused` plumbing. The **hidden-tab** behaviour survives — `document.hidden` still skips a poll while the tab is backgrounded and `visibilitychange` still fires one on return; that is idle hygiene, not the operator's pause, and the TODO names the button only.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/pages/disks.html`, `src/dashboards/web/pages/llm.html`, `src/dashboards/web/pages/system.html`, `src/dashboards/web/pages/ups.html` — delete the line-25 `<button class="ghost" id="pause" type="button" aria-pressed="false">⏸ Pause</button>` from each of the four (the only served occurrences — `/`, `/certs`, `/settings`, `/login` have none; a grep of `web/pages` for `pause` must come back empty when done).
|
||||||
|
2. `src/dashboards/web/static/app.js` — delete the `#pause` binding block (lines 773–783: the button lookup, the `aria-pressed` paint, the click handler, and the `keydown` `Space` handler with its `event.preventDefault()`); in `poll()` (lines 587–615) delete the `_paused` state, `setPaused()`, the `paused` getter and the `state._paused` checks in `tick()`/`schedule()`/`visibilitychange` — the `document.hidden` guard and the `visibilitychange` refire stay, minus the `&& !state._paused` clause; the docstring at line 587 ("pausing when the tab is hidden or the user hit pause") drops the pause half.
|
||||||
|
3. `src/dashboards/web/static/dash.css` — line 203's comment (which lists "status dots, pause, `⚷ Certs`, `⚙ Account`, theme" as the 360 px wrap's tenants) drops the pause mention.
|
||||||
|
4. `tests/test_api.py` — add the served-bytes pin: `id="pause"` is **absent** from all eight served pages (a return of the button fails a test) and `⏸` appears nowhere in the served scripts.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: `tests/test_api.py` — the absence pin above; the existing poller tests (the `Dash.poll` hidden-tab behaviour, if any pin it — grep `hidden` in the test files first) stay green
|
||||||
|
- Coverage: **>90%** on this task's new/modified code
|
||||||
|
- Browser verification (invariant 10): `/` and `/system` — no pause control in the masthead, the `Space` key does nothing on the page body; hide the tab (CDP `Emulation.setDocumentHidden` or the window manager) and confirm the poll loop idles, then refire on return (the phase-18 verification script pattern)
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `grep -ri pause src/dashboards/web/` comes back empty (code, markup and comments all clean)
|
||||||
|
- [ ] a hidden tab skips polls and a visible tab fires one on return (verified, not deduced)
|
||||||
|
- [ ] `tests/test_api.py` green including the new absence pin
|
||||||
|
- [ ] full test suite green
|
||||||
|
- [ ] no behavior change in completed work: the range chips, the event pagers, the status dots and every other masthead control behave exactly as before
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# Task 02 — GPU data as its own values; GPU-less hosts omit the slot
|
||||||
|
|
||||||
|
**Phase:** `21_page_cleanups` · **Source:** `TODO.md:12` — "The GPU data is wrapped poorly - Radeon 8060S Graphics · VRAM 180.2 MiB / 512.0 MiB · pool 79% · 69 °C is too much info for such small text and should be its own values. Boxes without a GPU should not report N/A but instead should omit these values."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The `/system` hero card's GPU slot stops stuffing model · VRAM · pool · temp into one small sub-line: each becomes its own labelled value on the card, at a readable size. A host with no GPU (gateway, the no-GPU capture) no longer renders an `n/a` hero slot — the slot is **omitted** and the hero row simply has five. The API is untouched: `gpus[]` keeps every field (`model`, `vram_used_bytes`, `vram_total_bytes`, `mem_used_pct`, `temp_c`), and the per-card history (`gpu.busy_pct@N`, `gpu.temp_c@N`) and the collapsed GPU chart are unchanged.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/static/page-system.js` — replace `gpuSub()` (lines 150–156) with a `gpuValues(gpu)` that renders four labelled cells — **Model** (mono, the `gpu.model` string), **VRAM** (`used / total` via the existing `bytes()` formatter, `n/a` if either side is absent), **Pool** (`mem_used_pct` %), **°C** (`temp_c`) — laid out as a compact `.gpu-values` grid under the busy hero (the hero itself keeps only the `busy_pct` number + its label; colour rules stay: the GPU is uncoloured, phase 06's "a busy GPU is the job" rule — no severity token, no bar). A missing field on a **present** card prints `n/a` in its own cell (rule 2: the field is absent, the card is real); a `null` verdict cell draws faint, the existing `Dash.hero` convention.
|
||||||
|
2. `src/dashboards/web/static/page-system.js` — `renderGpuSlot()` (lines 164–188): when `gpus` is empty, do **not** render the slot's fallback `n/a` content — render nothing (empty the slot's container and drop its hero styling so the row reflows to five heroes; the `data-k="gpus"` wrapper can stay as an empty, zero-footprint element or be removed — pick one and pin it). The multi-GPU case: one value grid per card, the cards' `GPU i` labels unchanged.
|
||||||
|
3. `src/dashboards/web/static/dash.css` — the `.gpu-values` grid: a compact 2×2 (or 4×1 on wide) of labelled cells reusing the existing token vocabulary (`.metric`/`.faint`/`.mono` shapes, no new palette — the app's colour rule), sized so a 20-character model string wraps rather than truncates.
|
||||||
|
4. `tests/test_api.py` — rewrite the six-heroes pin (line 1425): a host whose snapshot carries `gpus` still renders exactly six slots **in the existing order** (CPU, MEM, TEMP, DISK, GPU, NET); a host with no GPU renders exactly five with **no** `data-k="gpus"` slot (or the pinned empty variant) and no `n/a` in the hero row; the `gpu-values` cells (Model / VRAM / Pool / °C) are present in the served script. `tests/test_system_contract.py` and `tests/card_contract.py` stay green untouched (API fields unchanged).
|
||||||
|
5. `AGENTS.md` — (Documentation & Consistency pass) the `/system` paragraph's "a missing number or verdict — gateway's CPU temp, any host's absent GPU — prints `n/a` **uncoloured**" is split: the CPU-temp half keeps the `n/a` rule, the GPU half becomes the omission rule (the TODO is the decision), and the dense-sub-line description of the GPU slot is rewritten to the four-value layout.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: `tests/test_api.py` — the rewritten hero pin over the three captures (discrete GPU, APU, no-GPU) **and** over HTTP (the `self_test`-style app harness the contract tests use); `tests/test_system_contract.py` + `tests/card_contract.py` green untouched
|
||||||
|
- Coverage: **>90%** on this task's new/modified code
|
||||||
|
- Browser verification (invariant 10): `/system` against the live fleet — deskwork's Strix Halo card shows the four values at readable size (the TODO's quoted string is gone from the sub-line), gateway's card shows five heroes with no GPU slot and no `n/a`; the collapsed GPU charts still draw for the GPU hosts; screenshots in `data/logs/`
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] a GPU host's card shows Model / VRAM / Pool / °C as four labelled values; the dense sub-line string from the TODO is gone
|
||||||
|
- [ ] a GPU-less host renders five heroes with no GPU slot and no `n/a` in the hero row (served-bytes pin + screenshot)
|
||||||
|
- [ ] the GPU slot stays uncoloured (no `ok|warn|fail` token in the new markup — the phase-06 rule)
|
||||||
|
- [ ] `tests/test_api.py` + `tests/test_system_contract.py` + `tests/card_contract.py` green
|
||||||
|
- [ ] full test suite green
|
||||||
|
- [ ] no behavior change in completed work: the hero order on GPU hosts, the `gpu.busy_pct@N` history, the GPU chart slot create/destroy and the "More detail" disclosure are untouched
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 03 — The `/certs` agents table scrolls horizontally
|
||||||
|
|
||||||
|
**Phase:** `21_page_cleanups` · **Source:** `TODO.md:9` — "In the certs page I can't horizontal scroll in the agents table to see the certs installed on the hosts."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The `#agents-table` on `/certs` becomes horizontally reachable at phone width: its right-hand columns (the serial, expiry, `san` list and the Rotate / Revoke / bundle controls) are reachable by scrolling instead of being clipped. The repo's own mechanism is the `.scroll` modifier — `dash.css` gives `table.data` a scroll container **only** with that class (the same rule gap 12 measured on `/`'s fleet table), so this is the class, not a second mechanism. Row contents, the `COLUMNS` contract and every served pin are unchanged.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/pages/certs.html` — line 81: `<table class="data" id="agents-table">` gains the `scroll` modifier (`class="data scroll"`), or is wrapped in `<div class="scroll">` — check which form `dash.css`'s selector actually matches (grep `\.scroll` in the file first; use the one it styles, do not invent a third).
|
||||||
|
2. `src/dashboards/web/static/dash.css` — if the `.scroll` selector matches `table.data.scroll`, nothing to add; if it matches a `div.scroll > table.data` wrapper, verify the wrapper's max-height/overflow behaviour against the card's padding so the scroll container does not clip the row's own bottom border (measure in a browser, do not deduce). No new token, no new palette.
|
||||||
|
3. `tests/test_api.py` — the served-HTML pin: `#agents-table` (or its wrapper, whichever shipped) carries the `.scroll` mechanism in the served bytes; the `COLUMNS`-resolution test (the one that reads the served `COLUMNS` table against a live `GET /api/certs` row in both directions) stays green untouched.
|
||||||
|
4. `src/dashboards/web/static/page-certs.js` — lines 55–56 (`table`/`body` lookups): no change expected — the id and the `tbody` selector are untouched by the wrapper/class; verify the page still renders its rows (a wrapper that breaks `getElementById('agents-table')` would be caught by the existing render path).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: `tests/test_api.py` — the pin above; the unchanged `COLUMNS` contract test
|
||||||
|
- Coverage: **>90%** on this task's new/modified code (one class / one wrapper; the gate applies to the changed test file)
|
||||||
|
- Browser verification (invariant 10): at 360 px on `/certs`, the table's scroll container is reachable — a CDP check that the container's `scrollWidth > clientWidth` and that scrolling to the end exposes the last column (the controls column), plus a screenshot mid-scroll in `data/logs/`; at 1280 px the table renders as before (no scrollbar on a wide viewport with three short rows)
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] at 360 px the `#agents-table` scrolls horizontally and its last column is reachable (screenshot in `data/logs/`)
|
||||||
|
- [ ] at 1280 px the table is visually unchanged
|
||||||
|
- [ ] `tests/test_api.py` green including the new pin and the unchanged `COLUMNS` test
|
||||||
|
- [ ] full test suite green
|
||||||
|
- [ ] no behavior change in completed work: the row contents, the status chips, the Rotate / Revoke / bundle controls and the `restart_required` outcome line behave exactly as before
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Phase 22 — /disks hides hosts without smartctl
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L13 — "VMs that don't have smartctl enabled disks should not show up. Gateway shouldn't even be a host on the disks page."
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** `src/dashboards/api.py` (`disks_router()` at line 791 — `/api/disks/snapshot` answering one entry per configured host; the `/api/summary` disks block at lines 640–704 — `disks_hosts` built from `job.state.payload or _disks_stub(...)` at line 640, `fleet_disk_summary(disks_hosts)` at line 693; `_disks_stub()` at line 771 — `smartctl_available: None` for a never-answered host), `src/dashboards/sources/disks.py` (`smartctl_available()` at line 142 — the three-way flag: `True` a disk answered, `False` the agent said the binary is not installed, `None` an agent predating the pass or `--no-smart`), `src/dashboards/web/static/page-disks.js` (the per-host cards + the `fleet_disk_summary` header, hidden below two hosts), `tests/test_api.py` (the `/api/disks/*` pins — "one entry per configured host", the `smartctl_available` three-way, "with and without targets configured", the `fleet_disk_summary` key paths)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
A host whose agent answered `smartctl_available: false` (the binary is not installed — gateway's VM, no smartmontools behind `vda`) is not a host on the `/disks` page at all: it is absent from `/api/disks/snapshot`, from the fleet block and from `/api/summary`'s disks rows, and the fleet header counts only the hosts that are shown. The **three-way** flag itself is unchanged in `sources/disks.py` (the agent still reports it, the `/disks` page's `null`-case why-line still exists for `None` hosts, and an **unreachable** host keeps its dimmed stub row — a dead agent is a different fact from a missing binary, invariant 3). The filter is one API-level decision, so the source, the store and the health records see nothing new.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `21_page_cleanups` (todo) — both phases touch the `/disks` page markup (task 21.01 removes the page's `#pause` button); landing 21 first keeps this phase's diff confined to the host list.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_hide_smartctl_false_hosts.md` — the one API-level visibility gate (explicit `False` only) over `/api/disks/snapshot`, `/api/summary`'s disks block and the fleet summary; the pins rewritten
|
||||||
|
2. `02_verification.md` — browser + API verification against the live fleet: gateway absent, deskwork/driveripper intact, fleet header consistent, screenshots
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: `tests/test_api.py` — the rewritten `/api/disks/snapshot` pin (one entry per configured host **whose agent can read its drives** — a `smartctl_available: false` payload is absent, a `None` payload is present, an `unreachable` stub is present), the `/api/summary` disks-block pin (fleet counts over the visible hosts, per-host rows for the visible hosts only), the unchanged `smartctl_available` three-way pin in `sources/disks.py`'s own tests (`tests/test_disks.py` or wherever the flag is pinned — grep `smartctl_available` in `tests/` first)
|
||||||
|
- Coverage: **>90%** on new/modified code
|
||||||
|
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**. AGENTS.md's `sources/disks.py` bullet (the three-way flag's `False` arm — "so the page can say *install smartmontools*") and the `/disks` UI paragraph (the `.banner.info` why-line for `false`) are rewritten: the `false` arm's page-side consequence is now *absence*, and the three-way flag keeps its agent-side meaning; the "gateway answers `smartctl_available: false`" fleet fact gains the "and is therefore not a host on the disks page" clause.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `/api/disks/snapshot` and `/api/summary`'s disks block carry no entry for a host whose agent answered `smartctl_available: false`; a `None` host and an `unreachable` stub both remain (pinned both ways in `tests/test_api.py`)
|
||||||
|
- [ ] the `/disks` page on the live fleet shows deskwork + driveripper only, the fleet header counts 2 hosts, and the per-host cards are unchanged (screenshot in `data/logs/`)
|
||||||
|
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
|
- [ ] no behavior change in completed work: the agent's SMART pass, the `disks:<host>` job's health record, the stored `disk.*` series, the `None`-case why-line and the unreachable stub are all untouched
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 01 — The one visibility gate: `smartctl_available` is `False` only
|
||||||
|
|
||||||
|
**Phase:** `22_disks_hide_unconfigured` · **Source:** `TODO.md:13` — "VMs that don't have smartctl enabled disks should not show up. Gateway shouldn't even be a host on the disks page."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
One API-level helper decides disk-page visibility: a host entry is shown when its payload's `smartctl_available` is **not explicitly `False`** — `True` (a disk answered) and `None` (nobody knows: an agent predating the pass, `--no-smart`, or a never-answered stub) stay, `False` (the agent said the binary is not installed) does not. The gate runs over the assembled per-host list in both endpoints, so `fleet_disk_summary()` (which sums whatever it is handed) sees only the visible hosts and its `hosts_total`/`hosts_live` count exactly the page's rows. `sources/disks.py` keeps the three-way flag verbatim — the agent still reports it and the `None`-case why-line still renders.
|
||||||
|
|
||||||
|
- ASSUMPTION: **explicit `False` only** is the hide condition. `None` stays visible (an agent that cannot say is not an agent that said *no* — the three-way flag's whole point), and an `unreachable` stub stays visible (invariant 3's rule: a dead agent must not read as "never configured"; its `smartctl_available` is `None` by `_disks_stub`'s shape anyway). The operator's words — "don't have smartctl **enabled**" / "shouldn't **even be a host**" — are the `false` arm; if a `None` host should also vanish, that is a one-clause change to the helper, not a redesign.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/api.py` — add one small helper next to `_disks_stub` (line 771): `_disk_visible(host: Mapping) -> bool` — `host.get("smartctl_available") is not False` (the explicit-`False` check, spelled out so a `None` payload passes; a docstring naming the three arms).
|
||||||
|
2. `src/dashboards/api.py` — `/api/disks/snapshot` (in `disks_router()`, line 791): filter the assembled per-host list through `_disk_visible` before the response is built (the response shape is unchanged — a shorter list, same entry shape).
|
||||||
|
3. `src/dashboards/api.py` — the `/api/summary` disks block (lines 640–704): `disks_hosts` (line 640) is filtered through the same helper **before** `fleet_disk_summary(disks_hosts)` (line 693) and before the per-host row loop (lines 695–704), so the fleet counts and the rows agree with `/api/disks/snapshot` by construction — one helper, two call sites, no second copy of the predicate.
|
||||||
|
4. `tests/test_api.py` — rewrite the affected pins: the `/api/disks/snapshot` "one entry per configured host" test gains a `smartctl_available: false` host (a `disks:<host>` job whose payload the test harness holds, the `self_test`-style swap the `nut` job's pin uses) and asserts it is **absent** while a `None` host and an `unreachable` stub are **present**; the `/api/summary` disks-block pin asserts the fleet counts over the visible hosts only (e.g. three configured hosts, one `false` → `hosts_total: 2`); the "without targets configured" leg stays green untouched.
|
||||||
|
5. `tests/` — locate the `smartctl_available` three-way pin (grep `smartctl_available` in `tests/`; `tests/test_disks.py` or the agent/disks source tests) and confirm it is **untouched** by this change (the flag's computation is unchanged; only the API's reading of it gained a gate).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: `tests/test_api.py` — the pins above, both directions (a `false` host absent from **both** endpoints; a `None`/`unreachable` host present in both); the unchanged `fleet_disk_summary` unit tests in `tests/` (the function is untouched — it sums what it is handed)
|
||||||
|
- Coverage: **>90%** on this task's new/modified code
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `/api/disks/snapshot` and `/api/summary` agree on the visible host set, and neither lists a `smartctl_available: false` host (pinned)
|
||||||
|
- [ ] a `None` host and an `unreachable` stub both render in both endpoints (pinned)
|
||||||
|
- [ ] `fleet_disk_summary` counts exactly the visible hosts (pinned)
|
||||||
|
- [ ] `uv run pytest` green, `ruff` + `pyright` clean
|
||||||
|
- [ ] full test suite green
|
||||||
|
- [ ] no behavior change in completed work: `sources/disks.py`, the `disks:<host>` job, the stored `disk.*` series and the health records are byte-identical in behaviour
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# Task 02 — Live verification: gateway gone, the rest intact
|
||||||
|
|
||||||
|
**Phase:** `22_disks_hide_unconfigured` · **Source:** `TODO.md:13` — "VMs that don't have smartctl enabled disks should not show up. Gateway shouldn't even be a host on the disks page."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The gate is proven on the live fleet, read-only: the dev server (invariant 10: `./scripts/dev-server.sh restart`, sign in with `testpass`) answers `/api/health` with all nine sources `ok`, the `/disks` page shows deskwork and driveripper only, and every number that used to render still renders. No fleet change is made — this task dials the fleet and screenshots it.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `./scripts/dev-server.sh restart` — the dev server up on http://127.0.0.1:8090; `./scripts/dev-server.sh status` — the nine-source table: `host:driveripper`, `host:deskwork`, `host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `nut`, `llm:gufo`, `llm:vllm` all `ok` (gateway's **agent** still answers — the job is `ok`; only the *page* hides it, which is the point: the `disks:gateway` health record must not be touched by the API filter, and `status` is the proof).
|
||||||
|
2. Signed-in `GET /api/disks/snapshot`: no `gateway` entry, `deskwork` and `driveripper` present with their `smartctl_available: true` payloads; `GET /api/summary`: the disks block's `hosts_total` is 2, the per-host rows are the two, and the `system` block still lists all three hosts (the filter is `/disks`-scoped only — a `system` host list that lost gateway would be the over-reach the pin must catch).
|
||||||
|
3. Browser on `/disks` (invariant 10): the fleet header reads the two-host sum; gateway has no card and no why-line; deskwork's `nvme0n1` and driveripper's 16 drives render exactly as before the phase (the live SMART numbers from the fleet record); a screenshot in `data/logs/`.
|
||||||
|
4. The `None`-arm check: no live host is currently `None` (all three agents post-date the SMART pass), so the `None` arm is proven by the task-01 pins only — state that in the notes rather than fabricating a live case.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Coverage: n/a (verification task; the gate's coverage is task 01's)
|
||||||
|
- The read-only rule: no `POST` to the fleet, no `upsd` command, no unit change — a live test that writes is not a test (AGENTS.md testing policy)
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `./scripts/dev-server.sh status` — nine sources `ok`, including `disks:gateway` (the job is healthy; only the page hides it)
|
||||||
|
- [ ] `/api/disks/snapshot` has no gateway entry; `/api/summary`'s system block still lists gateway (screenshot or transcript in `data/logs/`)
|
||||||
|
- [ ] the `/disks` page shows the two live hosts with their pre-phase numbers (screenshot in `data/logs/`)
|
||||||
|
- [ ] the full suite + `bash .agents/validate.sh` green
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# Phase 23 — LLM rate responsiveness
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L8 — "The LLM monitor has a strange delay and a 'ramp up/ramp down' effect that makes it look innacurate. Data reporting should be accurate to as small a delta as possible."
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** `src/dashboards/app.py` line 122 (`build_hub` builds **one** `RateEstimator(span=settings.rate_window)` and hands it to both `host_jobs` (line 124) and `llm_jobs` (line 130)), `src/dashboards/config.py` lines 197–199 (`llm_interval` 2 s, `llm_timeout` 3 s, `rate_window` 10 s), `src/dashboards/collectors.py` line 477 (`llm_jobs(settings, client, rates)` — the estimator is a parameter, so a second one is a wiring change, not a signature redesign), `src/dashboards/rates.py` (line 55 `MIN_WINDOW_SECONDS = 0.5`, line 61 `MAX_WINDOW_RATIO = 2.0`, the phase-13 jump gate — all untouched by this phase), `src/dashboards/web/static/page-llm.js` (line 39 `pollMs: 2000`, the `Δ<window>` labels printed from the payload's `window`), `src/dashboards/sources/llm.py` (the rate cells carry `computed:Δ<series>/Δt` + the honest `Δ<window>`; the window the estimator *used* travels with the value — rule 2), the mechanism the TODO describes (gap 9 / phase 13 in AGENTS.md: a 10 s window over a 2 s poll ramped the hero up over ~5 polls when a slot started and decayed over ~5 polls when it ended — the "ramp up/ramp down" the operator sees; the sub-8× skew family and the stall are **already** closed by phases 13/14 and stay closed: this phase changes the window size, nothing else)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The LLM throughput rates are measured over a window as small as the app's own cadence allows — the LLM source gets its **own** `RateEstimator` whose span is the LLM poll interval (2 s), so a slot start reads steady within two polls instead of ramping over ten, while the `host` source keeps the 10 s `RATE_WINDOW_SECONDS` window its 10 s cadence needs (a 2 s window over a 10 s poll would refuse every window — the estimator's floor and ceiling would answer `n/a` forever). The phase-13 protections (the 0.5 s floor, the `2 × span` ceiling, the 8× jump gate with its independent-anchor confirmation) apply to the smaller window unchanged; the `Δ<window>` label prints the smaller window it actually used, so the number always says what it is.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- — (none) — the phase touches `app.py`/`collectors.py` wiring and the LLM page; it is scheduled after `22_disks_hide_unconfigured` only to keep the fleet-changing phase 24 last.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_llm_window_per_poll.md` — the second estimator: `build_hub` hands `llm_jobs` a `RateEstimator(span=settings.llm_interval)`, the host keeps the 10 s one
|
||||||
|
2. `02_window_pins.md` — the pins: LLM rate cells label the 2 s window, host cells keep 10 s, the phase-13 gates still hold over the smaller window
|
||||||
|
3. `03_live_verification.md` — the live proof: a slot start reads steady within two polls (no 10 s ramp), the `Δ2.0s` label travels with the value, screenshots
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: `tests/test_llm.py` (or wherever the `parse_gufo_metrics`/`parse_vllm_metrics` tests live — grep `rate_window`/`RateEstimator` in `tests/` first) — the window-value pins for both sources, the phase-13 tests (floor refusal, ceiling, jump gate, the stale-anchor tests) run **unchanged** and green; `tests/test_api.py`'s `/api/llm/snapshot` shape pins stay green (the shape is unchanged — only the `window` values move)
|
||||||
|
- Coverage: **>90%** on new/modified code
|
||||||
|
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**. AGENTS.md's `app.py`/`poller.py` bullets (the "one `RateEstimator`" sentence) and `rates.py`'s docstring context gain the two-estimator fact; gap 9's "the estimator no longer turns a slow fetch into 0.2 s arithmetic" history stays — this phase is the window-size decision on top of it, and the `RATE_WINDOW_SECONDS=10` line in `.env.example` stays (it is the host's knob now, and gains a comment saying so if it has none).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `/api/llm/snapshot` rate cells carry a `window` ≈ 2 s (the label prints `Δ2.0s` on the page) and `/api/system/snapshot` rate cells keep ≈ 10 s (pinned both ways)
|
||||||
|
- [ ] the phase-13 tests (floor, ceiling, jump gate, stale anchor) pass unchanged
|
||||||
|
- [ ] live: a slot start reads steady within two polls and a slot end decays within two — the 10 s ramp is gone (screenshots + the stored 1 m rollup in `data/logs/`)
|
||||||
|
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
|
- [ ] no behavior change in completed work: the host rates, the estimator's gates, the rollups and the `Δ<window>` labelling are all unchanged in kind
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Task 01 — The LLM source gets its own 2 s-window estimator
|
||||||
|
|
||||||
|
**Phase:** `23_llm_rate_responsiveness` · **Source:** `TODO.md:8` — "The LLM monitor has a strange delay and a 'ramp up/ramp down' effect that makes it look innacurate. Data reporting should be accurate to as small a delta as possible."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`build_hub` builds **two** estimators: the host's keeps `RateEstimator(span=settings.rate_window)` (10 s — its 10 s cadence needs the wide window; a 2 s span over a 10 s poll would sit the window-empty fallback against the `2 × span` ceiling and answer `n/a` every poll), and the LLM's is `RateEstimator(span=settings.llm_interval)` (2 s — one poll interval: the smallest window the 2 s cadence can honestly fill). `llm_jobs` (which already takes the estimator as a parameter, `collectors.py` line 477) receives the LLM one; `HostCollector` keeps the 10 s one. No new env var: the LLM window **tracks** `LLM_INTERVAL_SECONDS`, so the one existing knob stays the single source of truth for the LLM cadence-and-window pair.
|
||||||
|
|
||||||
|
- ASSUMPTION: **no new env var** — `RATE_WINDOW_SECONDS` (10, `.env.example` line 85) becomes the host's knob and the LLM window is `llm_interval` by construction. A 2 s window is 4× above the 0.5 s `MIN_WINDOW_SECONDS` floor, so the floor never engages on a healthy cadence; the 8× jump gate stays engaged (a real burst from idle still holds one poll for confirmation), and the poll cadence itself is **unchanged** — gufo and vLLM see exactly the same request rate as before (the TODO asks for a smaller *delta*, not more load on the engines).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/app.py` — `build_hub` (lines 122–130): `rates` becomes `host_rates = RateEstimator(span=settings.rate_window)`; new `llm_rates = RateEstimator(span=settings.llm_interval)`; `host_jobs(..., HostCollector(host_rates), ...)` unchanged in shape; `llm_jobs(settings, client, llm_rates)` takes the new one. The two estimators are keyed on disjoint series names (rule 2's key is the resolved name + label set), so one shared object was never load-bearing — but the docstring/comment at line 122 (if it says "one estimator for all sources") is rewritten to the two-estimator shape.
|
||||||
|
2. `src/dashboards/collectors.py` — line 477: no signature change expected (the parameter is already there); verify the LLM jobs thread `rates` through to `parse_gufo_metrics`/`parse_vllm_metrics` (they already take it, `sources/llm.py` lines 189/243) — if any LLM path constructs its own estimator, it takes the injected one (one estimator per source, the phase's invariant).
|
||||||
|
3. `src/dashboards/config.py` — no change expected (the knobs exist); if a comment on `rate_window` (line 199) implies it is the one global knob, it gains the "host's knob; the LLM window is `llm_interval`" clause.
|
||||||
|
4. `tests/` — the `build_hub`-level test (grep `build_hub` in `tests/` — the `tests/test_api.py` wiring pins) gains one assertion pair: the estimator the LLM jobs receive has `span == settings.llm_interval` and the host collector's has `span == settings.rate_window` (spy the two objects, the way the `agent_tls_map` pin spies its entry).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the wiring pin above; the existing LLM parse tests (fixture-driven, `tests/fixtures/`) run with the injected estimator — their assertions on the `window` value, if any hard-code 10.0, are updated to the new 2.0 (grep `10.0`/`window` in `tests/test_llm.py` first); the host parse tests keep 10.0
|
||||||
|
- Coverage: **>90%** on this task's new/modified code
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `build_hub` hands the LLM jobs a 2 s-span estimator and the host a 10 s-span one (pinned by the spy)
|
||||||
|
- [ ] the fixture-driven LLM tests pass with the 2 s window; the host tests keep 10 s
|
||||||
|
- [ ] `uv run pytest` green, `ruff` + `pyright` clean
|
||||||
|
- [ ] full test suite green
|
||||||
|
- [ ] no behavior change in completed work: the host's measured rates, the estimator's gates and the engines' poll cadence are untouched
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 02 — The pins: 2 s LLM window, 10 s host window, gates intact
|
||||||
|
|
||||||
|
**Phase:** `23_llm_rate_responsiveness` · **Source:** `TODO.md:8` — "…Data reporting should be accurate to as small a delta as possible."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The window decision is pinned on the surface that shows it, in both directions (the repo's testing-policy rule): a LLM rate cell answers its `window` ≈ the LLM interval with the `Δ<window>` label traveling with the value, a host rate cell keeps ≈ 10 s, and the phase-13 gate tests — the 0.5 s floor refusal, the `2 × span` ceiling, the 8× jump hold with its independent-anchor confirmation, and the stale-scrape-body regression — run **unchanged** and green, because they are estimator properties, not window-size properties.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `tests/test_llm.py` (or the file the LLM parse tests live in — grep first) — the two window pins: (a) a gufo metrics scrape driven through `parse_gufo_metrics` with a `RateEstimator(span=2.0)` (the `build_hub` shape, task 01's) answers the rate cell with a `window` in `(0.5, 2.0]` on a healthy 2 s cadence and the cell's `source` string still reads `computed:Δ<series>/Δt`; (b) the same test shape with `span=10.0` on the host's series (or the existing host fixture tests) keeps its `window` ≈ 10 s — the two sources' windows are **different numbers in the same payload shape**, and a pin that asserts only one of them is the gap this task closes.
|
||||||
|
2. `tests/test_rates.py` (or wherever the phase-13 tests live) — run them unchanged: the floor test (a sub-0.5 s window answers `(None, dt)`), the ceiling test (`2 × span`), the jump-gate test (≥8× holds one poll, publishes on a *different-anchor* confirmation) and the two stale-body regression tests. If a test constructs its estimator with an explicit `span` that now disagrees with the app's shape, it is **not** edited to match — the estimator's constants are app-independent and the tests' own spans are theirs (the app's shape is task 01's pin's job).
|
||||||
|
3. `tests/test_api.py` — the `/api/llm/snapshot` shape pins stay green (the shape is unchanged; only `window` values move). If a pin hard-codes a `window` number in the LLM payload, it is the one place it is updated — to a range assertion (`0.5 < window <= 2.0 × llm_interval`), not a literal, so a future cadence change does not rot it.
|
||||||
|
4. `tests/test_config.py` — confirm `rate_window`'s pin (the 10 s default) stays; no new setting was added, so no new pin is needed (the "no new env var" ASSUMPTION is what the absence of a new setting is).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the four pin groups above
|
||||||
|
- Coverage: **>90%** on this task's new/modified code
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] a LLM rate cell's `window` is pinned to the 2 s shape and its label prints `Δ2.0s` (or the honest sub-window when the floor engages)
|
||||||
|
- [ ] a host rate cell's `window` is pinned to the 10 s shape
|
||||||
|
- [ ] the phase-13 gate tests pass unchanged
|
||||||
|
- [ ] `uv run pytest` green, `ruff` + `pyright` clean
|
||||||
|
- [ ] full test suite green
|
||||||
|
- [ ] no behavior change in completed work: the payload shapes, the `source` strings and the estimator constants are untouched
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 03 — Live verification: the ramp is gone
|
||||||
|
|
||||||
|
**Phase:** `23_llm_rate_responsiveness` · **Source:** `TODO.md:8` — "The LLM monitor has a strange delay and a 'ramp up/ramp down' effect that makes it look innacurate."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The smaller window is proven on the live engines, read-only: with the dev server up (invariant 10) and a decode workload running on the engines, the `/llm` hero reaches steady state within **two** polls of the slot starting (pre-phase it ramped over ~five) and decays within two of it ending, every value carrying its `Δ2.0s` label, and the stored 1 m rollups staying in the fleet's measured band (gap 9's pass line: anything above ~300 tok/s is arithmetic, not work). No engine is written to — the workload is whatever the operator's engines are doing, or a deliberately short local decode the operator starts; a live test that loads a shared engine is not a test.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `./scripts/dev-server.sh restart`, sign in with `testpass`; `./scripts/dev-server.sh status` — the nine-source table `ok` (a restart also proves the two-estimator wiring survives a real boot, not just the in-memory test store).
|
||||||
|
2. Signed-in polling of `/api/llm/snapshot` at 2 s (a short script, the `scripts/watch-store-perf.py` pattern): capture the decode tok/s cell's `value` + `window` for ~60 s while a slot is busy, then again for ~30 s after it frees. The pass shape: within two polls of the transition the value is within 8× of steady (the jump gate's own ratio — a held value is the designed behaviour, and a *confirmation* within the next poll is the honest publish; a value still ramping at poll 5 is the pre-phase defect returning).
|
||||||
|
3. The `Δ` label check (gap 9's rule, restated): **zero** values printed without their `Δ<window>` label; the label reads the window actually used (`Δ2.0s` on a healthy cadence, the honest sub-window — never a bare number — when the floor or the stall engages).
|
||||||
|
4. The stored-rollup check (the phase-14 rule): the 1 m rollups of `llm.decode_tokens_per_second` for the window under test top out in the fleet's measured band (gap 9: 119.9 gufo / 171.7 vLLM were the real maxima; the 300 tok/s pass line stands) — a 1 m average set by a ramp would be the arithmetic this phase exists to remove.
|
||||||
|
5. Screenshots of `/llm` mid-burst and post-free in `data/logs/` (invariant 10), plus the captured series as a TSV beside them (the `data/logs/llm-decode-*.tsv` precedent).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Coverage: n/a (verification task; the pins are task 02's)
|
||||||
|
- The read-only rule: no `POST` to the fleet, no `upsd` command, no engine write — observing a workload the engines are already doing is the test
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] a slot start reads steady (within 8× of steady) within two polls, and a slot end decays within two — the 10 s ramp is gone (captured series in `data/logs/`)
|
||||||
|
- [ ] zero values printed without their `Δ<window>` label, the label reading the window actually used
|
||||||
|
- [ ] the 1 m rollups stay in the measured band (nothing above the 300 tok/s pass line)
|
||||||
|
- [ ] screenshots in `data/logs/`
|
||||||
|
- [ ] the full suite + `bash .agents/validate.sh` green
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# Phase 24 — Dev mTLS: no non-mTLS agent, ever
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "The client should never allow non-mtls communication. deskwork and gateway are still non-mtls according to the dashboard. During dev, the dev server should generate certs in a non-committed directory and then distribute them to clients during registration so we never need non-mtls servers."
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** phase 15 (complete) — the durable CA in `meta` (`certs.py`: five `certs.*` rows, the bundle stored verbatim), the agent's TLS mode (`agent/host_agent.py` 1.1.0: `--cert-bundle`, `CERT_OPTIONAL`, the live `# REVOKED` denylist, `agent.tls`/`agent.bundle_rev` in the snapshot), the collector's certificate dial (`collectors.py` `AgentTls` — an `https://` `HOST_TARGETS` entry dials with this app's client certificate), the bundle download route (`GET /api/certs/<name>/bundle`, session-gated, the stored bytes verbatim), the installer's `--cert-bundle` shape check (`deploy/install-agent.sh` lines 270–331 — the python3 validator runs **before** `need_root`, the bundle installs `0640 root:dashboards-agent` at line 385, the `--status` https probe), and the fleet record (AGENTS.md gap 11: no bundle is installed on any host, `HOST_TARGETS` is `http://` for every host, the installed agents run 1.0.0 so **1.1.0 ships together with the bundle, never the bundle alone**; gap 10: driveripper's unit is reproducible only with the repo script's `--as-root --caps CAP_SYS_RAWIO CAP_SYS_ADMIN --force` line — the Oct-18 sudo-granted snapshot must not be run; deskwork's `ops` has no installer grant, so its install goes through the operator's root). `scripts/dev-server.sh` (the dev server: reads `.env.dev`, nohup `dashboards`, `/api/health` readiness probe, `status` subcommand) and `data/` (gitignored — the non-committed directory the TODO names)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The dev loop stops producing `http://` agents. On start, the dev server exports every configured host's stored bundle to non-committed `data/certs/` (the CA is already durable in `meta` — "generate" means *materialise what the first boot minted*, never a second authority), and the installer gains `--fetch-bundle` so a client's **registration** — its install — pulls its bundle from the dashboard and installs it in the same run (the operator no longer curls a file across the network by hand). Then the fleet itself moves: agent 1.1.0 + bundle on deskwork, driveripper and gateway, `HOST_TARGETS` flipped to `https://` host by host (a URL flipped before its bundle is installed turns that host red by construction — the order is the safety), leaving the dashboard's nine sources `ok` with three `agent.tls: true` hosts and no `http://` agent URL anywhere.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- — (none) — the phase builds on complete phase 15 and phase 17; it is scheduled last on purpose, because it is the fleet-changing one.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_dev_bundle_export.md` — `scripts/dev-certs-export.py` + `dev-server.sh`: the dev server writes every configured host's bundle to `data/certs/<name>.bundle` (`0600`) on start; an unissued host says so and writes nothing
|
||||||
|
2. `02_installer_fetch_bundle.md` — `install-agent.sh --fetch-bundle <url> --cert-name <name> --session <token>`: registration pulls the bundle from the dashboard, the shape check runs on the fetched bytes before `need_root`, the token never reaches the token file or the unit
|
||||||
|
3. `03_fleet_rollout.md` — agent 1.1.0 + bundle on the three hosts (deskwork via the operator's root, driveripper with the gap-10 `--force` line, gateway with its sudoers grant), `HOST_TARGETS` flipped per host, the URL never flipped before the bundle
|
||||||
|
4. `04_verification.md` — the fleet is mTLS: `agent.tls`/`deployed_bundle_rev` agree for all three hosts, nine sources `ok`, no `http://` agent URL, the 16-driver SMART reads still live, screenshots
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: `tests/test_deploy_scripts.py` — the export helper against a `create_app(...)` hosted on a socket (the sandbox pattern the installer tests already use: fake `install`/`chown` where the installer runs); the `--fetch-bundle` legs (a good bundle fetched, installed and byte-identical; a malformed one refused **before any write**; the token absent from every written artefact)
|
||||||
|
- Coverage: **>90%** on new/modified code
|
||||||
|
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**. The README's *Protecting an agent, host by host* rollout becomes the `--fetch-bundle` recipe (the manual curl is demoted to the fallback), the dev section gains the `data/certs` export, AGENTS.md's fleet record and gap 11 resolve to "all three hosts mTLS since <date>" with the exact installed units, and the "bearer-token transport is the compatibility path, not the designed one" sentence becomes history. **No token, no PEM and no session value go into any tracked file** — the exported bundles live in gitignored `data/`, the fleet's `HOST_TARGETS` URLs carry no userinfo once flipped (invariant 6), and the task transcript names hosts, not credentials.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `./scripts/dev-server.sh restart` leaves `data/certs/<name>.bundle` for every configured host that has an issued bundle, `0600`, byte-identical to the stored row, and a one-line note for any unissued host (pinned + verified)
|
||||||
|
- [ ] `sudo deploy/install-agent.sh --fetch-bundle <url> --cert-name <host> --session <token> ...` installs a byte-identical bundle in the sandbox, refuses a malformed one before `need_root`, and leaves no token in the token file, the unit or the log (pinned)
|
||||||
|
- [ ] all three hosts run agent 1.1.0 from an installed bundle, `HOST_TARGETS` is `https://` for every host with no userinfo, and `./scripts/dev-server.sh status` reads nine sources `ok` with `agent.tls: true` and `deployed_bundle_rev == bundle_rev` on all three (screenshots of `/certs` in `data/logs/`)
|
||||||
|
- [ ] driveripper's 16 drives still read live on `/disks` after the `--force` unit rewrite (the gap-10 unit shape is reproduced exactly)
|
||||||
|
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
|
- [ ] no behavior change in completed work: the token-mode path, the revocation wiring, the bundle route and the agent's token-mode bytes are all unchanged
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 01 — The dev server exports the bundles to `data/certs/`
|
||||||
|
|
||||||
|
**Phase:** `24_dev_mtls_rollout` · **Source:** `TODO.md:3` — "During dev, the dev server should generate certs in a non-committed directory and then distribute them to clients during registration…"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
On start, the dev server materialises the **stored** CA's per-host bundles in non-committed `data/certs/`: one `0600` file per configured host with an issued bundle (`data/certs/<name>.bundle`, byte-identical to the `meta` row the bundle route serves — the CA is durable in `meta`, so "generate" is an export of what the first boot already minted, never a second authority; `data/` is gitignored, so nothing committed ever carries a key). A configured host with **no** issued bundle (the invariant-3 shape: `issued: false`) writes no file and prints one line naming the host and the `/certs` route that issues it. The export is a small standalone script the dev server calls, so the test suite drives it against a `create_app(...)` hosted on a socket without a nohup'd process.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `scripts/dev-certs-export.py` — a standalone helper (stdlib `urllib`, the way `install-agent.sh`'s validator uses `python3`): arguments `--url` (the dev server's base URL), `--password` (the sign-in password — the dev server reads it from the same `.env.dev` it already loads, so the value comes from the environment, never a tracked file), `--names` (the configured host names, space-separated — read from `HOST_TARGETS`' names, the same parsing `parse_targets` does, passed in by the caller rather than re-parsed inside), `--out-dir` (defaults to `<root>/data/certs`). Behaviour: sign in via `POST /api/auth/login` (the public route; the token travels in `Authorization` for the bundle GETs), then per name `GET /api/certs/<name>/bundle` — a `{"configured": true, "issued": false}` answer writes **no file** and prints the one-line note; a configured host that 404s (`{"configured": false}`) is impossible by construction (names come from `HOST_TARGETS`) but is handled as the same no-file line, not a crash; a good bundle is written `<out-dir>/<name>.bundle` at `0600` (created via `os.open`, not `open()`, so the mode is the file's birth mode). The script exits 0 when every name resolved to either a file or the no-issued note, and non-zero on a sign-in failure — a dev server that cannot sign in says so in `dev-server.sh`'s start output, never silently.
|
||||||
|
2. `scripts/dev-server.sh` — in `start()`, after the readiness probe: call the export (`.venv/bin/python scripts/dev-certs-export.py --url http://127.0.0.1:$PORT --password "$DASH_ADMIN_PASSWORD" --names <the HOST_TARGETS names> --out-dir $ROOT/data/certs`), printing its per-host lines; in `status`, print the `data/certs/` listing (names + sizes) as the "bundles exported" block. The export failure is a **warning line**, not a `start` failure — a bundle export that cannot sign in is a dev-convenience failure, and the server itself is up (the readiness probe already proved it); the one line says what to check.
|
||||||
|
3. `tests/test_deploy_scripts.py` — the pin: host `create_app(settings with admin_password="testpass", pbkdf2_iterations=1000, one or two configured targets, an in-memory store with an issued bundle for one host and none for the other)` on `127.0.0.1:0` via `uvicorn` (the project's own dependency — `__main__.py`'s pattern), run the script against it, and assert: the issued host's file exists at `0600` and is **byte-identical** to `GET /api/certs/<name>/bundle` from the same app; the unissued host wrote **no file** and the output carries its one-line note; a wrong `--password` exits non-zero and writes nothing.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the pin above (the sandbox pattern `tests/test_deploy_scripts.py` already uses for the installer)
|
||||||
|
- Coverage: **>90%** on this task's new/modified code
|
||||||
|
- Real-boot check: `./scripts/dev-server.sh restart` on the dev box prints the per-host export lines and `data/certs/` holds the files (the dev box is the test bed — invariant 10)
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `./scripts/dev-server.sh restart` leaves `data/certs/<name>.bundle` per issued host, `0600`, byte-identical to the stored row; an unissued host is a one-line note and no file (pinned + verified on the dev box)
|
||||||
|
- [ ] `status` prints the exported-bundles block
|
||||||
|
- [ ] no bundle, key or PEM is ever written outside `data/` (the gitignored tree) — a grep of the tracked files for the export's output path confirms it lands only under `data/`
|
||||||
|
- [ ] `uv run pytest` green, `ruff` + `pyright` clean
|
||||||
|
- [ ] full test suite green
|
||||||
|
- [ ] no behavior change in completed work: the dev server's start/stop/status semantics, the readiness probe and the CA rows in `meta` are untouched (an export never mints — `certs.read_state` is the request-time read the bundle route uses, and the export dials that route, not `ensure_certs`)
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# Task 02 — `install-agent.sh --fetch-bundle`: registration pulls the bundle
|
||||||
|
|
||||||
|
**Phase:** `24_dev_mtls_rollout` · **Source:** `TODO.md:3` — "…then distribute them to clients during registration so we never need non-mtls servers."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The installer's registration moment — the install — is where the client gets its bundle: `--fetch-bundle <dashboard-url> --cert-name <name> --session <token>` fetches `GET /api/certs/<name>/bundle` from the dashboard **in the install run**, the existing python3 shape check (lines 270–331, which runs before `need_root`) validates the **fetched bytes**, and the rest of the `--cert-bundle` path (the `0640 root:dashboards-agent` install at line 385, the unit's `--cert-bundle` + `RuntimeDirectory` lines, the `--status` https probe) runs on them unchanged. The session token is a **fetch credential only**: it never reaches the agent's token file, the unit, the log or the installed bundle, and the fetch happens before `need_root` (a host whose dashboard refuses the fetch never gets a partial install — the same order the bundle validator already has).
|
||||||
|
|
||||||
|
- ASSUMPTION: the session travels on **argv** (`--session <token>`), matching how the operator already signs in for the manual curl the README's rollout documents; the token is visible in the `sudo` process list for the fetch's duration, which is the accepted cost on a single-admin box (the alternative — a `0600` session file — is a second artefact to clean and the operator's muscle memory is the curl recipe). The three flags are **all-or-nothing**: any one without the others is refused with a usage line before `need_root` and before any write.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `deploy/install-agent.sh` — argument parsing (near line 136): `--fetch-bundle) FETCH_URL="$2"; shift 2 ;;`, `--cert-name) CERT_NAME="$2"; shift 2 ;;`, `--session) SESSION_TOKEN="$2"; shift 2 ;;`; the all-or-nothing check (the three set together or not at all; `--fetch-bundle` and a local `--cert-bundle <file>` together is refused — two bundle sources are a contradiction, pick one) in the same pre-`need_root` region the bundle validator occupies (lines 160–331).
|
||||||
|
2. `deploy/install-agent.sh` — the fetch: a `python3` `urllib` block (the script already requires `python3` for the shape check — no new dependency, no `curl` assumption): `GET $FETCH_URL/api/certs/$CERT_NAME/bundle` with `Authorization: Bearer $SESSION_TOKEN` written to a `mktemp` file; the response body's shape is checked in the same block — a `{"configured": true, "issued": false}` body is refused with its own one-line reason (*"no bundle issued for <name> — issue it from /certs first"*), a `404`/`401`/`403` is refused naming the HTTP status, and only a 200 with a non-empty body falls through to the **existing** shape check (lines 270–331) run on the temp file; on any refusal the temp file is deleted (`trap`-covered, the way the script already cleans its unit temp file) and nothing is written. On success the temp file is handed to the existing `BUNDLE_SRC` path — the `0640` install, the unit lines and the `--status` probe all run unchanged on the fetched bytes.
|
||||||
|
3. `deploy/install-agent.sh` — token hygiene: `SESSION_TOKEN` is used only in the fetch block; it is never `echo`d (the script's existing log lines name the URL, never the header), never written to `$TOKEN_FILE`/`$CERT_FILE`/the unit, and is `unset` after the fetch; the `--help` block gains the three flags with the one-line recipe (the URL is the dashboard's, the name is the `HOST_TARGETS` name, the session is the signed-in token).
|
||||||
|
4. `tests/test_deploy_scripts.py` — the pins (the sandbox pattern: fake `install`/`chown`, a local `http.server` thread serving the bundle route's exact bytes — the `application/x-pem-file` attachment body a `create_app` bundle route would serve): (a) a good fetch → the installed `CERT_FILE` is **byte-identical** to the served bytes, `0640`, and the unit carries the `--cert-bundle` line; (b) a served `{"configured": true, "issued": false}` body → refusal with the one-line reason, **no** `install` call (the fake records it), temp file gone; (c) a served 401 → refusal naming the status, no write; (d) the **token absence** pin: the token string appears in no written artefact — not the token file, not the unit, not the script's captured stdout (the test serves the bundle with a token it chose and greps the sandbox for it); (e) the all-or-nothing and the two-bundle-source refusals, both before `need_root` (the sandbox's root check records the call order).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the five pin legs above
|
||||||
|
- Coverage: **>90%** on this task's new/modified code (the gate applies to the changed test file)
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `sudo deploy/install-agent.sh --fetch-bundle <url> --cert-name <name> --session <token> --bind … --port …` installs a byte-identical bundle in the sandbox, `0640`, with the unit's TLS lines (pinned)
|
||||||
|
- [ ] a malformed/missing/401 bundle is refused before `need_root` with its own one-line reason and no partial write (pinned)
|
||||||
|
- [ ] the session token appears in no written artefact — token file, unit, or captured stdout (pinned)
|
||||||
|
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||||
|
- [ ] full test suite green
|
||||||
|
- [ ] no behavior change in completed work: the local `--cert-bundle <file>` path, the `--status` probe, the re-run guard (task-17's diff-and-refuse) and the token file's mode are byte-identical in behaviour to before
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 03 — The fleet goes mTLS, host by host
|
||||||
|
|
||||||
|
**Phase:** `24_dev_mtls_rollout` · **Source:** `TODO.md:3` — "The client should never allow non-mtls communication. deskwork and gateway are still non-mtls according to the dashboard."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
All three agents run 1.1.0 from an installed bundle, and `HOST_TARGETS` is `https://` for every host. The order per host is the safety (gap 11's rule: **a URL flipped before its bundle is installed turns that host red by construction**): install agent 1.1.0 + bundle → restart the unit → the `--status` CA-only https probe answers → **then** the `HOST_TARGETS` line flips → the dev server restarts → `agent.tls` shows. 1.1.0 ships **together with** the bundle, never the bundle alone (the installed agents are 1.0.0 — gap 11's re-check). Hosts in order: gateway (the trivial one: `(ALL) NOPASSWD: ALL`, a VM with the standard unit), deskwork (no `ops` installer grant — its root half is the operator's action), driveripper (the gap-10 line, the one that must not be the Oct-18 snapshot).
|
||||||
|
|
||||||
|
- ASSUMPTION: the executor has **ssh as `ops`** from the dev box (the fleet record's probes ran from this laptop; the repo's dev workflow dials the fleet from here). If ssh is unavailable in the run environment, the task degrades to its written form: the exact command blocks are printed for the operator, the operator applies them, and task 04's verification runs against the applied fleet — the pass criteria are the same either way.
|
||||||
|
- ASSUMPTION: the flipped URL is **clean `https://<host>:9120` with no userinfo** — the TLS handshake is the auth, the token file line stays in the unit (the rollback path), and invariant 6's "no token in a tracked file" is satisfied by `.env.dev` itself being gitignored with the token now out of the URL. **Rollback** per host = restore the pre-flip URL line from the timestamped backup copy (made in `data/env-backup/` before any flip) and restart the dev server — the bundle stays installed, the unit keeps both its token file and the bundle line.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. **Prep (dev box):** `./scripts/dev-server.sh restart` (task 01's export leaves `data/certs/` populated); sign in, note the session token (it lives only in the operator's shell and the task transcript's redaction, never in a file); `cp .env.dev data/env-backup/.env.dev-<timestamp>` (gitignored).
|
||||||
|
2. **gateway** (`10.1.1.27:9120`, the standard unit): over ssh as `ops` — copy the repo's `agent/host_agent.py` (1.1.0) to `/usr/local/lib/dashboards-agent/host_agent.py`, then `sudo deploy/install-agent.sh --bind 10.1.1.27 --port 9120 --fetch-bundle http://127.0.0.1:8090 --cert-name gateway --session <token>` (the fetch dials the **dev server on the dev box** — the dashboard's bundle route; on a host that cannot reach the dev box, the same flag with the dashboard's LAN URL, or task 01's exported file, is the fallback; the unit gains the `--cert-bundle` + `RuntimeDirectory` lines and keeps its `--token-file` line), `systemctl restart dashboards-agent`, `./deploy/install-agent.sh --status` (the CA-only https probe answers). **Then** flip gateway's `.env.dev` line to `https://10.1.1.27:9120`, `./scripts/dev-server.sh restart`, confirm `host:gateway`/`disks:gateway` `ok` with `agent.tls: true`.
|
||||||
|
3. **deskwork** (`10.1.0.51:9120`, the operator's root half): `ops` has no installer grant on deskwork (the fleet record), so the executor prints the exact block — the agent-file copy, `sudo deploy/install-agent.sh --bind 10.1.0.51 --port 9120 --fetch-bundle <dashboard url> --cert-name deskwork --session <token>`, the restart, the `--status` probe — for the operator to run as root; the executor then verifies the unit state over ssh (`systemctl is-active`, `systemctl cat`), flips the URL line and confirms `host:deskwork`/`disks:deskwork` `ok` with `agent.tls: true` (if the operator's run and the flip land before the executor's verify, the verify is the check; the command block is the record).
|
||||||
|
4. **driveripper** (`10.1.0.126:9120`, the gap-10 line): over ssh as `ops` — copy the repo's `agent/host_agent.py` (1.1.0) to `/usr/local/lib/dashboards-agent/host_agent.py`, then the **repo's** script with the gap-10 line exactly: `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force --fetch-bundle <dashboard url> --cert-name driveripper --session <token>` (the `--force` is what lands the unit change — task 17's guard refuses a differing re-run without it, and the line reproduces the `User=root` + `CapabilityBoundingSet` unit that keeps all 16 SMART reads live; **never** the Oct-18 sudo-granted snapshot — it predates the flags and the guard), `systemctl restart dashboards-agent`, the `--status` probe, **then** flip the URL line and confirm `host:driveripper`/`disks:driveripper` `ok` with `agent.tls: true` and the 16 drives still reading (a `smart: null` row here is the `--force` rewrite having changed the unit — stop and diff the unit against the fleet record's shape before anything else).
|
||||||
|
5. **Fleet state:** after all three flips, `.env.dev`'s `HOST_TARGETS` has no `http://` agent entry and no userinfo token; the three units still carry their `--token-file` lines (the rollback path); the transcript in `data/logs/` names hosts, ports and revs — no token, no PEM.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Coverage: n/a (fleet task; the pins are tasks 01–02's)
|
||||||
|
- The read-only rule applies to the **verification** half: no `upsd` command, no engine write, no `POST /api/ups/command`; the installer's own writes are the change this task exists to make
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] all three units run the 1.1.0 agent file from an installed bundle (`systemctl cat` + `--status` CA-only probe per host, transcript in `data/logs/`)
|
||||||
|
- [ ] `.env.dev`'s `HOST_TARGETS` is `https://` for all three hosts, no userinfo; the pre-flip backup exists in `data/env-backup/`
|
||||||
|
- [ ] `./scripts/dev-server.sh status` — nine sources `ok` with `agent.tls: true` on all three (the per-host flips never left the fleet red longer than the host's own restart — the per-host order is the proof)
|
||||||
|
- [ ] driveripper's 16 drives still read live on `/disks` after the `--force` rewrite
|
||||||
|
- [ ] the full suite + `bash .agents/validate.sh` green
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# Task 04 — The fleet is mTLS: the end state, verified
|
||||||
|
|
||||||
|
**Phase:** `24_dev_mtls_rollout` · **Source:** `TODO.md:3` — "…so we never need non-mtls servers."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The end state is proven read-only, the way phase 15's task-05 final pass proved its composition: the dashboard dials all three agents with its client certificate, the issued-vs-deployed revisions agree, nothing non-mTLS is left in the config, and the things the bundle install must not have broken (driveripper's 16 SMART reads, deskwork's GPU row) still work. No live revocation is performed — the revocation wiring is phase 15's e2e proof (`data/logs/verify-15-tls-e2e.py`), and a live revoke of a working fleet is a change, not a check.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `./scripts/dev-server.sh status` — the nine-source table `ok` (`host:driveripper`, `host:deskwork`, `host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `nut`, `llm:gufo`, `llm:vllm`); `disks:gateway` is `ok` with the page hiding gateway (phase 22's gate) — the job healthy, the page scoped.
|
||||||
|
2. Signed-in `GET /api/certs`: all three rows `status: "active"`, `deployed_bundle_rev == bundle_rev` (the "install the new bundle" sentence is gone for every host), `agent` block from the `system` payload: `agent.tls: true` + `agent.bundle_rev` present on all three (the task-03 passthrough — a token-mode snapshot grows nothing, so the fields' **presence** is the mTLS proof, rule 2's shape).
|
||||||
|
3. Signed-in `GET /api/system/snapshot` + `GET /api/disks/snapshot`: the `agent` blocks on all three hosts; gateway's row absent from `/disks` (phase 22); driveripper's 16 drives with live SMART numbers (the `--force` unit rewrite is the gap-10 shape — a `smart: null` row here fails the task); deskwork's `nvme0n1` row and its GPU card on `/system` (the 1.1.0 agent file's reads intact).
|
||||||
|
4. The config: `grep` of `.env.dev` — no `http://` in `HOST_TARGETS`, no `@` userinfo in any target URL; `data/env-backup/` holds the pre-flip copy (the rollback path exists on disk).
|
||||||
|
5. The negative control, **without touching the fleet**: `data/logs/verify-15-tls-e2e.py`'s transcript (phase 15) is the standing proof that a revoked serial is refused on `/health` too; this task re-runs **none** of it live — it names the transcript as the control and checks only that `tests/test_collectors.py`'s revocation-before-dial pin and `tests/test_api.py`'s `build_hub` wiring pin are green (they run in the suite).
|
||||||
|
6. Screenshots (invariant 10) in `data/logs/`: `/certs` (the three active rows with matching revs), `/disks` (the two visible hosts, driveripper's 16 live rows), `/system` (deskwork's GPU card) — the "the dashboard says all three are mTLS" proof is a screenshot, not a curl.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Coverage: n/a (verification task)
|
||||||
|
- The read-only rule: no `POST` to the fleet, no `upsd` command, no revocation, no unit change — the task reads the fleet and screenshots it
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] nine sources `ok`; `deployed_bundle_rev == bundle_rev` and `agent.tls: true` for all three hosts (pinned by the API checks, shown by the screenshots)
|
||||||
|
- [ ] `.env.dev` has no `http://` agent URL and no userinfo; the pre-flip backup exists
|
||||||
|
- [ ] driveripper's 16 SMART reads and deskwork's GPU card are live in the served payloads
|
||||||
|
- [ ] screenshots of `/certs`, `/disks`, `/system` in `data/logs/`
|
||||||
|
- [ ] the revocation pins (`tests/test_collectors.py`, `tests/test_api.py`) green in the suite
|
||||||
|
- [ ] the full suite + `bash .agents/validate.sh` green; AGENTS.md's fleet record and gap 11 updated by the phase's Documentation & Consistency pass to the mTLS end state
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# Phase 25 — Masthead control placement: badges everywhere, right-justified, clock rightmost
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3–L5 — "hosts disks engines UPS disappear when in the certs/account page" · "I made a mistake earlier, I did want hosts disks engines UPS badges and certs/account/dark right justified don the navbar" · "The time should show up as the farthest right item on the navbar"
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** phase 20 (complete) — the masthead is three rows at every width (row 1 brand + `#nav-toggle` + the five tabs, row 2 the `#nav-status` per-source dots, row 3 the `.masthead-right` controls), not sticky (`position: relative`, `z-index: 40`), with the ≤640px dropdown sliding on `nav.tabs.is-open`; phase 18 — the hamburger reveal through `--nav-toggle-display`. The dots are built by `Dash.initChrome()` in `src/dashboards/web/static/app.js` (the `if (nav)` block on `#nav-status`, one chip per `system`/`disks`/`llm`/`ups` dashboard from `/api/health`, refreshed every 15 s), and exist in the markup only of the five dashboard pages — `certs.html` and `settings.html` ship no `#nav-status` (the comment in `settings.html` says a session page "has no fleet reachability to report"), which is what L3 names. The served-CSS pins for the bar are in `tests/test_api.py`: `test_the_masthead_is_three_left_justified_rows_at_every_width` (the two `justify-content: flex-start` declarations, the two `flex-basis: 100%` row breaks, the `min-height`/`gap`), `test_the_mobile_masthead_block_declares_no_second_bar_height`, `test_the_masthead_scrolls_away_and_stays_the_dropdown_anchor`; the one-loop control-order audit is `test_every_page_links_to_both_admin_pages` (clock → `⚷ Certs` → `⚙ Account` → theme toggle on all eight signed-in pages), and the two pins that assert the admin pages have **no** dots are in `test_the_settings_page_is_not_one_of_the_dashboards` and `test_the_certs_page_is_a_page_and_not_a_sixth_dashboard`. `tests/test_auth_api.py` asserts `/login` carries no `#nav-status` — unchanged by this phase.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The four per-source badges (hosts · disks · engines · UPS) render on **every** page that renders the masthead — including `/certs` and `/settings` — and the two right-hand rows (the dots, the controls) are right-justified, with the clock the farthest-right item on the bar. The three-row shape at every width is preserved; only the alignment and the control order move, and the clock's position is what L5 asks for.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- — (none) — the phase builds on complete phase 20 and phase 18; it is scheduled first of the three appended phases on purpose, because the mobile audit phase (27) measures the masthead in the shape this phase leaves it in.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_status_dots_on_admin_pages.md` — `#nav-status` on `/certs` and `/settings` (markup only — `initChrome` already builds the dots when the container exists); the two no-dots pins flip to dots-present
|
||||||
|
2. `02_right_justify_masthead_rows.md` — `justify-content: flex-end` on `#nav-status` and `.masthead-right`, the comment block re-worded, the three-rows pin re-pinned to the new alignment
|
||||||
|
3. `03_clock_rightmost.md` — the clock moves to the end of the control row on all eight signed-in pages; the one-loop order pin updated; the phase's real-browser verification (badges on the admin pages, both rows right-justified, clock rightmost, at 1280 px and 360 px)
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the served-byte pins in `tests/test_api.py` — the flipped admin-page pins (task 01), the re-pinned row-alignment pin (task 02), the updated control-order pin (task 03); `tests/test_auth_api.py`'s `/login`-has-no-dots pin must stay green untouched
|
||||||
|
- Coverage: **>90%** on new/modified code (the new code is the pins themselves; the change is CSS and template markup)
|
||||||
|
- Real-browser check (invariant 10): a `data/logs/25-task03-*.cjs` script in the `20-task01-rows.cjs` convention, run against the restarted dev server at 1280 px and 360 px on all seven tabbed pages plus `/login` (unsigned), with screenshots in `data/logs/`
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
|
||||||
|
- The final pass updates the now-stale sentences in `AGENTS.md`: "The masthead is three **left-justified** rows at every width" (alignment), "A page with no dots (`/certs`, `/settings`) renders two rows" (gone — all masthead pages carry the dots), "the two links therefore sit on all eight signed-in pages in one order (clock → `⚷ Certs` → `⚙ Account` → theme toggle)" (new order: `⚷ Certs` → `⚙ Account` → theme toggle → clock), and the "left-justified" wording in the `dash.css` header comment. `README.md` is checked for the same sentences; none is expected.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `/certs` and `/settings` render the four badges, populated from `/api/health` (pinned + verified in a browser against the live dev fleet)
|
||||||
|
- [ ] the dots row and the controls row are right-justified at every width; the three-row shape and the ≤640px dropdown are unchanged (pinned + measured in a browser)
|
||||||
|
- [ ] the clock is the farthest-right item on the bar on all eight signed-in pages (pinned + measured)
|
||||||
|
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
|
- [ ] no behavior change in completed work: the row-break mechanism (`flex-wrap` + the two `flex-basis: 100%`), the non-sticky bar, the dropdown's slide and its four close paths, and the dots' refresh semantics are all untouched
|
||||||
+27
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 01 — The badges render on the admin pages too
|
||||||
|
|
||||||
|
**Phase:** `25_masthead_control_placement` · **Source:** `TODO.md:3–4` — "hosts disks engines UPS disappear when in the certs/account page" · "I made a mistake earlier, I did want hosts disks engines UPS badges and certs/account/dark right justified don the navbar"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`#nav-status` exists in the `.masthead-right` of `/certs` and `/settings`, exactly as it does in the five dashboard pages, so `Dash.initChrome()` builds the four badges (`hosts`/`disks`/`engines`/`UPS`) there without any change to `app.js` — the "disappear when in the certs/account page" symptom is the missing container, and the fix is the container.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/pages/certs.html` — add `<div id="nav-status" class="gap" aria-live="polite"></div>` as the **first** child of `<div class="masthead-right">` (before the `<span class="clock">`), the same position the five dashboard pages use (`index.html:23` is the reference).
|
||||||
|
2. `src/dashboards/web/pages/settings.html` — the same addition, and delete the `<!-- No `#nav-status`: a session page has no fleet reachability to report, and `Dash.initChrome()` builds the dots only when the container exists. -->` comment — the premise it states is what the operator overruled (the badges report the fleet, and the fleet is the point of the site, on every page).
|
||||||
|
3. `src/dashboards/web/static/app.js` — **no change** (stated so the executor does not "improve" it): `initChrome()`'s `if (nav)` block already builds the four chips and refreshes them from `/api/health` every 15 s whenever `#nav-status` exists; `setDot`'s `idle` state already covers a dashboard the page's config does not run.
|
||||||
|
4. `tests/test_api.py` — flip the two pins, one assertion each: in `test_the_settings_page_is_not_one_of_the_dashboards`, `assert 'id="nav-status"' not in html` becomes `in html` (message re-worded: the session page grew the badges the operator asked for); in `test_the_certs_page_is_a_page_and_not_a_sixth_dashboard`, the same flip, and the docstring sentence "no `#nav-status` (a CA page has no fleet reachability to report)" is re-worded to the new fact. `test_the_certs_page_is_a_page_and_not_a_sixth_dashboard`'s other assertions (zero `aria-current`, the tab strip, the head) are untouched, and `tests/test_auth_api.py`'s assertion that **`/login`** has no `#nav-status` is untouched — `/login` still renders the brand link and the theme toggle only.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the two flipped pins above; `test_the_header_status_dot_picks_up_the_disks_job` and `test_the_disks_tab_is_in_the_masthead_of_every_page` stay green unchanged
|
||||||
|
- Coverage: **>90%** on this task's new/modified code (markup + pins)
|
||||||
|
- The badges populating on the two admin pages is the phase's real-browser check (task 03), run against the live dev fleet so the dots are `ok` green, not grey `idle`
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GET /certs` and `GET /settings` (signed in) both contain `<div id="nav-status" class="gap" aria-live="polite">` as the first child of `.masthead-right` (pinned)
|
||||||
|
- [ ] in a browser, both admin pages show the four badges and they answer the same `ok`/`warn`/`fail` state the dashboard pages show (measured, screenshots in `data/logs/`)
|
||||||
|
- [ ] `/login` (unsigned) still carries no `#nav-status` (`tests/test_auth_api.py` green)
|
||||||
|
- [ ] `uv run pytest` green, `ruff` + `pyright` clean
|
||||||
|
- [ ] no behavior change in completed work: `initChrome` is byte-identical, and the badge refresh cadence and `idle` semantics are untouched
|
||||||
+29
@@ -0,0 +1,29 @@
|
|||||||
|
# Task 02 — The dots row and the controls row are right-justified
|
||||||
|
|
||||||
|
**Phase:** `25_masthead_control_placement` · **Source:** `TODO.md:4` — "I made a mistake earlier, I did want hosts disks engines UPS badges and certs/account/dark right justified don the navbar"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Phase 20 task 01 made the two right-hand rows of the masthead flush left; this task moves them flush **right**. The mechanism is two declarations — `justify-content: flex-end` on `#nav-status` and on `.masthead-right` — and nothing else: the row breaks (the two `flex-basis: 100%` items inside the `flex-wrap: wrap` bar), the `min-height`, the gaps and the ≤640px block all stay exactly as they are, so the bar is still three rows at every width and the phone's wrapped control row still wraps *within itself*.
|
||||||
|
|
||||||
|
- ASSUMPTION: the three-row shape is preserved at every width and only the alignment changes. A return to the old single-row bar is not the ask — the row breaks are what make the bar fit at 360 px (phase 18/20 measured that), and the operator's word is "right justified in the navbar", not "one row".
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/static/dash.css` — two declarations: `#nav-status`'s `justify-content: flex-start` becomes `flex-end`, and `.masthead-right`'s `justify-content: flex-start` becomes `flex-end`. `margin-inline-start: auto` stays **absent** from `.masthead-right` (the old single-row right-pinning is gone; `flex-end` on a full-basis row is the alignment now).
|
||||||
|
2. `src/dashboards/web/static/dash.css` — re-word the comments that state the old alignment, so a future reader is not sent to a left-justified bar that no longer exists: the header comment block (the "three left-justified rows at every width" sentence, and its row-2/row-3 descriptions), the `Row 3:` comment above `.masthead-right` (which currently cites the old `TODO.md` word for `flex-start` — the new sentence states the operator overruled it: the badges and the cluster are right-justified), the `Row 2:` comment above `#nav-status`, and the ≤640px block's comment "Row 3 alignment is the base rule's `flex-start`, restated nowhere" (now `flex-end`). The `#nav-toggle { margin-inline-start: auto }` line in the mobile block is **unchanged**: the hamburger sits at the right of row 1 (brand + toggle + tabs), which is where a phone's thumb already is — it was never part of the right-justified cluster.
|
||||||
|
3. `tests/test_api.py` — rename `test_the_masthead_is_three_left_justified_rows_at_every_width` to `test_the_masthead_is_three_rows_right_justified_at_every_width` and change its two alignment assertions from `== "flex-start"` to `== "flex-end"` (the dots line and the cluster line), re-wording the assertion messages and the docstring ("two of them flush left" → "two of them flush right"). Every other assertion in the test stays: `flex-wrap: wrap`, the `flex-direction: row` check, the no-`height`/`min-height: var(--header-h)` pair, the gap, the two `flex-basis: 100%` row breaks, and the "no `margin-inline-start` on the cluster" guard — the row-break mechanism is exactly what must survive the re-justify.
|
||||||
|
4. `tests/test_api.py` — `test_the_mobile_masthead_block_declares_no_second_bar_height` and `test_the_masthead_scrolls_away_and_stays_the_dropdown_anchor` stay green **unchanged** (neither reads `justify-content`); confirm they do not need a word change in their docstrings before moving on.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the re-pinned alignment pin above; the two untouched mobile/sticky pins as the guard that the row mechanism survived
|
||||||
|
- Coverage: **>90%** on this task's new/modified code (CSS + pin)
|
||||||
|
- The measured half (the rows actually render right-justified) is the phase's real-browser check (task 03)
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the served stylesheet reads `justify-content: flex-end` on both `#nav-status` and `.masthead-right`, with `margin-inline-start` still absent from the cluster (pinned)
|
||||||
|
- [ ] the ≤640px block still declares `#nav-status { flex-wrap: nowrap; }` and the wrapping cluster, and still declares no second bar height (existing pins green)
|
||||||
|
- [ ] in a browser at 1280 px and 360 px, the dots row and the controls row are flush with the bar's right padding edge and the bar is still three rows (measured, screenshots in `data/logs/`)
|
||||||
|
- [ ] `uv run pytest` green, `ruff` + `pyright` clean
|
||||||
|
- [ ] no behavior change in completed work: the dropdown slide, the hamburger reveal and the dots' content are untouched
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 03 — The clock is the farthest-right item, and the phase's browser proof
|
||||||
|
|
||||||
|
**Phase:** `25_masthead_control_placement` · **Source:** `TODO.md:5` — "The time should show up as the farthest right item on the navbar"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
On a right-justified control row, the farthest-right item is the **last** one in markup order, so the clock moves from the first control (before `⚷ Certs`) to the last control (after the theme toggle). The new order on all eight signed-in pages is `⚷ Certs` → `⚙ Account` → `☾ Dark` → clock. This is the phase's verification task as well: the layout change is CSS + markup, so the served-byte pins cannot say the rows actually render right-justified — this task measures them in a real browser (invariant 10).
|
||||||
|
|
||||||
|
- ASSUMPTION: "farthest right" is achieved by markup order (the clock is the last child of `.masthead-right`), not by a CSS `order` declaration — the one-loop order pin compares markup, and a `order:` property the pin cannot see is a drift waiting to happen.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. The seven page templates that render `.masthead-right` — `src/dashboards/web/pages/index.html`, `system.html`, `disks.html`, `llm.html`, `ups.html`, `settings.html`, `certs.html` — move the line `<span class="clock" id="clock">--:--:--</span>` from its current position (first control, before the Certs link) to **after** the `<button … id="theme-toggle">` element, so the control row reads `⚷ Certs` → `⚙ Account` → `☾ Dark` → clock. The `<div id="nav-status">` stays the first child (it breaks onto row 2 before the controls). All seven templates must end byte-identical in the control-row region apart from the page-specific comment — `test_every_page_links_to_both_admin_pages` is a one-loop audit precisely because the eight markups are supposed to be the same.
|
||||||
|
2. `src/dashboards/web/static/dash.css` — re-word the header comment's row-3 line ("row 3 the `.masthead-right` controls (clock, `⚷ Certs`, `⚙ Account`, theme)") to the new order. No declaration changes.
|
||||||
|
3. `tests/test_api.py` — in `test_every_page_links_to_both_admin_pages`: update the docstring's order sentence (clock → Certs → Account → theme toggle ⇒ `⚷ Certs` → `⚙ Account` → theme toggle → clock); keep the two existing `before_theme` assertions (Certs and Account still sit before the theme toggle, Certs before Account — both true in the new order); add the new one: `html.index('id="clock"') > html.index('id="theme-toggle")` on every page (the clock is the last control — on a `flex-end` row that is the farthest-right item). The `/login` half of the test (no links, no toggle) is untouched.
|
||||||
|
4. `data/logs/25-task03-verify.cjs` — the browser proof, in the `20-task01-rows.cjs` convention (`NODE_PATH=<playwright node_modules> node data/logs/25-task03-verify.cjs`, `DASH_BASE`/`DASH_PW` env, `check()` accumulator, non-zero exit on any failure): sign in, then on all seven tabbed pages at **1280 px** and **360 px** — (a) group the bar's children into rows by bounding box and assert the three-row shape; (b) the `#nav-status` row and the controls row are flush with the bar's right padding edge (their right edges equal the bar's content right edge within 1 px, and they are **not** flush left — the old layout's right edge was far from the bar's); (c) within the controls row, the clock's right edge is the rightmost of the four controls and the order reads Certs, Account, theme, clock; (d) `#nav-status` on `/certs` and `/settings` is populated (four chips, none `idle`-grey against the live dev fleet — the task 01 check). `/login` (unsigned) at both widths: brand + theme toggle only, no dots, no clock, no horizontal overflow. Screenshots `data/logs/masthead-25-<page>-<width>.png` (the seven tabbed pages at both widths; the script names them, the pass line is every `check()` green).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the extended `test_every_page_links_to_both_admin_pages` (the order pin), with `test_the_masthead_is_three_rows_right_justified_at_every_width` (task 02) and the two task-01 flipped pins as the surrounding contract
|
||||||
|
- Coverage: **>90%** on this task's new/modified code (markup + pin + script)
|
||||||
|
- Real-browser check: the `.cjs` pass above against the restarted dev server (`./scripts/dev-server.sh restart` first, per invariant 10), the dev fleet answering so the badges are green
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] on all eight signed-in pages the control row reads `⚷ Certs` → `⚙ Account` → `☾ Dark` → clock in the markup, with the clock after the theme toggle (pinned)
|
||||||
|
- [ ] the `.cjs` pass is all-green at 1280 px and 360 px: three rows, both right-hand rows flush right, clock rightmost, badges populated on `/certs` and `/settings` (screenshots in `data/logs/`)
|
||||||
|
- [ ] `/login` (unsigned) is unchanged: no badges, no clock, no links (existing `tests/test_auth_api.py` pin green)
|
||||||
|
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
|
- [ ] `AGENTS.md`'s now-stale masthead sentences are updated (the phase overview's Documentation & Consistency list) and `README.md` checked
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# Phase 26 — UPS events: power events collapsible, connection errors always visible
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L6 — "The phase I ran earlier collapsed all the events in the UPS tab. That wasn't the point. I wanted the power loss events in a collapsible list, not events like \"NUT server NutError: cannot reach upsd at 10.1.0.126:3051 (timed out after 4s)\" in a collapsible list."
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** phase 19 (complete) put the **whole** `/ups` event feed behind one collapsed `details.events-box` (no `open` attribute, lazy: `refresh()` fetches the feed only while `state.events.open`, the first expand fetches, a fresh re-expand paints from the `<ul>`'s kept children, `painted`/`fetchedAt` stamped on the success path alone, the `Dash.eventsPager` window of 50 inside the bounded 320px scroll). The operator's complaint is that the classification is wrong, not that a collapse exists: a poll failure like `NUT server NutError: cannot reach upsd … (timed out after 4s)` is hidden behind a summary nobody opens, while the power events it was meant to tuck away are the only ones the operator actually wants collapsed. The feed's **kinds** are fixed and known: the poller (`collectors.py:490` — `Job("nut", "ups", "NUT server", …)`, events under the dashboard name `ups`) writes `unreachable` / `reachable` / `recording` / `store-error` (`poller.py:127/157/164/190` — the quoted `NutError` line is the `unreachable` message); the `NutMonitor` transition bookkeeping writes `status` (a state change lands at `collectors.py:393`) and `test` (the self-test verdict, `collectors.py:409`); `api.py` writes `action` for every command sent through `POST /api/ups/command` (the fixed-text `action` events, `api.py:999/1014`). The served-byte pins for the current shape are in `tests/test_api.py`: `test_ups_power_events_feed_sits_behind_a_collapsed_disclosure` (the `details.events-box`, no `open`, `#events` inside), `test_ups_events_are_fetched_only_while_the_disclosure_is_open` (the lazy gate — **this phase inverts it on purpose**), `test_the_ups_event_rows_keep_the_date_time_and_severity_shape` (the row shape), `test_the_ups_events_disclosure_is_a_quiet_disclosure_not_a_new_component` (the `details.events-box` CSS shape — unchanged by this phase). The page is `src/dashboards/web/static/page-ups.js` (`refresh()` around line 439, `refreshEvents()` around line 445, the disclosure's toggle handler in the `DOMContentLoaded` block) over the card in `src/dashboards/web/pages/ups.html:75-78`.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`/ups` renders **two** lists from the one unchanged feed fetch: the power events — state transitions (including the power-loss transitions), self-test verdicts and sent commands — stay behind the collapsed `details.events-box` disclosure, and the connection/error events (`unreachable`/`reachable`/`recording`/`store-error`) are a card that is **always visible** (hidden only while it has no rows to show). The split is page-side, keyed on the event's `kind`; `/api/events` gains no vocabulary.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- — (none) — the phase builds on complete phase 19; it is scheduled before `27_mobile_tables_and_audit` on purpose, because that phase's mobile audit captures `/ups` in the shape this phase leaves it in (two cards, one of them hidden-when-empty).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_page_side_event_split.md` — the `kind` split in `page-ups.js` (power kinds `status`/`test`/`action`, everything else errors) and the removal of the phase-19 lazy gate: the feed is fetched on every poll, because a visible list that ages is worse than one more 200-event fetch per interval
|
||||||
|
2. `02_render_both_lists_paged.md` — the always-visible `#events-errors` card (above the disclosure, hidden while empty) and both lists paged through the shared `Dash.eventsPager`
|
||||||
|
3. `03_repin_and_browser_verify.md` — the pin re-pins (the disclosure stays, the lazy gate inverts, the row shape extends to the error list, the kind set is pinned so a silent upstream rename breaks a test) and the phase's real-browser verification
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the re-pinned and new pins in `tests/test_api.py` (task 03); the split itself is pinned on the served script (the kind set) and the served markup (the error card, `hidden`, not inside a `<details>`)
|
||||||
|
- Coverage: **>90%** on new/modified code
|
||||||
|
- Real-browser check (invariant 10): a `data/logs/26-task03-*.cjs` script in the `20-task01-rows.cjs` convention, run against the restarted dev server: the disclosure shut by default, the error card's visibility matching the feed's error-kind rows, both lists' rows matching a direct `/api/events?source=ups&limit=200` split, screenshots in `data/logs/`
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
|
||||||
|
- The final pass updates the now-stale sentences in `AGENTS.md`: the event-feed paragraph's lazy-fetch block ("`refresh()` fetches the feed only while `state.events.open` is true, so a card nobody is looking at costs the poll loop nothing" and the first-expand/fresh-re-expand sentences) is re-written to the new fact (the one fetch serves two lists, always; the disclosure gates only the power list's visibility), and the pin description of `test_ups_events_are_fetched_only_while_the_disclosure_is_open` is re-pointed at its inverted successor. `README.md` is checked for the same sentence; none is expected.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] a poll failure (e.g. `NUT server NutError: cannot reach upsd …`) is visible on `/ups` without opening anything, and the power events stay behind the collapsed disclosure (pinned + verified in a browser)
|
||||||
|
- [ ] the error card is hidden while it has no rows and appears with its first error row; both lists share the one `/api/events?source=ups` fetch and the shared pager (pinned)
|
||||||
|
- [ ] a failed fetch leaves both lists on their last good rows (the stale-last-good rule) — pinned where phase 19 pinned its
|
||||||
|
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
|
- [ ] no behavior change in completed work: the `details.events-box` CSS shape, the pager's page/scroll semantics, and the row's time/host/severity shape are untouched
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Task 01 — The feed splits on `kind`, and the lazy gate comes out
|
||||||
|
|
||||||
|
**Phase:** `26_ups_event_split` · **Source:** `TODO.md:6` — "I wanted the power loss events in a collapsible list, not events like \"NUT server NutError: cannot reach upsd at 10.1.0.126:3051 (timed out after 4s)\" in a collapsible list."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`page-ups.js` splits the one unchanged feed fetch into two lists: the **power** events — `status` (the state transitions, including the power-loss transitions the operator names), `test` (the self-test verdicts) and `action` (the commands sent through the confirm dialog) — and **everything else** (the poller's `unreachable`/`reachable`/`recording`/`store-error`, the quoted `NutError` line included), which is the always-visible error list. The phase-19 lazy gate goes out with the split: a visible list that ages is worse than one more 200-event fetch per interval, so `refresh()` fetches on every poll.
|
||||||
|
|
||||||
|
- ASSUMPTION: "power events" is the **known** set of three kinds (`status`/`test`/`action`); any other kind is an error-row by default, because the operator's complaint is that errors must never be the hidden list, and a new kind landing in the hidden list would be the same bug re-shipped.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/static/page-ups.js` — at the top of the file, `const POWER_EVENT_KINDS = new Set(["status", "test", "action"]);` with a comment naming the three writers (`collectors.py:393` `status`, `collectors.py:409` `test`, `api.py` `action` — so a rename on the source side is visible at the site of the drift). In `refreshEvents()`: after the one `fetchJSON(\`/api/events?limit=${Dash.EVENT_FEED_LIMIT}&source=ups\`)` (the call itself is **unchanged** — no `page`/`limit`/`kind` vocabulary, the phase-19 rule), `const power = data.events.filter((e) => POWER_EVENT_KINDS.has(e.kind));` and `const errors = data.events.filter((e) => !POWER_EVENT_KINDS.has(e.kind));`.
|
||||||
|
2. `src/dashboards/web/static/page-ups.js` — in `refresh()`, replace `if (state.events.open) await refreshEvents();` with `await refreshEvents();` and re-word the comment above it (the "a shut `<details>` must cost the poll loop nothing" rationale is what the operator overruled: the error list is visible, so the feed is needed whether or not the disclosure is open). In the `DOMContentLoaded` block, delete the `eventsBox` toggle handler's fetch logic — with an always-fresh list the `toggle` event no longer needs to do anything, and `state.events` (with its `open`/`painted`/`fetchedAt` fields) is deleted entirely: the disclosure now gates only the power list's **visibility** (the native `<details>` behaviour), and a failed fetch leaves both lists on their last good rows through `refreshEvents()`'s existing `try { … } catch (err) { /* next poll */ }` — the success-path stamps go away with the state they stamped, and the stale-last-good rule is the `catch` block, which stays.
|
||||||
|
3. `src/dashboards/web/static/page-ups.js` — the split is the **only** change to the fetch path: the `row` renderer, `eventText` handling and the `runtime_seconds` footnote stay byte-identical (task 02 points them at both lists), and the `state` object loses `events` with no other field touched.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the split and the gate removal are pinned on the served script in task 03 (`tests/test_api.py`) — the kind set, the unconditional `refreshEvents()` call, the deleted `state.events`
|
||||||
|
- Coverage: **>90%** on this task's new/modified code
|
||||||
|
- The behavioural proof (a real `unreachable` row visible without opening the disclosure) is the phase's real-browser check (task 03)
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `page-ups.js` defines `POWER_EVENT_KINDS = new Set(["status", "test", "action"])` and `refresh()` calls `refreshEvents()` unconditionally (pinned on the served script)
|
||||||
|
- [ ] `state.events` and the toggle handler's fetch logic are gone; a failed fetch leaves both lists on their last good rows (pinned where phase 19 pinned its)
|
||||||
|
- [ ] the `/api/events` call is byte-identical to the phase-19 one (no new query vocabulary — pinned)
|
||||||
|
- [ ] `uv run pytest` green, `ruff` + `pyright` clean
|
||||||
|
- [ ] no behavior change in completed work: the poll cadence, the confirm dialog, the hero row and the charts are untouched
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 02 — The error card: always visible, hidden only when empty
|
||||||
|
|
||||||
|
**Phase:** `26_ups_event_split` · **Source:** `TODO.md:6` — "I wanted the power loss events in a collapsible list, not events like \"NUT server NutError: cannot reach upsd at 10.1.0.126:3051 (timed out after 4s)\" in a collapsible list."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The error list is a card of its own, **above** the power-events card, built from the existing vocabulary (`section.card`, `ul.events`, the pager's strip) with no new CSS component: visible from first paint while it has rows, `hidden` while it has none, and paged by the same `Dash.eventsPager` the power list uses (its `WeakMap` state is keyed by the `<ul>`, so the two lists page independently — the pager's existing contract).
|
||||||
|
|
||||||
|
- ASSUMPTION: the error section sits **above** the Power events disclosure — the visibility complaint is about *seeing* the errors, and the collapsible list keeps its current position in the card stack.
|
||||||
|
- ASSUMPTION: the error card is `hidden` while it has zero rows — a permanent empty "Connection & errors" card is the noise the page's empty-state rules exist to avoid, and its first `unreachable` row is what makes it appear.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/pages/ups.html` — above the existing `<section class="card" style="margin-top:.85rem">` (the disclosure card, `ups.html:74-79`), add the error card: `<section class="card" id="ups-errors-card" hidden>` with `<header><h3>Connection & errors</h3><span class="spacer"></span><span class="faint">upsd reachability and this process's own failures</span></header>` and `<div class="body"><ul class="events" id="events-errors"><li class="muted">loading…</li></ul></div>`. The `hidden` attribute is the resting state (before the first successful fetch there are no rows); `style="margin-top:.85rem"` moves to the **error** card's position only in the sense that the disclosure card keeps its existing attribute — a `hidden` card is `display: none`, so the disclosure card's own `margin-top` still gives the gap to whatever is above it when the error card is absent, and no rule is needed for the two-cards case beyond the disclosure card's existing attribute (verify in the browser that the gap is `.85rem` in both states; if the hidden card's own margin would double it, drop the error card's `margin-top` and let the stack's own gap rule do the work — the page's sections carry the shared `.85rem` gap).
|
||||||
|
2. `src/dashboards/web/static/page-ups.js` — in `refreshEvents()`, after the split (task 01): render the power list exactly as today — `Dash.eventsPager($('#events'), power, { label: 'Power events', … })` with its existing `empty` sentence and `row` renderer — and the error list — `Dash.eventsPager($('#events-errors'), errors, { label: 'Connection & errors', empty: '<li class="muted" style="grid-template-columns:1fr">No connection errors — upsd is answering and this process is recording.</li>', row: <the same `row` function as the power list, extracted to one const so the two lists cannot drift> })`; then `$('#ups-errors-card').hidden = errors.length === 0;`. The `loading…` seed row in the error card is overwritten on the first successful fetch and never shows, because the card is `hidden` until that fetch lands with rows (a first fetch with zero error rows keeps it hidden — no flash).
|
||||||
|
3. `src/dashboards/web/static/dash.css` — **no new rules**: the card, the `ul.events` bounded 320px scroll and the `.events-pager` strip are all existing vocabulary (the phase-19 rule a second opinion about the driver's word is the bug — the same applies to a second style for a second list). If step 1's gap verification needs a rule, the only allowed one is a margin on `#ups-errors-card` itself, pinned in task 03.
|
||||||
|
4. The `row` renderer's `runtime_seconds` footnote stays on both lists (an `action` row's runtime is the power list's; an error row simply has no `event.data.runtime_seconds` and prints none — no branch added).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: task 03's pins — the error card in the served markup (`hidden`, not inside a `<details>`), the kind set, the single `row` const, the `hidden = errors.length === 0` line
|
||||||
|
- Coverage: **>90%** on this task's new/modified code
|
||||||
|
- The two-cards layout, the gap in both states and the error card's first-appear are the phase's real-browser check (task 03)
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `/ups` shows the `Connection & errors` card above the Power events disclosure, paged, sharing the row shape (pinned + measured in a browser)
|
||||||
|
- [ ] the card is `hidden` while the feed has no non-power kinds and appears with its first error row, the gap to the disclosure card being `.85rem` in both states (measured)
|
||||||
|
- [ ] both lists page independently (pager state per `<ul>`) and a failed fetch leaves both on their last good rows (pinned)
|
||||||
|
- [ ] no new CSS selector is introduced for the error card (pinned)
|
||||||
|
- [ ] `uv run pytest` green, `ruff` + `pyright` clean
|
||||||
|
- [ ] no behavior change in completed work: the disclosure's CSS shape and the power list's empty sentence are untouched
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Task 03 — The pins re-pin, and the browser proves the split
|
||||||
|
|
||||||
|
**Phase:** `26_ups_event_split` · **Source:** `TODO.md:6`
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The phase's pins: the disclosure shape stays pinned (the power list is still behind the collapsed `details.events-box`), the lazy-gate pin inverts to its replacement (the fetch is no longer gated on the disclosure), the row-shape pin extends to the error list, and the kind set is pinned on the served script so a silent rename on the source side breaks a test before an operator sees a misclassified row. Then the real-browser verification (invariant 10).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `tests/test_api.py` — `test_ups_power_events_feed_sits_behind_a_collapsed_disclosure`: keep every existing assertion (one `details.events-box`, no `open` attribute, `#events` inside it) and extend: `id="events-errors"` is present in the served page and is **not** inside any `<details>` (the error list must not be collapsible — the operator's whole complaint), and the error card carries the `hidden` attribute in the template.
|
||||||
|
2. `tests/test_api.py` — `test_ups_events_are_fetched_only_while_the_disclosure_is_open` becomes its inversion, renamed `test_ups_events_are_fetched_on_every_poll` (the phase-19 lazy gate is what the operator overruled; the old test failing is the point of this phase): the served script's `refresh()` region (the same `boot.index("…")` slice the old test uses) shows `refreshEvents()` called **unconditionally**, and the served script carries no `state.events` and no `toggle`-handler fetch. The stale-last-good pin (a failed fetch leaves the rows on screen) moves onto the `refreshEvents()` `catch` block, the way phase 19 pinned it.
|
||||||
|
3. `tests/test_api.py` — `test_the_ups_event_rows_keep_the_date_time_and_severity_shape`: the existing `#events` assertions stay, and the same shape (the `<time>` element, the host chip with its `sev` class, the `eventText`-cleaned message, the optional runtime footnote) is asserted for `#events-errors` through the shared `row` const. New pin: the served script defines `POWER_EVENT_KINDS` as exactly `status`, `test`, `action` (a set literal read out of the script text — a kind added to or removed from the set is a classification the operator did not make). New pin: `$('#ups-errors-card').hidden = errors.length === 0` (or its exact equivalent line) — the hidden-when-empty behaviour is the one state this page may claim about a list it cannot see.
|
||||||
|
4. `tests/test_api.py` — `test_the_ups_events_disclosure_is_a_quiet_disclosure_not_a_new_component`: unchanged, and extended with one guard — no new CSS selector for the error card in the served stylesheet (the card is `section.card` + `ul.events` + `.events-pager` vocabulary only; a second style for a second list is the phase-19 anti-pattern).
|
||||||
|
5. `data/logs/26-task03-verify.cjs` — the browser proof, in the `20-task01-rows.cjs` convention (`DASH_BASE`/`DASH_PW`, `check()` accumulator, non-zero exit on failure), against the restarted dev server (`./scripts/dev-server.sh restart` first): sign in; fetch `/api/events?source=ups&limit=200` **directly** and split by the three power kinds; load `/ups` and assert — (a) the disclosure is shut by default and the power list renders only on expand; (b) the error card is visible if and only if the direct fetch has non-power-kind rows; (c) each list's DOM row count and first row's text match the direct-fetch split (the `eventText` cleaning applied); (d) a failed-fetch leg is not needed against live hardware — the stale-last-good rule is pinned in step 2's pin instead. Screenshots `data/logs/ups-event-split-<state>.png` (the error card visible with rows, and hidden when the feed has none — if the live feed has no error rows, the hidden state is the one measured, and the visible state is measured by the DOM from the direct-fetch split, not by staging rows).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the pins of steps 1–4
|
||||||
|
- Coverage: **>90%** on this task's new/modified code
|
||||||
|
- Real-browser check: step 5's `.cjs` pass, screenshots in `data/logs/`
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the four pin groups above are green, with `test_the_ups_events_disclosure_is_a_quiet_disclosure_not_a_new_component` passing unchanged in its original assertions
|
||||||
|
- [ ] the `.cjs` pass is all-green against the live dev server (screenshots in `data/logs/`)
|
||||||
|
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
|
- [ ] `AGENTS.md`'s now-stale lazy-fetch sentences are updated (the phase overview's Documentation & Consistency list) and `README.md` checked
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# Phase 27 — Mobile: every table scrolls, and the whole app is audited at phone width
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L7 — "None of the tables are horizontally scrollable on mobile. Fix that and make sure everything works on mobile. Do a full mobile compatible audit."
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** AGENTS.md gap 12 (measured on 2026-10-20): at 360 px, `/`'s fleet table measures 450–510 px inside a 320 px `section.card`, and `.card { overflow: hidden }` clips the tail columns (`WORST DISK`, `GPU BUSY`, `TEMP`, `UP`) with no scroll container to reach them — `table.data` only scrolls with its `.scroll` modifier (`dash.css:884`: `display: block; overflow: auto; max-height: 320px`), which **no page currently uses** (phase 21's `.scroll` on the `/certs` table is still in `todo/` and has not landed). Gap 12 names the fix as "a responsive-table pass (`.scroll` on the wide tables, or a per-host card layout at ≤640px)" and records it as a phase, not a cleanup, because it touches pinned page shape on `/`. The table inventory is fixed and small: two tables in static markup (`index.html:61` `#fleet-table`, `certs.html:85` `#agents-table`) and five in JS-built markup (`page-system.js:194` the GTt table, `:257` the filesystems table, `:269`/`:274` the two top-process tables, all inside the "More detail" disclosure; `page-ups.js:278` the raw NUT-variables table, inside its disclosure). Phase 18's standing guarantee — the document no longer scrolls sideways at a phone width (360 px where it was 750 px) — must survive this phase, so the fix is **inside** the tables, never the document. The ≤640px breakpoint is the repo's mobile boundary (the hamburger, the bar's row-breaks). The browser-proof convention is the `data/logs/*.cjs` scripts (`20-task01-rows.cjs` is the template: `NODE_PATH=<playwright node_modules> node …`, `DASH_BASE`/`DASH_PW`, a `check()` accumulator, non-zero exit on failure, screenshots in `data/logs/`). Phase 26 (scheduled before this one) is what the audit will capture on `/ups`: two cards, the error one hidden-when-empty.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Every `table.data` on every page scrolls **horizontally inside its own container** when it is wider than the card (a phone's 320 px), with no new document-level sideways scroll, and the whole app — all eight signed-in pages plus `/login` — is audited at 360 px (1280 px as the control) by a re-runnable script whose pass is the phase's proof.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `26_ups_event_split` (todo) — the audit captures `/ups` in the shape 26 leaves it in (the two cards and the hidden-when-empty error card are what the 360 px pass must not clip); running 27's audit before 26 lands would measure the pre-split single card.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_scrollable_tables_on_mobile.md` — the one ≤640px rule that makes every `table.data` horizontally scrollable (horizontal-only, no height cap), pinned on the served CSS
|
||||||
|
2. `02_mobile_audit_script.md` — `data/logs/27-task02-mobile-audit.cjs`: the re-runnable sweep of all nine pages at 360 px and 1280 px — no horizontal document scroll, no clipped table (the gap-12 failure mode), the masthead intact, screenshots
|
||||||
|
3. `03_fix_mobile_issues_found.md` — run the audit, fix every finding, re-run until every page passes both widths; the final screenshot set is the phase's proof
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the served-CSS pin (task 01); the audit script itself is the integration evidence (tasks 02–03) — the repo has no JS test runner, so a `.cjs` pass with screenshots is the established browser-proof shape (invariant 10)
|
||||||
|
- Coverage: **>90%** on new/modified code (the new code is the pin; the change is one CSS rule)
|
||||||
|
- Real-browser check (invariant 10): the task-02 script, run against the restarted dev server, is the phase's verification — not an add-on to it
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
|
||||||
|
- The final pass resolves AGENTS.md **gap 12**: the clipping paragraph becomes history ("the gap-12 clipping is closed by `27_mobile_tables_and_audit`: every `table.data` scrolls horizontally inside its own container at ≤640px, measured by `data/logs/27-task02-mobile-audit.cjs`"), the "a phase rather than a cleanup" sentence goes with it, and the mobile-nav paragraphs that quote the 360 px document width are checked for consistency with the audit's no-document-scroll assertion. `README.md` is checked for the same sentence; none is expected.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] at 360 px, every `table.data` wider than its container scrolls horizontally inside the container — the gap-12 fleet-table clipping is gone (pinned + measured by the audit)
|
||||||
|
- [ ] the audit pass is all-green on all nine pages at 360 px and 1280 px: no horizontal document scroll (phase 18's guarantee intact), masthead intact, screenshots in `data/logs/`
|
||||||
|
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
|
- [ ] no behavior change in completed work: at 1280 px the tables render exactly as before (no scrollbar where the table fit, no layout shift)
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 01 — Every table scrolls horizontally at the mobile breakpoint
|
||||||
|
|
||||||
|
**Phase:** `27_mobile_tables_and_audit` · **Source:** `TODO.md:7` — "None of the tables are horizontally scrollable on mobile. Fix that…"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
One ≤640px rule makes **every** `table.data` horizontally scrollable inside its own card when it is wider than the card, with no per-table markup change: the five JS-built tables and the two static ones all carry the class already, so a class-level rule reaches all seven at once. The mechanism is the one the existing `.scroll` modifier proves — `display: block` keeps the table's internal layout in an anonymous box, `overflow-x: auto` gives a scrollbar only where the table overflows.
|
||||||
|
|
||||||
|
- ASSUMPTION: the rule is **horizontal-only** (`overflow-x`, not `overflow: auto`) and carries **no** `max-height` — the `.scroll` modifier's 320px height cap is its vertical-use contract, and a phone getting a height-capped table on top of a horizontal one is two new failure modes bought to fix one.
|
||||||
|
- ASSUMPTION: the rule lives in the existing `@media (max-width: 640px)` block (the repo's mobile boundary, the same block the hamburger and the bar's gap live in), applied to `table.data` generally rather than adding the `.scroll` class per table — the per-table class path is what gap 21 does for one table, and seven hand-placed classes are the drift this phase exists to avoid. The all-widths `table.data.scroll` rule (`.css:884`) is untouched.
|
||||||
|
- ASSUMPTION: the `.scroll`-shape pass is taken over gap 12's alternative (a per-host card layout at ≤640px) — a re-layout of `/`'s fleet table into cards touches pinned page shape far more than an overflow rule does, and the operator's word is "horizontally scrollable", which is the `.scroll` shape.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/static/dash.css` — inside the existing `@media (max-width: 640px)` block (the "mobile nav" section, beside the `.masthead`/`#nav-status` mobile rules), add: `table.data { display: block; overflow-x: auto; }` with a comment stating the contract: horizontal-only, no height cap, every `table.data` (static or JS-built) at the phone breakpoint, the document itself never scrolling sideways (phase 18's guarantee) and the all-widths `table.data.scroll` rule untouched.
|
||||||
|
2. `src/dashboards/web/static/dash.css` — verify the rule reaches the tables inside disclosures: the "More detail" tables (`page-system.js:194/257/269/274`) and the NUT-variables table (`page-ups.js:278`) sit in `display: none` containers until expanded; `overflow-x: auto` on a `display: block` table inside a revealed `details` is the same box model (no extra rule needed — but the audit's task-02 expanded-disclosure leg is what proves it, so do not skip expanding those two disclosures in the script).
|
||||||
|
3. `tests/test_api.py` — pin the served CSS the way `test_the_mobile_masthead_block_declares_no_second_bar_height` reads its block: in the ≤640px block, `table.data` declares `display: block` **and** `overflow-x: auto`, and declares **no** `max-height` (the height cap must not migrate into the mobile rule). The all-widths `table.data.scroll` rule's declarations are asserted unchanged in the same test.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the served-CSS pin above
|
||||||
|
- Coverage: **>90%** on this task's new/modified code (one rule + pin)
|
||||||
|
- The behavioural proof (the gap-12 table actually scrolls at 360 px) is the task-02 audit
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the ≤640px block declares `table.data { display: block; overflow-x: auto }` with no `max-height`, pinned on the served stylesheet
|
||||||
|
- [ ] the `table.data.scroll` rule (`.css:884`) is byte-identical to before
|
||||||
|
- [ ] at 360 px in a browser, `/`'s fleet table scrolls horizontally inside its card (the tail columns `WORST DISK`/`GPU BUSY`/`TEMP`/`UP` reachable) and the document does not scroll sideways (measured by the task-02 audit)
|
||||||
|
- [ ] `uv run pytest` green, `ruff` + `pyright` clean
|
||||||
|
- [ ] no behavior change in completed work: at 1280 px no table shows a scrollbar and no layout shifts (measured by the audit's control width)
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 02 — The re-runnable mobile audit script
|
||||||
|
|
||||||
|
**Phase:** `27_mobile_tables_and_audit` · **Source:** `TODO.md:7` — "…make sure everything works on mobile. Do a full mobile compatible audit."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`data/logs/27-task02-mobile-audit.cjs` — the full mobile compatibility audit as a re-runnable script in the repo's established convention (`20-task01-rows.cjs` is the template: `NODE_PATH=<playwright node_modules> node data/logs/27-task02-mobile-audit.cjs`, `DASH_BASE`/`DASH_PW` env with the dev defaults, a `check(name, ok, detail)` accumulator, non-zero exit on any failure). It sweeps **all nine pages** — the eight signed-in pages (`/`, `/system`, `/disks`, `/llm`, `/ups`, `/settings`, `/certs`) plus `/login` (unsigned, the only anonymous page) — at **360 px** (the phone width gap 12 measured at) with **1280 px** as the control, and its all-green pass is the phase's proof.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `data/logs/27-task02-mobile-audit.cjs` — setup: sign in at `DASH_BASE` (the `20-task01-rows.cjs` sign-in pattern, `testpass` default), then per page × width, the check set:
|
||||||
|
- **No horizontal document scroll** (phase 18's standing guarantee — this phase must not break it): `document.documentElement.scrollWidth <= window.innerWidth + 1` and `document.body.scrollWidth <= window.innerWidth + 1`.
|
||||||
|
- **No clipped table** (the gap-12 failure mode, the one this phase exists for): for every `table.data`, if `el.scrollWidth > el.clientWidth + 1` then the computed `overflow-x` is `auto` or `scroll` (the table scrolls **inside** its own container); a wide table whose `overflow-x` is `visible` (clipped by the card's `overflow: hidden`) fails with the table's id. At 1280 px the same check runs as the control (a desktop table that overflows — none should — would fail there too).
|
||||||
|
- **Every element fits the viewport**: a generic overflow scan — no element with `getBoundingClientRect().right > window.innerWidth + 1` other than a `table.data` inner box already accounted for (a hero row, chart or card spilling off the phone is the "make sure everything works on mobile" half the table check does not name).
|
||||||
|
- **Masthead intact** (360 px leg): the three-row shape by bounding-box grouping (the `20-task01-rows.cjs` `PROBE()`), every control's box inside the viewport, and the ≤640px dropdown still opening directly under the bar (open `#nav-toggle`, measure `nav.tabs`'s top against the bar's bottom, close it).
|
||||||
|
- **Expanded-disclosure leg** (360 px): on `/system` expand one host's "More detail" `details` and on `/ups` open the NUT-variables disclosure, re-run the table check on the newly revealed tables (the `display: none` containers of task 01's step 2), then close them.
|
||||||
|
- **`/ups` two-card leg** (360 px, phase 26's shape): the error card (when present) and the disclosure card both fit, the error card's `hidden` state matching the feed as task 26 pinned.
|
||||||
|
- Screenshots: `data/logs/mobile-27-<page>-<width>.png` per page × width (nine × two = eighteen, plus the two expanded-disclosure states), each `check()` line printed and accumulated.
|
||||||
|
2. `data/logs/27-task02-mobile-audit.cjs` — the 1280 px control leg doubles as the "no behavior change" proof: the same element-fits and table checks at desktop width, so a rule that shifts a desktop table (a spurious `display: block` at 1280 px, a layout shift) fails the control as loudly as the phone leg fails the phone.
|
||||||
|
3. The script exits non-zero on any failure and prints the failure lines (the `20-task01-rows.cjs` convention); it is **run twice** as part of this task: once as written (expecting the pre-task-03 findings, if any, to surface) and once after task 03's fixes — the first run's output is the findings list task 03 works from, and both runs' output is the phase's evidence. (The script is in gitignored `data/logs/` — evidence, not a tracked pin; the tracked pin is task 01's served-CSS one.)
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the script's check set is the integration evidence; task 01's served-CSS pin is the tracked half
|
||||||
|
- Coverage: **>90%** on new/modified code (the script is evidence, not shipped code — no coverage line of its own, the pin carries the gate)
|
||||||
|
- Real-browser check: this **is** the real-browser check (invariant 10), run against the restarted dev server (`./scripts/dev-server.sh restart` first) with the live fleet answering so the tables have real rows
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the script exists, runs in the documented one-liner, signs in, sweeps nine pages × two widths, and exits non-zero on any failure
|
||||||
|
- [ ] its check set covers: document scroll, per-table clipping (the gap-12 mode), the generic element-fits scan, the masthead + dropdown at 360 px, the two expanded-disclosure legs, and the `/ups` two-card leg
|
||||||
|
- [ ] the first run's output is the findings list (task 03's input) and its screenshots are in `data/logs/`
|
||||||
|
- [ ] `uv run pytest` green, `ruff` + `pyright` clean (the script is `node --check`-clean — `bash .agents/validate.sh`'s node check covers the `web/static` scripts, and this one is checked by hand as part of the run)
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# Task 03 — Fix every finding until the audit is all-green at both widths
|
||||||
|
|
||||||
|
**Phase:** `27_mobile_tables_and_audit` · **Source:** `TODO.md:7` — "…make sure everything works on mobile. Do a full mobile compatible audit."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The audit's first run (task 02) is the findings list; this task closes it. Every finding is a CSS/markup fix — the data layer, the API and the page scripts' logic are out of scope (a table that does not fit is a layout fact, and the fix is the layout, never the data) — and the phase is done when the audit passes **every page at both widths** with the screenshots in `data/logs/`.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. Run the task-02 script against the restarted dev server and work the findings list top down. The expected findings, in the order the audit surfaces them:
|
||||||
|
- **`/` fleet table** (gap 12, the one the phase exists for): task 01's rule is the fix — verify the tail columns (`WORST DISK`, `GPU BUSY`, `TEMP`, `UP`) are reachable by scrolling the table, not the document. If the rule is already in place from task 01 and the table still clips, the card's `overflow: hidden` is clipping the table's **scrollbar track**: the fix is `table.data` keeping `display: block; overflow-x: auto` (task 01's pin) — a `card`-level change is the fallback, and it needs a pin of its own if used.
|
||||||
|
- **`/certs` agents table**: same rule, same check; note phase 21 (still `todo/`) is scheduled to add the `.scroll` class to this table — this task's rule makes the class redundant at ≤640px but harmless at all widths, so 21 landing later is a no-op collision, not a conflict (state that in the fix note if the table is already covered).
|
||||||
|
- **`/system` "More detail" tables** (filesystems, top processes, GTt): the expanded-disclosure leg of the audit; a table inside the `details` that still clips gets the same class-level fix (no per-table markup — the class is already there).
|
||||||
|
- **`/ups` raw NUT-variables table**: same, through its disclosure.
|
||||||
|
- **Anything the generic element-fits scan names that is not a table** (a hero row, a chart, a card spilling off the phone): fix the specific element — the fix is named per element in the findings note, and a fix that hides a control to make the width work is a refusal (the phase-18 rule: nothing is hidden to make the arithmetic work).
|
||||||
|
2. `tests/test_api.py` — for every finding whose fix is a **markup** change (an `id` added, a wrapper introduced, a class added to a static table), add the served-byte pin in the same pass; a CSS-only fix is covered by task 01's pin. No pin is added for a fix that only touches `data/logs/` evidence.
|
||||||
|
3. Re-run the task-02 script until **every check on all nine pages passes at 360 px and 1280 px**; the final run's output and its eighteen-plus screenshots are the phase's proof (the final-run screenshot set supersedes the first run's in `data/logs/`).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: task 01's served-CSS pin + any new markup pin from step 2
|
||||||
|
- Coverage: **>90%** on new/modified code
|
||||||
|
- Real-browser check: the final all-green audit run (invariant 10), the screenshots in `data/logs/`
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the audit is all-green on all nine pages at 360 px and 1280 px, with the final-run screenshots in `data/logs/`
|
||||||
|
- [ ] every finding from the first run is closed by a named fix (CSS/markup only) or a stated refusal (a control the operator needs, not hidden)
|
||||||
|
- [ ] the gap-12 fleet-table clipping is gone: the tail columns reachable by the table's own scrollbar, the document not scrolling sideways (measured)
|
||||||
|
- [ ] `AGENTS.md` gap 12 is resolved per the phase overview's Documentation & Consistency list, and `README.md` checked
|
||||||
|
- [ ] `uv run pytest` green, coverage not down, `ruff` + `pyright` clean
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
|
- [ ] no behavior change in completed work: the 1280 px control leg is all-green (no desktop layout shift, no spurious scrollbars)
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# Phase 28 — The agent reads the gateway's proxy configs and its live sockets
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "Create a 'gateway' page that visualizes my incoming connections based on the gateway host's nginx and haproxy configs. I want it to show animated pipes with data coming in (IPs, ports, domains, etc.) and data going out (IPs, ports, domains, etc.)." (the agent half)
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** `agent/host_agent.py` (the stdlib-only single-file agent, `Collector._safe` best-effort sections, `snapshot()`, the SCHEMA docstring, `AGENT_VERSION` 1.1.0, the `--no-smart` opt-out and the three-way availability precedent it sets), `tests/test_agent.py` (the in-process harness over a fake `root` tree, the real-`/proc` test, the smartctl-capture fixture pins), `scripts/capture-fixtures.sh` (the ssh-capture convention: plain command first, `sudo -n` fallback, refuse to overwrite good fixtures with unreadable output), `tests/fixtures/` (the verbatim-capture ground-truth directory), the gateway host's live state verified 2026-10-21 (read-only): `nginx` and `haproxy` both `active`, `/etc/haproxy/haproxy.cfg` (6 `frontend` blocks — `bind :::22 v4v6` et cetera — each with one `default_backend`; 6 `backend` blocks, each one `server <name> <domain>:<port> check inter 5s`), `/etc/nginx/nginx.conf` (a catch-all `listen 80` / `server_name _` block plus `include /etc/nginx/conf.d/*.conf;` and `include /etc/nginx/stream.d/*.conf;`), `/etc/nginx/conf.d/http.conf` (server blocks: `listen 4443 ssl proxy_protocol` / `listen 8443 ssl`, `server_name <domain>`, `location / { proxy_pass http://<domain>:<port>; }`), `/etc/nginx/stream.d/stream.conf` (a `map $ssl_preread_server_name $targetBackend { … }` block and `proxy_pass $targetBackend` with `ssl_preread on`), all three files world-readable and readable by the `dashboards-agent` user, `/proc/net/tcp` readable the same way. The dashboard side (source, wiring, page) is phase 29; this phase makes the data exist on the wire.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`host_agent.py` 1.2.0 serves three new **additive** snapshot sections on the gateway host — `haproxy` (frontends + binds, backends + server targets, the frontend→backend routing), `nginx` (server blocks with `listen` + `server_name`, `upstream`, `proxy_pass` including the `$var`-to-`map` join, `map` blocks) and `sockets` (LISTEN + ESTABLISHED from `/proc/net/tcp{,6}`) — each best-effort (a missing config degrades its own section, never the snapshot), `--no-proxy` opting out silently, the real configs captured verbatim as fixtures, the rollout onto the gateway host done, and the sections proven live over mTLS. Nothing changes for token-mode or pre-1.2.0 consumers: an old agent's snapshot is byte-identical in every old field.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `27_mobile_tables_and_audit` (todo) — not a code dependency (different files); ordered after it so the fleet rollout in task 05 does not race the mobile-audit's browser passes on the shared dev box
|
||||||
|
- `15_agent_mtls` / `24_dev_mtls_rollout` (complete) — the installed-bundle install path (`deploy/install-agent.sh --cert-bundle`), the `--status` CA-only probe, and the gap-10 re-stage-first rule the rollout follows
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_haproxy_section.md` — the agent reads + parses `haproxy.cfg` into the additive `haproxy` section, three-way availability, `--no-proxy` opt-out, never fails the snapshot
|
||||||
|
2. `02_nginx_section.md` — the agent reads the main config + one level of `include <glob>` (bounded) into the additive `nginx` section, with the `proxy_pass $var` → `map` join
|
||||||
|
3. `03_socket_section.md` — the agent parses `/proc/net/tcp` + `/proc/net/tcp6` (bounded) into the additive `sockets` section: LISTEN + ESTABLISHED with local/remote ip:port
|
||||||
|
4. `04_version_and_capture.md` — `AGENT_VERSION` 1.2.0, the SCHEMA docstring, `scripts/capture-gateway-config.sh` (real configs → `tests/fixtures/`), the live probe that the sections answer on the gateway host
|
||||||
|
5. `05_rollout_gateway.md` — re-stage `/home/ops/agent-install/`, install 1.2.0 on the gateway host, restart the unit, verify the sections over mTLS — read-only live verification, transcript in `data/logs/`
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/test_agent.py` — every new section over a fake `root` tree (present config → parsed shape; missing file → `available: false` + exactly one errors line; unreadable → `null` + one errors line; `--no-proxy` → sections absent and no errors line; a corrupt config degrades its own section and the snapshot still succeeds with the other sections intact)
|
||||||
|
- Fixture: `tests/fixtures/gateway_haproxy.cfg` + `gateway_nginx_*.conf` + `gateway_proc_net_tcp{,6}.txt` are verbatim captures from the gateway host (invariant 7); the parser tests run against them, so a config drift fails a test before a dashboard does
|
||||||
|
- Coverage: **>90%** on the new/modified code (the three collectors + the parsers)
|
||||||
|
- Live (read-only, invariant 10): task 04's probe over the mTLS dial, and task 05's post-restart verification with its transcript
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
|
||||||
|
- The final pass (task 05) updates `AGENTS.md`'s agent bullet (the new sections, `AGENT_VERSION` 1.2.0, the `--no-proxy` line) and the `SCHEMA` docstring is the self-documenting half on the wire. `README.md` is checked for any agent-version sentence.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest` green (the hermetic suite, no network), coverage not down
|
||||||
|
- [ ] a 1.2.0 agent over a captured-config root tree answers all three sections; over a tree without them it answers `available: false`/`null` with one errors line each and the snapshot's other sections are intact
|
||||||
|
- [ ] `AGENT_VERSION` is 1.2.0 and the SCHEMA docstring names the three new sections
|
||||||
|
- [ ] the fixture files are verbatim captures (the capture script's refusal-to-overwrite-with-unreadable pin holds)
|
||||||
|
- [ ] the gateway host runs 1.2.0 (`systemctl is-active dashboards-agent` → `active`, the file's `AGENT_VERSION` reads 1.2.0) and the sections answer over mTLS — transcript in `data/logs/`
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
|
- [ ] no behavior change in completed work: a 1.2.0 agent's snapshot is identical to 1.1.0's in every pre-existing field for a non-gateway host (the new keys are the only difference, and `--no-proxy` removes even those)
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 01 — The agent reads and parses the haproxy config
|
||||||
|
|
||||||
|
**Phase:** `28_gateway_agent_proxy` · **Source:** `TODO.md:3` — "Create a 'gateway' page that visualizes my incoming connections based on the gateway host's nginx and haproxy configs…"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`host_agent.py` gains a `haproxy` snapshot section: the agent reads the haproxy config (path an argument, default `/etc/haproxy/haproxy.cfg`), parses its `frontend` blocks (`name`, each `bind` address:port, `default_backend`/`use_backend` references) and `backend` blocks (`name`, each `server` line's name + target host:port), and serves them as a new additive section. The section follows the smart precedent exactly: a missing file is `available: false` plus **one** errors line for the pass, a file that cannot be read or parsed is `available: null` plus one errors line, `--no-proxy` opts out silently, and no failure of this section ever fails the snapshot or `/health`.
|
||||||
|
|
||||||
|
- ASSUMPTION: the parsed shape is `haproxy: {available: true|false|null, config: str (the path), frontends: [{name, binds: [{addr, port}], default_backend: str|null}], backends: [{name, servers: [{name, target, port}]}]}` — the frontend→backend routing is a name the **dashboard** joins (sources/gateway.py, phase 29), the agent only reports what the config says.
|
||||||
|
- ASSUMPTION: `bind` lines parse the last token's `addr:port` (the captured shape is `bind :::22 v4v6` — the address token is the one that contains `:port` after the word `bind`; a `bind` line without a parsable address:port is skipped with no error, because haproxy allows non-address bind forms and the section degrades to fewer binds rather than failing).
|
||||||
|
- ASSUMPTION: `server` lines parse `server <name> <target>[:port] <rest>` — `check inter 5s` and the like are the ignored rest; a `target` without a port (a bare upstream name) is kept with `port: null` (absent is null, never 0 — the module's own rule).
|
||||||
|
- ASSUMPTION: `use_backend <name>` lines inside a frontend are collected as `backends: [str]` on that frontend (the captured config uses `default_backend` only; the parser covers the captured shape plus `use_backend` because haproxy answers two names for the same routing fact, the way `resolve_command()` does for NUT verbs); unknown directives are ignored, never errors.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `agent/host_agent.py` — module docstring: add the three-way availability + `--no-proxy` rule to the design-rules section (the "SMART is a best-effort add-on" shape, adapted: one errors line for a missing binary becomes one errors line for a missing/unreadable config).
|
||||||
|
2. `agent/host_agent.py` — `Collector.__init__`/arguments: a `haproxy_config` path argument (default `/etc/haproxy/haproxy.cfg`) and the `--no-proxy` flag on the CLI (the `--no-smart` precedent: a flag that removes the section entirely, no errors line), threaded into `snapshot()`.
|
||||||
|
3. `agent/host_agent.py` — `Collector.haproxy()` + a module-level `parse_haproxy_config(text) -> dict` (the `parse_smart_output` idiom: a pure function over text so tests feed it fixtures without a filesystem): the block scanner (a `frontend`/`backend` line starts a block, an indented line belongs to it, a top-level line ends it) and the `bind`/`server`/`default_backend`/`use_backend` token parsers per the assumptions above.
|
||||||
|
4. `agent/host_agent.py` — `snapshot()`: `self._safe(self.haproxy, "haproxy", errors)` added to the returned document (additive key; a 1.1.0 consumer sees no difference, a 1.2.0 consumer with `--no-proxy` sees no `haproxy` key at all — the same absence the smart section takes).
|
||||||
|
5. `tests/test_agent.py` — the in-process harness cases: the captured-shape fixture parses to the exact expected frontends/backends (assert the `:::22` → `addr: "::"`, `port: 22` split and the six `default_backend` names); a `root` tree without the file → `available: false` + exactly one errors line naming the missing path (the smartctl "not found" pin's shape — the line must still say the file is missing, `tests/test_agent.py` pins that wording); an unreadable file (a mode-000 file in the fake tree, where the harness allows it) → `available: null` + one errors line; a corrupt file (garbage text) → `available: null` + one errors line and the snapshot's `cpu`/`memory` sections still intact; `--no-proxy` → no `haproxy` key and no errors line.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the five harness cases above + `parse_haproxy_config` over `tests/fixtures/gateway_haproxy.cfg` (task 04 records it; write the test against the captured shape now — the real file's content is known: 6 frontends, 6 backends, the exact `bind :::NNNN v4v6` and `server <name> <domain>:<port> check inter 5s` lines)
|
||||||
|
- Coverage: **>90%** on `parse_haproxy_config` + `Collector.haproxy`
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/test_agent.py -q` green, including the new haproxy cases and every pre-existing agent test
|
||||||
|
- [ ] a snapshot over a tree with the config carries the `haproxy` section with all six frontends and six backends; over a tree without it, `available: false` + one errors line; the snapshot never fails because of it
|
||||||
|
- [ ] `--no-proxy` removes the section with no errors line (pinned)
|
||||||
|
- [ ] `uv run ruff check agent tests` and `uv run pyright` clean
|
||||||
|
- [ ] no behavior change in completed work: every pre-existing snapshot field byte-identical on a host without haproxy (the `haproxy` key is the only addition)
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 02 — The agent reads and parses the nginx configs
|
||||||
|
|
||||||
|
**Phase:** `28_gateway_agent_proxy` · **Source:** `TODO.md:3` — "Create a 'gateway' page that visualizes my incoming connections based on the gateway host's nginx and haproxy configs…"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`host_agent.py` gains an `nginx` snapshot section: the agent reads the main nginx config (path an argument, default `/etc/nginx/nginx.conf`) **and follows its `include <path-or-glob>` lines one level** (the captured main config includes `/etc/nginx/conf.d/*.conf` and `/etc/nginx/stream.d/*.conf`), parses `server` blocks (`listen` address:port pairs, `server_name`), `upstream` blocks, `proxy_pass` directives (a literal `scheme://host:port` **and** a `proxy_pass $var` whose target the agent resolves against the file's `map` blocks), and the `map` blocks themselves. Same three-way availability and best-effort rules as task 01: missing main file → `available: false` + one errors line; unreadable/unparseable → `null` + one line; `--no-proxy` removes the section; the snapshot never fails because of it.
|
||||||
|
|
||||||
|
- ASSUMPTION: the parsed shape is `nginx: {available: true|false|null, config: str, files: [str (main + the included paths actually read)], servers: [{listen: [{addr, port}], server_names: [str], proxy_passes: [{location: str, scheme: str|null, target: {host, port}|null, variable: str|null}]}], upstreams: [{name, servers: [{host, port}]}], maps: [{variable: str, entries: [{key, target: {host, port}|null}]}]}` — a `proxy_pass $targetBackend` whose `$targetBackend` matches a `map` block's variable is served with `variable` set and `target: null` (the map decides per-connection at runtime; the **dashboard** joins the variable to the map, the agent reports both facts and never merges them, because a merged target would be a number the wire did not publish).
|
||||||
|
- ASSUMPTION: include-following is **one level, bounded** — `include` lines in the main config are read (glob expanded), but `include` lines inside an included file are not followed (the captured tree is one level deep; a deeper include is a config the operator changed, and the section degrades to `files` naming only what was read rather than erroring). Bounds: at most 8 included files and 1 MB of included text combined; exceeding a bound drops the excess includes with one errors line (the smart pass-budget shape: the pass reports what it did not cover) and the section stays `available: true`.
|
||||||
|
- ASSUMPTION: `listen` parsing takes the first token after `listen` (the captured shapes: `listen 4443 ssl proxy_protocol;`, `listen [::]:4443 ssl;`, `listen 80;`) — `host:port`, `[host]:port`, or bare `port` (→ `addr: null`); the `ssl`/`proxy_protocol`/`http2` flags are not parsed (they change how the bytes travel, not what the page's pipes show).
|
||||||
|
- ASSUMPTION: `server_name _` (the catch-all) is kept as the literal `_` entry — the dashboard renders it as "any domain", the agent does not interpret it.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `agent/host_agent.py` — `Collector.__init__`/arguments: a `nginx_config` path argument (default `/etc/nginx/nginx.conf`), threaded through like `haproxy_config`; `--no-proxy` covers this section too (it already exists from task 01).
|
||||||
|
2. `agent/host_agent.py` — a module-level `parse_nginx_config(main_text, included: dict[str, str]) -> dict` (pure over text, the `parse_haproxy_config` idiom) with: a brace-block scanner (nginx directives nest by `{}`, unlike haproxy's indent — the scanner is the task's core); `server`/`upstream`/`map` block extraction; the `listen`/`server_name`/`proxy_pass`/`include`/`server`-inside-upstream token parsers per the assumptions above.
|
||||||
|
3. `agent/host_agent.py` — `Collector.nginx()`: read the main config through the existing `read_text` (which returns `None` on unreadable — the section's `null` arm), expand each `include` line (a glob through `glob.glob`, the literal path otherwise), apply the 8-file/1 MB bounds, call the parser, and return the section dict with `files` naming the main + each included path actually read.
|
||||||
|
4. `agent/host_agent.py` — `snapshot()`: `self._safe(self.nginx, "nginx", errors)` added to the returned document (additive, task 01's shape).
|
||||||
|
5. `tests/test_agent.py` — the in-process harness cases: the captured main + `conf.d/http.conf` + `stream.d/stream.conf` (as a fake `root` tree under `etc/nginx/`) parse to the expected servers (assert the `4443`/`8443` listen ports, the `server_name` list, the `proxy_pass` targets including the `127.0.0.1:8081` literal), the `stream.conf` `map` block (assert the variable name `$targetBackend` and a representative entry), and the `proxy_pass $targetBackend` served with `variable: "targetBackend"` + `target: null`; a tree without the main file → `available: false` + one errors line; a corrupt main file → `null` + one line, snapshot intact; the include bound (a main config including 9+ files) → `available: true`, `files` the first 8, one errors line naming the dropped count.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the five harness cases above + `parse_nginx_config` unit cases for the token parsers (`listen [::]:4443` → `addr: "::"`, bare `80` → `addr: null`, a `proxy_pass http://host:port` with a scheme, an `upstream` block's `server` lines)
|
||||||
|
- Coverage: **>90%** on `parse_nginx_config` + `Collector.nginx`
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/test_agent.py -q` green, including the new nginx cases and every pre-existing agent test
|
||||||
|
- [ ] a snapshot over the captured-config tree carries the `nginx` section with every `server` block, the `map`, and the variable-form `proxy_pass` served unmerged
|
||||||
|
- [ ] the one-level include bound is pinned (9th+ include dropped with one errors line, section still `available: true`)
|
||||||
|
- [ ] `uv run ruff check agent tests` and `uv run pyright` clean
|
||||||
|
- [ ] no behavior change in completed work: every pre-existing snapshot field byte-identical on a host without nginx
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 03 — The agent reads the live sockets
|
||||||
|
|
||||||
|
**Phase:** `28_gateway_agent_proxy` · **Source:** `TODO.md:3` — "I want it to show animated pipes with data coming in (IPs, ports, domains, etc.) and data going out (IPs, ports, domains, etc.)." (the live half — the configs say *where* the pipes go; this task says *what is flowing now*)
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`host_agent.py` gains a `sockets` snapshot section: the agent parses `/proc/net/tcp` and `/proc/net/tcp6` (both readable by the agent user, verified on the gateway host) into the LISTEN sockets (the gateway's open frontends, with their local address:port) and the ESTABLISHED sockets (local and remote address:port — the live end-to-end pairs the dashboard joins to the config). Same best-effort rules: an unreadable `/proc/net/tcp` degrades the section to `null` with one errors line, `--no-proxy` removes it, and the snapshot never fails because of it. The section is a raw fact list — **no** config join happens in the agent (a socket does not know its domain; the dashboard's `sources/gateway.py` does the join, phase 29).
|
||||||
|
|
||||||
|
- ASSUMPTION: the parsed shape is `sockets: {available: true|false|null, listens: [{family: "v4"|"v6", addr: str|null, port: int}], established: [{family, local: {addr, port}, remote: {addr, port}}]}` — `addr` is the dotted/colon form (v4) or the expanded `x:x:…` form (v6, the `/proc` 4-byte-group little-endian spelling unrolled); a `0.0.0.0`/`::` listener keeps its literal wildcard address (it is the fact the dashboard's inbound join keys on — the port, with the wildcard addr marking "any address").
|
||||||
|
- ASSUMPTION: only the two states the page needs are reported — `0A` (LISTEN) into `listens`, `01` (ESTABLISHED) into `established` — every other state is skipped (a TIME_WAIT ocean is noise the pipes do not show, and skipping it keeps the section small).
|
||||||
|
- ASSUMPTION: the read is bounded at 4096 lines per file (the header line excluded from the count); a file longer than the bound reports what it read and one errors line naming the truncation (the smart pass-budget shape), `available` stays `true`.
|
||||||
|
- ASSUMPTION: the hex→address conversion is a pure module-level `parse_proc_net_tcp(text, family) -> (listens, established)` (the `parse_smart_output` idiom), so the captured-`/proc` fixtures test the parser without a live filesystem.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `agent/host_agent.py` — module docstring: the `sockets` design line (two states only, the 4096-line bound, the best-effort rule) beside the smart one.
|
||||||
|
2. `agent/host_agent.py` — module-level `parse_proc_net_tcp(text: str, family: str) -> tuple[list[dict], list[dict]]`: the column split (the `/proc/net/tcp` columns: `sl local_address rem_address st …`; `local_address`/`rem_address` are `HEXADDR:HEXPORT`, v4 a 32-char hex little-endian word, v6 four 32-char groups in the same order), the `st` filter (`01`/`0A` only), and the address unrolling per the assumptions above.
|
||||||
|
3. `agent/host_agent.py` — `Collector.sockets()`: read `/proc/net/tcp` and `/proc/net/tcp6` through `read_lines` (an unreadable file is `None` → the `null` arm; both unreadable → `null` + one combined errors line), apply the line bound, call the parser, and return the section dict.
|
||||||
|
4. `agent/host_agent.py` — `snapshot()`: `self._safe(self.sockets, "sockets", errors)` added to the returned document (additive, tasks 01/02's shape).
|
||||||
|
5. `tests/test_agent.py` — the parser unit cases over a captured `/proc/net/tcp` fragment (a LISTEN on `:::443` and one on `0.0.0.0:80`, an ESTABLISHED v4 pair, an ESTABLISHED v6 pair, a TIME_WAIT line asserted **absent** from both lists) + the harness cases: a fake tree whose `proc/net/tcp{,6}` carry the fragment → the section's exact lists; a tree without the files → `null` + one errors line, snapshot intact; the 4096-line bound (a 5000-line file) → `available: true`, the first 4095 data rows, one errors line naming the truncation.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the parser unit cases + the three harness cases above
|
||||||
|
- Coverage: **>90%** on `parse_proc_net_tcp` + `Collector.sockets`
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/test_agent.py -q` green, including the new socket cases and every pre-existing agent test
|
||||||
|
- [ ] a snapshot over a tree with the captured `/proc` files carries the `sockets` section with the exact LISTEN/ESTABLISHED lists; the TIME_WAIT line is absent (pinned)
|
||||||
|
- [ ] the 4096-line bound is pinned (truncation line, section still `available: true`)
|
||||||
|
- [ ] `uv run ruff check agent tests` and `uv run pyright` clean
|
||||||
|
- [ ] no behavior change in completed work: every pre-existing snapshot field byte-identical on a host where `/proc/net/tcp` is unreadable (the section is `null`, nothing else moved)
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 04 — Agent 1.2.0, the SCHEMA, and the gateway-config captures
|
||||||
|
|
||||||
|
**Phase:** `28_gateway_agent_proxy` · **Source:** `TODO.md:3` — the ground-truth half: invariant 7 (verify against the deployed upstream; never guess a metric name) applied to the three new sections
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The agent's version and self-documentation catch up with tasks 01–03 (`AGENT_VERSION` 1.2.0, the SCHEMA docstring names the three sections), and the capture script records the gateway host's real `haproxy.cfg`, nginx configs, and `/proc/net/tcp{,6}` as verbatim fixtures — so the parser tests run against the deployed shape, not a memory of it, and a config drift fails a test before a dashboard does.
|
||||||
|
|
||||||
|
- ASSUMPTION: the fixtures **keep the real service domain names** (e.g. `nextcloud.reeseapps.com`) — unlike the smartctl capture's Serial/WWN scrub, the domains are the data the `/gateway` page exists to show (inbound domain, outbound target) and the page is session-gated; a scrubbed fixture would be a fixture that tests a config the gateway does not run. The capture script therefore scrubs nothing; what it refuses is the other direction (overwriting a good fixture with unreadable output, the `capture-fixtures.sh` rule).
|
||||||
|
- ASSUMPTION: `AGENT_VERSION` goes 1.1.0 → 1.2.0 (a new additive section is a minor bump, the 1.0.0 → 1.1.0 TLS bump's shape); the bundle's `# BUNDLE-REV` is untouched (the agent file is not the bundle — no CA fact moved).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `agent/host_agent.py` — `AGENT_VERSION = "1.2.0"` and the SCHEMA docstring: the three new section shapes (the `haproxy`/`nginx`/`sockets` blocks, one line each, the way `smart` is documented) plus the three-way availability + `--no-proxy` note.
|
||||||
|
2. `scripts/capture-gateway-config.sh` — a new capture script (the `capture-fixtures.sh` convention, ssh-alias driven: `GATEWAY_SSH` default `gateway-ops`, `ssh` alias `gateway-ops`): scp/capture `/etc/haproxy/haproxy.cfg` → `tests/fixtures/gateway_haproxy.cfg`; `/etc/nginx/nginx.conf`, `/etc/nginx/conf.d/*.conf`, `/etc/nginx/stream.d/*.conf` → `tests/fixtures/gateway_nginx_main.conf` / `gateway_nginx_conf_d_*.conf` / `gateway_nginx_stream_d_*.conf`; `cat /proc/net/tcp /proc/net/tcp6` → `tests/fixtures/gateway_proc_net_tcp.txt` / `gateway_proc_net_tcp6.txt`. The `capture-fixtures.sh` refusal rule: a capture that comes back empty or with an ssh/permission error is **not** written over an existing fixture (one line naming the failure, exit 1). No sudo (all five reads are world-readable by the agent user — verified 2026-10-21).
|
||||||
|
3. Run the capture once (the operator's box reaches the gateway host) and commit the fixtures; `tests/test_agent.py`'s task 01–03 cases already assert against the captured shape.
|
||||||
|
4. `tests/test_deploy_scripts.py` — pin the script the way `capture-fixtures.sh` is pinned: it dials the ssh alias (not a hardcoded IP), scrubs nothing (a domain grep over its own source asserts no scrub step exists), and the refusal-to-overwrite branch is named.
|
||||||
|
5. Live probe (read-only, invariant 10): `ssh gateway-ops` — `systemctl is-active dashboards-agent`, then a `python3` one-liner (the `--status` probe idiom, CA-only over the installed bundle's CA) that dials the **current** 1.1.0 agent and confirms the old sections still answer (the no-behavior-change control), and confirms the new sections are **absent** from the 1.1.0 snapshot (the three-way `predates` arm's ground truth). The probe writes `data/logs/28-task04-probe.out`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/test_deploy_scripts.py` pins for the capture script; the existing fixture-pinned parser tests (tasks 01–03) run against the real captures
|
||||||
|
- Coverage: **>90%** on the new script's Python-side branches (the script is bash; its pins are the `test_deploy_scripts.py` shape)
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `AGENT_VERSION` reads `1.2.0` and the SCHEMA docstring names all three new sections (pinned by a `tests/test_agent.py` assert on the constant)
|
||||||
|
- [ ] `bash scripts/capture-gateway-config.sh` (or the operator's run) leaves the seven fixture files in `tests/fixtures/`, verbatim, with the refusal rule holding
|
||||||
|
- [ ] `tests/test_agent.py` green against the real captures (a config drift in the fixture fails a test)
|
||||||
|
- [ ] `data/logs/28-task04-probe.out` records: the gateway's installed agent is 1.1.0, the old sections answer, the three new sections are absent
|
||||||
|
- [ ] `uv run ruff check scripts` (if the script has a Python leg) / `bash -n scripts/capture-gateway-config.sh` clean, `uv run pyright` clean
|
||||||
|
- [ ] no behavior change in completed work: the 1.1.0 agent's served snapshot is unchanged (the probe proves it)
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 05 — Roll 1.2.0 onto the gateway host, verify over mTLS
|
||||||
|
|
||||||
|
**Phase:** `28_gateway_agent_proxy` · **Source:** `TODO.md:3` — the deployment half: the sections only exist on the wire once the gateway host runs 1.2.0 (the phase-24 rollout's per-host safety order, gap-10's re-stage-first rule)
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The gateway host runs `host_agent.py` 1.2.0 with the three new sections live over mTLS, installed by the same granted-path procedure the fleet uses — re-stage `/home/ops/agent-install/` first (the gap-10 rule: a path grant is only as current as the file it points at), run the installer, restart the unit, and verify the sections answer. The install is **read-only on the dashboard's side** (no config change, no `HOST_TARGETS` change — the gateway is already in `HOST_TARGETS` and dials fine on 1.1.0), and the verification transcript lands in `data/logs/`.
|
||||||
|
|
||||||
|
- ASSUMPTION: the gateway host's unit is **unchanged by this rollout** — `--no-proxy` is not passed (the sections are on by default), so the installer's re-run guard sees an identical unit render and stays silent (the phase-17 task-03 silent path); no `--force` needed.
|
||||||
|
- ASSUMPTION: the restart is the `systemctl restart dashboards-agent` the grant's `systemctl restart *` line allows (the gateway's `ops` has `(ALL) NOPASSWD: ALL` — the fleet record's `deploy/agent-sudoers` shape).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. Re-stage: `ssh gateway-ops` — copy the repo's `agent/host_agent.py` (1.2.0, md5 recorded) and `deploy/install-agent.sh` (the repo's current sha256, recorded) into `/home/ops/agent-install/` (the directory is `ops`-owned, no root needed — the phase-24 task-03 procedure), leaving any existing snapshot beside it as `<name>-<date>-backup` (a file sudo will not run).
|
||||||
|
2. Install: run the granted path `sudo /home/ops/agent-install/deploy/install-agent.sh --bind 10.1.1.27 --port 9120` (the gateway's existing unit arguments — read them first with `systemctl cat dashboards-agent` and reproduce them exactly, plus `--cert-bundle` if the unit has the line) — the re-run guard prints nothing (identical unit) and the agent file is updated in place.
|
||||||
|
3. Restart: `sudo systemctl restart dashboards-agent`; `systemctl is-active dashboards-agent` → `active`.
|
||||||
|
4. Verify (read-only): the `--status` CA-only probe (the `python3` `ssl` one-liner over the installed bundle's CA, `--bind 10.1.1.27 --port 9120`) answers; then a `python3` one-liner that dials over mTLS (the dashboard's client cert, the stored CA — the `tests/test_live_fleet.py` dial idiom) and asserts: `agent_version` is `1.2.0`, the three new keys are present, `haproxy.available` is `true` with 6 frontends, `nginx.available` is `true` with ≥ 9 server blocks and the `$targetBackend` map, `sockets.available` is `true` with a non-empty `listens` (port 443 among them), and the pre-existing sections (`cpu`, `memory`, `network`) still answer.
|
||||||
|
5. `data/logs/28-task05-rollout.md` — the transcript: the md5/sha256 of what was staged where, the `systemctl cat` before/after (diff empty), the probe output, the one-liner's assertions, and the `haproxy`/`nginx`/`sockets` section summaries (counts only, no config text — invariant 6: no config body in the note).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Live (read-only, invariant 10): the verification one-liner is the phase's proof; no dashboard process is touched, no `HOST_TARGETS` line moves
|
||||||
|
- The hermetic suite is untouched by this task (no code change) — `uv run pytest` is re-run as the gate, not for this task's logic
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the gateway host's installed `host_agent.py` is the repo's 1.2.0 (md5 matches) and `systemctl is-active dashboards-agent` → `active`
|
||||||
|
- [ ] the `--status` CA-only probe answers; the mTLS one-liner asserts all five facts (version, three sections present + available, pre-existing sections intact)
|
||||||
|
- [ ] `data/logs/28-task05-rollout.md` exists with the transcript and no config text
|
||||||
|
- [ ] the dashboard's `host:gateway` / `disks:gateway` jobs still read `ok` in `./scripts/dev-server.sh status` (the 1.2.0 snapshot's new keys cost the old sources nothing)
|
||||||
|
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||||
|
- [ ] no behavior change in completed work: the gateway's pre-existing snapshot fields are unchanged (the one-liner's control assertions prove it)
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
# Phase 29 — The gateway dashboard page: animated pipes from config + live sockets
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "Create a 'gateway' page that visualizes my incoming connections based on the gateway host's nginx and haproxy configs. I want it to show animated pipes with data coming in (IPs, ports, domains, etc.) and data going out (IPs, ports, domains, etc.)." (the dashboard half)
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** phase 28 (the `haproxy`/`nginx`/`sockets` agent sections, `AGENT_VERSION` 1.2.0, the gateway host rolled out), `src/dashboards/config.py` (`Settings`, `_env`, the `nut_enabled`/`actions_allowed` gate properties), `src/dashboards/sources/disks.py` (the "a source of its own reading the same agents" precedent — the `disks:<host>` job dials the same `/snapshot`), `src/dashboards/poller.py` (job loop, health records, stale/last-good), `src/dashboards/app.py` (`build_hub` — the job list wired once for production and the injected-store test path; the `agent_tls_map` the `host:`/`disks:` jobs share per host), `src/dashboards/api.py` (the snapshot/history route pairs, the `404 {"configured": false}` gate that runs before any dial, `_history_batch`, the `/api/summary` additive blocks), `src/dashboards/alerts.py` (the "an unconfigured source never alerts" rule), `src/dashboards/store.py` (the `sample(source, host, metric, res, ts)` + rollup/retention story — invariant 9), `src/dashboards/web/pages/` + `static/` (the five dashboards, the five-tab masthead pinned by `tests/test_api.py`, `app.js`'s `initChrome` badges, `Dash.hero`/`DashCharts` — no framework, no CDN, invariant 5), `tests/test_api.py` (the served-bytes pins), `tests/test_config.py`, `tests/test_live_fleet.py` (the read-only live suite).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The fifth dashboard page — `/gateway` — renders the gateway host's connection topology as animated pipes: for every configured frontend (haproxy `bind`, nginx `listen`, the stream `443`) an **inbound** pipe (client → gateway: the port, the `server_name` domains, the live connection count and client addresses) and, fanning out from the gateway, one **outbound** pipe per configured target (haproxy `server`, nginx `proxy_pass`, the stream `map` entries: the target host:port, the live count). The page is a **source of its own** (`gateway`, its own poller job on the same `/snapshot` the `host:` job dials, its own interval/timeout/health/events), its counts are stored as gauges (invariant 9's rollup/retention story) and graphed from `/api/gateway/history`, and it keeps every invariant the fleet already keeps: unconfigured is `404` without a dial (invariant 3), a dead source ages instead of zeroing (invariant 4), no count is a rate (rule 2 — no estimator, no `Δ<window>`), no count carries a verdict (like `net.*`), and an old agent (1.1.0, no sections) reads as "predates gateway support" — dimmed, `n/a`, never red.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `28_gateway_agent_proxy` (todo) — the three agent sections on the wire and the gateway host running 1.2.0 (the live half of this phase's verification dials what 28 installed)
|
||||||
|
- `27_mobile_tables_and_audit` (todo) — the page's 360 px behaviour is audited by 27's script shape (the table, if any, scrolls inside its card)
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_config_and_source.md` — `GATEWAY_HOST` in `Settings` + the gate properties; `sources/gateway.py`: the config + sockets join into inbound/outbound pipes, gauges only, the three-way agent support
|
||||||
|
2. `02_wiring.md` — the `gateway` poller job in `build_hub` (30 s), health, the alerts rules, the `/api/summary` block, the `/api/config` entry
|
||||||
|
3. `03_api_routes.md` — `/api/gateway/snapshot` + `/api/gateway/history`: the configured gate before any dial (spy-pinned), history through `_history_batch`, unknown names as empty series
|
||||||
|
4. `04_page.md` — `/gateway` + `page-gateway.js`: the animated pipes (hand-rolled SVG + CSS, no CDN, `prefers-reduced-motion`, `aria-label`), the live counts, the three degradation arms
|
||||||
|
5. `05_nav_and_summary.md` — the sixth tab on every signed-in page (the pinned masthead pins updated + browser-measured), the `/api/summary` pin extended
|
||||||
|
6. `06_docs_and_verify.md` — README/AGENTS (the page row, the `GATEWAY_HOST` row, the fleet fact), the browser screenshot pass against live data, the read-only live verification
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/test_gateway_source.py` (new — the join over fixture snapshots: an inbound join by listen port, the domain-target port-only join labelled "by port", the raw-IP exact join, the `$var`→`map` join, the 1.1.0-predates arm, the never-answered stub arm); `tests/test_config.py` (`GATEWAY_HOST` default, the gate property); `tests/test_api.py` (the route pins, the summary block, the nav pins)
|
||||||
|
- Coverage: **>90%** on `sources/gateway.py` + the new config field
|
||||||
|
- Real-browser check (invariant 10): task 05's measurement + task 06's screenshots against the live dev server
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
|
||||||
|
- Task 06 updates `AGENTS.md`'s page table (`/gateway` row), the `config.py` bullet (`GATEWAY_HOST`), and the metric-naming paragraph (`gateway.*` gauges); `README.md` gains the page's how-to row. The invariant 9 story: the `gateway.*` series roll up raw → 1 m → 1 h like every other gauge.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest` green (the hermetic suite), coverage not down; `uv run pytest --cov` holds the 94% floor
|
||||||
|
- [ ] `GATEWAY_HOST` unset (or a name not in `HOST_TARGETS`) → `/api/gateway/snapshot` answers `404 {"configured": false}` with no dial (spy-pinned) and the page says "not configured"
|
||||||
|
- [ ] a 1.1.0-style snapshot (no sections) → the page's "agent predates gateway support" arm, dimmed, every number `n/a`, the job still `ok` (the invariant-3 shape, pinned)
|
||||||
|
- [ ] the gateway host (1.2.0, post-phase-28) → the page renders the six haproxy frontends, the nginx server blocks and the stream map, with live counts that match a direct `/proc/net/tcp` read (task 06's verification)
|
||||||
|
- [ ] `bash .agents/validate.sh` green
|
||||||
|
- [ ] no behavior change in completed work: the nine existing sources read `ok` in `./scripts/dev-server.sh status`, the five existing dashboards render as before, the `/api/summary` pre-existing blocks are byte-identical in shape (the `gateway` block is additive)
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 01 — `GATEWAY_HOST` and the gateway source
|
||||||
|
|
||||||
|
**Phase:** `29_gateway_dashboard_page` · **Source:** `TODO.md:3` — "…visualizes my incoming connections based on the gateway host's nginx and haproxy configs… data coming in (IPs, ports, domains, etc.) and data going out (IPs, ports, domains, etc.)."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`Settings` names which `HOST_TARGETS` entry is the gateway (`GATEWAY_HOST`, default `gateway`), and `sources/gateway.py` projects one agent snapshot into the page's data: the **inbound** pipes (each configured frontend — a haproxy `bind` port, a nginx `listen` port, the stream `443` — with its `server_name` domains and the live ESTABLISHED count + client addresses keyed on the listen port) and the **outbound** pipes (each configured target — a haproxy `server`, a nginx `proxy_pass`, a stream `map` entry — with the live ESTABLISHED count). The counts are gauges: **no rate** (no `RateEstimator`, no `window_seconds`, no `Δ<window>` — a count is a count, rule 2), **no verdict** (no `severity` dict on any count — `severity.py` rates no connection count, the `net.*` precedent), and the three-way agent support degrades a pre-1.2.0 agent to "predates gateway support" (invariant 3), never red.
|
||||||
|
|
||||||
|
- ASSUMPTION: **inbound join**: an ESTABLISHED socket belongs to frontend `P` when its **local port** is `P`'s listen port; the client is the socket's **remote** address. A socket whose local port matches no configured frontend is `unmatched` (reported, not dropped — the page shows "N other connections", the operator's fact that the gateway talks to something the configs don't name).
|
||||||
|
- ASSUMPTION: **outbound join**: an ESTABLISHED socket belongs to target `T` when its **remote** address matches — a raw-IP target on an exact (ip, port) match; a **domain target on a port-only match, labelled `by_port: true`** in the payload (the agent never does DNS — invariant 7's "never guess" applied to the join: a port match against a domain is a real socket fact plus a config fact, and the label says which half is which). A socket whose remote matches no target is `unmatched` too.
|
||||||
|
- ASSUMPTION: the payload shape is `{configured: true, supported: true, host: str, haproxy: {frontends: [{name, port, backend: str|null}]}, nginx: {servers: [{port, server_names: [str]}], map_targets: [{key, host, port}]}, inbound: [{port, source: "haproxy"|"nginx", domains: [str], connections: int, clients: [str]}, outbound: [{target: {host, port, name: str|null, by_port: bool}, connections: int, source: str|null}], unmatched: {inbound: int, outbound: int, other: int}, sockets: {listens: int, established: int}}` — `supported: false` (a 1.1.0-style snapshot) carries the same keys as empty/nulls plus `supported: false`, and a never-answered host is the stub the `/api/system` stubs use (`unreachable: true`, every count `null`, invariant 4).
|
||||||
|
- ASSUMPTION: the `$var`→`map` join: a `proxy_pass $targetBackend` whose variable matches a `map` block contributes the map's entries as `map_targets` (key = the domain, the target = the entry's host:port) and **no** literal outbound pipe (the per-connection target is the map's job; the page renders the map entries as the fan-out — the 17-entry `stream.conf` map is exactly the "data going out (domains)" the TODO names).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/config.py` — `Settings.gateway_host: str = _env("GATEWAY_HOST", "gateway")`, `gateway_interval_seconds` (default `30.0`, the `DISKS_INTERVAL_SECONDS` idiom) and `gateway_timeout_seconds` (default `10.0`), and the `gateway_enabled` property (`True` iff `gateway_host` is among the `HOST_TARGETS` names — the `nut_enabled` shape); the `any_source_enabled` property includes it.
|
||||||
|
2. `src/dashboards/sources/gateway.py` — new: `parse(snapshot: dict, host: str) -> dict` (the `sources/disks.py` projection idiom — pure over the snapshot the poller fetched, no I/O of its own): the section readers (an absent section → the `supported: false` arm; `available: false`/`null` → a `why` line in the payload, the invariant-3 arms), the inbound/outbound joins per the assumptions above, the `unmatched` tallies, and the `samples` list: one gauge per distinct listen port (`gateway.in_conns@<port>`) and one per distinct target host (`gateway.out_conns@<host>`; a `by_port` domain target's series is keyed on the host, the port is in the payload not the name — the `@stream` scheme, invariant 9's naming).
|
||||||
|
3. `tests/test_config.py` — `GATEWAY_HOST` default `gateway`, an explicit value winning, `gateway_enabled` true/false over the target list (the `nut_enabled` test shape).
|
||||||
|
4. `tests/test_gateway_source.py` — new: the join over fixture-shaped snapshots (reuse the phase-28 fixtures' parsed shape — the test builds the snapshot dict the agent serves): the inbound join (a socket on local `:443` lands in the stream frontend's pipe with its client address; a socket on an unconfigured port is `unmatched.inbound`); the outbound join (a raw-IP target's exact match; a domain target's port-only match with `by_port: true`; no DNS call anywhere — a `socket.getaddrinfo` spy must record nothing); the `$var`→`map` join (the 17-entry map → 17 `map_targets`, no literal pipe); the `supported: false` arm (a 1.1.0-style snapshot without the sections); the never-answered arm is the API's job (task 03), not the source's.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/test_config.py` + `tests/test_gateway_source.py` as above
|
||||||
|
- Coverage: **>90%** on `sources/gateway.py` + the new config fields
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GATEWAY_HOST` defaults to `gateway` and `gateway_enabled` is true only for a configured name (pinned)
|
||||||
|
- [ ] `parse()` over a 1.2.0 fixture snapshot yields the exact inbound/outbound lists (counts, clients, `by_port`, the map fan-out) — a rename in the agent's section keys fails a test
|
||||||
|
- [ ] the `getaddrinfo` spy records **nothing** across the whole source (pinned — the join never dials DNS)
|
||||||
|
- [ ] the `supported: false` arm renders the payload's `why` and null counts, never a red verdict (pinned)
|
||||||
|
- [ ] `uv run pytest` green, `uv run ruff check src tests` and `uv run pyright` clean
|
||||||
|
- [ ] no behavior change in completed work: the pre-existing `Settings` fields and the `host`/`disks`/`llm`/`nut` sources are byte-identical in behaviour
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 02 — The gateway job: poller, health, alerts, summary, config
|
||||||
|
|
||||||
|
**Phase:** `29_gateway_dashboard_page` · **Source:** `TODO.md:3` — the source is a **job of its own** (the `disks:<host>` precedent): its own interval, timeout, health record, and events, so a wedged config read ages the pipes instead of blinking `/system`
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The `gateway` source runs as its own poller job (the `disks:` shape — one job dials the same `/snapshot` the `host:` job of the gateway host reads, with the gateway's own interval/timeout), so the page ages on its own cadence and a failed poll keeps the last good pipes dimmed instead of zeroing them (invariant 4). The alerts rules keep an unconfigured source silent (invariant 3) and a dead source to exactly one line (the `disks` echo rule — the gateway job dials the same agent, so its failure carries no new information while the `host:` job is the voice; a gateway job failing *while its host answers* — the sections wedged — is the only voice that fault has). `/api/summary` gains the additive `gateway` block, `/api/config` gains the gateway row.
|
||||||
|
|
||||||
|
- ASSUMPTION: the job is `gateway` (one job, the `nut` shape — a single source, not per-host: the page is about the one gateway host), `DISKS`-interval defaults from task 01 (30 s / 10 s), events under the dashboard name `gateway`.
|
||||||
|
- ASSUMPTION: the **no-verdict alert rule**: the source's payload carries no `severity`, so `alerts.py` gains no threshold for `gateway` — the only alerts a `gateway` job raises are reachability (stale > 3× its 30 s interval → `warn`, stopped answering → `fail`, the `disks`-echo suppression applied: a dead gateway job while its `host:` job is also dead adds no line).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/app.py` — `build_hub`: the `gateway` job (one, not per host) wired the `disks:` job is wired — the same `AgentTls` entry the gateway host's `host:`/`disks:` jobs use (so a revocation fails all three with one sentence, the task-03 `agent_tls_map` shape), its `parse` the `sources.gateway.parse` + `samples` pass-through, no `RateEstimator` (the source takes none — task 01). The job exists **only** when `settings.gateway_enabled` (the `disks` job's configured gate: no job at all when the source is unconfigured, invariant 3).
|
||||||
|
2. `src/dashboards/alerts.py` — the `gateway` block: the reachability rules per the assumption above; the `HREF` table gains `/gateway`; **no** verdict rule (asserted: a `gateway` payload with a fabricated `severity` dict still produces no verdict alert — the module may not have a threshold of its own, the phase-09 disk rule's shape).
|
||||||
|
3. `src/dashboards/api.py` — `/api/summary`: the additive `gateway` block (`{configured, supported, host, inbound: int (Σ connections), outbound: int, map_targets: int, unmatched: int, states: {gateway: str}}` — a configured job with no payload yet is a stub, the `system.hosts` stub shape); `/api/config`: the gateway row (`GATEWAY_HOST`'s name + the `Target.display_url` of the entry it names, redacted — invariant 6's boundary).
|
||||||
|
4. `tests/test_api.py` — the `build_hub` wiring pins (the job present iff `gateway_enabled`; the shared `AgentTls` entry — a revocation written into `meta` fails the `gateway` job **before any dial**, the `test_a_revoked_agent_fails_both_of_its_sources_without_a_dial` shape extended to three sources); the `/api/summary` `gateway` block pin (with and without the source configured — unconfigured: the block is `configured: false` and **absent no dial happens**, the spy-pinned shape); the alerts pins (unconfigured → no `gateway` alert; dead job + dead host → one line, not two; a wedged-sections failure while the host answers → one line naming `gateway`).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the wiring/alerts/summary pins above
|
||||||
|
- Coverage: **>90%** on the modified `app.py`/`alerts.py`/`api.py` branches
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] a configured build carries exactly one `gateway` job on the gateway host's shared TLS entry; an unconfigured build carries none (pinned)
|
||||||
|
- [ ] a revocation in `meta` fails the `gateway` job before any dial (spy-pinned, the `host:`/`disks:` pair's shape)
|
||||||
|
- [ ] `/api/summary`'s `gateway` block is pinned with and without configuration; the pre-existing blocks are unchanged
|
||||||
|
- [ ] the alerts pins hold: unconfigured silent, dead-source one line, no verdict rule
|
||||||
|
- [ ] `uv run pytest` green, `uv run ruff check src tests` and `uv run pyright` clean
|
||||||
|
- [ ] no behavior change in completed work: the pre-existing jobs, alerts, and summary blocks behave exactly as before
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Task 03 — `/api/gateway/snapshot` and `/api/gateway/history`
|
||||||
|
|
||||||
|
**Phase:** `29_gateway_dashboard_page` · **Source:** `TODO.md:3` — the page's data routes, in the app's established shape: the configured gate before any dial (invariant 3), history through the one batched door (invariant 9 / phase-14 task 02)
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`/api/gateway/snapshot` answers the source's payload (task 01's shape: the config, the joined inbound/outbound pipes, the unmatched tallies, `supported`) for the `GATEWAY_HOST` entry — with a configured-but-never-answered host as the `unreachable` stub (every count `null`, invariant 4's page-side half) and `GATEWAY_HOST` unset/unknown as `404 {"configured": false}` **before anything that could reach the network** (no client, no fetch, no job — spy-pinned both ways, the `tests/test_api.py` disks-pair shape). `/api/gateway/history` serves the `gateway.in_conns@<port>` / `gateway.out_conns@<host>` series through the same `_history_batch()` → `store.history_many()` door the other four history routes use (phase-14 task 02 — one batched decision, the same `{source, host, metric, range_seconds, points}` document, unknown names as empty series).
|
||||||
|
|
||||||
|
- ASSUMPTION: the route pair is read-only — a fourth data page grows no write (invariant 1's shape: the app's only POST that dials an upstream stays `/api/ups/command`, and the `tests/test_api.py` route-verb enumeration gains the two GETs to the read-only list).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/api.py` — the `gateway` router: `GET /api/gateway/snapshot` (the configured gate first — `settings.gateway_enabled` false → `404 {"configured": false}` with **no** client construction and **no** `fetch_snapshot` call, spy-pinned; configured → the job's last payload through `sources.gateway.parse`'s never-answered-stub arm) and `GET /api/gateway/history` (the `hosts=`/`metrics=`/`range=` query shape the other three history routes take, answered through `_history_batch`).
|
||||||
|
2. `src/dashboards/api.py` — the route-verb enumeration test's read-only list gains the two paths (the task-01 assumption's pin).
|
||||||
|
3. `tests/test_api.py` — the pins, the disks pair's shape: (a) `GATEWAY_HOST` unset → `404 {"configured": false}`, the `httpx.AsyncClient.get` spy and the `collectors.fetch_snapshot` spy record **nothing**; (b) `GATEWAY_HOST` a name not in `HOST_TARGETS` → the same refusal, the same no-dial proof; (c) configured, the job's payload present → the snapshot's exact keys (the task-01 shape — a rename fails a test); (d) configured, no payload yet → the `unreachable` stub (every count `null`); (e) the history route answered through `_history_batch` — points read straight out of the store with `query()` (a third path, in neither the loop nor the batch, fails the test — the phase-14 task-02 pin shape) for `gateway.in_conns@443` and a `gateway.out_conns@<host>` series, with an unknown name answering as an empty series.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the five pins above
|
||||||
|
- Coverage: **>90%** on the new routes
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] unconfigured (unset or unknown name) → `404 {"configured": false}` with the two spies recording nothing (pinned both arms)
|
||||||
|
- [ ] the never-answered stub is pinned (`unreachable: true`, every count `null`)
|
||||||
|
- [ ] the history route's points equal the store's rows for the `gateway.*` series; unknown names → empty series (pinned)
|
||||||
|
- [ ] the two GETs are in the read-only enumeration; no new POST exists (pinned)
|
||||||
|
- [ ] `uv run pytest` green, `uv run ruff check src tests` and `uv run pyright` clean
|
||||||
|
- [ ] no behavior change in completed work: the pre-existing snapshot/history routes answer byte-identical shapes
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 04 — The `/gateway` page: animated pipes
|
||||||
|
|
||||||
|
**Phase:** `29_gateway_dashboard_page` · **Source:** `TODO.md:3` — "I want it to show animated pipes with data coming in (IPs, ports, domains, etc.) and data going out (IPs, ports, domains, etc.)."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The `/gateway` page renders the joined payload (task 01) as animated pipes: one **inbound** pipe per configured frontend — `client → gateway` — labelled with the listen port, the `server_name` domains (haproxy's L4 frontends label their port only), the live connection count, and up to a few client addresses; and, fanning out from the gateway node, one **outbound** pipe per configured target — `gateway → target` — labelled with the target host:port (and `· by port` on a `by_port` match), its live count, and its source (the haproxy backend / nginx location / stream-map key it came through). The animation is **decorative at a fixed speed** — moving dashes whose direction is the data flow — because a count is not a rate (rule 2): the page never claims a tok/s-shaped number from it. No count is coloured by a verdict (the `net.*` precedent: the stylesheet section carries no `ok`/`warn`/`fail` token), and the three degradation arms render their own sentence: unconfigured ("not configured — set `GATEWAY_HOST`"), a 1.1.0 agent ("agent predates gateway support — install 1.2.0", dimmed, `n/a`), a never-answered host (dimmed `.stale`, `n/a`, the row kept, invariant 4).
|
||||||
|
|
||||||
|
- ASSUMPTION: the render is **one card per source family** (haproxy / nginx-https / nginx-stream / unmatched), each card a hand-rolled SVG: a left column of inbound pipe stubs, the gateway node mid-card (one rounded rect, its label the host name), a right column of outbound pipe stubs — the pipe is a `<path>` whose `stroke-dasharray`/`stroke-dashoffset` CSS animation runs only while the pipe has ≥ 1 live connection (a zero-connection pipe is a static faint line + the count `0`, because animating a pipe nobody is using would be a shape claiming a flow the count denies — the sparkline's "a shape is not a verdict" rule applied to motion); `prefers-reduced-motion: reduce` renders every pipe static (no animation at all, the repo's standing rule).
|
||||||
|
- ASSUMPTION: the labels are `textContent`-only (no `innerHTML` for anything upstream-provided — a config domain is upstream text; the `DashCharts.escapeHtml` idiom via the page's own escape), and every SVG pipe carries `role="img"` + an `aria-label` built from its inputs (the battery-glyph convention: the pipe carries information the markup must say in words) — e.g. `"inbound pipe, port 4443, nextcloud.reeseapps.com, 3 connections"`.
|
||||||
|
- ASSUMPTION: the page is the app's standing chrome — the masthead's sixth tab (task 05), the four `#nav-status` badges (the `gateway` job is the fifth dot, `app.js`'s one-dot-per-job rule), the theme toggle, the window chips **absent** (the page's history is the count trend — one `LineChart` per card, the `Charts` disclosure's `/system` shape: collapsed, lazy-built, one `/api/gateway/history` request naming every series the card could draw, `ariaLabel` on the plot), and the page's single `faint` footnote: the `by port` join sentence (the only why-line the operator needs to read a count).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/pages/gateway.html` — the page: the shared `<head>`/masthead (task 05's tab slot), the four badge container, the card grid (the four cards' skeletons with their empty-state sentences), the `#nav-status`, and no `id` the script does not reference (the orphan-`id` audit's shape).
|
||||||
|
2. `src/dashboards/web/static/page-gateway.js` — the page script (the `page-disks.js` idiom): `state` keyed per card; `renderPipes(card, pipes, direction)` — the SVG builder (the `DashCharts` escape discipline, the `aria-label` per the assumption, the zero-connection static arm, the `by port` sub-label); the degradation arms (the unconfigured / predates / never-answered sentences, the `.stale` dim, `n/a` for every count); the `Charts` disclosure's one-history-request rule (the `/system` shape: collapsed default, lazy build, a cached response from another window re-fetched); the `unmatched` card (the "N other connections" sentence — the operator's fact that the gateway talks to something the configs do not name).
|
||||||
|
3. `src/dashboards/web/static/dash.css` — the pipe rules: `.pipe` (the `<path>` stroke, the `stroke-dasharray`), `@keyframes pipe-flow` (the dashoffset drift), `.pipe.still` (the zero-connection arm), and the `@media (prefers-reduced-motion: reduce)` block naming **both** the animating and the still states (the phase-18 reduced-motion shape — naming only one leaves the other animating).
|
||||||
|
4. `tests/test_api.py` — the served-bytes pins: the pipe `@keyframes` + the reduced-motion rule (both states, inside the file's one such block); the zero-connection `.still` arm; the page's absence of any `ok|warn|fail` token in the gateway card section (the no-verdict pin, the phase-06 GPU shape); the `aria-label` on every `LineChart` the page builds (the standing chart-audit pin, extended); the orphan-`id` audit over the served page; the CDN-reference absence (invariant 5).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the served-bytes pins above (the repo has no JS test runner — a `.cjs` browser pass is the behavioural proof, task 06)
|
||||||
|
- Coverage: **>90%** on the new Python-side branches (the pins); the JS is pinned on served bytes + the browser pass
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the page renders the four cards with their pipes, labels (port / domains / host:port / `· by port` / counts / clients), and the `unmatched` card — against a fixture-driven dev store
|
||||||
|
- [ ] a zero-connection pipe is static (`.still`), a ≥ 1-connection pipe animates, `prefers-reduced-motion` renders all static (pinned + measured by task 06's browser pass)
|
||||||
|
- [ ] the three degradation arms print their own sentence, dimmed where the invariant says, every count `n/a` (pinned)
|
||||||
|
- [ ] no verdict token in the gateway card section; no CDN reference; every chart labelled (pinned)
|
||||||
|
- [ ] `uv run pytest` green, `uv run ruff check src tests` and `uv run pyright` clean, `node --check` on the new script
|
||||||
|
- [ ] no behavior change in completed work: the pre-existing pages render byte-identical
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 05 — The sixth tab, and the `/api/summary` pin
|
||||||
|
|
||||||
|
**Phase:** `29_gateway_dashboard_page` · **Source:** `TODO.md:3` — the page is one of the dashboards: the masthead's tab row gains `Gateway` on every signed-in page, and the overview's fleet probe names the new source
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The masthead's tab row — five tabs on all seven signed-in pages (`/`, `/system`, `/disks`, `/llm`, `/ups` + `/settings`/`/certs`'s no-`aria-current` five) — gains the sixth, `Gateway`, in one order on every page, and the pinned markup audits move from a five-tab assertion to a six-tab one (the pins are the contract; they move on purpose, with the browser measurement as the proof). `app.js`'s one-dot-per-job rule makes the `gateway` job the fifth `#nav-status` badge with no JS change (the container already exists; the job is what the badge reads). The `/api/summary` pin (task 02's additive block) is completed here with the page-side half: the overview renders the `gateway` block's one line (the "N inbound · M outbound" count, linked to `/gateway` via the block's own href), and the no-behavior-change control is the five pre-existing dashboards rendering as before.
|
||||||
|
|
||||||
|
- ASSUMPTION: the tab order is `Overview · System · Disks · LLM · UPS · Gateway` (the new tab last — a sixth tab inserted mid-row would move the `aria-current` targets the five pinned pages assert, and a moved tab is a broken-button read on every page at once).
|
||||||
|
- ASSUMPTION: the tab is `aria-current="page"` on `/gateway` only (the standing rule: the five dashboards carry it on their own page, `/settings`/`/certs` carry the five tabs with **no** `aria-current` — the sixth tab inherits the same shape on the two admin pages).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/pages/*.html` — the six tab pages (`index`, `system`, `disks`, `llm`, `ups`, `gateway`) gain the `Gateway` tab (the `aria-current` on `gateway.html` only); `settings.html` and `certs.html` gain the same sixth tab with no `aria-current` (the two admin pages' shape — the phase-20 task-04 "both links on both admin pages" rule extended to the tab row).
|
||||||
|
2. `tests/test_api.py` — the tab pins move: the five-tab assertion becomes the six-tab one (the `MARKUP_PAGES` loop's tuple gains `Gateway`, the order pin per the assumption, the `aria-current` pin per the assumption); the `test_every_page_links_to_both_admin_pages`-shape loop stays one loop over the pages (the phase-25 task-03 pin's shape — one order, one loop).
|
||||||
|
3. `src/dashboards/web/static/page-index.js` — the overview's `gateway` line: the block's one line in the fleet panel's shape (count + the `/gateway` link via the block's `href`), hidden-when-unconfigured (the `disks` block's invariant-3 shape — an unconfigured source never renders a row, never a red one).
|
||||||
|
4. `tests/test_api.py` — the `/api/summary` pin completed: the `gateway` block's keys pinned (task 02) **and** the overview line's rendering pinned on the served `page-index.js` (the line reads the block's keys — a rename fails a test, the `SLOT_FIELDS` device shape).
|
||||||
|
5. `data/logs/29-task05-tabs.cjs` — the browser measurement (the `25-task03-verify.cjs` shape): eight signed-in pages × two widths (1280 / 360) — six tabs present in order, the `aria-current` on `/gateway` only, the five pre-existing tabs' `aria-current` unchanged, the sixth `#nav-status` badge present and populated on `/gateway`, the 360 px dropdown lists all six, no document-level sideways scroll (phase 18's guarantee), screenshots `data/logs/masthead-29-*.png`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the served-markup pins (the moved tab audit, the `aria-current` pin, the overview-line pin)
|
||||||
|
- Real-browser check (invariant 10): the task-05 script is the proof — eight pages × two widths
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] six tabs on all eight signed-in pages in the pinned order, `aria-current` on `/gateway` only (pinned + measured)
|
||||||
|
- [ ] the `gateway` job is the fifth `#nav-status` badge (measured on `/gateway`, populated from `/api/health`)
|
||||||
|
- [ ] the overview renders the `gateway` line when configured and hides it when unconfigured (pinned)
|
||||||
|
- [ ] the 360 px dropdown lists all six tabs; no document-level sideways scroll (measured)
|
||||||
|
- [ ] `uv run pytest` green, `node --check` on the touched scripts, screenshots in `data/logs/`
|
||||||
|
- [ ] no behavior change in completed work: the five pre-existing dashboards' tab row is byte-identical apart from the added tab (the served-bytes pin proves it)
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 06 — Docs, and the live verification of the rendered pipes
|
||||||
|
|
||||||
|
**Phase:** `29_gateway_dashboard_page` · **Source:** `TODO.md:3` — the definition of done: UI changes verified in a real browser with a screenshot against **live** data (invariant 10), and the docs updated when a page, a config row, or a fleet fact changed
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The docs carry the new page (the `AGENTS.md` table row, the `config.py` bullet's `GATEWAY_HOST`, the metric-naming paragraph's `gateway.*` gauges, the `README.md` how-to row), and the phase's proof is the read-only live pass: the dev server (1.2.0 gateway host, post-phase-28) renders `/gateway` with the real topology — the six haproxy frontends, the nginx server blocks, the stream map — and the page's live counts **match a direct `/proc/net/tcp` read** on the gateway host (the join is proven end-to-end, not just against fixtures). The pass writes its transcript and screenshots to `data/logs/`.
|
||||||
|
|
||||||
|
- ASSUMPTION: the verification is **read-only** (the `tests/test_live_fleet.py` rule): the pass dials the dashboard's public routes over the session, reads `/proc/net/tcp` on the gateway host via the ssh alias, and compares — it starts no workload, sends no command, writes nothing to the fleet.
|
||||||
|
- ASSUMPTION: the count comparison is **port-bucketed, not exact-timestamped**: the pass reads the page's inbound count for port `P` and the host's ESTABLISHED-on-`P` count within one poll window, and asserts they are equal **or** differ by ≤ 1 (a connection can close between the two reads — the one-socket tolerance is the honest bound; a divergence > 1 fails the pass).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `AGENTS.md` — the page table row (`/gateway` — "which domains are coming in on which port, and where the traffic goes — the gateway host's nginx + haproxy configs joined to its live sockets" — source: "the same agents, `haproxy`/`nginx`/`sockets` sections of `GET /snapshot` (agent 1.2.0)"); the `config.py` bullet's `GATEWAY_HOST` line (default `gateway`, the gate property); the metric-naming paragraph's `gateway.in_conns@<port>` / `gateway.out_conns@<host>` gauge line (no rate, no verdict — the `net.*` sentence applied).
|
||||||
|
2. `README.md` — the page's how-to row (the `GATEWAY_HOST` setup, the not-configured arm, the 1.2.0 agent requirement) in the standing page-row shape.
|
||||||
|
3. `data/logs/29-task06-live.cjs` — the read-only live pass (the `24-task04-verify.cjs` shape): sign in, `GET /api/gateway/snapshot` (assert `configured: true`, `supported: true`, the six haproxy frontends, the nginx server blocks, the stream map's 17 entries), `GET /gateway` (the six tabs' page, the four cards rendered, the pipes' `aria-label`s present), then `ssh gateway-ops` → `cat /proc/net/tcp /proc/net/tcp6` → the port-bucketed count comparison per the assumption (inbound on each configured listen port; the `unmatched.inbound` tally against the port complement), and the `by_port` spot-check (one domain target's count equals the ESTABLISHED-on-that-port count).
|
||||||
|
4. `data/logs/29-task06-live.out` + screenshots `data/logs/29-task06-{gateway,card-haproxy,card-stream,charts-expanded}.png` — the transcript (the assertion table, the count comparison, **no** config body or token in it — invariant 6) and the screenshots (the phase's proof, invariant 10).
|
||||||
|
5. The 1.1.0 arm's live proof is the **dev-store** one (not a fleet rollback — a fleet arm is the operator's call): the dev store pointed at a 1.1.0-style fixture snapshot renders the "predates" arm, screenshot `data/logs/29-task06-predates.png`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Live (read-only, invariant 10): the task-06 pass is the phase's proof; the 1.1.0 arm is dev-store-driven
|
||||||
|
- The hermetic suite is re-run as the gate (`uv run pytest --cov`), not for this task's logic
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `AGENTS.md`'s page table, the `config.py` bullet, and the metric-naming paragraph name `/gateway` + `GATEWAY_HOST` + the `gateway.*` gauges (a grep over both files finds each)
|
||||||
|
- [ ] `README.md` has the page's how-to row
|
||||||
|
- [ ] the live pass's transcript records: the six frontends, the nginx blocks, the 17 map entries, the port-bucketed count comparison (divergence ≤ 1 on every compared port), the `by_port` spot-check
|
||||||
|
- [ ] the four screenshots + the predates-arm screenshot are in `data/logs/`, and the transcript carries no config body and no token (invariant 6, grepped)
|
||||||
|
- [ ] `uv run pytest --cov` holds the 94% floor, `bash .agents/validate.sh` green
|
||||||
|
- [ ] no behavior change in completed work: the nine pre-existing sources read `ok` in `./scripts/dev-server.sh status` and the five pre-existing dashboards render as before (the pass's control leg)
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
# Phase 30 — The gateway connection map: two config-derived tables, joined lines, green = in the config
|
||||||
|
|
||||||
|
**Source:** operator request, 2026-10-23 (post-phase-29 review of `/gateway`)
|
||||||
|
**Story:** n/a
|
||||||
|
**Context:** `src/dashboards/sources/gateway.py` (the join this phase rewrites), `agent/host_agent.py` **1.2.0** (its `haproxy` / `nginx` / `sockets` sections — the *only* upstream this phase reads, and it needs **no agent change**), `tests/fixtures/gateway_haproxy.cfg` + `gateway_nginx_main.conf` + `gateway_nginx_conf_d_http.conf` + `gateway_nginx_stream_d_stream.conf` + `gateway_proc_net_tcp{,6}.txt` (the verbatim captures the graph is pinned to), `src/dashboards/api.py` (`/api/gateway/*`, the `/api/summary` `gateway` block), `src/dashboards/alerts.py` (the reachability-only rule this phase changes by decision), `src/dashboards/web/pages/gateway.html` + `static/page-gateway.js` + `static/dash.css` (the pipe vocabulary), `tests/test_gateway_source.py`, `tests/test_api.py` (the served-bytes pins this phase deliberately rewrites).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`/gateway` becomes the **connection map** it was meant to be: **two tables — nginx and haproxy** — in which every row is a connection **that exists in a config file**, drawn as a line that physically joins its inbound end to its outbound end (haproxy straight across; nginx branching out of `:443` to the services the stream `map` routes to), each line **green because the config says it exists**, each line carrying the **switch** shape from the operator's sketch (built, visible, not yet wired). The socket table is demoted to a number beside the line instead of deciding its colour or its existence; the `stream map` card and the `unmatched` card are gone; anything the two configs do not describe is **not reported at all**, and the warnings that remain are **config-integrity** problems — a backend no frontend routes to, a `proxy_pass $var` with no `map` — surfaced as a warnings panel on the page and one alert row on `/`.
|
||||||
|
|
||||||
|
## Scope decisions taken with the operator (binding for this phase)
|
||||||
|
|
||||||
|
| # | Decision | Consequence |
|
||||||
|
|---|----------|-------------|
|
||||||
|
| 1 | **Green means "in the config."** | Every configured connection draws green. Colour is an **existence state**, not a verdict: `severity.py` still has no `gateway.*` row and no payload key carries `severity`. Red is the **off** state (a route switched out of the config), styled now, unreachable until the write path exists. |
|
||||||
|
| 2 | **Motion means "a socket is open this instant."** | The dash animates while that pipe's live count is ≥ 1 and sits still otherwise. No rolling/recent-peak derivation, no speed tied to volume. |
|
||||||
|
| 3 | **The switch is built, not wired.** | Real `<button role="switch">` per connection, `aria-checked="true"` (in config = on), `aria-disabled="true"` (not `disabled`, which would drop it from the a11y tree), no click handler, a `title` naming why. The UPS controls are the precedent for a gated control; the write path is a **later phase**. |
|
||||||
|
| 4 | **Live counts stay, demoted.** | The inbound trunk carries the real ESTABLISHED count + client addresses; the 24 h per-card chart stays; the header keeps tallies. A leaf endpoint written as a **name** shows `—` (never `0`, never a fake `n/a` column). |
|
||||||
|
| 5 | **Only the unambiguous socket join survives.** | Inbound = the socket's local port is a configured listen port. Outbound = the socket's remote `(address, port)` **equals a target the config wrote as an address**. `by_port`, `unmatched.inbound`, `unmatched.outbound`, `unmatched.groups`, `partial`, `unmatched_listeners` are **deleted**. |
|
||||||
|
| 6 | **Anything outside the two configs is ignored.** | `9120` (this app's own agent), `2222`, DHCP/mDNS/whatever else listens is neither drawn, nor counted, nor warned about. `sockets.listens` / `sockets.established` stay in the payload as the kernel's raw totals for programmatic consumers only. |
|
||||||
|
| 7 | **The warnings list is config integrity.** | Seven kinds (`03_config_warnings.md`) plus three explicit refusals where the payload cannot tell broken from merely unreadable; all derived from config text, all actionable. Panel on `/gateway`, and **one** `warn` row in `/api/summary`'s `alerts` — which changes the "a `gateway` alert is reachability only" rule in `alerts.py` and in `AGENTS.md`. |
|
||||||
|
|
||||||
|
**Out of scope:** any change to `agent/host_agent.py` (1.2.0 already publishes everything needed, including `upstreams` and `maps`), any write path or config edit, any new dependency, any new sample family (the existing `gateway.in_conns@<port>` / `gateway.out_conns@<host>:<port>` names stay, so stored history keeps its meaning).
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
- `28_gateway_agent_proxy` (complete) — agent 1.2.0's `haproxy` / `nginx` / `sockets` sections and the captured fixtures.
|
||||||
|
- `29_gateway_dashboard_page` (complete) — the `gateway` poller job, `/api/gateway/*`, the page's chrome, the six-tab masthead. This phase **reshapes** that payload and page; it does not remove the job, the routes, the tab, or the gauges.
|
||||||
|
- `27_mobile_tables_and_audit` (complete) — the 360 px audit script `data/logs/27-task02-mobile-audit.cjs` must still exit 0 with `/gateway`'s new rows in it.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
|
||||||
|
1. `01_the_connection_graph.md` — `sources/gateway.py`: the config-derived graph — `listeners[]` + `connections[]` (haproxy `frontend → backend → server`, nginx vhost `server_name → proxy_pass`, stream `proxy_pass $var → map` entries, `proxy_pass <pool> → upstream` servers), with no counts yet.
|
||||||
|
2. `02_the_join_simplified.md` — the surviving socket join (inbound by local port, outbound by exact address only), the shared-endpoint rule, the deletion of `outbound` / `by_port` / `unmatched` / `partial`, the gauges re-pointed, and `/api/summary`'s `gateway` block plus the overview's one line reshaped in the same commit.
|
||||||
|
3. `03_config_warnings.md` — `warnings()`: the seven config-integrity kinds (and three refusals), empty on this fleet's captured configs (a pinned assertion), plus one `warn` overview row from `alerts.py`.
|
||||||
|
4. `04_page_two_tables.md` — `gateway.html` + `page-gateway.js` + structural CSS: exactly **two** panels (nginx, haproxy), trunk → leaf rows, the warnings panel on the shared `ul.alerts` vocabulary, four header tallies, the degradation arms kept.
|
||||||
|
5. `05_lines_and_switches.md` — `dash.css` + `page-gateway.js`: the connecting line (straight across for haproxy, branched for nginx), the green `--gate-on`/`--gate-off` existence pair, `live` motion, the switch built to the sketch and pinned as unwired. The `ok|warn|fail`-in-the-pipe-section pin is **restated**, not deleted.
|
||||||
|
6. `06_docs_and_verify.md` — `AGENTS.md` (the `sources/gateway.py`, `api.py`, `alerts.py`, metric-naming, UI-conventions and gap-13 passages) + `README.md`, the browser pass at 1440 px and 360 px with screenshots, the mobile-audit re-run, and the read-only live verification against the real ingress.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Unit: `tests/test_gateway_source.py` is **rewritten around the graph** — every connection the captured configs declare is present and no other; the join identity is restated as *"the inbound counts sum to the ESTABLISHED sockets whose local port a config names"* (the old `inbound + outbound + unmatched == the table` identity dies with `unmatched`); `socket.getaddrinfo` stays spied and **never called**; the six warning kinds each get a synthetic broken config and the captured configs pin **zero** warnings. `tests/test_api.py`'s gateway pins are rewritten in the same tasks that change the payload/page — including `GATEWAY_PAGE_FIELDS`, `GATEWAY_FIELDS`, the stub/answer key-parity pin, the no-verdict/no-window walk, and the colour pin (restated per decision 1).
|
||||||
|
- Integration: the page renders from a fixture-driven dev store; the warnings panel appears only when a warning exists; the chart disclosure stays collapsed-lazy-one-request-per-card.
|
||||||
|
- Coverage: **>90%** on new/modified Python (`sources/gateway.py`, `alerts.py`, `api.py`).
|
||||||
|
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's) and `bash .agents/validate.sh` is green, coverage floor included.
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**. For this phase that specifically means: `AGENTS.md`'s `sources/gateway.py` bullet (the payload keys, the join, the deleted `unmatched`/`by_port`/`partial`), its `api.py` `/api/gateway/*` paragraph and the `/api/summary` `gateway` block field list, its `alerts.py` bullet (the gateway alert is no longer reachability-only), the `/gateway` UI-conventions paragraph (two tables, joined lines, green = in the config, the unwired switch), the metric-naming paragraph, and **gap 13** (the clipped-pipe-label gap, which the new row layout either resolves or restates honestly).
|
||||||
|
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `/gateway` serves and renders **exactly two** cards, `nginx` and `haproxy`; no `stream map` card, no `unmatched` card, no third table.
|
||||||
|
- [ ] Every haproxy row draws its line **straight across** from `:<bind port>` to its `server` target; the nginx `:443` trunk **branches** to each `map` entry (16 on this fleet) and each http vhost trunk to its `proxy_pass` target — measured in the browser pass, not asserted in prose.
|
||||||
|
- [ ] Every configured line is **green**; it animates only while its live count ≥ 1; both states still static under emulated `prefers-reduced-motion: reduce`. No `severity` key at any depth of the payload; no `severityFrom`/`severityFor` in the page script.
|
||||||
|
- [ ] One switch per connection, `role="switch"` + `aria-checked="true"` + `aria-disabled="true"`, no click handler bound, `title` naming the missing write path — and a test that fails if it becomes clickable without a handler or stops saying it is unwired.
|
||||||
|
- [ ] `POST` nothing: the verb-enumeration pin still finds `/api/ups/command` as the app's only upstream write, and `/gateway` dials only `/api/gateway/snapshot` + `/api/gateway/history`.
|
||||||
|
- [ ] `GET /api/gateway/snapshot` on the live ingress: `warnings == []`, every one of the 6 haproxy frontends connected to its `server`, `:443` connected to the 17 `map` entries, `:4443`/`:8443` connected to their `proxy_pass` targets, and no `unmatched` / `by_port` / `partial` key anywhere.
|
||||||
|
- [ ] A `gateway` `warn` row appears in `/api/summary.alerts` **iff** `warnings` is non-empty, and disappears when they are; a stale/stopped `gateway` job still alerts exactly as before.
|
||||||
|
- [ ] `uv run pytest` (incl. `--cov` ≥ 94 floor), `uv run ruff check src tests agent`, `uv run pyright`, `node --check` on the changed scripts, `bash .agents/validate.sh` — all green.
|
||||||
|
- [ ] `data/logs/30-*.png` screenshots at 1440 px and 360 px; `data/logs/27-task02-mobile-audit.cjs` still exits 0 with `/gateway` in its sweep.
|
||||||
|
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass).
|
||||||
|
- [ ] No behaviour change in completed work: the other five dashboards and `/settings`/`/certs` render as before; `host:*`, `disks:*`, `llm:*`, `nut` still read `ok` in `./scripts/dev-server.sh status` (10 sources, `gateway` included).
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
# Task 01 — The connection graph: `listeners` and `connections`, derived from config text only
|
||||||
|
|
||||||
|
**Phase:** `30_gateway_connection_map` · **Source:** operator request 2026-10-23 — "the dotted line connections need to connect to their counterparts… for nginx you'll need to branch the lines from `:443` to the services they connect to… this is just 'exists in config therefore it is a connection'"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`sources/gateway.py` learns the thing it never published: **which outbound target belongs to which inbound listener**. It gains `connections[]` — one entry per connection the two configs actually declare, each carrying its inbound port, its outbound endpoint, and the config's own selector for it (a haproxy backend name, an nginx `server_name`, a stream `map` key) — and renames `inbound` to `listeners`, because "inbound" was a direction and this is a thing the page draws a trunk for. Task 01 adds the graph and renames; task 02 deletes the guessing socket join. **No agent change**: everything needed is already in agent 1.2.0's `haproxy` / `nginx` sections, including `upstreams` and `maps`.
|
||||||
|
|
||||||
|
## The graph, stated exactly
|
||||||
|
|
||||||
|
**`listeners[]`** — one per configured listen port, same claim rule as today's `inbound`, same `metric` string:
|
||||||
|
|
||||||
|
```
|
||||||
|
{"port": 443, "source": "nginx", "domains": ["_"], "connections": 0|int|None,
|
||||||
|
"clients": [...], "metric": "gateway.in_conns@443"}
|
||||||
|
```
|
||||||
|
|
||||||
|
**`connections[]`** — one per `(source, in_port, out_host, out_port)`, selectors merged, in config order:
|
||||||
|
|
||||||
|
```
|
||||||
|
{"id": "nginx:443->reeseapps-homepage.reeselink.com:443",
|
||||||
|
"source": "nginx", # the card it belongs to: "haproxy" | "nginx"
|
||||||
|
"in": {"port": 443},
|
||||||
|
"out": {"host": "reeseapps-homepage.reeselink.com", "port": 443,
|
||||||
|
"name": None, # the haproxy `server` name when there is one
|
||||||
|
"kind": "map", # "backend" | "server_name" | "pool" | "map"
|
||||||
|
"selectors": ["homepage.reeseapps.com"], # the config's own word(s) that reach this target
|
||||||
|
"address": False}, # True when the config wrote an IP literal
|
||||||
|
"connections": None, # filled by task 02; task 01 always None
|
||||||
|
"metric": "gateway.out_conns@reeseapps-homepage.reeselink.com:443"}
|
||||||
|
```
|
||||||
|
|
||||||
|
Derivation, per config family:
|
||||||
|
|
||||||
|
1. **haproxy** — `frontend → routes → backend → server`. For every frontend row (one per `bind` port, `routes` = `default_backend` then each `use_backend` name, which `haproxy_frontends()` already builds — note the agent's own `frontend.backends` holds **only** the `use_backend` names and keeps `default_backend` separate, which is why that insert exists) and every `server` of each routed backend block: a connection with `in.port = the bind port`, `out = {host: server.target, port: server.port, name: server.name, kind: "backend", selectors: [backend name]}`. This needs `haproxy_targets()` to stop dropping the backend it was declared in — change its rows to `{name, host, port, backend}`.
|
||||||
|
2. **nginx http vhost** — for every `server` **block**, for every `proxy_passes` entry with a concrete `target` and a port: one connection **per `listen` port of that block**, `out.selectors = the block's server_names` verbatim (the catch-all `_` included), `kind: "server_name"`.
|
||||||
|
3. **nginx pool** — a `proxy_pass http://<name>` with **no port** names an `upstream` block. When an `upstreams[]` entry has that name: one connection **per upstream `server` host:port**, `kind: "pool"`, `selectors: [<pool name>]`. When none does: **no connection** (that is task 03's `nginx_pool_undeclared`).
|
||||||
|
4. **nginx stream map** — a `proxy_pass` with `variable` set: find the `maps[]` block whose `variable` matches after `lstrip("$")`. For each of its `entries` with a target, one connection **per `listen` port of the block that wrote the `proxy_pass`**, `kind: "map"`, `selectors: [entry.key]`. The key `default` is kept **verbatim** — it is the config's own word for "everything nobody mapped", not a guess. A map no `proxy_pass` names contributes nothing (the `$connection_upgrade` rule, unchanged), and an entry with a null target contributes nothing (task 03 warns about it).
|
||||||
|
5. **Never** invent a connection for what the payload cannot decide: a `proxy_pass` with neither `target` nor `variable` (a unix socket, a bare word) yields nothing and **no warning**; a haproxy frontend whose only routing is a fetch expression (`use_backend %[req.hdr(host),map_dom(...)]`, which the agent reports as no name) yields its **listener** and **no connections** — the page draws that trunk alone, and `AGENTS.md` rule 2 is why it is not a warning: "no name" and "no routing directive" are one value in this payload.
|
||||||
|
|
||||||
|
Merge rule: identical `(source, in_port, host, port)` is **one** connection whose `selectors` are the union in first-seen order (one vhost's two names arriving at one pool is one line, not two). Never merge across `in_port` — `:4443 → keycloak:8080` and `:8443 → keycloak:8080` are two pipes on two socket spaces (the `minecraft` 25565/25566 lesson).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/sources/gateway.py`
|
||||||
|
- `haproxy_targets()`: rows gain `backend`; docstring says the backend name is what makes the frontend→server join possible.
|
||||||
|
- New `nginx_server_blocks(section)` returning one row per **server block** — `{ports: [int], names: [str], proxies: [entry]}` — so the block's `listen`, `server_name` and `proxy_pass` stay associated (today's `nginx_listen_ports()` merges names across blocks and loses exactly the fact this task needs; keep it if the echo still needs it, or derive the echo from `nginx_server_blocks()` — do not keep two readers of the same block).
|
||||||
|
- New `config_connections(snapshot_haproxy, snapshot_nginx) -> list[dict]` implementing rules 1–5 + the merge, pure and clock-free, with the module docstring gaining a **"The connection graph"** section stating the four derivations, the merge rule, and the two refusals (rule 5).
|
||||||
|
- `parse()`: rename the `inbound` payload key to `listeners` (its rows keep every field they had, including `metric`), add `payload["connections"] = config_connections(...)`, and keep `outbound` and `unmatched` **untouched for this task** (task 02's business). `connections[].connections` stays `None` here — say so in a comment so task 02's owner sees where the count lands.
|
||||||
|
- `_skeleton()`: add `"listeners": []` and `"connections": []` (replacing `"inbound": []`) so `stub()` grows the same key set — the key-parity test fails otherwise.
|
||||||
|
2. `src/dashboards/api.py` — the mechanical rename in every reader of the payload: `/api/gateway/snapshot` (it passes the payload through plus `states`), the `/api/summary` `gateway` block (its `inbound`/`outbound` totals now read `payload["listeners"]` / `payload["outbound"]`; **keep the block's field names as they are** — task 04 reshapes that block deliberately, and this task must not change `/api/summary`'s shape).
|
||||||
|
3. `src/dashboards/web/static/page-gateway.js` — the minimum that keeps the page honest while task 05 rewrites it: `familyPipes()` reads `snapshot.listeners` instead of `snapshot.inbound`, and `renderHeader()`'s inbound tally does too. Nothing else about the drawing changes.
|
||||||
|
4. `tests/test_gateway_source.py` — keep every existing assertion working under the rename, and **add the graph pins** (this task's real product):
|
||||||
|
- `test_every_connection_the_captured_configs_declare_is_drawn` — against the real `gateway_snapshot` fixture, **32** connections total, asserted as the **set** of `(in_port, out.host, out.port)` so a missing or extra one fails by name:
|
||||||
|
- **6** haproxy, one `server` each: `:22 → reeseapps-gitea.reeselink.com:2222` (`gitea`), `:2000 → stackexpected-gitea.reeselink.com:2222`, `:5201 → web.reeselink.com:5201`, `:8080 → unifi-external.reeselink.com:8080`, `:25565 → minecraft.reeselink.com:25565`, `:25566 → minecraft.reeselink.com:25566`
|
||||||
|
- **17** on nginx `:443` from the stream map — the 16 `*.reeselink.com:443` entries **plus** `default → 127.0.0.1:4443`
|
||||||
|
- **6** on nginx `:4443` (nextcloud `:11000`, gitea `:3000`, jellyfin `:8096`, bricktracker `:3333`, keycloak `:8080`, emmaleaf `:80`) and **3** on `:8443` (nextcloud `:11001`, `127.0.0.1:8081`, keycloak `:8080`), each with its own `server_name` in `selectors`
|
||||||
|
- `:80` (nginx, `server_name _`) is a **listener with no connection** — it declares no `proxy_pass`. That is this fleet's live instance of refusal rule 5 and it gets its own assertion: the trunk exists, `connections` names port 80 nowhere.
|
||||||
|
- `test_a_haproxy_connection_carries_the_backend_and_the_server_name` — `out.name == "gitea"`, `out.kind == "backend"`, `selectors == ["gitea-2222"]`.
|
||||||
|
- `test_the_stream_default_entry_is_kept_verbatim` — a connection whose `selectors == ["default"]` to `127.0.0.1:4443`, `out.address is True`.
|
||||||
|
- `test_a_pool_names_its_upstream_servers_and_an_undeclared_pool_draws_nothing` — synthetic nginx config with `upstream backend_pool { server 10.0.0.1:3000; server 10.0.0.2:3000; }` + `proxy_pass http://backend_pool;` → two connections `kind == "pool"`; and `proxy_pass http://no_such_pool;` → no connection at all (and **no** invented host).
|
||||||
|
- `test_two_names_one_target_one_port_is_one_connection` (selectors merged, first-seen order) and `test_the_same_target_on_two_listen_ports_is_two_connections` (the `admin.keycloak` / `keycloak` pair from the real `http.conf`).
|
||||||
|
- `test_a_frontend_that_routes_by_expression_is_a_listener_with_no_connection` and `test_a_proxy_pass_with_neither_target_nor_variable_draws_nothing` — the two refusals.
|
||||||
|
- `test_no_connection_is_ever_invented_for_a_port_no_config_names` — walk every `connections[].in.port` and assert it is in `listeners`.
|
||||||
|
5. `tests/test_api.py` — the rename only: `GATEWAY_PAGE_FIELDS`' `inbound…` paths → `listeners…`, the stub/answer key-parity pin's expected key set, `test_the_gateway_stub_and_the_answer_share_one_key_set`, `test_the_gateway_snapshot_carries_no_verdict_and_no_window` (walk `connections` too), `test_gateway_snapshot_answers_the_pipes_the_projection_joined`, and `/api/summary`'s `GATEWAY_FIELDS` **unchanged in names** (only its derivation moved).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the pins above; `uv run pytest tests/test_gateway_source.py tests/test_api.py -q` then the full suite. Coverage **>90%** on `sources/gateway.py`'s new branches (all four derivations, both refusals, the merge, the undeclared-pool arm).
|
||||||
|
- `uv run ruff check src tests agent && uv run pyright` clean; `node --check src/dashboards/web/static/page-gateway.js`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GET /api/gateway/snapshot` on the dev server (live ingress) returns `listeners` (10 rows today), `connections` (**32** on the captured configs — 6 haproxy + 17 stream-map + 9 vhost; the *test* asserts the captured fixture's set, the live read is a recorded observation, not an assertion, because `/etc/nginx` on a router changes), and `outbound` still present.
|
||||||
|
- [ ] Every connection's `in.port` appears in `listeners`; every haproxy connection names its backend; every stream connection names its map key; `default` survived.
|
||||||
|
- [ ] `stub()` and `parse()` answer the same key set (pinned).
|
||||||
|
- [ ] `uv run pytest` **100% passing**, coverage not down; ruff + pyright + `node --check` clean.
|
||||||
|
- [ ] `README.md`/`AGENTS.md` — no user-visible change yet (the page still draws the old cards); note nothing here, task 04/05 do the doc updates.
|
||||||
|
- [ ] No behaviour change in completed work: the other pages byte-identical, the 10 sources `ok`.
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
# Task 02 — Delete the guessing socket join: inbound by port, outbound by exact address, nothing else counted
|
||||||
|
|
||||||
|
**Phase:** `30_gateway_connection_map` · **Source:** operator request 2026-10-23 — "This isn't live tracking, this is just 'exists in config therefore it is a connection'… I don't know what the unmatched section is" + the two follow-up decisions: *"exact-address match stays"* and *"ignore anything not in haproxy or nginx configs"*
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The socket table stops deciding the page's shape. `sources/gateway.py` keeps **exactly two** sound attributions — a socket's **local port** names a configured listener (inbound), and a socket's remote `(address, port)` **equals a target the config wrote as an address** (outbound) — and ignores every other socket: not counted, not totalled, not warned about. `outbound[]`, `unmatched`, `unmatched.groups`, `partial`, `unmatched_listeners`, `by_port` and `MAX_GROUP_HOSTS` leave the payload; `AGENTS.md`'s "the join is the whole honesty of the module" passage is rewritten in task 06 against this file.
|
||||||
|
|
||||||
|
**Why the deleted arms went** (state this in the module docstring, because the reason is the rule): a config writes `reeseapps-brain.reeselink.com:443` and `/proc` reports `10.1.1.106:443`. Nothing in this stack resolves a name (pinned by the `socket.getaddrinfo` spy), and 16 map entries share port 443, so the old `by_port` arm could only credit a port — which the module itself refused to attribute and dumped in `unmatched.outbound`. On the live router that residual was **mostly this app talking to itself**: the "listeners no config names" were `9120` (the dashboards agent's own port) and `2222`. A residual that is mostly other software is not a signal, and a card of it teaches the operator to ignore the panel.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/sources/gateway.py`
|
||||||
|
- `join_sockets(listeners, endpoints, established)` — rewrite to the two arms above and **return nothing**; it stamps counts in place. Signature docs state the new identity: *"every socket is counted at most once; inbound + outbound ≤ `established`; a socket that matches neither is not counted at all, by decision."* Keep the `continue`-after-count discipline (a loopback hop is two `/proc` rows and must not be drawn down one pipe twice).
|
||||||
|
- Build `endpoints: dict[(host, port), {"connections": int, "routes": int}]` from the **address-written** `connections` only (`out.address is True`); `routes` counts how many connections claim the endpoint. After the pass, stamp `conn["connections"] = endpoint["connections"]` and `conn["shared"] = endpoint["routes"] > 1`. A name-written connection keeps `connections: None` — the truth, not a zero.
|
||||||
|
- Delete: `outbound` / `_outbound_pipe()` / `_groups()` / `MAX_GROUP_HOSTS` / `live_listen_ports()` (its only job was the unmatched direction test) / the `unmatched` key from `_skeleton()`. Add `payload["endpoints_live"]`? **No** — nothing needs it; the page reads the count off each connection.
|
||||||
|
- `samples()`: gauges from `listeners[].metric` (`gateway.in_conns@<port>`) and `connections[].metric` (`gateway.out_conns@<host>:<port>`) whenever the value is a number. Consequence to state in the docstring **and** in task 06's `AGENTS.md` pass: **name-keyed `gateway.out_conns@<domain>:<port>` series stop being written**, because the number they carried was the port's, not the target's; existing rows age out under the unchanged retention policy, so no schema or prune change (invariant 9).
|
||||||
|
- Module docstring: replace "The join, and where its honesty lives" with the two surviving arms + the ignore rule + the loopback-hop note; keep "No DNS, ever" and "No verdict" verbatim (they are unchanged rules).
|
||||||
|
2. `src/dashboards/api.py`
|
||||||
|
- `/api/gateway/snapshot`: passes the payload through plus `states` — update to the new key set (drop `unmatched`), nothing else.
|
||||||
|
- `/api/summary`'s `gateway` block becomes exactly: `{configured, host, href, supported, unreachable, listeners, connections, live_in, live_out, map_targets, states}` — `listeners`/`connections` are the row counts, `live_in`/`live_out` are the sums of the counts that answered (**`None`, never `0`,** when the socket table did not answer or nothing answered), `map_targets` stays the stream-map entry count, `href` still comes from `alerts.HREF`. `unmatched`, `unmatched_listeners` and `partial` are **deleted**. (`warnings` joins this block in task 03.)
|
||||||
|
- `/api/config`'s `gateway` row is unchanged (it never named a pipe field).
|
||||||
|
3. `src/dashboards/web/static/page-index.js` — the one line under the fleet table reads the new block: `gateway <listeners> ports · <connections> routes · <live_in> live`, every figure `n/a` when the block says `null`, the shared `.stale` dim when `unreachable`, and **nothing at all** when `configured: false` (invariant 3, unchanged). `href` comes from the block; the page invents no route and no number.
|
||||||
|
4. `src/dashboards/web/static/page-gateway.js` — **compatibility only**, so the page stays truthful while task 04 rebuilds it: the outbound column reads `snapshot.connections` filtered by `source` (map each to the shape the existing row renderer wants, or inline a small renderer — do not add new visual vocabulary here), the `by port` chip and its `title` are **deleted**, the `unmatched` card is deleted from the JS and its `<article>` from `gateway.html`, the header's `unmatched` tally cell is deleted, and `sumConnections()` reads `listeners` / `connections`. Any deleted id must leave no orphan container (the orphan-`id` audit).
|
||||||
|
5. `tests/test_gateway_source.py` — rewrite the join pins:
|
||||||
|
- **delete** `test_a_domain_target_is_matched_on_port_alone_and_labelled_by_port`, `test_a_port_shared_by_several_names_resolves_none_of_them`, `test_the_group_host_list_is_bounded_and_reports_the_real_total`, `test_a_listening_port_no_config_claims_stays_visible`, `test_a_connection_on_a_port_no_config_names_is_reported_not_dropped`.
|
||||||
|
- **add** `test_a_name_target_is_never_credited_for_a_socket_on_its_port` (the 16-way `:443` fan-out keeps `connections: None`, and the sockets are not counted anywhere — assert the sum over `connections` is unchanged by their presence), `test_a_port_no_config_names_is_neither_counted_nor_reported` (add a LISTEN row on 9120 to a synthetic table: nothing in the payload mentions it), `test_an_endpoint_written_as_an_address_is_matched_exactly_and_carries_its_count` (no `by_port` key exists any more — assert its absence), `test_two_connections_sharing_an_endpoint_say_so_and_repeat_the_count_never_split_it` (`shared: true`, both carry the endpoint's number), `test_the_loopback_hop_counts_once_on_each_side` (`:4443` inbound and `127.0.0.1:4443` outbound each take their own `/proc` row, and no row is counted twice).
|
||||||
|
- **restate** `test_the_live_counts_reconcile_with_the_socket_table` as: every inbound count's port is a configured listener, `sum(listeners) + sum(endpoint counts) <= established`, and `sockets.listens` / `sockets.established` equal the fixture's raw row counts. Keep `test_no_join_ever_asks_a_resolver` exactly as it is.
|
||||||
|
6. `tests/test_api.py` — `GATEWAY_PAGE_FIELDS` and `GATEWAY_FIELDS` to the new key sets; the stub/answer key-parity pin; `test_the_gateway_snapshot_carries_no_verdict_and_no_window` walks `connections` too (recursive: no `severity`, no `window_seconds` at any depth); `test_the_summary_gateway_block_is_the_payloads_sums_not_the_pages_arithmetic`, `…_is_the_stub_of_a_source_that_never_answered`, `…_of_the_captured_ingress_reads_the_live_join`; `test_a_dead_gateway_host_is_one_overview_line_not_three` (the `disks`/`gateway` echo-suppression rule is unchanged — a dead host's `gateway` job still adds nothing new); `test_the_gateway_counts_grow_no_verdict_alert_even_with_verdicts_in_the_payload` now injects `severity` into `listeners` **and** `connections`; the overview-line pin `test_the_overview_line_reads_the_gateway_block_it_claims` resolves `GATEWAY_FIELDS` against a live document and `page-index.js`'s readers; `/gateway`'s markup pins for the deleted unmatched card and `by port` chip (the **absence** of both, now).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the pins above; the full suite plus the **live** read (`uv run pytest -m live --live`) confirms `gateway` still reads `ok` and the snapshot's `listeners` counts match `ss -tan` on the router for configured ports.
|
||||||
|
- Coverage: **>90%** on `sources/gateway.py` (both arms, the shared-endpoint arm, the sockets-silent arm), `api.py`'s changed block, `page-index.js` pinned on served bytes.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `grep -rn "unmatched\|by_port\|MAX_GROUP_HOSTS" src/ tests/` finds nothing outside a comment explaining what was removed and why.
|
||||||
|
- [ ] `GET /api/gateway/snapshot` (live): no `unmatched`, no `outbound`, no `by_port`; `connections[].connections` is a number **only** for the address-written endpoints (`127.0.0.1:4443`, `127.0.0.1:8081` today) and `null` for every name.
|
||||||
|
- [ ] `GET /api/summary` → `gateway` block has exactly the 11 fields listed above; `unmatched` / `unmatched_listeners` / `partial` are gone; the overview page's one line prints ports / routes / live and vanishes when unconfigured.
|
||||||
|
- [ ] Nothing in the payload mentions port `9120` or any other port the two configs do not name (live check + pinned).
|
||||||
|
- [ ] `uv run pytest` **100% passing** incl. `--cov` ≥ 94 floor; ruff + pyright + `node --check` on both changed scripts.
|
||||||
|
- [ ] `AGENTS.md`'s `sources/gateway.py` / `api.py` `/api/gateway/*` / `/api/summary` passages no longer describe `unmatched`/`by_port`/`partial` (full rewrite lands in task 06; this task must not leave a sentence that **contradicts** the shipped payload).
|
||||||
|
- [ ] No behaviour change in completed work: the five other dashboards and the two admin pages unchanged, 10 sources `ok`.
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
# Task 03 — Config-integrity warnings: seven kinds, from the config text, on the page and one row on `/`
|
||||||
|
|
||||||
|
**Phase:** `30_gateway_connection_map` · **Source:** operator request 2026-10-23 — "I don't know what the unmatched section is, there should be nothing unmatched, but if there is that should be called out in a **list of warnings to address** like other things in this dashboard project" + the follow-up decision *"Ignore anything not in haproxy or nginx configs"* and *"Config integrity, yes"*
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
With everything outside the two configs ignored (task 02), the only warning left worth an operator's attention is a config that **does not connect to itself**. `sources/gateway.py` gains `warnings(snapshot)` — seven kinds, each derived from config text alone, each actionable — the payload carries them, `/api/summary` counts them, and `alerts.py` raises **one** `warn` row for them. That row is the deliberate change to the documented "a `gateway` alert is reachability only" rule, and task 06 rewrites that rule in `AGENTS.md` against this file.
|
||||||
|
|
||||||
|
## The seven kinds
|
||||||
|
|
||||||
|
Each warning is `{"id", "kind", "subject", "message"}`; `id = f"{kind}:{subject}"` is the page's stable DOM key; `message` is operator-facing prose written in this file's voice, carrying the config's own words verbatim.
|
||||||
|
|
||||||
|
| kind | condition | message shape |
|
||||||
|
|------|-----------|---------------|
|
||||||
|
| `haproxy_route_dangling` | a frontend's `default_backend` or any `use_backend` name no `backend` block declares | `frontend haproxy-22 routes to backend "gitea-2222", which no backend block declares` |
|
||||||
|
| `haproxy_backend_orphan` | a backend that declares ≥ 1 `server` and appears in no frontend's routes | `backend "old-tunnel" has 2 servers and no frontend routes to it` |
|
||||||
|
| `haproxy_backend_empty` | a backend block with no `server` line at all | `backend "gitea-2222" declares no server — it routes nowhere` |
|
||||||
|
| `nginx_server_unlistened` | an nginx `server` block with `server_name` and/or `proxy_pass` and **no** `listen` | `nginx server "nextcloud.example.com" proxy_passes but never listens` |
|
||||||
|
| `nginx_pool_undeclared` | a `proxy_pass http://<name>` with no port and no matching `upstream` block (task 01 draws no connection for it) | `nginx listen 4443 proxy_passes to pool "backend_pool", which no upstream block declares` |
|
||||||
|
| `nginx_variable_unmapped` | a `proxy_pass $var` with no `map` block defining that variable | `nginx listen 443 proxy_passes $targetBackend, which no map block defines` |
|
||||||
|
| `nginx_map_route_missing` | an entry in a **referenced** map whose target is null | `map $targetBackend entry "x.example.com" names no address to route to` |
|
||||||
|
|
||||||
|
**Three deliberate non-warnings** (each gets a comment where the check would have gone, because the absence is the rule):
|
||||||
|
|
||||||
|
- an nginx `server` block with a `listen` and no `proxy_pass` — it may be a static or redirect vhost, and this payload cannot tell the two apart, so a warning would invent a fault;
|
||||||
|
- a haproxy `frontend` whose only routing is a fetch expression (`use_backend %[req.hdr(host),map_dom(…)]`, which agent 1.2.0 reports as **no name**) — "no name" and "no routing directive" are one value on the wire, so warning about it blames a config this agent cannot read; making this distinguishable is an agent change (a `routes_dynamic` flag), not a source-level guess;
|
||||||
|
- a `proxy_pass` with neither `target` nor `variable` (a unix socket, a bare word) — legitimate, undrawable, not broken.
|
||||||
|
|
||||||
|
A section is judged **only** when its own `available` is `True`. `predates`, `no file`, and `unreadable config` produce **no** warnings — a host that cannot be read has not been found guilty of anything (invariant 3).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/sources/gateway.py`
|
||||||
|
- New pure `warnings(haproxy_section, nginx_section) -> list[dict[str, str]]` implementing the table above with the three refusals, deterministic order (haproxy kinds in config order, then nginx kinds in config order), deduplicated by `id`.
|
||||||
|
- `parse()`: `payload["warnings"] = warnings(haproxy, nginx)` when supported, `[]` in the predates arm and whenever the socket/config halves did not answer; `_skeleton()` gains `"warnings": []` so `stub()` matches (the key-parity pin).
|
||||||
|
- Docstring: a short **"What this module warns about"** section — the seven kinds, the three refusals, and the sentence *"a warning here is a config that does not connect to itself; nothing outside the two configs is reported at all."*
|
||||||
|
2. `src/dashboards/alerts.py`
|
||||||
|
- `build(...)` gains `gateway_payload: Mapping[str, Any] | None = None` (keyword, last, defaulted, so every existing caller and test keeps working).
|
||||||
|
- One row, emitted **only when the payload exists, the job is currently answering, and `warnings` is non-empty**: `{id: "gateway:config", severity: "warn", source: "gateway", message: f"{n} proxy config {problem/problems} to address — {first['message']}", href: HREF["gateway"], at: <payload ts when it has one>}`. When the `gateway` job is failing or stale the reachability row it already produces is the **only** line (the existing "a dead source gets exactly one line" rule), so a dead agent never yields two.
|
||||||
|
- This module still holds **no thresholds** and no `severity.py` row: a config warning is a structural fact, not a graded number. Update this module's docstring rule list — it currently says the gateway source has "no verdict rule and none is wanted"; it now says: *reachability plus config-integrity, no graded number, ever.*
|
||||||
|
3. `src/dashboards/api.py` — pass the gateway payload to `alerts.build(...)` from `/api/summary`; add `warnings` (the count) to the `gateway` block; `GATEWAY_FIELDS` pin extended in the same commit.
|
||||||
|
4. `src/dashboards/web/static/page-index.js` — the overview's one gateway line appends ` · N to address` **only when `warnings > 0`**, so a clean ingress's line does not grow a permanent zero.
|
||||||
|
5. `tests/test_gateway_source.py` — one test per kind, each on a **synthetic** config written into the fake `--root` tree the `gateway_snapshot` fixture uses (or a hand-built section dict where the agent's parser would only be in the way, which is legitimate for a pure function over sections — say which you used in the test docstring), plus:
|
||||||
|
- `test_the_captured_configs_produce_no_warnings` — the real fixture asserts `warnings == []` **and** the exact reason per kind (this is the pin that keeps the seven checks from being a lie about this fleet);
|
||||||
|
- `test_an_unreadable_or_absent_section_warns_about_nothing` (`no_haproxy` / `no_nginx` / `predates` modes),
|
||||||
|
- `test_a_listen_with_no_proxy_pass_is_not_a_warning` and `test_a_frontend_routing_by_fetch_expression_is_not_a_warning` (the refusals, pinned in the direction that they produce **no** warning while still producing their listener),
|
||||||
|
- `test_warnings_are_deterministic_deduplicated_and_json_serialisable`.
|
||||||
|
6. `tests/test_alerts*.py` (whichever file holds the `alerts.build` suite today) — the row appears for one warning and for seven; **one** row, not one-per-warning; `severity == "warn"`; `href == "/gateway"`; a failing/stale `gateway` job yields its reachability row and **not** the config row; an unconfigured gateway yields nothing (invariant 3); the `disks`/`gateway` echo-suppression table is unchanged.
|
||||||
|
7. `tests/test_api.py` — `/api/summary` with a mutated gateway payload (the `self_test_app` precedent: swap the collector so a poll cannot overwrite what the test set) yields exactly one new `warn` row linking to `/gateway`, and the same mutation with an empty `warnings` list yields none. **Both directions**, per the project's testing policy.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: as listed; the alert row's two directions are mandatory.
|
||||||
|
- Coverage: **>90%** on `warnings()`, the `alerts.build()` branch, and the `api.py` wiring.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GET /api/gateway/snapshot` on the live ingress answers `warnings: []`, and `GET /api/summary` carries **no** `gateway:config` alert row.
|
||||||
|
- [ ] Injecting one synthetic dangling `use_backend` into the fixture produces exactly one `haproxy_route_dangling` warning, one `warn` row on `/api/summary` whose `href` is `/gateway`; removing it produces none (both pinned).
|
||||||
|
- [ ] A `gateway` job that is failing produces its reachability row and **no** config row (pinned).
|
||||||
|
- [ ] `test_the_captured_configs_produce_no_warnings` is green and names why each of the seven kinds cannot fire on this fleet.
|
||||||
|
- [ ] `uv run pytest` **100% passing**, coverage not down; ruff + pyright clean.
|
||||||
|
- [ ] `AGENTS.md`'s `alerts.py` bullet updated in this task (it documents a rule this task changes) — task 06 re-checks the whole file.
|
||||||
|
- [ ] No behaviour change in completed work: no other source's alert rows change shape or count, the overview's two-panel structure is intact.
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
# Task 04 — Two panels, not four: the nginx and haproxy connection tables, grouped trunk → leaves
|
||||||
|
|
||||||
|
**Phase:** `30_gateway_connection_map` · **Source:** operator request 2026-10-23 — "the stream map shouldn't be its own table, it's part of nginx. There should be only 2 tables: nginx and haproxy"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
`/gateway` becomes two panels and one warnings list. The `stream map` card is gone — a `map` is part of nginx, so its 17 entries are drawn as the leaves of nginx's `:443` trunk — and the `unmatched` card is gone (task 02 deleted the data behind it). Each card lists its **listeners**, and under each listener the **connections** that listener's config declares, so the structure of the page *is* the structure of the file. This task ships the markup and the render pass with the connectors present but plainly styled; task 05 gives them the green line, the elbows and the switches.
|
||||||
|
|
||||||
|
> **On the word "table":** these are `.card` panels with a header row and one row per connection, **not** `<table>` elements. A `<table>` cannot draw an elbow from `:443` to sixteen rows, and `AGENTS.md` pins `table.data` to the horizontal-scroll rules (phase 27) which a graph row does not want. The visual reading is a table — heading, aligned rows, a right-hand number column — and the orphan-`id` / every-`<table>`-carries-`data` pins stay green because no `<table>` is introduced.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/pages/gateway.html`
|
||||||
|
- `#gate-cards` holds **exactly two** `<article class="card gate-card">`: `data-gate="nginx"` then `data-gate="haproxy"` (the operator named nginx first, and it is the bigger fan-out). Delete the `map` and `unmatched` articles and every id/class only they used.
|
||||||
|
- Each card body: `<div class="gate-map" data-k="map">` (groups injected here) + the `data-k="empty"` sentence + the existing `details.charts-box` block, unchanged in shape.
|
||||||
|
- New warnings panel, **above** the cards, shipped `hidden`:
|
||||||
|
```html
|
||||||
|
<section class="card" id="gate-warnings" hidden>
|
||||||
|
<header><h3>Warnings to address</h3><span class="chip" id="warnings-count"></span></header>
|
||||||
|
<div class="body"><ul class="alerts" id="warnings-list"></ul></div>
|
||||||
|
</section>
|
||||||
|
```
|
||||||
|
Its rows use the overview's vocabulary verbatim — `<li class="warn"><span class="src">haproxy</span><span class="msg">…</span></li>` — because "like other things in this dashboard project" is the instruction and `ul.alerts` *is* that thing. Clean means **hidden**, not an empty card.
|
||||||
|
- `#tallies` becomes four cells: `ports` (configured listeners), `connections` (config-declared connections), `live` (ESTABLISHED sockets on configured ports), `warnings`. `#t-listeners` / `#t-established` / `#t-unmatched` and their JS are deleted; `#sub` keeps `host · agent · N connections`.
|
||||||
|
- `<footer class="pagefoot">`: delete the `by port` sentence. Keep/rewrite three, each one sentence: the animation rule; **"only an endpoint the config wrote as an address can be matched to a socket — a name is never resolved, so its line carries no number"**; and **"the switch shows what the config says; toggling a route needs a config write path and is not wired up yet."**
|
||||||
|
2. `src/dashboards/web/static/page-gateway.js`
|
||||||
|
- `CARDS = [{key: 'nginx', …}, {key: 'haproxy', …}]` and no third entry; `pruneCards` unchanged in behaviour.
|
||||||
|
- New `groupsFor(snapshot, key)` → `[{port, listener, connections: [...]}]`, ordered: ports ascending, connections in payload order. A listener with **no** connections still produces a group (this fleet's `:80` vhost with no `proxy_pass` is the live case) and its leaf list renders the one faint sentence `no target named in the config` — a refusal, never a blank row.
|
||||||
|
- `renderGroup(card, group)` builds once and patches afterwards (the `page-disks.js` / slot-grid discipline: keyed `Map` per card, `appendChild` reorders rather than rebuilds, so a 17-row fan-out is not re-created every 30 s). Row structure:
|
||||||
|
```
|
||||||
|
<li class="gate-link" data-id="{connection.id}">
|
||||||
|
<span class="gate-elbow" aria-hidden="true"></span>
|
||||||
|
<span class="gate-switch-slot"></span> <!-- task 05 fills this -->
|
||||||
|
<svg class="pipe on">…</svg> <!-- task 05 styles it -->
|
||||||
|
<span class="gate-target">
|
||||||
|
<span class="pipe-name mono">{host}:{port}</span>
|
||||||
|
<span class="gate-selector faint">{selectors joined} · {kind}</span>
|
||||||
|
</span>
|
||||||
|
<span class="pipe-count">{count or '—'}</span>
|
||||||
|
</li>
|
||||||
|
```
|
||||||
|
Trunk: `:{port}` + the live count (`14 live` / `n/a`) + up to `CLIENTS_SHOWN` client addresses + the domains count (`16 names`, full list in `title`). Every upstream string (`server_name`, `proxy_pass`, map key, client address) goes out through `textContent`/`title` only, and the row's `title` + `aria-label` carry the **whole** endpoint even when CSS ellipsises it (gap 13's rule, re-measured in task 06).
|
||||||
|
- A count that is `null` prints `—` with `title` = `not attributed: the config names this endpoint as a name, and nothing here resolves names` — never `0`, and not a repeated `n/a` column of twelve identical words. A number prints through `countText`.
|
||||||
|
- `renderWarnings(snapshot)`: fills `#warnings-list` with one `li.warn` per payload warning (`src` = the config family word, `msg` = `warning.message`), sets `#warnings-count` to `N` , and keeps `#gate-warnings` `hidden` when the list is empty. The page renders `warnings` and never re-derives one.
|
||||||
|
- `renderHeader()` → the four tallies; the `.stale` dim and the three degradation arms (`why` banner, `unreachable` dim, `404 {configured:false}` → `#gate-empty` naming `GATEWAY_HOST` and agent **1.2.0**) keep their exact current behaviour. A failed poll still re-draws nothing (last good map stays).
|
||||||
|
- `cardStreams()` (the chart series list) reads `listeners` + `connections` for that source; one `/api/gateway/history` request per card per refresh, plot built only while the disclosure is open, `ariaLabel` on every plot — the phase-29 discipline, unchanged.
|
||||||
|
3. `src/dashboards/web/static/dash.css` — the **structure** only in this task: `.gate-map` (grid, `min-width: 0`), `.gate-group` (two columns: trunk `minmax(0, 9.5rem)` + leaves `minmax(0, 1fr)`), `.gate-leaves` (list-style none, `display: grid`, gap), `.gate-link` (`display: grid`, `align-items: center`, columns for elbow / switch / pipe / target / count), `.gate-target` (`min-width: 0`), `.gate-selector` (11 px, ellipsis), `.gate-trunk` (`min-width: 0`), `@media (max-width: 640px)` stacking the group to one column with the trunk above its leaves. No colours, no dashed borders, no switch rules yet — task 05's job, so the two are reviewable separately.
|
||||||
|
4. `tests/test_api.py` — rewrite the page-shape pins in the same commit:
|
||||||
|
- `test_the_gateway_page_is_a_card_per_source_family_with_a_reconciliation_row` → **`test_the_gateway_page_is_exactly_two_connection_panels_and_a_warnings_list`**: exactly two `[data-gate]` articles and they are `nginx` + `haproxy` (assert the list, ordered, so a third card fails); no `[data-gate="map"]`, no `[data-gate="unmatched"]`; `#gate-warnings` + `#warnings-list` exist and ship `hidden`; the four tally ids exist and the three deleted ones do not; every id on the page is referenced by the script or the stylesheet (the standing orphan audit must stay green by inspection of its own list).
|
||||||
|
- `test_the_gateway_page_prints_every_field_the_projection_publishes` → `GATEWAY_PAGE_FIELDS` resolves against a live payload, now including `connections[].in.port`, `connections[].out.host/port/name/kind/selectors/address`, `connections[].connections`, `warnings[].kind/subject/message`, `listeners[].domains/clients/connections`.
|
||||||
|
- `test_the_gateway_page_renders_each_degradation_arm_in_the_payloads_own_words` — still all four arms, on the new markup.
|
||||||
|
- `test_the_gateway_page_keeps_its_last_drawing_when_a_poll_fails` — unchanged behaviour, new selectors.
|
||||||
|
- **new** `test_the_stream_map_is_rendered_inside_nginx_and_nowhere_else` — the served script contains no `map` card key and the live payload's map-sourced connections are drawn under `source: "nginx"`; and `test_a_listener_with_no_declared_target_says_so_instead_of_drawing_nothing` (the `:80` arm), `test_warnings_render_in_the_overviews_alert_vocabulary_and_hide_when_clean` (both directions), `test_an_unattributed_endpoint_prints_a_dash_and_never_a_zero`.
|
||||||
|
- `test_no_page_narrates_its_own_pipeline` — the new footer sentences must pass the existing deleted-sentence list (they are operator-facing why-lines, not pipeline narration) and no removed sentence comes back.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: served-bytes pins above (no JS runner in this repo — task 06's browser pass is the behavioural proof).
|
||||||
|
- Coverage: the changed Python is nil-or-minimal here; the pins are the product. `node --check src/dashboards/web/static/page-gateway.js` clean.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `/gateway` renders **two** panels; the nginx panel's `:443` group lists 17 leaves on the live config, `:4443` six, `:8443` three, haproxy six groups of one, and `:80` shows its refusal sentence.
|
||||||
|
- [ ] No `stream map` card, no `unmatched` card, no `by port` chip anywhere (grep + pin).
|
||||||
|
- [ ] `#gate-warnings` is hidden with the live (clean) configs and appears with rows when a warning is injected into a fixture payload.
|
||||||
|
- [ ] No sideways document scroll at 360 px with the 17-row fan-out open (measured now, re-measured in task 06).
|
||||||
|
- [ ] `uv run pytest` **100% passing**, coverage not down; ruff + pyright + `node --check` clean.
|
||||||
|
- [ ] README/AGENTS: the `/gateway` UI paragraph gains "two panels, nginx + haproxy, map is nginx's" in this task if the sentence already contradicts the shipped page; task 06 does the full pass.
|
||||||
|
- [ ] No behaviour change in completed work: the other five dashboards + `/settings` + `/certs` byte-identical, 10 sources `ok`.
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
# Task 05 — The line and the switch: green because it is in the config, joined trunk to leaf, unwired by design
|
||||||
|
|
||||||
|
**Phase:** `30_gateway_connection_map` · **Source:** operator request 2026-10-23 — "the orange dashes look great but they should be **green if the connection is active**. 'Active' means it exists in the nginx config… the dotted line connections need to connect to their counterparts. This is easy for haproxy — just draw the line straight across. For nginx you'll need to branch the lines from `:443` to the services they connect to" + the sketch (`/tmp/pi-clipboard-…png`: green boxes + horizontal pill = on, red boxes + vertical pill = off) + "build the switches in now but don't wire them up"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Every connection row draws a **green dashed line from its inbound end to its outbound end** — straight across for a haproxy frontend with one `server`, **branched** from the `:443` trunk into one tick per map entry for nginx — with the **switch** from the sketch sitting in the middle of it. Colour is an **existence state** taken from the config (green = in the config, red = out of it), never a verdict on a number; motion still means "a socket is open this instant" (decision 2). The switch is real markup, real ARIA, red-and-green styled, **and unwired**: `aria-disabled="true"`, no handler, a `title` that says so.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `src/dashboards/web/static/dash.css`
|
||||||
|
- In the `:root` block that declares `--ok`/`--fail`, add the **existence pair** with a comment naming what it is *not*:
|
||||||
|
```css
|
||||||
|
/* The gateway's existence pair — deliberately NOT the severity vocabulary. Green means the
|
||||||
|
route is written in a proxy config; red means it is out of the config. No gateway figure
|
||||||
|
is graded (`severity.py` has no `gateway.*` row), so nothing here may reach for --ok,
|
||||||
|
--warn or --fail directly: the pair is one indirection whose name says what it answers. */
|
||||||
|
--gate-on: var(--ok);
|
||||||
|
--gate-off: var(--fail);
|
||||||
|
```
|
||||||
|
(The standing "no rule reaches for a token `:root` does not declare" pin sees both new tokens; per-theme `--ok` substitution happens on the root element, so light/dark both follow without a second declaration.)
|
||||||
|
- Rewrite the `gateway pipes` section's colour half:
|
||||||
|
```css
|
||||||
|
.pipe.on .pipe-path { stroke: var(--gate-on); }
|
||||||
|
.pipe.off .pipe-path { stroke: var(--gate-off); }
|
||||||
|
.pipe.live .pipe-path { animation: pipe-flow 1.2s linear infinite; }
|
||||||
|
.pipe.still .pipe-path { stroke-dasharray: 4 8; } /* a route with nothing on it right now */
|
||||||
|
```
|
||||||
|
`.pipe-path`'s base rule keeps **no** `animation` and **no** verdict colour; the `live`/`still` classes stay (their meaning is now "a socket is open this instant" — restate that in the comment, because the class name no longer says "idle route").
|
||||||
|
- The connectors, all of it pure CSS (no JS coordinate maths, no `ResizeObserver`, nothing measured at runtime):
|
||||||
|
```css
|
||||||
|
.gate-group { position: relative; display: grid; column-gap: 0;
|
||||||
|
grid-template-columns: minmax(0, 9.5rem) minmax(0, 1fr); }
|
||||||
|
.gate-leaves { position: relative; padding-inline-start: 26px; }
|
||||||
|
.gate-link { position: relative; }
|
||||||
|
/* each row carries its own half of the branch, so the spine starts at the first branch and
|
||||||
|
stops at the last without a mask, a magic height or a background-coloured eraser */
|
||||||
|
.gate-link::before { content: ""; position: absolute; left: -26px; top: 0; height: 50%;
|
||||||
|
border-left: 2px dashed var(--border-strong); }
|
||||||
|
.gate-link::after { content: ""; position: absolute; left: -26px; top: 50%; bottom: 0;
|
||||||
|
border-left: 2px dashed var(--border-strong); }
|
||||||
|
.gate-link:first-child::before, .gate-link:last-child::after { display: none; }
|
||||||
|
.gate-elbow { align-self: center; height: 0; border-top: 2px dashed var(--border-strong); }
|
||||||
|
.gate-trunk { position: relative; } /* its own tick reaches the spine at right: 0 */
|
||||||
|
/* straight across: a group with one leaf is not a branch, so it draws no vertical at all */
|
||||||
|
.gate-group.solo .gate-link::before, .gate-group.solo .gate-link::after { display: none; }
|
||||||
|
```
|
||||||
|
`.gate-group.solo` is added by the JS when a group has exactly one connection (every haproxy group on this fleet). The `.gate-link` grid columns are `26px auto minmax(2rem, 4.5rem) minmax(0, 1fr) auto` — elbow, switch, the animated run, the target, the count.
|
||||||
|
- The switch, keyed on `[aria-checked]` and **never** on `:enabled` (a not-yet-wired switch must still look green when it is on), and tinted from the same pair so the gateway owns one colour vocabulary:
|
||||||
|
```css
|
||||||
|
.gate-switch { width: 34px; height: 18px; padding: 2px; border-radius: 999px;
|
||||||
|
border: 1px solid var(--gate-on); display: inline-flex; align-items: center;
|
||||||
|
background: color-mix(in srgb, var(--gate-on) 14%, transparent);
|
||||||
|
cursor: not-allowed; }
|
||||||
|
.gate-switch-thumb { width: 12px; height: 12px; border-radius: 50%;
|
||||||
|
background: var(--gate-on); transform: translateX(16px);
|
||||||
|
transition: transform .16s ease; }
|
||||||
|
.gate-switch[aria-checked="false"] { border-color: var(--gate-off);
|
||||||
|
background: color-mix(in srgb, var(--gate-off) 14%, transparent); }
|
||||||
|
.gate-switch[aria-checked="false"] .gate-switch-thumb { transform: translateX(0); }
|
||||||
|
```
|
||||||
|
`color-mix` (already the sheet's idiom) is why no `--ok-soft` / `--fail-soft` appears in a gateway rule: the soft fill derives from the existence pair rather than borrowing the verdict tints.
|
||||||
|
- The existing `@media (prefers-reduced-motion: reduce)` block gains `.gate-switch-thumb { transition: none; }` beside its two `.pipe` rules (it must name every animated thing on the page — naming one and leaving the thumb sliding is the phase-18 lesson).
|
||||||
|
- `@media (max-width: 640px)`: the group stacks (task 04's rule) and the spine's `top` reaches up into the group gap so the trunk above still reads as connected; the switch and its row do not shrink and nothing scrolls sideways.
|
||||||
|
2. `src/dashboards/web/static/page-gateway.js`
|
||||||
|
- `gateLink()` builds the row in task 04's order and fills `.gate-switch-slot`:
|
||||||
|
```html
|
||||||
|
<button class="gate-switch" type="button" role="switch" aria-checked="true" aria-disabled="true"
|
||||||
|
title="Toggling a route is not wired up yet — it needs a config write path. This switch shows what the config says.">
|
||||||
|
<span class="gate-switch-thumb"></span>
|
||||||
|
</button>
|
||||||
|
```
|
||||||
|
`aria-checked` is `true` because the row exists at all (drawn ⇒ in the config); the `false` styling is present and unreachable until the write phase. `aria-label` names the endpoint, the selector and the config family (`"nextcloud.reeseapps.com → nextcloud.reeselink.com:11000, in the nginx config, on"`), since a switch with no accessible name is silence. **No click handler is bound, anywhere.**
|
||||||
|
- `renderGroup()` toggles `.solo` on the group from its own leaf count, and toggles `live`/`still` **and** `on`/`off` on the row's `<svg class="pipe">` (`on` always today; `off` is a one-line seam for the write phase).
|
||||||
|
- `pipeAria()` gains the state word and keeps every existing fact (port / target / domains / clients / count), so an unattributed count is still announced as unknown and never as zero.
|
||||||
|
3. `tests/test_api.py` — **restate, do not delete**, the colour pin:
|
||||||
|
- `test_no_gateway_count_is_coloured_by_a_verdict` keeps every current assertion (no `--ok|--warn|--fail|.ok|.warn|.fail|severity` inside the `gateway pipes` section, no `severityFrom`/`severityFor`/`.severity` in the script, no verdict class in the markup) and its docstring is rewritten to say the pipes/switches colour from `--gate-on`/`--gate-off`, that the pair is an **existence** state and not the severity vocabulary, and that the one place `.warn` legitimately appears is the warnings panel's `ul.alerts` rows (which are `alerts.py`'s verdict, not a number's).
|
||||||
|
- **new** `test_a_connection_is_green_because_it_is_in_the_config` — `.pipe.on .pipe-path` stroke is `var(--gate-on)`, `.pipe.off .pipe-path` is `var(--gate-off)`, `:root` declares both, and `.pipe-path`'s base rule declares no `stroke` verdict; the script sets `on` unconditionally for a drawn connection.
|
||||||
|
- **new** `test_haproxy_draws_straight_across_and_nginx_branches` — served CSS: `.gate-group.solo` hides `.gate-link::before`/`::after`; the branch halves and `.gate-elbow` exist; and the script's `.solo` toggle reads the group's own leaf count.
|
||||||
|
- **new** `test_the_switch_is_built_and_explicitly_unwired` — `role="switch"`, `aria-checked`, `aria-disabled="true"` (and **not** `disabled`, which would drop it from the a11y tree), the `title` names the missing write path, and no `addEventListener` in the file names `gate-switch` / binds a click on it. This is the pin that fails the day someone ships a dead-looking-live control.
|
||||||
|
- `test_the_pipe_animation_stills_in_both_states_under_reduced_motion` — both pipe states **and** `.gate-switch-thumb` are named in the file's one reduced-motion block.
|
||||||
|
- `test_a_pipe_moves_only_while_it_carries_a_connection` — unchanged claim, new class pairs (`live` animates, `still` does not, colour comes from `on`/`off`).
|
||||||
|
- The orphan-`id` audit and the CDN-reference pin stay green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: served-CSS + served-script pins above; the behavioural proof is task 06's browser pass (computed stroke colour, animation running, elbow geometry, 360 px).
|
||||||
|
- Coverage: no new Python; `node --check` clean; `bash .agents/validate.sh` (which runs `node --check` over `web/static`) green.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] On live data: every haproxy row's line runs **straight** from `:22`-style trunk to its `server` with no vertical segment; the nginx `:443` trunk has a spine reaching every one of its 17 ticks; each row has exactly one switch.
|
||||||
|
- [ ] Every drawn pipe computes to the green of `--gate-on` in both themes; a row with ≥ 1 live connection animates and one with 0 or `null` does not; with `prefers-reduced-motion: reduce` emulated, nothing moves (pipes and thumb).
|
||||||
|
- [ ] `grep -n "disabled" src/dashboards/web/static/page-gateway.js` shows `aria-disabled` and no `disabled` attribute; no click listener exists for the switch; the `title` sentence is on every switch.
|
||||||
|
- [ ] The severity vocabulary is still absent from the pipes section, the script and the markup — and its absence is pinned with the restated docstring, not removed.
|
||||||
|
- [ ] `uv run pytest` **100% passing**, coverage not down; ruff + pyright + `node --check` clean.
|
||||||
|
- [ ] No behaviour change in completed work; the other pages byte-identical.
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user