phase: 12_nut_self_test_verdict
Final state verified. Report: **Verified (phase 12 — all 3 tasks already complete; this was the verification pass)** - Re-ran the wire capture live: `ups.test.result` = `No test initiated` (apc) / `Done and passed` (cyberpower) — matches `LIVE_SELF_TEST_WORDS` + both fixtures; `/api/summary` has **no** self-test row, `/api/ups/snapshot` reads `cyberpower ok | Done and passed`, `apc unknown | …run a battery test`; both screenshots read back correct. - Fixed a defect found in prior work while checking the "unrelated live noise" note: `NutMonitor.collect()` reused one upsd socket after a *partially-read* reply, so a 4 s stall produced bogus `expected 'BEGIN LIST VAR', got 'RW apc battery.charge.low "10"'` errors (same hazard on `INSTCMD`, where a late `OK` could be read as a different command's success). `NutClient` now marks a socket whose reply stopped early as `UNUSABLE`, naming the command that stalled; a complete `ERR …` line still leaves it usable (the per-UPS tolerance `collect()` depends on). +3 hermetic pins against the fake upsd; live re-check (09:33–09:44) now logs one line per stall, no cascade. **Tests / lint / coverage (all green)** - `bash .agents/validate.sh` → **exit 0**; `uv run pytest --cov --cov-fail-under=94` → **1172 passed, 11 skipped**, coverage **98.63 %** (floor 94); `uv run ruff check src tests agent` clean; `uv run pyright` 0 errors; `node --check` clean. Modified modules: `sources/nut.py` 95 %, `collectors.py` 97 % (no new line uncovered). - `uv run pytest -m live --live` → **8 passed / 3 failed**, all three the pre-existing `driveripper` agent-transport failures AGENTS.md records (verified still true: unit `inactive`, no installed file, nothing on `:9120`). - `./scripts/dev-server.sh status`: `nut ok`, `llm:* ok`, `host/disks:deskwork|gateway ok`; the two `driveripper` jobs DOWN. **Completion criteria** - Live capture recorded + asserts `ok` — met (re-confirmed from the wire today). - `/ups` renders `ok`, overview silent, screenshots present — met. - pytest / cov / ruff / pyright / validate — met (above). - No other `health.checks[]` behaviour changed — met (diff confined to the `test_result` branch; all prior nut/ups-contract tests unmodified). **Documentation**: README verified consistent (self-test wording, config table cross-checked against `config.py`, all documented commands/flags exist) — no changes needed beyond phase tasks' edit. AGENTS.md fixes: stale test count (1140 → 1172), `nvme0n1` power-on hours 19 158 → 19 160, gap 4 store-error claim corrected (last stored `store-error` 2026-10-18 06:33, none in the previous 24 h), new `collectors.py`/`sources/nut.py` socket-hygiene note, new gap-1 bullet on the upsd stall, plus live evidence appended to `data/logs/NOTES-self-test.md`. **Deviations**: made no live *changes* (read-only probes only) — `driveripper`'s agent install needs root (`ops` has no installer/visudo grant) and a battery test stays the operator's call; the upsd 4 s stall itself is upstream-side. `uv pip list --outdated` still advisory (certifi/fastapi/starlette) — dependencies untouched this phase. **Next pending phase**: `13_llm_page_and_decode_rate`.
This commit is contained in:
@@ -0,0 +1,55 @@
|
||||
# Phase 12 — A passed battery self-test is news, not a warning
|
||||
|
||||
**Source:** `TODO.md` L3 — ""Last self-test: Done and passed" still shows up as a warning in the overview and UPS section"
|
||||
**Story:** n/a
|
||||
**Context:** `src/dashboards/sources/nut.py` (`assess()` — the explicit `health.checks[]` list, the "Last self-test" check at the `test_result` branch), `src/dashboards/alerts.py` (the overview alert rule that reads `health.checks[]`), `tests/test_nut.py` (the two existing "Last self-test" assertions, the `SimUps` test-result tests), `tests/fixtures/nut_apc.vars` / `nut_cyberpower.vars` (verbatim `LIST VAR` captures), `src/dashboards/sources/nut_sim.py` (the simulator's `PASSED`/`FAILED` window), `src/dashboards/web/pages/ups.html` + `src/dashboards/web/static/page-ups.js` (the check list the `/ups` card renders), `data/logs/` (screenshots)
|
||||
|
||||
## Objective
|
||||
|
||||
The `Last self-test` check grades the driver's own word about the last battery test, and a
|
||||
passed test — the real drivers answer **`Done and passed`** — must read `ok`, not `warn`.
|
||||
The current word lists (`"ok", "passed", "pass", "completed"` / `"failed", "fail", "error",
|
||||
"aborted", "test incomplete"`) let the one phrasing this fleet's hardware actually uses fall
|
||||
through to `warn`, so a *healthy* UPS carries a warning on the overview and the UPS page.
|
||||
After this phase the verdict follows the test's outcome: pass → `ok`, fail → `fail`,
|
||||
never-tested / unset → `unknown`, and the overview's alert feed stops naming it.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `05_ups_hero_battery_glyph` (complete) — the `health.checks[]` contract and the `/ups`
|
||||
check list this phase changes.
|
||||
- `11_admin_login` (todo) — **no shared files**: 11 touches `app.py`/`config.py`/`store.py`/
|
||||
the auth surface; this phase touches `sources/nut.py` + its tests + fixtures. Either order
|
||||
runs; the only coupling is the shared `tests/test_nut.py` — apply whichever runs second on
|
||||
top of the file state at that time.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_pin_self_test_wording.md` — capture the real drivers' `ups.test.result` values and pin the current (wrong) verdict in a failing test, so the fix has a before/after.
|
||||
2. `02_verdict_map_for_test_results.md` — the `assess()` word map: pass phrasings → `ok`, fail phrasings → `fail`, "No test initiated"/unset → `unknown` (never `warn`); fixtures + `tests/test_nut.py` + the simulator's wording stay consistent.
|
||||
3. `03_no_false_warning_overview_and_ups.md` — end-to-end: a passed test produces no alert on `/api/summary` and an `ok` chip on `/ups`; pinned by API test and a live screenshot.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: `tests/test_nut.py` — the full word map (each pass phrasing `ok`, each fail phrasing
|
||||
`fail`, `No test initiated` and unset `unknown`, case/whitespace-insensitive), driven from
|
||||
the verbatim fixture wording where it exists and synthetic `UpsReading`s elsewhere.
|
||||
- Integration: `tests/test_api.py` — a `health.checks[]` entry with a passed self-test
|
||||
produces no alert in the `/api/summary` payload; a failed one does.
|
||||
- The `SimUps` test-result tests (`tests/test_nut.py`) keep passing: the simulator's
|
||||
`PASSED`/`FAILED` window words were already in the right tiers.
|
||||
- Coverage: **>90%** on new/modified code; the overall gate (94%) must hold.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `printf 'LIST VAR apc ups.test.result\n' | nc <driveripper> 3051` (run from a host
|
||||
that can reach upsd — the laptop still cannot, per the fleet facts) captured and
|
||||
recorded in `tests/test_nut.py` / the fixture; the captured word asserts `ok`
|
||||
- [ ] the "Last self-test" check renders `ok` (dimmed, no warning class) on `/ups` and does
|
||||
not appear in the overview's Warnings panel; screenshots in `data/logs/`
|
||||
(`self-test-passed-overview.png`, `self-test-passed-ups.png`)
|
||||
- [ ] `uv run pytest` green, `uv run pytest --cov --cov-fail-under=94` green,
|
||||
`uv run ruff check src tests agent && uv run pyright` clean, `bash .agents/validate.sh`
|
||||
green
|
||||
- [ ] no behavior change in completed phases: every other `health.checks[]` check (power
|
||||
path, charge, runtime, battery age/temperature, load, alarms) grades exactly as before
|
||||
@@ -0,0 +1,50 @@
|
||||
# Task 01 — Pin the self-test word the real drivers use
|
||||
|
||||
**Phase:** `12_nut_self_test_verdict` · **Source:** `TODO.md:3` — ""Last self-test: Done and passed" still shows up as a warning in the overview and UPS section"
|
||||
|
||||
## Objective
|
||||
|
||||
Prove the bug in a test before touching the fix: the `Last self-test` check in
|
||||
`assess()` must be shown to grade the drivers' real wording (`Done and passed`) as
|
||||
`warn`, and the verbatim `ups.test.result` value must be captured from the running
|
||||
upsd so the fixture/test asserts the hardware's actual sentence, not a memory of it
|
||||
(AGENTS.md rule 7 — verify against the deployed upstream; the factory state both UPSes
|
||||
answer is `No test initiated`, pinned in `AGENTS.md`'s fleet facts).
|
||||
|
||||
## Work
|
||||
|
||||
1. `tests/test_nut.py` — add the pinning test **first**: build the `UpsReading` the way
|
||||
the two existing "Last self-test" tests do (the `SimUps`/`vars` helpers in the file),
|
||||
with `test_result="Done and passed"`, and assert the current behaviour
|
||||
(`check["status"] == "warn"`) with a comment that the *fix* flips it to `ok`. This is
|
||||
the red test the whole phase turns green.
|
||||
2. Capture the real value — from a host that can reach upsd (the laptop cannot yet,
|
||||
per the fleet facts; `driveripper` itself, or the deployed dashboard's network):
|
||||
`printf 'LIST VAR apc ups.test.result\nLIST VAR cyberpower ups.test.result\n' | nc
|
||||
<upsd-host> 3051` — record the exact words in `tests/test_nut.py` (an assertion per
|
||||
UPS where the value is a stable driver constant) and note the capture date. If a test
|
||||
has been run on the hardware, the value is the driver's pass word; if not, it is
|
||||
`No test initiated` — either way it goes into the test, because the verdict map must
|
||||
cover both.
|
||||
3. `tests/fixtures/nut_apc.vars` / `nut_cyberpower.vars` — if the live `ups.test.result`
|
||||
line differs from the captured fixture (a test has since run), re-capture with
|
||||
`./scripts/capture-fixtures.sh` (the script scrubs serials; the test-result line is
|
||||
not a secret) so the fixtures stay verbatim ground truth.
|
||||
4. Confirm the symptom end-to-end while it is live: `curl -s
|
||||
http://127.0.0.1:8090/api/summary` (signed in, per phase 11) shows the alert row for
|
||||
the `Last self-test` check, and `/api/ups/snapshot`'s `health.checks[]` carries the
|
||||
`warn` entry — the two surfaces the TODO names.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: the pinning test (red now, green after task 02).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest tests/test_nut.py -k self_test` shows the new test **failing** on
|
||||
the current code (or passing if the wording already graded `ok` — in which case
|
||||
record that and task 02 narrows to the `No test initiated` branch only)
|
||||
- [ ] the real `ups.test.result` word is captured from the running upsd and recorded in
|
||||
the test suite with its capture date
|
||||
- [ ] fixtures re-captured if the live value differs from the recorded one
|
||||
@@ -0,0 +1,62 @@
|
||||
# Task 02 — Grade the test outcome, not the phrasing
|
||||
|
||||
**Phase:** `12_nut_self_test_verdict` · **Source:** `TODO.md:3` — ""Last self-test: Done and passed" still shows up as a warning in the overview and UPS section"
|
||||
|
||||
## Objective
|
||||
|
||||
The `Last self-test` branch in `assess()` maps the driver's word to a verdict with two
|
||||
hardcoded tuples, and any phrasing outside both falls through to `warn` — which is how
|
||||
`Done and passed` (the real drivers' pass word) and `No test initiated` (the factory
|
||||
state) both read as warnings. Rewrite the branch as an explicit outcome map: pass
|
||||
phrasings → `ok`, fail phrasings → `fail`, "no test has run" phrasings and unset →
|
||||
`unknown`. A warning for a *passed* test is the bug; a warning for a machine that never
|
||||
ran a test is noise this phase removes too.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/sources/nut.py` — the `test_result` branch in `assess()`:
|
||||
- pass tier: the existing `("ok", "passed", "pass", "completed")` **plus**
|
||||
`done and passed` (the live word captured in task 01, kept lowercase/trimmed —
|
||||
keep the existing `strip().lower()` normalisation);
|
||||
- fail tier: unchanged (`"failed", "fail", "error", "aborted", "test incomplete"`);
|
||||
- new `unknown` tier: `no test initiated` (and the empty-string case) — reuse the
|
||||
existing `None`-branch sentence ("no self-test result reported — run a battery
|
||||
test") so the two "never tested" states say the same thing;
|
||||
- the fall-through for any *other* phrasing stays `warn` (an unrecognised word is
|
||||
the one thing worth a warning — a driver inventing a new word is drift worth
|
||||
noticing), and the check keeps printing the raw word plus the `test_date` suffix.
|
||||
- ASSUMPTION: `No test initiated` (the factory state on both UPSes, per AGENTS.md
|
||||
fleet facts) maps to `unknown`, not `warn` — a pack that has never been
|
||||
self-tested is a suggestion to run one, not a fault; the `None` branch already
|
||||
says exactly that, so the two states read identically.
|
||||
2. `tests/test_nut.py` — the task-01 red test now asserts `ok`; add the full map:
|
||||
every pass phrasing (including the captured live word) → `ok`, every fail phrasing
|
||||
→ `fail`, `No test initiated` / empty / unset → `unknown` with the run-a-test
|
||||
sentence, an unknown phrasing (e.g. `mystery`) → `warn`; case and whitespace
|
||||
variations of the live word.
|
||||
3. `tests/fixtures/nut_apc.vars` / `nut_cyberpower.vars` — if the captured live value
|
||||
(task 01) is the factory `No test initiated`, the fixtures already carry it; if a
|
||||
test has run, the fixture carries `Done and passed` — either way the fixture-driven
|
||||
tests in `tests/test_ups_contract.py` / `test_nut.py` must pass against the new map
|
||||
(the fixture's word is part of the hero contract's check list).
|
||||
4. `src/dashboards/sources/nut_sim.py` — no code change expected: the simulator's
|
||||
`PASSED` / `FAILED` window words are already in the pass/fail tiers; run the
|
||||
`SimUps` test-result tests to prove it. If the simulator ever answers a *third*
|
||||
word, add it to the map — do not special-case the simulator.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: the full word-map table in `tests/test_nut.py` (the tiers above, plus
|
||||
`test_date` suffix still printed on the pass/fail detail line).
|
||||
- Regression: every pre-existing `test_nut.py` / `test_ups_contract.py` test passes
|
||||
unmodified in intent (the fixtures' words land in the right tier).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest tests/test_nut.py tests/test_ups_contract.py` green, including the
|
||||
task-01 test now asserting `ok` for `Done and passed`
|
||||
- [ ] no phrasing the two real drivers answer (verified against the task-01 capture)
|
||||
falls into `warn`
|
||||
- [ ] the `unknown` tier's detail sentence is the existing "run a battery test" one, so
|
||||
unset and never-initiated read the same
|
||||
+60
@@ -0,0 +1,60 @@
|
||||
# Task 03 — No false warning on the overview and the UPS page
|
||||
|
||||
**Phase:** `12_nut_self_test_verdict` · **Source:** `TODO.md:3` — ""Last self-test: Done and passed" still shows up as a warning in the overview and UPS section"
|
||||
|
||||
## Objective
|
||||
|
||||
Prove the fix end-to-end on the two surfaces the TODO names: a passed self-test must
|
||||
produce **no alert** in `/api/summary` (the overview's Warnings panel) and an `ok`
|
||||
entry in `/api/ups/snapshot`'s `health.checks[]` (the `/ups` check list), while a
|
||||
*failed* test still alerts — the check keeps its reason for existing.
|
||||
|
||||
## Work
|
||||
|
||||
1. `tests/test_api.py` — the alert-absence pin: an app (the shared `create_app`
|
||||
test harness, `auth_contract` seeding) whose UPS payload carries a `Last self-test`
|
||||
check with a passed word answers `/api/summary` with **no** alert row naming it
|
||||
(the `alerts` list is the thing the overview renders — a row there is the warning
|
||||
the TODO sees); the same harness with a `fail`-tier word **does** produce the
|
||||
alert, so the test pins both directions.
|
||||
2. `src/dashboards/web/static/page-ups.js` / `src/dashboards/web/pages/ups.html` — no
|
||||
code change expected (the check list renders `health.checks[]` verdicts verbatim);
|
||||
verify the `ok` chip styling for the "Last self-test" row is the existing dim path
|
||||
(`ok` stays quiet ink, per the shared verdict vocabulary). If the row needs no
|
||||
markup, say so in the completion note — do not restyle.
|
||||
3. Live verification (AGENTS.md invariant 10 — UI changes are screenshotted): the dev
|
||||
server (`./scripts/dev-server.sh status` first; the source must be `ok`), signed in
|
||||
with the dev password:
|
||||
- `curl -s http://127.0.0.1:8090/api/summary` — no `Last self-test` alert while the
|
||||
live UPSes report a passed (or never-initiated) result;
|
||||
- the `/` overview: Warnings & errors panel shows no self-test row;
|
||||
- the `/ups` page: the "Last self-test" row renders the driver's word with the `ok`
|
||||
(or `unknown`) class.
|
||||
4. `data/logs/` — `self-test-passed-overview.png` and `self-test-passed-ups.png`
|
||||
(the overview with the Warnings panel visible, and the `/ups` card with the check
|
||||
list visible). If the live UPSes are at the factory `No test initiated`, the
|
||||
screenshots show the `unknown` dim state — the *absence of the warning* is the
|
||||
point, in either state.
|
||||
- ASSUMPTION: no battery test is run on the live hardware to force a `Done and
|
||||
passed` state — a self-test puts real load on a battery things are plugged into,
|
||||
which stays the operator's call (the AGENTS.md known-gaps rule). The hermetic
|
||||
`fail`/`ok` pins in `tests/test_api.py` carry the behaviour; the live screenshots
|
||||
carry whichever state the hardware currently reports.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Integration: the two-direction alert pin in `tests/test_api.py`.
|
||||
- Live: the two surfaces checked per step 3, screenshots recorded.
|
||||
- Coverage: **>90%** on this task's new/modified code (mostly test code — the overall
|
||||
94% gate must hold).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `curl -s http://127.0.0.1:8090/api/summary | grep -i "self-test"` returns nothing
|
||||
while the live `health.checks[]` entry is `ok`/`unknown`, and returns the alert
|
||||
row when a `fail`-tier word is injected (the `tests/test_api.py` pin)
|
||||
- [ ] `/` renders no self-test row in Warnings & errors; `/ups` renders the check with
|
||||
its verdict class — both screenshotted in `data/logs/`
|
||||
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||
- [ ] the `Dev server stays up` rule held: `./scripts/dev-server.sh status` shows the
|
||||
`ups` source `ok` before and after the screenshots
|
||||
Reference in New Issue
Block a user