phase: 12_nut_self_test_verdict
Final state verified. Report: **Verified (phase 12 — all 3 tasks already complete; this was the verification pass)** - Re-ran the wire capture live: `ups.test.result` = `No test initiated` (apc) / `Done and passed` (cyberpower) — matches `LIVE_SELF_TEST_WORDS` + both fixtures; `/api/summary` has **no** self-test row, `/api/ups/snapshot` reads `cyberpower ok | Done and passed`, `apc unknown | …run a battery test`; both screenshots read back correct. - Fixed a defect found in prior work while checking the "unrelated live noise" note: `NutMonitor.collect()` reused one upsd socket after a *partially-read* reply, so a 4 s stall produced bogus `expected 'BEGIN LIST VAR', got 'RW apc battery.charge.low "10"'` errors (same hazard on `INSTCMD`, where a late `OK` could be read as a different command's success). `NutClient` now marks a socket whose reply stopped early as `UNUSABLE`, naming the command that stalled; a complete `ERR …` line still leaves it usable (the per-UPS tolerance `collect()` depends on). +3 hermetic pins against the fake upsd; live re-check (09:33–09:44) now logs one line per stall, no cascade. **Tests / lint / coverage (all green)** - `bash .agents/validate.sh` → **exit 0**; `uv run pytest --cov --cov-fail-under=94` → **1172 passed, 11 skipped**, coverage **98.63 %** (floor 94); `uv run ruff check src tests agent` clean; `uv run pyright` 0 errors; `node --check` clean. Modified modules: `sources/nut.py` 95 %, `collectors.py` 97 % (no new line uncovered). - `uv run pytest -m live --live` → **8 passed / 3 failed**, all three the pre-existing `driveripper` agent-transport failures AGENTS.md records (verified still true: unit `inactive`, no installed file, nothing on `:9120`). - `./scripts/dev-server.sh status`: `nut ok`, `llm:* ok`, `host/disks:deskwork|gateway ok`; the two `driveripper` jobs DOWN. **Completion criteria** - Live capture recorded + asserts `ok` — met (re-confirmed from the wire today). - `/ups` renders `ok`, overview silent, screenshots present — met. - pytest / cov / ruff / pyright / validate — met (above). - No other `health.checks[]` behaviour changed — met (diff confined to the `test_result` branch; all prior nut/ups-contract tests unmodified). **Documentation**: README verified consistent (self-test wording, config table cross-checked against `config.py`, all documented commands/flags exist) — no changes needed beyond phase tasks' edit. AGENTS.md fixes: stale test count (1140 → 1172), `nvme0n1` power-on hours 19 158 → 19 160, gap 4 store-error claim corrected (last stored `store-error` 2026-10-18 06:33, none in the previous 24 h), new `collectors.py`/`sources/nut.py` socket-hygiene note, new gap-1 bullet on the upsd stall, plus live evidence appended to `data/logs/NOTES-self-test.md`. **Deviations**: made no live *changes* (read-only probes only) — `driveripper`'s agent install needs root (`ops` has no installer/visudo grant) and a battery test stays the operator's call; the upsd 4 s stall itself is upstream-side. `uv pip list --outdated` still advisory (certifi/fastapi/starlette) — dependencies untouched this phase. **Next pending phase**: `13_llm_page_and_decode_rate`.
This commit is contained in:
+264
-2
@@ -9,6 +9,7 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
@@ -16,6 +17,7 @@ import pytest_asyncio
|
||||
from dashboards.sources import nut
|
||||
from dashboards.sources.nut import NutClient, NutError, UpsReading, normalise_celsius
|
||||
from dashboards.sources.nut_sim import NutSimulator, SimUps, from_env
|
||||
from ups_contract import parse_capture
|
||||
|
||||
VAR_ROWS = {
|
||||
"battery.charge": "100",
|
||||
@@ -47,10 +49,18 @@ class FakeUpsd:
|
||||
"""Just enough of upsd's protocol to exercise the client, over a real socket."""
|
||||
|
||||
def __init__(self, *, require_auth: bool = False, close_early: bool = False,
|
||||
hang: bool = False) -> None:
|
||||
hang: bool = False, partial_list: str | None = None,
|
||||
deny: tuple[str, ...] = ()) -> None:
|
||||
self.require_auth = require_auth
|
||||
self.close_early = close_early
|
||||
self.hang = hang
|
||||
# Answer this `LIST` kind with `BEGIN` + one row and then go silent: the row is already
|
||||
# in the client's buffer when its read times out, which is the exact state a slow upsd
|
||||
# leaves a shared connection in.
|
||||
self.partial_list = partial_list
|
||||
# Answer these `LIST` kinds with a complete `ERR` line — a refusal is the whole reply,
|
||||
# so a client may (and must) keep using the connection afterwards.
|
||||
self.deny = deny
|
||||
self.received: list[str] = []
|
||||
self.authenticated = False
|
||||
self._server: asyncio.AbstractServer | None = None
|
||||
@@ -84,7 +94,16 @@ class FakeUpsd:
|
||||
|
||||
if verb == "LIST":
|
||||
kind, _, target = rest.partition(" ")
|
||||
if kind == "UPS":
|
||||
if kind in self.deny:
|
||||
send("ERR ACCESS-DENIED")
|
||||
elif kind == self.partial_list:
|
||||
# One row of the real shape, then silence: no END, so the client times out
|
||||
# with this row sitting in its socket buffer.
|
||||
send(f"BEGIN LIST {kind}")
|
||||
send(f'VAR "{_strip(target)}" "ups.status" "OL"' if kind == "VAR"
|
||||
else f'{kind} "{_strip(target)}" "beeper.mute"')
|
||||
return
|
||||
elif kind == "UPS":
|
||||
send("BEGIN LIST UPS")
|
||||
send('UPS "cyberpower" "Cyber Power Systems PR1500LCDRT2U"')
|
||||
send("END LIST UPS")
|
||||
@@ -301,6 +320,77 @@ async def test_a_silent_upsd_times_out_instead_of_hanging() -> None:
|
||||
await server.stop()
|
||||
|
||||
|
||||
async def test_a_reply_that_stops_early_makes_the_socket_refuse_the_next_command() -> None:
|
||||
"""A stalled `LIST` is reported as the stall it was, not as a protocol mystery.
|
||||
|
||||
This fleet's upsd answers every poll within `NUT_TIMEOUT_SECONDS` except every so often,
|
||||
and the lines it had already sent stay in the socket buffer. `collectors.NutMonitor`
|
||||
tolerates a failed `LIST CMD`/`LIST RW` and reads the next UPS on the same connection, so
|
||||
those leftovers used to be taken for that command's `BEGIN` and the poll died saying
|
||||
`expected 'BEGIN LIST VAR', got 'RW apc battery.charge.low "10"'` — a sentence about the
|
||||
wire protocol, when the truth was a 4-second stall two commands earlier (the pair of
|
||||
messages is in the dev `event` table, which is how this got noticed).
|
||||
"""
|
||||
server = FakeUpsd(partial_list="CMD")
|
||||
port = await server.start()
|
||||
try:
|
||||
async with NutClient("127.0.0.1", port, timeout=0.3) as client:
|
||||
with pytest.raises(NutError) as stalled:
|
||||
await client.list_cmds("apc")
|
||||
assert stalled.value.code == "TIMEOUT"
|
||||
with pytest.raises(NutError) as next_command:
|
||||
await client.list_vars("apc")
|
||||
finally:
|
||||
await server.stop()
|
||||
assert next_command.value.code == "UNUSABLE"
|
||||
assert "did not answer" in str(next_command.value), "the cause is named, not the symptom"
|
||||
assert "LIST CMD" in str(next_command.value), "and it names the command that stalled"
|
||||
|
||||
|
||||
async def test_an_instant_command_that_never_answers_spends_the_connection() -> None:
|
||||
"""The guard is on the write path too, where it matters most.
|
||||
|
||||
`instcmd` is the only thing in this app that dials a write. If upsd answers it after we
|
||||
gave up, that late `OK` sits on the socket, and a client allowed to ask another question
|
||||
would read a *different* command's success as the answer to it — the one mistake the four
|
||||
gates cannot survive, since it would print `OK` beside a `load.off` nobody confirmed.
|
||||
"""
|
||||
server = FakeUpsd(hang=True)
|
||||
port = await server.start()
|
||||
try:
|
||||
async with NutClient("127.0.0.1", port, timeout=0.3,
|
||||
username="monitor", password="pw") as client:
|
||||
with pytest.raises(NutError) as hung:
|
||||
await client.instcmd("apc", "beeper.mute")
|
||||
assert hung.value.code == "TIMEOUT"
|
||||
with pytest.raises(NutError) as next_command:
|
||||
await client.instcmd("apc", "load.off")
|
||||
finally:
|
||||
await server.stop()
|
||||
assert next_command.value.code == "UNUSABLE"
|
||||
assert "INSTCMD" in str(next_command.value)
|
||||
|
||||
|
||||
async def test_an_upsd_error_line_leaves_the_connection_usable() -> None:
|
||||
"""The other half: `ERR …` is a *complete* reply, so the tolerance in `collect()` stands.
|
||||
|
||||
One UPS answering `LIST CMD` with `ACCESS-DENIED` must cost that UPS its command buttons
|
||||
and nothing else — the same poll keeps reading the other battery. A guard that poisoned the
|
||||
connection on a refusal would turn a per-UPS refusal into a dead UPS page.
|
||||
"""
|
||||
server = FakeUpsd(deny=("CMD",))
|
||||
port = await server.start()
|
||||
try:
|
||||
async with NutClient("127.0.0.1", port, timeout=2.0) as client:
|
||||
with pytest.raises(NutError) as denied:
|
||||
await client.list_cmds("apc")
|
||||
assert denied.value.code == "ACCESS-DENIED"
|
||||
variables = await client.list_vars("apc")
|
||||
finally:
|
||||
await server.stop()
|
||||
assert variables["ups.status"] == "OL CHRG"
|
||||
|
||||
|
||||
# ---- reading semantics ---------------------------------------------------- #
|
||||
|
||||
def reading(**overrides: str) -> UpsReading:
|
||||
@@ -407,6 +497,178 @@ def test_missing_test_history_is_unknown_not_ok() -> None:
|
||||
assert check["status"] == "unknown"
|
||||
|
||||
|
||||
# ---- the self-test word the hardware actually answers (phase 12 pin) ------- #
|
||||
|
||||
#: `ups.test.result` exactly as the two real UPSes answered it, captured from the running
|
||||
#: upsd on **2026-10-19** from the dev box (which reaches driveripper's 3051 — the transport
|
||||
#: phase 10 re-verified) with::
|
||||
#:
|
||||
#: printf 'LIST VAR apc ups.test.result\nLIST VAR cyberpower ups.test.result\n' |
|
||||
#: nc <upsd-host> 3051
|
||||
#:
|
||||
#: These are the sentences the "Last self-test" check has to grade, and they are not the
|
||||
#: words its pass tuple was written for (AGENTS.md rule 7 — verify against the deployed
|
||||
#: upstream): the CyberPower has had a test run and answers `Done and passed`, the APC has
|
||||
#: not and still answers the factory `No test initiated`. The same two words are what
|
||||
#: `/api/summary` was putting in the Warnings panel on the capture date, one per UPS.
|
||||
#: Re-captured with `./scripts/capture-fixtures.sh` the same day, so `nut_*.vars` carries
|
||||
#: them verbatim; if a future capture changes one, the first test below fails and the word
|
||||
#: map and this table get updated together.
|
||||
LIVE_SELF_TEST_WORDS = {
|
||||
"apc": "No test initiated",
|
||||
"cyberpower": "Done and passed",
|
||||
}
|
||||
|
||||
|
||||
def self_test_check(health: dict) -> dict:
|
||||
return next(check for check in health["checks"] if check["name"] == "Last self-test")
|
||||
|
||||
|
||||
def test_the_captured_self_test_words_are_what_the_fixtures_carry() -> None:
|
||||
"""The capture and the verbatim fixture must not drift apart.
|
||||
|
||||
`assess()` is asserted against `LIVE_SELF_TEST_WORDS` below, so a fixture re-captured
|
||||
after somebody runs a real battery test has to land in the same table — the day the two
|
||||
disagree, the verdict is being tested against a memory of the hardware instead of the
|
||||
hardware (`AGENTS.md` rule 7).
|
||||
"""
|
||||
for ups, word in LIVE_SELF_TEST_WORDS.items():
|
||||
capture = (Path(__file__).parent / "fixtures" / f"nut_{ups}.vars").read_text()
|
||||
assert f'VAR {ups} ups.test.result "{word}"' in capture, (
|
||||
f"nut_{ups}.vars no longer answers the captured word {word!r} — re-run "
|
||||
f"./scripts/capture-fixtures.sh and update LIVE_SELF_TEST_WORDS with the diff"
|
||||
)
|
||||
assert parse_capture(ups).get("test_result") == word
|
||||
|
||||
|
||||
def test_a_passed_self_test_reads_ok_and_not_a_warning() -> None:
|
||||
"""The task-01 pin, now green: the CyberPower's pass word must read `ok`.
|
||||
|
||||
Before phase 12 `assess()` compared the driver's word against
|
||||
`("ok", "passed", "pass", "completed")`, so `Done and passed` fell through to `warn` and
|
||||
the overview printed a warning for a battery that passed.
|
||||
"""
|
||||
health = nut.assess(parse_capture("cyberpower"), now=1_767_225_600.0)
|
||||
check = self_test_check(health)
|
||||
assert check["status"] == "ok", (
|
||||
f"the driver reported {check['detail']!r} and it graded {check['status']!r}"
|
||||
)
|
||||
assert check["detail"].startswith(LIVE_SELF_TEST_WORDS["cyberpower"]), (
|
||||
"the driver's own sentence is the useful part of this line")
|
||||
|
||||
|
||||
def test_a_self_test_that_was_never_initiated_is_not_a_warning() -> None:
|
||||
"""The second half of the same pin: the APC's factory state is not a fault.
|
||||
|
||||
A pack nobody has ever self-tested is a suggestion to run one, so it reads `unknown`
|
||||
with the unset branch's sentence instead of `warn`.
|
||||
"""
|
||||
health = nut.assess(parse_capture("apc"), now=1_767_225_600.0)
|
||||
check = self_test_check(health)
|
||||
assert check["status"] == "unknown", (
|
||||
f"never tested ({check['detail']!r}) must read like the unset case, not "
|
||||
f"{check['status']!r}"
|
||||
)
|
||||
assert check["detail"] == nut.NEVER_TESTED_DETAIL
|
||||
assert "run a battery test" in check["detail"]
|
||||
|
||||
|
||||
#: The whole `Last self-test` outcome map: the driver's word -> the verdict `assess()` gives.
|
||||
#: The pass tier holds every pass word this fleet's drivers, the simulator, and the older
|
||||
#: NUT drivers answer — including `Done and passed`, task 01's live capture and the one the
|
||||
#: old tuple missed. The fail tier is unchanged: a test that ran and failed is the reason the
|
||||
#: check exists. `unknown` holds both "no test has ever run" states. `warn` is the deliberate
|
||||
#: fall-through for a word this map has never seen — a driver inventing a verdict is upstream
|
||||
#: drift worth a line, and the check prints that word verbatim so it can be added.
|
||||
SELF_TEST_WORD_MAP = {
|
||||
"ok": "ok",
|
||||
"passed": "ok",
|
||||
"pass": "ok",
|
||||
"completed": "ok",
|
||||
LIVE_SELF_TEST_WORDS["cyberpower"]: "ok",
|
||||
"FAILED": "fail",
|
||||
"fail": "fail",
|
||||
"Error": "fail",
|
||||
"aborted": "fail",
|
||||
"test incomplete": "fail",
|
||||
LIVE_SELF_TEST_WORDS["apc"]: "unknown",
|
||||
"": "unknown",
|
||||
"mystery": "warn",
|
||||
"DONE AND FAILED": "warn",
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("word,expected", sorted(SELF_TEST_WORD_MAP.items()))
|
||||
def test_the_self_test_check_grades_the_outcome_and_not_the_phrasing(word, expected) -> None:
|
||||
check = self_test_check(nut.assess(reading(**{"battery.test.result": word}),
|
||||
now=1_767_225_600.0))
|
||||
assert check["status"] == expected, f"{word!r} graded {check['status']!r}"
|
||||
if expected == "unknown":
|
||||
assert check["detail"] == nut.NEVER_TESTED_DETAIL
|
||||
else:
|
||||
assert check["detail"].startswith(word), "the raw word is what the operator reads"
|
||||
assert "2026-09-01" in check["detail"], "the test_date suffix stays on the line"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("variant", [
|
||||
"Done and passed", "DONE AND PASSED", "done and passed", " Done and passed ",
|
||||
])
|
||||
def test_the_live_pass_word_is_matched_case_and_whitespace_insensitively(variant) -> None:
|
||||
assert nut.self_test_outcome(variant) == "ok"
|
||||
check = self_test_check(nut.assess(reading(**{"battery.test.result": variant}),
|
||||
now=1_767_225_600.0))
|
||||
assert check["status"] == "ok"
|
||||
|
||||
|
||||
def test_never_tested_says_the_same_sentence_as_no_variable_at_all() -> None:
|
||||
"""`No test initiated` and a driver exporting nothing are one sentence, not two.
|
||||
|
||||
Both are "run a battery test": a check that invented a second phrasing for the same
|
||||
nothing would be two warnings to learn instead of one thing to do.
|
||||
"""
|
||||
values = {key: value for key, value in VAR_ROWS.items()
|
||||
if not key.startswith("battery.test")}
|
||||
unset = self_test_check(nut.assess(nut.UpsReading(name="apc", vars=values, commands=[]),
|
||||
now=1_767_225_600.0))
|
||||
never = self_test_check(nut.assess(reading(**{
|
||||
"battery.test.result": LIVE_SELF_TEST_WORDS["apc"]}), now=1_767_225_600.0))
|
||||
assert unset["status"] == never["status"] == "unknown"
|
||||
assert never["detail"] == unset["detail"] == nut.NEVER_TESTED_DETAIL
|
||||
|
||||
|
||||
def test_no_word_the_real_drivers_answer_lands_in_warn() -> None:
|
||||
"""Whatever the two boxes answer must be a word the map knows.
|
||||
|
||||
`warn` is the fall-through, so a captured word reaching it means the overview is warning
|
||||
about a battery this code has simply never read before — the exact bug this phase fixes.
|
||||
"""
|
||||
for ups, word in LIVE_SELF_TEST_WORDS.items():
|
||||
check = self_test_check(nut.assess(parse_capture(ups), now=1_767_225_600.0))
|
||||
assert check["status"] != "warn", f"{ups} answers {word!r} and it graded warn"
|
||||
assert check["status"] == nut.self_test_outcome(word)
|
||||
|
||||
|
||||
def test_the_outcome_map_holds_only_verdicts_the_pages_render() -> None:
|
||||
"""A typo'd tier in the table would reach the browser as an unknown CSS class."""
|
||||
assert set(nut.SELF_TEST_OUTCOMES.values()) <= {"ok", "fail", "unknown"}
|
||||
assert nut.self_test_outcome(None) == "unknown", "no variable is not a verdict"
|
||||
|
||||
|
||||
def test_the_simulator_only_answers_words_the_map_grades() -> None:
|
||||
"""The model's `PASSED` / `FAILED` were already in the right tiers (task 02 item 4).
|
||||
|
||||
If the simulator ever learns a third word it goes into `SELF_TEST_OUTCOMES`; it gets no
|
||||
special case here, because a model the health check grades differently from the hardware
|
||||
is a dev box that lies about `/ups`.
|
||||
"""
|
||||
sim = SimUps("apc", "Smart-UPS 1500", "APC", "AS-1")
|
||||
now = time.time()
|
||||
assert nut.self_test_outcome(sim.test_result) == "ok"
|
||||
assert nut.self_test_outcome("FAILED") == "fail"
|
||||
card = nut.payload(UpsReading(name=sim.name, vars=sim.vars(now), commands=[]), now=now)
|
||||
assert self_test_check(card["health"])["status"] == "ok"
|
||||
|
||||
|
||||
def test_alarms_drive_the_overall_verdict() -> None:
|
||||
health = nut.assess(reading(**{"ups.alarm": "BATT.FAIL", "battery.temperature": "27.4"}),
|
||||
now=1_767_225_600.0)
|
||||
|
||||
Reference in New Issue
Block a user