keyrelay/.agents/HW_VERIFICATION.md

HW_VERIFICATION.md — the device half of this roadmap's verification

The host gate (.agents/validate.sh) proves what can be proven without a board: the pure-module suites under sanitisers, the page and tool suites, both firmware builds, repo hygiene. It deliberately contains no -t upload, no curl to a device, no network fetch — an upload in a gate is a device side effect on somebody else's schedule (AGENTS.md).

That leaves claims that only a device can answer. AGENTS.md's standard: "A feature that lives on the device is not done when the source builds — it is done when the device is running it and something observed it." This file pairs every such claim in phases 02–05 with the exact command that observes it and the observation that counts, and states the rule that keeps a unit from ever blocking on a board nobody is holding.

bash tools/hw_probe.sh (printed at the end of every gate run, exit status deliberately ignored) says what is reachable right now — serial nodes, keyrelay.local, http/https, uv/node, target-side evdev nodes — without side effects and without blocking. It informs; it never gates and never records an outcome by itself.

Baseline (measured, phase 01 task 01 gate run)

Later phases quote a delta against these numbers, never a fresh guess.

metric	value
bridge RAM	26.7 % (87 592 / 327 680 B)
bridge Flash	36.1 % (1 135 293 / 3 145 728 B)
console RAM	16.8 % (55 200 / 327 680 B)
console Flash	33.6 % (1 056 081 / 3 145 728 B)
host checks	3406 (3162 C++ + 160 relayctl + 84 page), green in the normal and the SANITIZE=1 run
served page	48 598 bytes — the raw-string length the gate prints every run; a reflashed device must serve exactly this many

Phase 02 additionally quotes the heap cost of one and two live TLS sessions as a delta against the bridge RAM/heap line above.

Claims that need hardware

Outcome vocabulary, written into the row in the phase's commit by whoever ran the command: ok · skipped (no device) · failed: <what>. Blank = not run yet. A row nobody ran is recorded skipped (no device) at phase close — an unobserved claim is never back-filled with a guess (ASSUMPTION: unattended runs may have a board attached that nobody is watching, and a second machine for watch_target.py is rarer than either).

Every phase that touches the keystroke path (02–05 regression baseline)

claim	command	expected observation	outcome
keystrokes reach the target	python3 tools/watch_target.py --seconds 60 on the target, driven from the page or tools/relayctl.py	one press and release per key with timestamps; still held absent from the summary	skipped (no passphrase) — task 04, 2026-11-09: the board is reachable on :443 and the TLS/wss:// transport rows below are measured, but this board's control passphrase is not the default (status says dflt:0) and a verification run will not change a credential on a device it cannot put back. To close it: console.py token <known> on a board you are willing to re-provision, then drive wss://
a chord is the keys its label claims	python3 tools/relayctl.py --host keyrelay.local chord 04 2b while watch_target.py runs	LAlt press then Tab press — and not LCtrl. The watcher is the only thing that says what the target received (README: Alt+Tab was Ctrl+Alt+A on the wire once)
served page == built page	curl -s http://keyrelay.local/ | wc -c (after phase 02 lands: curl -sk https://keyrelay.local/ | wc -c)	byte-equal to the page bytes: line the gate prints. A mismatch means the page in flash is stale — reflash and reload any open tab	ok — re-measured 2026-11-09 over TLS after the phase-02 flash: curl -sk https://10.2.0.22/ = 48598 and curl -s http://keyrelay.local/ = 48598, both = the gate's count. Task 04 re-ran it over TLS three times in a row: 48 598 bytes each time with sha256 equal to the raw string in src/web_page.h (f7aae9e5…c6f7) — the page now leaves through kr::Conn at 1460 B per write and 5840 B per tick (34 chunks over ~9 loop() iterations) instead of one WiFiClient::print, so byte-equality across a multi-tick send is the transport claim. Original — measured 2026-11-09: curl -s http://keyrelay.local/ | wc -c = 48598 = gate's count (board on the LAN, pre-TLS, serving over http as expected). Re-run by the phase 01 final pass, one capped attempt each: 200 / 48 598 bytes, 404 on /nope, 401 on /status without the token, zero src=/href= in the served bytes. The first cold probe returned nothing (http_code=000, glibc does not cache mDNS) and the retry worked — that retry is the documented behaviour, not a second attempt at the check
silence disarms	arm, kill the controller process without a clean close (or close the tab), wait kNoClientTimeoutMs (8 s), python3 tools/relayctl.py --host keyrelay.local status	armed: 0, and watch_target.py shows releases for anything held. Test the timer, not the socket close (README bug 10)	skipped (no passphrase), same reason as the row above. What task 04 did pin is the property this row depends on and that the transport rewrite could silently have moved: _lastInboundMs is stamped in exactly one place, gated on _phase == WebSocket && _authed, and neither the TLS handshake, a /status poll nor a viewer refresh reaches it — asserted structurally in test_relayctl (the controller clock is stamped in exactly one place, no HTTP path can refresh the controller clock)
nothing outlives the link	hold a key and a mouse button, then drop the link	every key up, every button up; mouse summary shows no still held

Phase 02 — HTTPS + wss://

Every row below was produced by one capped command on the board at 10.2.0.22 / keyrelay.local, joined to the LAN, flashed with bridge. Two rows are honestly skipped: the passphrase-dependent ones (arming/typing over a pinned session need a credential the operator owns) and the rollback's device leg; one row is failed and fixed in the tree with its device leg still open (the certificate's SAN, found at the final pass) — see the rows themselves for what would close them and why it was not run.

claim	command	expected observation	outcome
TLS handshake completes	curl -sk https://keyrelay.local/status or openssl s_client -connect keyrelay.local:443 </dev/null	200 + status JSON / a session whose only verify failure is self-signed. HTTP 000 / connection refused = listener not up	ok — measured 2026-11-09 on the flashed bridge: openssl s_client negotiates TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384; curl -sk https://10.2.0.22/ → 48 598 bytes (= the gate's count); /status without the token over TLS → 401, path still gated. hw_probe's https line now reports HTTP 200 (TLS listener is up)
fingerprint printed once, on serial	serial console boot log after nvs erase; then python3 tools/console.py cert (opens the port — one attempt, capped)	SHA-256 fingerprint appears exactly once at first boot, on demand afterwards; never the key; survives reflashing (NVS)	ok (minus the nvs erase leg) — measured 2026-11-09: captured boot log shows [tls] identity loaded from NVS (reprint with \cert`)with **no** fingerprint on it, andconsole.py certprints it on demand; the identity survived two reflashes and three reboots with the same fingerprint.nvs erase→ first-boot printing was not re-run (the regeneration path had just been observed end-to-end). **Finding:** the *first* boot under the new firmware rotated the fingerprint once for a reason the boot log could not show (no host had the CDC open yet);loadTlsIdentity()/storeTlsIdentity() now print the failing gate and the record **lengths** (stored records cert=… key=… fp=… -- shape/selfCheck, NVS put refused: …, read-back mismatch: …`) so a rotation can never again be a mystery — every later boot loads cleanly. **(That first
"rotation" is corrected two rows down: it was the openssl -ext artifact, not a regenerated
identity. The instrumentation stays worth having — it is what makes the next claim provable.)**
pinned fingerprint == handshake cert	openssl s_client -connect keyrelay.local:443 </dev/null 2>/dev/null | openssl x509 -noout -fingerprint -sha256	same hex as the serial console printed; mismatch = two identities, investigate before pinning	ok — measured 2026-11-09: console cert 61:54:19:30:ed:df:e5:52:d1:ee:18:6f:cc:f9:57:c5:41:47:d4:15:61:26:1e:11:aa:49:4f:07:79:b1:af:f9 == the x509 -fingerprint -sha256 of the live handshake, and the pair still matched after two further reboot cycles. Corrected at the phase-02 final pass, same day: the earlier mismatch recorded as C8:76:F1:69:… was not a rotation — it is what openssl x509 -noout -fingerprint -sha256 prints when -ext <oid> is passed in the same invocation, on OpenSSL 4.0.3, for this same certificate. Reproduced byte for byte: the peer certificate captured with s_client, written with -outform der, hashes to 61541930…b1aff9; the same file run through -fingerprint -sha256 -ext subjectAltName prints C8:76F1…E6D2 and “No extensions in certificate”. So: take the fingerprint with -fingerprint -sha256 alone (or sha256sum of -outform der) and never with -ext in the same command, or a stable identity looks like a rotation. Re-measured at the final pass (console.py cert, one capped attempt, and s_client + `openssl x509 -outform der
:80 is redirect-only	curl -s -o /dev/null -w '%{http_code} %{redirect_url}' http://keyrelay.local/	3xx to https://, and no page bytes on plaintext	ok — measured 2026-11-09 on the task-04 flash, one capped curl each: / → 301 Location: https://10.2.0.22/; /index.html → 301 to the same path over https://; /nope → 301 to / (an unknown target is a 404 on 443, and an arbitrary request target is never echoed into a Location); /status?token=… → 426 with Upgrade: TLS/1.2, HTTP/1.1, no Location, and the body verified free of the token (grep -c of the value → 0). No plaintext path returns page bytes any more
plaintext /ws refused, not downgraded	a ws:// upgrade attempt on :80 (e.g. curl -i -N with upgrade headers, or relayctl.py pointed at plaintext)	refused loudly (per README error style), never a silent plaintext session	ok — measured 2026-11-09: curl -i -N with Upgrade: websocket on :80 → 426 Upgrade Required, Connection: Upgrade, Upgrade: websocket, TLS/1.2, HTTP/1.1, body plaintext websocket refused; reconnect over wss:// (https://<host>) — no 101, no session; the board's own counter moved (link controller=no rejected=1). relayctl's plaintext path now raises the same sentence via handshake_refusal() rather than the old "board refused the websocket upgrade". Re-measured at the phase-02 final pass (the bodies were reworded after that flash): /ws upgrade → 426, Upgrade: TLS/1.2, HTTP/1.1, Connection: close, Content-Length: 96, body keyrelay: this controller channel is TLS-only -- reconnect over wss://, the page is on https://; /status?token=<dummy> → 426, Content-Length: 140, no Location: (grep -c location: → 0) and the value not echoed (grep -c <dummy> → 0); / → 301 to https://keyrelay.local/ with a 77-byte redirect body and no page bytes; over :443: /status without the token → 401, /nope → 404, / → 200 / 51 626 B / sha256 dd16e195…f04e9 == the locally assembled page
TLS session heap cost	build RAM delta from the gate output + live free-heap readout the phase's status/serial path exposes, at 0 / 1 / 2 live sessions	quoted as a delta against the baseline table above; two sessions must fit without allocation failure	ok — measured 2026-11-09 from the serial [tls] sessions=N heap=… lines: boot [tls] https :443 listening heap=187936; 1 session ≈ 145.5 k, 2 sessions ≈ 107.9 k (second session ≈ −37.6 k: the mbed TLS record buffers), close returns heap to 168–185 k. Task 04 (transport view on both ports: 2 KB :80 redirect ring + 1460 B page chunk + 64 B tRing per slot; gate RAM 97 240 → 99 616) measured in one run sessions=1 140 092 → sessions=2 109 228 → close one 144 836 → close all 182 256: second session ≈ −31 k, and the memory comes back inside the band above. 20 further connect/disconnect cycles on :443 ran in the same pass. Original 20 connect/disconnect cycles: the sessions=1 readings held at 145.5–149.8 k with no downward drift — no leak. Slot cap enforced live: with two sessions held, a third got a fatal TLS alert (handshake_failure, 40) in 0.0 s while both listeners kept serving. Stall check: a connection that completed the handshake and sent nothing did not block loop() — the full page still served over a second connection during the stall, and the stalled socket was closed by the ~10 s deadline
-D KR_TLS=0 rollback behaves like today	build the rollback env, flash once, curl -s http://keyrelay.local/ | wc -c + page connects over ws://	page and ws:// work exactly as pre-phase-02; README documents the rollback	build leg ok, device leg skipped (not flashed) — task 06, 2026-11-09. The rollback is now a first-class environment, [env:rollback], built by .agents/validate.sh on every run alongside bridge and console: RAM 28.5 % (93 456 B) / Flash 36.4 % (1 146 137 B) against the TLS bridge 30.4 % (99 616 B) / 39.6 % (1 245 353 B), so TLS costs +6 160 B RAM and +99 216 B flash. (Final pass, same day: after the SAN fix bridge is 99 616 B / 1 246 145 B — RAM unmoved, +792 B of flash — and the rollback image is byte-for-byte unchanged at 93 456 B / 1 146 137 B, because the fix lives entirely inside #if KR_TLS; TLS now costs +100 008 B of flash.) That the flag took is measured, not assumed: xtensa-esp-elf-nm .pio/build/rollback/firmware.elf | grep -c mbedtls_ssl_ → 0 (115 in bridge), … | grep -c mbedtls_x509write → 0 (15), strings .pio/build/rollback/firmware.elf | grep -c "pin this certificate" → 0 (1). The device half was deliberately not run: flashing the rollback and back is two write cycles on a board that is a live HID keyboard into the machine running this pipeline, to_bootloader.py strands it in ROM download mode if anything goes wrong after that point, and no criterion for task 06 needed it. To close it, in order: uv run --python 3.14 --with platformio python -m platformio run -e rollback -t upload → timeout 10 curl -s http://keyrelay.local/ | wc -c (expect the same 51 626 the gate prints, on plaintext, with var KR_TLS_BUILD=0; from curl -s http://keyrelay.local/ | grep -o 'var KR_TLS_BUILD=[01];') → timeout 10 curl -s -o /dev/null -w '%{http_code}' https://keyrelay.local/ (expect 000/refused: no :443 exists any more) → boot log line [tls] TLS compiled out (KR_TLS=0) -- plaintext control channel → python3 tools/relayctl.py --plain --host keyrelay.local --token … status → then reflash the TLS build (run -e bridge -t upload) and confirm curl -sk serves 51 626 again
TLS handshake latency and page fetch over TLS	curl -sk -o /dev/null -w '%{time_appconnect} %{time_total}\n' https://keyrelay.local/ (and …/status), one capped run each	a handshake that completes in a time a browser operator will not mistake for a dead board, and a page that arrives whole over the chunked TLS send	ok — measured 2026-11-09 (task 06, read-only, board on the LAN): seven warm connections, handshake 0.349–0.423 s; page 51 626 B in 0.528–0.618 s total; /status 0.382–0.434 s total. The first connection after ~10 min of quiet took 3.94 s of handshake (one observation, recorded because it looks like a timeout from a browser and is a cold start). Both numbers are in README “Measured behaviour” with the command
served page carries the build marker and the pinning UI	curl -sk https://keyrelay.local/ | wc -c + grep -o 'var KR_TLS_BUILD=[01];' + sha256sum against the locally assembled page (chunks + KR_TLS=1 branch)	51 626 bytes, marker =1, certFp/pTls/clearWarn present, digest equal to the assembled TLS page	ok — measured 2026-11-09 after the task-05 flash: 51 626 bytes, var KR_TLS_BUILD=1;, all three new ids (6 matches), sha256 dd16e195…f04e9 byte-equal to the locally assembled page — the multi-tick chunked send still delivers the changed page intact, and curl -sI http://keyrelay.local/ is still a 301
relayctl.py pinning, end to end	--fingerprint 00…00 / the real fingerprint / --ca <exported self-signed> / --insecure / no argument / --plain, each timeout 15, dummy token	wrong pin aborts before sending; correct pin passes the pin and stops only at authentication; --ca verifies the self-signed cert with hostname checking on; --insecure warns on stderr every run; no argument → refusal at exit 2 without a socket; --plain reads the 426 as a TLS refusal, never as a passphrase problem	ok — measured 2026-11-09: wrong pin → certificate fingerprint mismatch … Nothing was sent (and the presented digest 61:54:19:30:ed:df:…:b1:af:f9 equals the serial cert value recorded above — the pin and the console agree on one identity across reflashes); correct pin, --insecure and --ca /tmp/board.pem all reached authentication and were refused there by E1 (the board's passphrase is the operator's — I did not guess it, so arming/typing over a pinned session is not device-confirmed in this row, three authfails total, well under the 5-try lockout); no argument → the four-option refusal, exit 2, no connection attempted; --plain → board refuses plaintext … wss:// on :443, not one word about the passphrase

| the certificate carries the name list cert prints | openssl s_client -connect keyrelay.local:443 </dev/null \| openssl x509 -noout -text, compared with console.py cert | the X509v3 Subject Alternative Name extension reads DNS:keyrelay.local, DNS:keyrelay, IP:192.168.4.1, so a client can check the name and not only the fingerprint | failed → fixed in the tree, device leg needs nvs erase. Found at the phase-02 final pass 2026-11-09 by reading the certificate instead of the code: -text printed the extension as 1.10.46.53.46.50.57.46.49.55 (not X509v3 Subject Alternative Name) and -ext subjectAltName said no extensions. A hex dump named the cause — the extension was framed 30 2f 06 09 32 2e 35 2e 32 39 2e 31 37 04 22 <the correct 34 bytes>: the OID was written from the ASCII text "2.5.29.17" where the DER content bytes 55 1d 11 belong, because mbedtls_x509write_crt_set_extension() takes the OID as DER content bytes. So the served certificate has the right SAN value under a label no client reads, i.e. no SAN: encryption and pinning work, hostname validation cannot (browsers match SAN, not CN). Fixed in src/tls_pem.h (kSanOidTlv/sanOidDer, sanExtensionFullDer pinned to the 43 bytes OpenSSL writes for these three names, derCarriesSan requiring OID and value), used by src/tls.cpp, 33 new host checks in test_tls_pem.cpp; the regression fixture is the bytes above, and derCarriesSan answers no to it while derContains says yes. README bug 11. To close the device leg (deliberately not run here: nvs erase also forgets the passphrase and the Wi-Fi credentials, which the verification run cannot put back): reflash bridge, nvs erase, reboot, then openssl s_client … \| openssl x509 -noout -text must show X509v3 Subject Alternative Name: DNS:keyrelay.local, DNS:keyrelay, IP:192.168.4.1, the new fingerprint must appear exactly once in the boot log, and console.py cert must print that same value with no !! the stored certificate does NOT carry that name list warning. Until then cert prints that warning for this board — which it now does, because the check looks at the OID. |

Phase 03 — session takeover

claim	command	expected observation	outcome
takeover releases the incumbent's held keys	two controllers on one board: incumbent holds a key, newcomer takes over (relayctl.py --take-over), all while watch_target.py runs on the target	watch_target.py shows the key up before the new session's first key-down — the ordering is the whole claim
the board comes back disarmed	python3 tools/relayctl.py --host keyrelay.local status immediately after the handover	armed: 0 across the handover; the newcomer must arm again on purpose; nothing in the handover path arms
a third controller is still refused	a third /ws upgrade while two sessions exist	E3 — takeover consumes the one spare slot, it does not open the door

Phase 04 — responsive page

claim	command	expected observation	outcome
page still byte-exact after the restructure	the generic curl … | wc -c row above	equals the gate's new page bytes: count (the count moves when the page does — that is fine; equality with the gate is the check)
one column on a phone, ARM thumb-reachable	open https://keyrelay.local/ on a phone at ~360 px	single column, no horizontal scroll, sticky ARM bar on screen, log collapsed and not pushing ARM off screen

Phase 05 — soft keyboard and touch pad

claim	command	expected observation	outcome
arm on a phone raises the soft keyboard, characters land	phone on the board's AP/network, tap ARM, type a word; watch_target.py on the target	phone keyboard appears; the watcher's typed: line reproduces exactly what was typed, one P=0: character per frame
a latched modifier chords correctly	latch Ctrl on the row, tap c; watch on the target	LCtrl press + C press — Ctrl+C on the wire, latch visibly cleared after
pad moves the pointer; lift releases everything	press-and-hold the pad, move, lift; hold a button meanwhile, then drop the link	movement dx … dy … lines that stop at lift; still held absent from the mouse summary; no button stuck

The blocking rule

A unit running any of the rows above:

1. Look first, side-effect-free. bash tools/hw_probe.sh (always exits 0, every probe capped, no serial port opened — opening /dev/ttyACM* toggles DTR/RTS and resets the board). It tells you whether an attempt is plausible.
2. One attempt, capped. Each device step runs at most once, wrapped in timeout (or the tool's own -m). No polling a port, no retry loops, no re-powering, no re-flashing "to see if it works". A timeout is an outcome (failed: timeout … or skipped (no device)), not a cue to try again.
3. Record what was observed. The outcome — ok, skipped (no device), failed: … — plus the exact command written goes into the row above, in the phase's commit. Never write ok for a command that did not run; the unattended-run contract is that the human picks these rows up later.
4. Never strand the board. tools/to_bootloader.py drops the board into ROM download mode deliberately, and a power cycle is the only way back. If a run leaves the board in download mode, its report says so explicitly — nobody may be there to notice.
5. Never block the gate. None of this is inside .agents/validate.sh. The gate stays device-free and network-free; the probe line at its end is context whose exit status is deliberately discarded.