keyrelay/.agents/HW_VERIFICATION.md

# HW_VERIFICATION.md — the device half of this roadmap's verification

The host gate (`.agents/validate.sh`) proves what can be proven without a board:
the pure-module suites under sanitisers, the page and tool suites, both firmware
builds, repo hygiene. It deliberately contains **no** `-t upload`, no `curl` to a
device, no network fetch — an upload in a gate is a device side effect on
somebody else's schedule (AGENTS.md).

That leaves claims that only a device can answer. AGENTS.md's standard:
*"A feature that lives on the device is not done when the source builds — it is
done when the device is running it and something observed it."* This file pairs
every such claim in phases 02–05 with the exact command that observes it and the
observation that counts, and states the rule that keeps a unit from ever blocking
on a board nobody is holding.

`bash tools/hw_probe.sh` (printed at the end of every gate run, exit status
deliberately ignored) says what is reachable *right now* — serial nodes,
`keyrelay.local`, http/https, `uv`/`node`, target-side evdev nodes — without
side effects and without blocking. It informs; it never gates and never records
an outcome by itself.

## Baseline (measured, phase 01 task 01 gate run)

Later phases quote a **delta against these numbers**, never a fresh guess.

| metric | value |
|---|---|
| `bridge` RAM | 26.7 % (87 592 / 327 680 B) |
| `bridge` Flash | 36.1 % (1 135 293 / 3 145 728 B) |
| `console` RAM | 16.8 % (55 200 / 327 680 B) |
| `console` Flash | 33.6 % (1 056 081 / 3 145 728 B) |
| host checks | **3406** (3162 C++ + 160 `relayctl` + 84 page), green in the normal **and** the `SANITIZE=1` run |
| served page | **48 598 bytes** — the raw-string length the gate prints every run; a reflashed device must serve exactly this many |

Phase 02 additionally quotes the heap cost of one and two live TLS sessions as a
delta against the `bridge` RAM/heap line above.

## Claims that need hardware

Outcome vocabulary, written into the row **in the phase's commit** by whoever ran
the command: `ok` · `skipped (no device)` · `failed: <what>`. Blank = not run
yet. A row nobody ran is recorded `skipped (no device)` at phase close — an
unobserved claim is never back-filled with a guess (ASSUMPTION: unattended runs
may have a board attached that nobody is watching, and a second machine for
`watch_target.py` is rarer than either).

### Every phase that touches the keystroke path (02–05 regression baseline)

| claim | command | expected observation | outcome |
|---|---|---|---|
| keystrokes reach the target | `python3 tools/watch_target.py --seconds 60` **on the target**, driven from the page or `tools/relayctl.py` | one press **and** release per key with timestamps; `still held` absent from the summary | skipped (no passphrase) — task 04, 2026-11-09: the board is reachable on `:443` and the TLS/`wss://` transport rows below are measured, but this board's control passphrase is not the default (`status` says `dflt:0`) and a verification run will not change a credential on a device it cannot put back. To close it: `console.py token <known>` on a board you are willing to re-provision, then drive `wss://` |
| a chord is the keys its label claims | `python3 tools/relayctl.py --host keyrelay.local chord 04 2b` while `watch_target.py` runs | `LAlt press` then `Tab press` — and **not** `LCtrl`. The watcher is the only thing that says what the target received (README: Alt+Tab was `Ctrl+Alt+A` on the wire once) | |
| served page == built page | `curl -s http://keyrelay.local/ \| wc -c` (after phase 02 lands: `curl -sk https://keyrelay.local/ \| wc -c`) | byte-equal to the `page bytes:` line the gate prints. A mismatch means the page in flash is stale — reflash **and** reload any open tab | ok — re-measured 2026-11-09 **over TLS after the phase-02 flash**: `curl -sk https://10.2.0.22/` = 48598 and `curl -s http://keyrelay.local/` = 48598, both = the gate's count. **Task 04** re-ran it over TLS three times in a row: 48 598 bytes each time with `sha256` equal to the raw string in `src/web_page.h` (`f7aae9e5…c6f7`) — the page now leaves through `kr::Conn` at 1460 B per write and 5840 B per tick (34 chunks over ~9 `loop()` iterations) instead of one `WiFiClient::print`, so byte-equality across a multi-tick send is the transport claim. Original — measured 2026-11-09: `curl -s http://keyrelay.local/ \| wc -c` = 48598 = gate's count (board on the LAN, pre-TLS, serving over http as expected). Re-run by the phase 01 final pass, one capped attempt each: 200 / 48 598 bytes, 404 on `/nope`, 401 on `/status` without the token, zero `src=`/`href=` in the served bytes. The first cold probe returned nothing (`http_code=000`, glibc does not cache mDNS) and the retry worked — that retry is the documented behaviour, not a second attempt at the check |
| silence disarms | arm, kill the controller process without a clean close (or close the tab), wait `kNoClientTimeoutMs` (8 s), `python3 tools/relayctl.py --host keyrelay.local status` | `armed: 0`, and `watch_target.py` shows releases for anything held. Test the *timer*, not the socket close (README bug 10) | skipped (no passphrase), same reason as the row above. What task 04 *did* pin is the property this row depends on and that the transport rewrite could silently have moved: `_lastInboundMs` is stamped in exactly one place, gated on `_phase == WebSocket && _authed`, and neither the TLS handshake, a `/status` poll nor a viewer refresh reaches it — asserted structurally in `test_relayctl` (`the controller clock is stamped in exactly one place`, `no HTTP path can refresh the controller clock`) |
| nothing outlives the link | hold a key and a mouse button, then drop the link | every key up, every button up; mouse summary shows no `still held` | |

### Phase 02 — HTTPS + `wss://`

Every row below was produced by one capped command on the board at `10.2.0.22` /
`keyrelay.local`, joined to the LAN, flashed with `bridge`. Two rows are honestly `skipped`:
the passphrase-dependent ones (arming/typing over a pinned session need a credential the
operator owns) and the rollback's device leg; one row is `failed` and fixed in the tree with
its device leg still open (the certificate's SAN, found at the final pass) — see the rows
themselves for what would close them and why it was not run.

| claim | command | expected observation | outcome |
|---|---|---|---|
| TLS handshake completes | `curl -sk https://keyrelay.local/status` or `openssl s_client -connect keyrelay.local:443 </dev/null` | 200 + status JSON / a session whose only verify failure is *self-signed*. `HTTP 000` / connection refused = listener not up | ok — measured 2026-11-09 on the flashed bridge: `openssl s_client` negotiates **TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384**; `curl -sk https://10.2.0.22/` → 48 598 bytes (= the gate's count); `/status` without the token over TLS → **401**, path still gated. `hw_probe`'s `https` line now reports `HTTP 200 (TLS listener is up)` |
| fingerprint printed **once**, on serial | serial console boot log after `nvs erase`; then `python3 tools/console.py cert` (opens the port — one attempt, capped) | SHA-256 fingerprint appears exactly once at first boot, on demand afterwards; **never** the key; survives reflashing (NVS) | ok (minus the `nvs erase` leg) — measured 2026-11-09: captured boot log shows `[tls] identity loaded from NVS (reprint with \`cert\`)` with **no** fingerprint on it, and `console.py cert` prints it on demand; the identity survived two reflashes and three reboots with the same fingerprint. `nvs erase` → first-boot printing was not re-run (the regeneration path had just been observed end-to-end). **Finding:** the *first* boot under the new firmware rotated the fingerprint once for a reason the boot log could not show (no host had the CDC open yet); `loadTlsIdentity()`/`storeTlsIdentity()` now print the failing gate and the record **lengths** (`stored records cert=… key=… fp=… -- shape/selfCheck`, `NVS put refused: …`, `read-back mismatch: …`) so a rotation can never again be a mystery — every later boot loads cleanly. **(That first
"rotation" is corrected two rows down: it was the `openssl -ext` artifact, not a regenerated
identity. The instrumentation stays worth having — it is what makes the next claim provable.)** |
| pinned fingerprint == handshake cert | `openssl s_client -connect keyrelay.local:443 </dev/null 2>/dev/null \| openssl x509 -noout -fingerprint -sha256` | same hex as the serial console printed; mismatch = two identities, investigate before pinning | ok — measured 2026-11-09: console `cert` `61:54:19:30:ed:df:e5:52:d1:ee:18:6f:cc:f9:57:c5:41:47:d4:15:61:26:1e:11:aa:49:4f:07:79:b1:af:f9` == the `x509 -fingerprint -sha256` of the live handshake, and the pair still matched after two further `reboot` cycles. **Corrected at the phase-02 final pass, same day:** the earlier mismatch recorded as `C8:76:F1:69:…` was **not a rotation** — it is what `openssl x509 -noout -fingerprint -sha256` prints when `-ext <oid>` is passed in the same invocation, on OpenSSL 4.0.3, for *this same* certificate. Reproduced byte for byte: the peer certificate captured with `s_client`, written with `-outform der`, hashes to `61541930…b1aff9`; the same file run through `-fingerprint -sha256 -ext subjectAltName` prints `C8:76F1…E6D2` *and* “No extensions in certificate”. So: take the fingerprint with `-fingerprint -sha256` alone (or `sha256sum` of `-outform der`) and never with `-ext` in the same command, or a stable identity looks like a rotation. Re-measured at the final pass (`console.py cert`, one capped attempt, and `s_client` + `openssl x509 -outform der | sha256sum`): console `61:54:19:30:…:b1:af:f9` == `61541930eddfe552…79b1aff9`, one identity, one board, no rotation since the instrumented boot. |
| `:80` is redirect-only | `curl -s -o /dev/null -w '%{http_code} %{redirect_url}' http://keyrelay.local/` | 3xx to `https://`, and **no page bytes** on plaintext | ok — measured 2026-11-09 on the task-04 flash, one capped `curl` each: `/` → **301** `Location: https://10.2.0.22/`; `/index.html` → **301** to the same path over `https://`; `/nope` → **301** to `/` (an unknown target is a 404 on 443, and an arbitrary request target is never echoed into a `Location`); `/status?token=…` → **426** with `Upgrade: TLS/1.2, HTTP/1.1`, **no `Location`**, and the body verified free of the token (`grep -c` of the value → 0). No plaintext path returns page bytes any more |
| plaintext `/ws` refused, not downgraded | a `ws://` upgrade attempt on `:80` (e.g. `curl -i -N` with upgrade headers, or `relayctl.py` pointed at plaintext) | refused loudly (per README error style), never a silent plaintext session | ok — measured 2026-11-09: `curl -i -N` with `Upgrade: websocket` on `:80` → **426 Upgrade Required**, `Connection: Upgrade`, `Upgrade: websocket, TLS/1.2, HTTP/1.1`, body `plaintext websocket refused; reconnect over wss:// (https://<host>)` — no `101`, no session; the board's own counter moved (`link controller=no rejected=1`). `relayctl`'s plaintext path now raises the same sentence via `handshake_refusal()` rather than the old "board refused the websocket upgrade". **Re-measured at the phase-02 final pass** (the bodies were reworded after that flash): `/ws` upgrade → **426**, `Upgrade: TLS/1.2, HTTP/1.1`, `Connection: close`, `Content-Length: 96`, body `keyrelay: this controller channel is TLS-only -- reconnect over wss://, the page is on https://`; `/status?token=<dummy>` → **426**, `Content-Length: 140`, **no `Location:`** (`grep -c location: → 0`) and the value not echoed (`grep -c <dummy> → 0`); `/` → **301** to `https://keyrelay.local/` with a 77-byte redirect body and **no page bytes**; over `:443`: `/status` without the token → **401**, `/nope` → **404**, `/` → 200 / 51 626 B / `sha256 dd16e195…f04e9` == the locally assembled page |
| TLS session heap cost | build RAM delta from the gate output + live free-heap readout the phase's `status`/serial path exposes, at 0 / 1 / 2 live sessions | quoted as a delta against the baseline table above; two sessions must fit without allocation failure | ok — measured 2026-11-09 from the serial `[tls] sessions=N heap=…` lines: boot `[tls] https :443 listening heap=187936`; **1 session ≈ 145.5 k, 2 sessions ≈ 107.9 k** (second session ≈ −37.6 k: the mbed TLS record buffers), close returns heap to 168–185 k. **Task 04** (transport view on both ports: 2 KB `:80` redirect ring + 1460 B page chunk + 64 B `tRing` per slot; gate RAM 97 240 → 99 616) measured in one run `sessions=1` `140 092` → `sessions=2` `109 228` → close one `144 836` → close all `182 256`: second session ≈ −31 k, and the memory comes back inside the band above. 20 further connect/disconnect cycles on `:443` ran in the same pass. Original 20 connect/disconnect cycles: the `sessions=1` readings held at 145.5–149.8 k with **no downward drift** — no leak. Slot cap enforced live: with two sessions held, a third got a **fatal TLS alert (handshake_failure, 40) in 0.0 s** while both listeners kept serving. Stall check: a connection that completed the handshake and sent nothing did **not** block `loop()` — the full page still served over a second connection during the stall, and the stalled socket was closed by the ~10 s deadline |
| `-D KR_TLS=0` rollback behaves like today | build the rollback env, flash once, `curl -s http://keyrelay.local/ \| wc -c` + page connects over `ws://` | page and `ws://` work exactly as pre-phase-02; README documents the rollback | **build leg ok, device leg skipped (not flashed)** — task 06, 2026-11-09. The rollback is now a first-class environment, `[env:rollback]`, built by `.agents/validate.sh` on every run alongside `bridge` and `console`: RAM 28.5 % (93 456 B) / Flash 36.4 % (1 146 137 B) against the TLS `bridge` 30.4 % (99 616 B) / 39.6 % (1 245 353 B), so TLS costs +6 160 B RAM and +99 216 B flash. *(Final pass, same day: after the SAN fix `bridge` is 99 616 B / 1 246 145 B — RAM unmoved, +792 B of flash — and the rollback image is **byte-for-byte unchanged** at 93 456 B / 1 146 137 B, because the fix lives entirely inside `#if KR_TLS`; TLS now costs +100 008 B of flash.)* That the flag *took* is measured, not assumed: `xtensa-esp-elf-nm .pio/build/rollback/firmware.elf \| grep -c mbedtls_ssl_` → **0** (115 in `bridge`), `… \| grep -c mbedtls_x509write` → **0** (15), `strings .pio/build/rollback/firmware.elf \| grep -c "pin this certificate"` → **0** (1). The device half was deliberately not run: flashing the rollback and back is two write cycles on a board that is a live HID keyboard into the machine running this pipeline, `to_bootloader.py` strands it in ROM download mode if anything goes wrong after that point, and no criterion for task 06 needed it. To close it, in order: `uv run --python 3.14 --with platformio python -m platformio run -e rollback -t upload` → `timeout 10 curl -s http://keyrelay.local/ \| wc -c` (expect the same 51 626 the gate prints, on plaintext, with `var KR_TLS_BUILD=0;` from `curl -s http://keyrelay.local/ \| grep -o 'var KR_TLS_BUILD=[01];'`) → `timeout 10 curl -s -o /dev/null -w '%{http_code}' https://keyrelay.local/` (expect 000/refused: no `:443` exists any more) → boot log line `[tls] TLS compiled out (KR_TLS=0) -- plaintext control channel` → `python3 tools/relayctl.py --plain --host keyrelay.local --token … status` → then reflash the TLS build (`run -e bridge -t upload`) and confirm `curl -sk` serves 51 626 again |
| TLS handshake latency and page fetch over TLS | `curl -sk -o /dev/null -w '%{time_appconnect} %{time_total}\n' https://keyrelay.local/` (and `…/status`), one capped run each | a handshake that completes in a time a browser operator will not mistake for a dead board, and a page that arrives whole over the chunked TLS send | ok — measured 2026-11-09 (task 06, read-only, board on the LAN): seven warm connections, **handshake 0.349–0.423 s**; page 51 626 B in **0.528–0.618 s** total; `/status` 0.382–0.434 s total. The first connection after ~10 min of quiet took **3.94 s** of handshake (one observation, recorded because it looks like a timeout from a browser and is a cold start). Both numbers are in README “Measured behaviour” with the command |
| served page carries the build marker and the pinning UI | `curl -sk https://keyrelay.local/ \| wc -c` + `grep -o 'var KR_TLS_BUILD=[01];'` + `sha256sum` against the locally **assembled** page (chunks + KR_TLS=1 branch) | 51 626 bytes, marker `=1`, `certFp`/`pTls`/`clearWarn` present, digest equal to the assembled TLS page | ok — measured 2026-11-09 after the task-05 flash: 51 626 bytes, `var KR_TLS_BUILD=1;`, all three new ids (6 matches), `sha256 dd16e195…f04e9` **byte-equal** to the locally assembled page — the multi-tick chunked send still delivers the changed page intact, and `curl -sI http://keyrelay.local/` is still a 301 |
| `relayctl.py` pinning, end to end | `--fingerprint 00…00` / the real fingerprint / `--ca <exported self-signed>` / `--insecure` / no argument / `--plain`, each `timeout 15`, dummy token | wrong pin aborts **before sending**; correct pin passes the pin and stops only at authentication; `--ca` verifies the self-signed cert with hostname checking on; `--insecure` warns on stderr every run; no argument → refusal at exit 2 without a socket; `--plain` reads the 426 as a TLS refusal, never as a passphrase problem | ok — measured 2026-11-09: wrong pin → `certificate fingerprint mismatch … Nothing was sent` (and the presented digest `61:54:19:30:ed:df:…:b1:af:f9` equals the serial `cert` value recorded above — the pin and the console agree on one identity across reflashes); correct pin, `--insecure` and `--ca /tmp/board.pem` all reached authentication and were refused there by `E1` (the board's passphrase is the operator's — I did not guess it, so **arming/typing over a pinned session is not device-confirmed in this row**, three `authfails` total, well under the 5-try lockout); no argument → the four-option refusal, exit 2, no connection attempted; `--plain` → `board refuses plaintext … wss:// on :443`, not one word about the passphrase |

| the certificate carries the name list `cert` prints | `openssl s_client -connect keyrelay.local:443 </dev/null \| openssl x509 -noout -text`, compared with `console.py cert` | the `X509v3 Subject Alternative Name` extension reads `DNS:keyrelay.local, DNS:keyrelay, IP:192.168.4.1`, so a client can check the name and not only the fingerprint | **failed → fixed in the tree, device leg needs `nvs erase`.** Found at the phase-02 final pass 2026-11-09 by reading the certificate instead of the code: `-text` printed the extension as `1.10.46.53.46.50.57.46.49.55` (not `X509v3 Subject Alternative Name`) and `-ext subjectAltName` said *no extensions*. A hex dump named the cause — the extension was framed `30 2f 06 09 32 2e 35 2e 32 39 2e 31 37 04 22 <the correct 34 bytes>`: the OID was written from the **ASCII text** `"2.5.29.17"` where the DER content bytes `55 1d 11` belong, because `mbedtls_x509write_crt_set_extension()` takes the OID as DER content bytes. So the served certificate has the right SAN *value* under a label no client reads, i.e. **no SAN**: encryption and pinning work, hostname validation cannot (browsers match SAN, not CN). Fixed in `src/tls_pem.h` (`kSanOidTlv`/`sanOidDer`, `sanExtensionFullDer` pinned to the 43 bytes OpenSSL writes for these three names, `derCarriesSan` requiring OID *and* value), used by `src/tls.cpp`, 33 new host checks in `test_tls_pem.cpp`; the regression fixture is the bytes above, and `derCarriesSan` answers no to it while `derContains` says yes. README bug 11. **To close the device leg** (deliberately not run here: `nvs erase` also forgets the passphrase and the Wi-Fi credentials, which the verification run cannot put back): reflash `bridge`, `nvs erase`, reboot, then `openssl s_client … \| openssl x509 -noout -text` must show `X509v3 Subject Alternative Name: DNS:keyrelay.local, DNS:keyrelay, IP:192.168.4.1`, the new fingerprint must appear exactly once in the boot log, and `console.py cert` must print that same value with **no** `!! the stored certificate does NOT carry that name list` warning. Until then `cert` prints that warning for this board — which it now does, because the check looks at the OID. |

### Phase 03 — session takeover

| claim | command | expected observation | outcome |
|---|---|---|---|
| takeover releases the incumbent's held keys | two controllers on one board: incumbent holds a key, newcomer takes over (`relayctl.py --take-over`), all while `watch_target.py` runs **on the target** | `watch_target.py` shows the key **up** *before* the new session's first key-down — the ordering is the whole claim | |
| the board comes back disarmed | `python3 tools/relayctl.py --host keyrelay.local status` immediately after the handover | `armed: 0` across the handover; the newcomer must arm again on purpose; nothing in the handover path arms | |
| a third controller is still refused | a third `/ws` upgrade while two sessions exist | `E3` — takeover consumes the one spare slot, it does not open the door | |

### Phase 04 — responsive page

| claim | command | expected observation | outcome |
|---|---|---|---|
| page still byte-exact after the restructure | the generic `curl … \| wc -c` row above | equals the gate's new `page bytes:` count (the count moves when the page does — that is fine; equality with the gate is the check) | |
| one column on a phone, ARM thumb-reachable | open `https://keyrelay.local/` on a phone at ~360 px | single column, no horizontal scroll, sticky ARM bar on screen, log collapsed and not pushing ARM off screen | |

### Phase 05 — soft keyboard and touch pad

| claim | command | expected observation | outcome |
|---|---|---|---|
| arm on a phone raises the soft keyboard, characters land | phone on the board's AP/network, tap ARM, type a word; `watch_target.py` **on the target** | phone keyboard appears; the watcher's `typed:` line reproduces exactly what was typed, one `P=0:` character per frame | |
| a latched modifier chords correctly | latch Ctrl on the row, tap `c`; watch on the target | `LCtrl press` + `C press` — Ctrl+C on the wire, latch visibly cleared after | |
| pad moves the pointer; lift releases everything | press-and-hold the pad, move, lift; hold a button meanwhile, then drop the link | `movement dx … dy …` lines that **stop at lift**; `still held` absent from the mouse summary; no button stuck | |

## The blocking rule

A unit running any of the rows above:

1. **Look first, side-effect-free.** `bash tools/hw_probe.sh` (always exits 0,
   every probe capped, no serial port opened — opening `/dev/ttyACM*` toggles
   DTR/RTS and resets the board). It tells you whether an attempt is plausible.
2. **One attempt, capped.** Each device step runs **at most once**, wrapped in
   `timeout` (or the tool's own `-m`). No polling a port, no retry loops, no
   re-powering, no re-flashing "to see if it works". A timeout is an outcome
   (`failed: timeout …` or `skipped (no device)`), not a cue to try again.
3. **Record what was observed.** The outcome — `ok`, `skipped (no device)`,
   `failed: …` — plus the exact command written goes into the row above, in the
   phase's commit. Never write `ok` for a command that did not run; the
   unattended-run contract is that the human picks these rows up later.
4. **Never strand the board.** `tools/to_bootloader.py` drops the board into ROM
   download mode *deliberately*, and a power cycle is the only way back. If a
   run leaves the board in download mode, its report says so explicitly —
   nobody may be there to notice.
5. **Never block the gate.** None of this is inside `.agents/validate.sh`. The
   gate stays device-free and network-free; the probe line at its end is context
   whose exit status is deliberately discarded.