Experimental/keyrelay
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: relative mouse on the HID interface, and the verifier to prove it

Browse Source 
The board now enumerates as a boot keyboard plus a relative mouse on the same
HID interface. The pointer is captured, never grabbed: Pointer Lock in the
page, so Esc (the browser's own, never forwarded) always frees it, and every
frame restates the whole button byte so a frame lost in the WiFi cannot leave
a button held. Losing the lock, the tab or the link runs the same release
funnel as a stuck key, and boot protocol or `mouse off` refuses mouse frames
with E11 rather than sending bytes a boot interface would misparse as keys.

src/mouse.h owns the report, src/output.h the queues and the funnel; the page,
relayctl.py and watch_target.py carry the button bits and the evdev decoding.
The verifier's own tables are pinned now — REL_WHEEL is 8, not 2, and the
first version guessed from the report's field order, so a working wheel looked
dead while the summary said no scroll happened.

LLM-Generated: true
This commit is contained in:
ducoterra
2026-11-08 23:27:42 -05:00
parent 9bc8d5213c
commit ece0313d02
19 changed files with 3135 additions and 324 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+304 -54
View File
@@ -1,19 +1,22 @@
# keyrelay — a WiFi-remote USB keyboard on the M5Stack AtomS3 Lite
# keyrelay — a WiFi-remote USB keyboard and mouse on the M5Stack AtomS3 Lite
Type on your laptop; the keystrokes land on a *different* machine, over the air, as
genuine USB HID input — including Enter and Ctrl+Alt+Del. The board enumerates on the
target as a plain boot-protocol keyboard, so the target needs no software, no driver
and no network. The controller is a web page served by the board itself.
genuine USB HID input — including Enter and Ctrl+Alt+Del. Capture the pointer and your
mouse moves the other machine's cursor. The board enumerates on the target as a
boot-protocol keyboard with a relative mouse on the same HID interface, so the target
needs no software, no driver and no network. The controller is a web page served by the
board itself.
```
your laptop target machine
┌──────────────────┐ ┌──────────────────┐
│ browser tab │ WiFi │ │
│ keydown/keyup ├────WebSocket──►│ AtomS3 Lite │
│ of the page │ (token) │ keyrelay │
└──────────────────┘ │ │
└────────┬─────────┘
│ USB HID boot keyboard
│ mousemove │ (token) │ keyrelay │
│ (Pointer Lock) │ │ │
└──────────────────┘ └────────┬─────────┘
│ USB HID: boot keyboard
│ + relative mouse
▼
whatever has keyboard focus:
a login prompt, a BIOS setup,
@@ -21,21 +24,33 @@ and no network. The controller is a web page served by the board itself.
```
```
disarmed LED dark nothing you type goes anywhere
disarmed LED dark nothing you type or move goes anywhere
armed LED solid red every keystroke in the tab goes to the target
```
The mouse is opt-in twice over: the bridge has to be armed *and* the page has to be
handed the pointer, which the browser only does when you click. `Esc` gives it back —
that one is the browser's own guarantee, not something this firmware has to honour.
* **Nothing types until you arm it.** Boot, reload, reconnect, focus loss, socket loss
and 8 s of silence all disarm, and disarming sends "all keys up".
and 8 s of silence all disarm, and disarming sends "all keys up" **and "all mouse
buttons up"** — to the target, as reports, not just to the board's own bookkeeping.
* **Arming needs a passphrase** (or a thumb on the board's button). The LED is readable
from across the room, so the state is not a secret between you and the screen.
* **The mouse is captured, not grabbed.** The page asks the browser for Pointer Lock —
the same thing a web game asks for — so the cursor cannot escape the tab while it is
being forwarded, and `Esc` always returns it. Losing the lock, the tab or the link
releases every button. Movement is relative: the board says how far, never where, so
there is no position for the two machines to disagree about.
* **The keystrokes are not stored, logged or replayable.** The board keeps counters and
the live key state, and nothing else — no sequence storage, no macro, no script, nothing
it can type later. Pasted text is typed from RAM and forgotten; the serial console has no
command that types anything at all. It is a keyboard, not a payload dropper.
* **955 checks** cover the key-state machine, the wire protocol, the text→keystroke map,
SHA-1/base64, the WebSocket codec and the controller page's own script before anything is
flashed, and the target-side verifier proves what actually arrived.
the live key and button state, and nothing else — no sequence storage, no macro, no
script, nothing it can type later, no gesture it can replay. Pasted text is typed from
RAM and forgotten; the serial console has no command that types or clicks anything at
all. It is a keyboard and a mouse, not a payload dropper.
* **3406 checks** cover the key-state machine, the mouse state, the report queues and the
release funnel, the wire protocol, the text→keystroke map, SHA-1/base64, the WebSocket
codec, the controller page's own script, the CLI's frames and the target-side verifier
itself — all before anything is flashed.
---
@@ -73,13 +88,21 @@ armed LED solid red every keystroke in the tab goes to the target
whose keyboard you would rather not wear out.
- Keeping an agent session live on a machine you are not typing into (that is the
sibling project [`../jiggler`](../jiggler), which is a simpler build of the same board).
- Anything where a KVM-over-IP would be right and you do not need video, mouse or
audio, and you have 8 dollars and a USB port to spare.
- Pointing at something on a machine you cannot reach — a BIOS menu item, a dialog in a
installer, a button in a kiosk app — with the cursor you can already see on that
machine's own monitor.
- Anything where a KVM-over-IP would be right and you do not need video or audio, and you
have 8 dollars and a USB port to spare.
**Bad fits — use something else**
- **You need the screen.** There is no video path. PiKVM or TinyPilot
do video + keyboard + mouse; this is keyboard only.
- **You need the screen.** There is no video path. You steer the target's cursor without
being able to see it, which is fine for a monitor you are looking at and hopeless for one
you are not. PiKVM or TinyPilot do video + keyboard + mouse.
- **You need to know where the cursor is.** Movement is relative — see
[Layout caveats](#layout-caveats). There is no absolute pointer, so "click the OK
button" is not a thing this can express, deliberately: an absolute pointer would give
the board a position to be wrong about.
- **You want unattended automation.** Deliberately impossible here (see above). If you
want scripted input on a machine you own, `xdotool`/`ydotool`/`uinput` on that machine
is the honest tool, and a Rubber Ducky is what this design is *not*.
@@ -105,6 +128,16 @@ focus loss, a silent controller or a detached target cancels the rest of a run m
What stays absent is the part that would change what this device *is* — something the
board remembers, assembles on its own, or fires without a human in the loop.
**The mouse is the same line, drawn in the same place.** There is no auto-click, no
double-click macro, no drag recorded and replayed, no "move to (x, y)", no idle wiggle
that keeps a screen unlocked, and nothing that moves the pointer because a timer said so.
`relayctl mouse click` and `mouse drag` exist because verifying that the button bits
arrive is a thing the operator has to be able to do, and each needs `--yes`; a movement
(`mouse move`) does not, because moving a cursor is reversible by definition. If nobody is
moving a mouse, nothing moves. The pointer is captured with the browser's own Pointer Lock,
so `Esc` — which the browser keeps for itself and never forwards — is an escape hatch that
does not depend on the firmware, the network or this page being correct.
Two consequences hold either way. The CLI's ordinary commands still press one key per
invocation, which is all verification needs; scripting 30 of them is possible and would
still be 30 deliberate actions with the LED on. And a captured board is a keyboard, not a
@@ -120,7 +153,7 @@ connector it already uses.
```bash
cd keyrelay
tests/run_tests.sh # 955 checks, no hardware needed
tests/run_tests.sh # 3406 checks, no hardware needed
pio run -e bridge # ~1.1 MB image, ~35% of the 3 MB app slot
python3 tools/to_bootloader.py # 1200-baud touch -> ROM download mode
pio run -e bridge -t upload # flash
@@ -194,34 +227,63 @@ Enforced in this order, each independent of the others:
3. **Armed is displayed on the device.** Solid red LED while armed and attached. If the
target is not attached the LED blinks red instead, because "armed" without a target
means keystrokes are being dropped, and that deserves to look wrong.
4. **Silence disarms.** No inbound frame for `kNoClientTimeoutMs` (8 s) → all keys up,
disarmed. Covers the laptop sleeping, the browser being killed, the WiFi dying, the
network going down — every way a controller can vanish without saying goodbye. The
browser sends a keepalive every 2 s and answers the board's own 15 s WebSocket pings,
4. **Silence disarms.** No inbound frame for `kNoClientTimeoutMs` (8 s) → all keys up and
all buttons up, disarmed. Covers the laptop sleeping, the browser being killed, the WiFi
dying, the network going down — every way a controller can vanish without saying goodbye.
The browser sends a keepalive every 2 s and answers the board's own 15 s WebSocket pings,
so a live tab holds it open indefinitely; `relayctl` does the same while it types. This
is checked in `loop()` against `ControlServer::lastInboundMs()`, and it is checked
*before* the typewriter is driven: it is the one rule that stops a run of pasted text
when the controller dies without a parting packet.
is checked in `loop()`, and it is checked *before* the typewriter is driven: it is the
one rule that stops a run of pasted text when the controller dies without a parting
packet.
**What counts as silence, and what does not.** The clock is moved only by the
*authenticated controller*: by its authentication, and by bytes on its WebSocket. An
accepted TCP connection is not a live operator, and neither is anything reading the
board — a monitoring script, a health check, a second tab pointed at `/status`, or
`relayctl listen` used to hold this timer at zero and keep the bridge armed
indefinitely with the last key still pressed (bug 10). A board armed by its own button
with no controller at all is deliberately left running: it has no controller to go
quiet. This backstop is not gated on the link still existing, and must never be — see
[Ten bugs](#ten-bugs-this-verification-caught-all-fixed).
5. **Focus loss releases every key.** The tab blurring or being hidden sends "all keys
up" and disarms (unless you tick *stay armed when this tab loses focus*, which still
releases the keys — you cannot keep a held Ctrl alive by tabbing away).
6. **The target taking the bus away disarms.** USB detach or host suspend clears the
report queue, releases the keys and disarms; `ARDUINO_USB_*` events drive it.
7. **A full queue is a stop, not a drop.** 48 queued state changes without a drain means
something is outrunning the USB path, so the bridge releases everything, reports `E7`
and disarms rather than typing out of step with what the operator sees.
7. **A full queue is a stop, not a drop.** 48 queued key states or 32 queued mouse
reports without a drain means something is outrunning the USB path, so the bridge
releases everything, reports `E7` and disarms rather than typing out of step with what
the operator sees. The queue is not a buffer to absorb loss: 255 queued reports once
sat through a disconnect, which is ~2 s of keystrokes delivered after the operator had
stopped asking.
8. **State, not events.** The controller restates the *whole* key state on every event
(see [Wire protocol](#wire-protocol)), so a lost frame cannot leave a modifier stuck
on the target. This is the single most important design decision in the project: every
"stuck Ctrl" horror story in HID injection tooling is an event-protocol bug.
"stuck Ctrl" horror story in HID injection tooling is an event-protocol bug. Mouse
buttons follow the same rule — every `M=` frame carries the complete button byte — and
movement is the one thing that is *not* state because a delta has no state to lose: a
dropped movement frame moves the cursor slightly less, never leaves anything stuck.
9. **A paste cancels; it never pauses.** `R` means "stop typing", not "hold on". The run
is dropped on release-all, disarm, target detach or suspend, queue overflow, focus
loss and silence — every fail-safe above routes through the same `releaseEverything()`
that clears the queue, raises the keys and discards what was still to type. The board
never holds a run that outlives the decision to send it.
that clears the queue, raises the keys, lets go of the mouse buttons and discards what
was still to type. The board never holds a run that outlives the decision to send it.
10. **Losing the pointer releases the pointer.** Pointer Lock going away, the tab
blurring, the tab being hidden, disarm, `R`, silence and target detach all send an
all-buttons-up mouse report. *stay armed when this tab loses focus* exempts the
keyboard only — a held mouse button is never something to keep across a tab switch.
The page stops capturing rather than queueing movement to send later.
The physical kill switch is the button: press it and the keys come up, no network, no
passphrase, no software required.
The physical kill switch is the button: press it and the keys come up and the buttons come
up, no network, no passphrase, no software required. `Esc` is a second one for the pointer
specifically, and it is the browser's, not ours.
**One release funnel, and it has to actually send.** Every item above ends in
`releaseEverything()`, which now queues an all-keys-up *and* an all-buttons-up report to
the target rather than only clearing the board's own idea of what is down. That is the
fixed ninth bug: a held key stayed held on the target after silence-disarm, target loss or
a reload, because the report that would have released it was never queued (see
[Measured behaviour](#ten-bugs-this-verification-caught-all-fixed)).
---
@@ -229,8 +291,9 @@ passphrase, no software required.
| actor | can | cannot |
|---|---|---|
| Anyone on the target (its own OS) | see a Logitech G815 keyboard in its device list; read the CDC console; type on itself | arm or drive the bridge remotely, detect that a second controller exists |
| Anyone on the board's network | fetch the page; open a socket; learn that a keyrelay exists | arm, send a key, send a chord, type a run of text, read `/status` — all need the passphrase |
| Anyone on the target (its own OS) | see a Logitech G815 keyboard and a pointer in its device list; read the CDC console; type on itself | arm or drive the bridge remotely, detect that a second controller exists |
| Anyone on the board's network | fetch the page; open a socket; learn that a keyrelay exists | arm, send a key, send a chord, type a run of text, move the pointer, click, read `/status` — all need the passphrase, and all but `/status` need the bridge armed |
| Whoever holds the laptop's mouse | the browser tab's pointer, only after a click on **Capture pointer**, and only while armed | move the target's pointer without the LED being red, or keep moving it after `Esc` — Pointer Lock is revoked by the browser and cannot be retaken without another click |
| Someone guessing the passphrase | 5 attempts before a 30 s lockout during which the token is not evaluated at all | learn anything from the failure: same error, same close, no timing oracle (constant-time compare) |
| The owner of the WiFi | join the network, then everything above | — the network *is* the perimeter; see below |
| You, holding the board | arm/disarm (button), reconfigure or reflash (console), stop everything (unplug) | — |
@@ -260,6 +323,15 @@ gated by what it reveals — `/status` needs the passphrase, the page needs it b
arm or type. If that ever goes too far, the `ap off`/network config is the blunt instrument;
there is no per-client ACL.
**The mouse widens what a compromised controller can do, so it is gated the same twice
over.** Arm plus Pointer Lock. It is also the reason there is a `mouse off` switch: the
composite descriptor is fixed when USB starts, so turning the pointer interface off is a
reboot, not a setting change — and a board that advertises no pointer cannot be made to
move one. When a target selects HID **boot protocol** (some BIOS setups, some KVMs) the
mouse is switched off by itself, because a 5-byte mouse report on a boot-protocol
interface would be parsed as keyboard data: the keyboard keeps working, the pointer stops,
and `E11` says why rather than the command silently doing nothing.
**Identity.** The board reports itself as `046d:c33f` "Logitech Gaming Keyboard G815",
matching the sibling `jiggler` firmware so the two are indistinguishable from the host
side and `/dev/input/by-id` names stay stable across reflashes. This makes the device
@@ -312,6 +384,19 @@ by the compositor and never reach the page. Three answers, in order of preferenc
not close the tab, `Ctrl+R` will not reload it and F11 will not fullscreen it. It also
means you cannot type into the page's own fields while armed — disarm first.
**While the pointer is captured, `Esc` belongs to the browser.** That is the point: it
revokes Pointer Lock, the page sees `pointerlockchange`, and every button goes up. What it
cannot do is disarm the bridge — the keyboard is still armed and still typing, so the status
line says so and you can press `R` or the board's button. Nothing else the laptop's
compositor grabs takes the pointer away: `Super` and `Alt+Tab` are inert while locked,
which is another reason capture is the design and a plain `mousemove` relay would not be.
**The laptop's own pointer settings apply to what gets forwarded.** Acceleration,
`libinput Accel Speed` and a high pointer speed on the laptop mean the same hand movement
sends different deltas going out, and the target then accelerates them again. Set the
laptop's pointer speed to the middle and disable acceleration while steering something you
cannot see the response curve of; the target's speed is what it is.
---
## Layout caveats
@@ -326,6 +411,22 @@ This is identical to what a physical KVM does, and the fixes are the same:
literal; `tools/relayctl.py` carries the same table and
`tests/test_relayctl.py` fails if the two drift).
**The pointer is relative, on purpose.** The board reports a boot mouse: buttons, ±127 per
axis per report, wheel and horizontal pan. It does not know where the target's cursor is,
and the target will not say, so there is no "move to" and no way to desynchronise — the
same property that makes `S=` the right model for keys. Two consequences to know before you
depend on it:
- **Acceleration is applied twice**, once by the laptop's driver before the page sees the
movement and once by the target's driver after the board sends it. The deltas that leave
the page are the laptop's, not your hand's, and the cursor moves by the target's idea of
those. Fine for steering; not fine for a pixel-exact drag.
- **The target clamps at its screen edges**, and the board cannot see that happen. Moving
5000 px right from the far right of the screen does nothing, and the board has no way to
know. This is why `mouse move` exists as a *verifiable* primitive and no macro of
movements is ever stored: relative motion is only meaningful while a human is watching
the screen it lands on.
ISO keyboards: `IntlBackslash` (0x32) is the extra key next to Left Shift on ISO and
`Backslash` (0x31) is the ANSI one; both are in the table. `AltRight` is `AltGr` and
travels as its own modifier bit, which is what layout engines on the target expect.
@@ -350,7 +451,7 @@ invisible to every compiler and a typo there fails in a browser, with a target l
|---|---|
| big ARM/DISARM bar | the state must be unmissable; it only changes when the board confirms |
| red page border + red LED while armed | two independent indicators of the same fact |
| status pills: network / target / link / armed / caps-num | "will my keystrokes land, and where" |
| status pills: network / target / link / armed / caps-num / **mouse** | "will my keystrokes land, and where", and whether the pointer is yours right now |
| **Ctrl+Alt+Del** with two-press confirm | the OS eats the real one; on Linux it reboots |
| VT1–VT7, Alt+Tab, Win, PrtSc, Menu | the chords the laptop's compositor steals |
| **⇦ snap left / snap right ⇨ / ⛶ fullscreen** | Super+Left, Super+Right, Super+PageUp — KDE window management, sent by the board so your laptop does not answer it with its own windows |
@@ -358,12 +459,25 @@ invisible to every compiler and a typo there fails in a browser, with a target l
| *stay armed if this tab loses focus* | off by default; for the one workflow that needs it |
| **Send text** box, its button and its *stop typing* | paste something and have it typed as keystrokes; the button says how many characters, refuses to be armed-into accidentally, and is the only path text takes |
| *drop characters I cannot type* | off by default: a paste that quietly loses a character is worse than one that refuses |
| held-keys readout, keys/chords/text/chords counters | so "it is typing" is a fact you can see |
| **Capture pointer / Release pointer** and the mouse pad | the pointer half of the bridge. Capture asks the browser for Pointer Lock, so the cursor cannot escape the tab while forwarded; the pad says what a relative mouse means before you discover it mid-drag |
| **Release buttons** under the pad | panic for the pointer, separate from the keyboard's because the two fail independently — held-button-then-lost-link is the mouse's stuck-key, and it is the same one release funnel |
| held-keys readout and the mouse line under it, keys/chords/text/chords/mouse counters | so "it is typing" and "it is moving" are facts you can see; the mouse line names the buttons held and the button byte, because "left is down" and "the target has 0x01" are different claims and only the second one is on the wire |
| log, *log each key sent*, **clear** | the board's errors verbatim plus one line per key pressed — never one per release, never a blank line, and a repeating line is counted rather than stacked |
The caps/num indicators are read *from the target* (its HID output reports drive them),
not guessed — a caps-lock desync between controller and target is a real failure mode of
remote keyboards and this design cannot have it.
remote keyboards and this design cannot have it. The mouse counters are read from the board
for the same reason: `mframes` counts the reports the board accepted, which is the number
`watch_target.py` on the target can be compared against.
**What capture does and does not do.** The pointer is captured only while the bridge is
armed, and only after a click; `Esc`, blur, a hidden tab, disarm, `R` and the board
reporting it cannot do mouse reports (boot protocol, or `mouse off`) all end it, each
releasing the buttons. The page never accumulates movement to send later: if the link is
down, movement is dropped, because a queue of pointer motion delivered after you stopped
moving is the pointer version of a stuck key. `mouse frames 0` in the status line while you
are moving means the frames are not getting there — that is a fault to look at, not noise,
so it is shown rather than smoothed.
---
@@ -381,7 +495,8 @@ and impossible for one malformed field to desynchronise what follows it.
| c → d | `C` | Ctrl+Alt+Del (fixed in firmware, not read off the wire) |
| c → d | `Y=<mods>:<u>,<u>…` | one synthetic chord, `kChordHoldMs` (40 ms) |
| c → d | `P=<ms>:<hex>` | type pasted text, `<ms>` ms per character (0 = board default) |
| c → d | `R` | release every key, stay armed — **and drop a text run in progress** |
| c → d | `M=<btn>:<dx>:<dy>:<wheel>:<pan>` | **the complete mouse button state** plus this report's relative movement |
| c → d | `R` | release every key *and every mouse button*, stay armed — **and drop a text run in progress** |
| c → d | `K` | keepalive |
| d → c | `OK` | accepted |
| d → c | `E<code>:<text>` | error (codes below) |
@@ -396,6 +511,21 @@ release are two frames and therefore two HID reports, so a fast tap cannot vanis
identical consecutive reports collapse, which is what keeps the queue short. A single
message resynchronises the target completely, which is why no key can get stuck.
**`M=` restates the buttons and carries one report's movement.** `<btn>` is the button byte
as two hex digits — `01` left, `02` right, `04` middle, `08` back, `10` forward, and any
combination — and is the *whole* button state every time, exactly like `S=`. A frame that
lifts one button restates the ones still held; there is no button-up event to lose, so no
button can get stuck on the target. The four signed decimal fields are this report's
movement, each `±127` (`kMouseDeltaMax`, the boot-mouse byte), and all five fields are
required — `M=01` is `E6`, not "left, no movement", because a controller that forgot three
fields is a controller whose next frame is also wrong. Movement is the one field that is
not state: losing a frame moves the cursor slightly less and leaves nothing behind.
The page chunks movement to 127 per axis and keeps the diagonal shape (a 200×200 move
leaves as 127,127 then 73,73, never as two axis-parallel legs), so a fast flick cannot turn
into a silently clamped one, and it flushes the pending movement under the *old* button
state before restating a new one — that ordering is what makes a drag a drag.
**`P=` is text, typed by the board.** `<hex>` is the bytes to type, so one frame carries
at most `kTextFrameMax` (120) characters and a longer paste is several frames; `<ms>` is
the pace asked for, and the board keeps it — expansion and timing live on the USB side
@@ -410,7 +540,13 @@ not a guess at timing.
Error codes: `1` not authenticated · `2` locked out · `3` another controller holds the
bridge · `4` not armed · `5` no USB host attached · `6` bad message · `7` queue overflow
(released and disarmed) · `8` text buffer full · `9` a character with no US-ANSI keystroke.
(released and disarmed) · `8` text buffer full · `9` a character with no US-ANSI keystroke ·
`10` the mouse interface is off on this board · `11` the target chose HID boot protocol, so
no mouse report can be sent safely.
`10` and `11` are two digits, which changed the encoder: `formatError` used to emit
`E` + `code % 10` and would have rendered both of them as `E0:` — a refusal that names
nothing. Codes now render in full, every single-digit code byte-identical to before.
The same protocol is what a future OS-wide capture client would speak — see
[Backlog](#backlog) — which is why it lives in `src/proto.h` instead of inside the server.
@@ -435,10 +571,17 @@ python3 tools/console.py /dev/ttyACM0 status # one command, print the reply
| `ap auto\|force\|off` | AP policy: when the station is down / always / never |
| `token <passphrase>` | control passphrase, 6–32 chars, never echoed back |
| `arm` / `disarm` | same as the button |
| `mouse on\|off` | whether the bridge enumerates a pointer interface at all. Stored, and it needs a **reboot**: the composite descriptor is fixed when USB starts, so this cannot change under a live target. Off means the pointer cannot be moved by anything, ever, until it is turned back on |
| `reboot`, `help` | |
There is deliberately **no command that types anything** — the console is configuration,
not an input path. Replies are printed with a blocking write (the global serial timeout is
`status` also says which of the two the running bridge is, and why: a line like
`mouse interface=yes target=report buttons=00 reports=41 queued=0`, where
`target=boot` means the target asked for boot protocol and the pointer is disabled by
itself until it re-selects report protocol.
There is deliberately **no command that types or clicks anything** — the console is
configuration, not an input path. That is why `mouse on|off` is the only mouse command and
it changes a capability rather than performing an action. Replies are printed with a blocking write (the global serial timeout is
0 so logging into a port nobody reads can never stall `loop()`); without that, multi-line
replies truncate at the 128-byte CDC TX ring, which is exactly what happened on the first
build and is why `pollConsole()` bumps the timeout around each reply.
@@ -451,22 +594,29 @@ build and is why `pollConsole()` bumps the timeout around each reply.
src/
config.h defaults: network, timeouts, caps, buffers (NVS overrides)
keystate.h pure modifier byte + pressed-set -> 8-byte HID report, rollover
mouse.h pure button byte + signed deltas -> 5-byte relative report; the
buttons are restated, not evented, exactly like keystate
text.h pure pasted text -> US-ANSI keystrokes, paced; run buffer and cancel
output.h pure the one place keystrokes, pasted runs, chords and mouse reports
become two ordered queues, and the one place anything is released
proto.h pure the wire protocol: parse/render commands, constant-time compare
sha1.h pure SHA-1 + base64 for the WebSocket handshake
ws_codec.h pure handshake request parsing + RFC 6455 frame codec
web_page.h the controller page, in PROGMEM
web_page.h the controller page, in PROGMEM (~48 KB now, with Pointer Lock)
control_server.{h,cpp} sockets: HTTP routes, WebSocket lifecycle, auth, heartbeat
main.cpp glue: USB identity/HID, WiFi, report queue, chords, text run,
main.cpp glue: USB identity/HID composite, WiFi, chords, text run,
LED, button, watchdogs, serial console
tools/
to_bootloader.py 1200-baud touch into ROM download mode (stdlib termios)
console.py serial console, one command or interactive (stdlib)
relayctl.py controller from a terminal: status/arm/disarm/key/chord/listen/paste
watch_target.py on the target: decode the board's evdev events, prove delivery
relayctl.py controller from a terminal: status/arm/disarm/key/chord/listen/
paste/mouse
watch_target.py on the target: decode the board's evdev events (keyboard *and*
mouse), prove delivery
tests/
run_tests.sh builds and runs everything (host g++ or a local container)
test_keystate.cpp test_proto.cpp test_text.cpp test_ws_codec.cpp
test_keystate.cpp test_mouse.cpp test_output.cpp test_proto.cpp test_text.cpp
test_ws_codec.cpp
test_relayctl.py test_page.js (node, skipped when absent)
AGENTS.md operating manual for whoever edits this next: the invariants, the three
tables that must agree, the build traps, the verification standard
@@ -476,7 +626,7 @@ TODO.md open items, kept with their root cause rather than deleted
If you are an agent or a new contributor, read `AGENTS.md` before changing anything: several
rules here are invisible to a compiler and to the test suite at the same time.
The five modules marked *pure* have no Arduino and no ESP-IDF include, so the identical
The seven modules marked *pure* have no Arduino and no ESP-IDF include, so the identical
code compiles and runs on a host under `-fsanitize=address,undefined`. That is not
convenience for its own sake: the sanitiser caught five real defects in the frame decoder
before the first flash (a `reset()` that left the reader latched on the previous frame, an
@@ -493,11 +643,31 @@ keydown/keyup -> full state S=.. -> Reader (RFC6455) -> proto::parse -> auth gat
-> ReportQueue (dedupe identical, overflow = release + disarm)
-> drainReports: tud_hid_n_ready(0) ? USBHIDKeyboard::sendReport : keep queued
-> HID interrupt endpoint, boot protocol, 8-byte report -> target
pointer move -> M=btn:dx:dy:wheel:pan -> same gate (armed + attached + mouse allowed)
-> MouseState::apply (button byte restated; idle frames produce no report at all)
-> a second ReportQueue, same dedupe and the same overflow rule
-> drainReports, after the keyboard queue -> report #2 on the *same* HID interface
fail-safe (any of: disarm, R, blur, silence, detach, suspend, overflow)
-> releaseEverything() -> OutputStage::releaseAll()
clears both queues, both trackers, the chord and the text run, THEN queues
all-keys-up and (if the target can take it) all-buttons-up
```
`USBHID::SendReport` in the core is synchronous (it waits on a semaphore for the transfer
to complete, up to 100 ms), so `drainReports()` takes at most `kMaxReportsPerLoop` = 8
reports per pass — bounded time in the USB path, and the control plane never waits on it.
reports per pass across **both** devices — bounded time in the USB path, and the control
plane never waits on it.
**Both devices live on one HID interface, which is a constraint with three consequences.**
The keyboard's `USBHIDKeyboard` is the first HID instance, so it claims the boot keyboard
subclass; the mouse is added as the second report on that same instance, which is why
`mouseDev` is created after `kbd` and never before. Because the descriptor is written when
`USB.begin()` runs, `mouse off` cannot take effect until a reboot. And because a
boot-protocol host parses every report on that interface as an 8-byte keyboard report, a
target that selects boot protocol gets the pointer switched off under it (`E11`) rather than
sending 5-byte mouse reports that would arrive as garbage keystrokes.
Chords (Ctrl+Alt+Del, VT switches) are synthetic: the press is pushed, and after the hold
the *tracked* state is pushed back, so keys that were genuinely held before a chord
survive it and the tracker is never corrupted by something the operator did not type.
@@ -529,12 +699,29 @@ SANITIZE=1 tests/run_tests.sh # + ASan/UBSan (recommended, and what found 5 b
| suite | checks | what it pins down |
|---|---|---|
| `test_keystate` | 40 | idempotence, transition preservation (a fast tap is two reports), 6-key limit, ErrorRollOver and recovery from it, dedupe, `clear()`, the exact Ctrl+Alt+Del bytes `05 00 4c 00 00 00 00 00` |
| `test_mouse` | 48 | the 5-byte report layout, every button bit where the HID spec says it is, idle frames producing **no** report, a button change with no movement still producing one, the button byte being restated rather than evented, `releaseAll` being idempotent, and that no frame can put a value outside one signed byte on the wire |
| `test_output` | 153 | **the release funnel.** Queues hold their order and collapse duplicates; releasing with keys held queues an all-keys-up report *when there is a target to receive it*; releasing drops a press that was queued but never sent; releasing with a mouse button held queues all-buttons-up; boot protocol or `mouse off` means no mouse report is queued **but the key release still is**; a pasted run and an in-flight chord are both cancelled; and overflow is detected on both queues. Regression-proven: reinstating the old `releaseEverything()` fails 23 of these |
| `test_proto` | 193 | every command form, strict rejection of malformed state specs, bounds (cap on keys, token length and text frames), render/parse round-trip, error framing, constant-time compare, **every chord button's bytes matched against the label printed on it**, and the page's text constants pinned to `config.h` |
| `test_text` | 459 | the whole printable-ASCII map walked (nothing may be unmapped, nothing may claim a key it has no glyph for), shifted glyphs and the digit row, what is refused rather than guessed at, all-or-nothing frame acceptance, pace arithmetic and clamping, the exact press/release report sequence for two characters, overlay on the operator's held keys, ErrorRollOver propagation, cancel, buffer-full, `P=` render/parse round-trip including the largest legal frame, and one end-to-end case that takes the exact frame the page sends and asserts the HID report sequence the target ends up seeing |
| `test_ws_codec` | 135 | SHA-1 against published vectors including the million-`a` case, base64 against RFC 4648, the RFC 6455 handshake example, frame encode/decode at the 125/126 and 64-bit boundaries, byte-at-a-time delivery, and rejection of unmasked, fragmented, reserved-bit, binary, over-long and oversized frames |
| `test_relayctl` | 81 | the Python client's codec, that its key table has not drifted from the page's, that its text validator agrees with the firmware's character map on all 128 ASCII codes, CRLF normalisation, and `TEXT_FRAME_MAX`/`MAX_PASTE_CHARS` pinned to `config.h` |
| `test_relayctl` | 160 | the Python client's codec, that its key table has not drifted from the page's, that its text validator agrees with the firmware's character map on all 128 ASCII codes, CRLF normalisation, `TEXT_FRAME_MAX`/`MAX_PASTE_CHARS` pinned to `config.h`, the mouse frame builder refusing a sixth button and any movement that does not fit one report, movement splitting that reproduces the original displacement exactly, click/drag/wheel frame *ordering*, every generated frame parseable by the firmware's own parser, and the button bits and frame pace agreeing between page, tool and `config.h` |
| `test_page` | 84 | the page's script, executed: blank and whitespace-only log lines never render and a repeating line is counted not stacked, character validation, CRLF, hex encoding, one frame for a short paste and exactly the right bytes, a busy board holding the next frame back, a 300-character paste reassembling to the text that went in, refusal of untypeable text and the opt-in drop, cancellation by stop/disarm, **the mouse: capture requires the bridge armed, movement becomes exactly one `M=` per report with nothing sent when idle, a 400×300 flick split so no frame exceeds one signed byte while the diagonal shape survives, the click ordering that makes a drag a drag, wheel pixels quantised to notches with the remainder carried, `Esc`/blur/disarm each releasing the buttons, `stay armed on blur` exempting the keyboard only, a board that reports `mouse:0` refusing capture and dropping one in progress — and every mouse frame the page emits parsed by the firmware's own parser** |
| `test_page` | 47 | the page's script, executed: blank and whitespace-only log lines never render and a repeating line is counted not stacked, character validation, CRLF, hex encoding, one frame for a short paste and exactly the right bytes, a busy board holding the next frame back, a 300-character paste reassembling to the text that went in, refusal of untypeable text and the opt-in drop, and cancellation by stop/disarm, plus every element id the script reaches for checked against the markup |
The verifier is verified too, because twice now it was the verifier that was wrong: node
classification (EV_REL means a pointing device — not the device's name, not the order the
glob found them in), movement coalescing one report into one line, net displacement
arithmetic, hi-res wheel events not counted as extra notches, and a stuck button being
called out in those words.
The last six checks in `test_relayctl` are **source pins**, and the suite says so out loud:
they assert that `loop()`'s silence disarm runs off `controllerSeen()` rather than
`linked()`, that `acceptNew()` never stamps the controller clock, and that `closeLink()`
stamps it only for a controller link. They cannot execute — that code needs a socket and a
board — but each one is a sentence a later edit can undo with no compiler error and no
behavioural test failing on the host, which is exactly how bug 10 survived. Reinstating
either half of it fails 3 of them.
Gaps, stated plainly: nothing here exercises real sockets, real WiFi associations, or real
USB transactions — those are the [end-to-end procedure](#end-to-end-verification) below.
The HTTP routing and auth-lifecycle code in `control_server.cpp` is the least-tested part
@@ -563,8 +750,20 @@ printf 'Hello from keyrelay\n' | \
python3 tools/relayctl.py --host keyrelay.local paste --yes # 19 characters
python3 tools/relayctl.py --host keyrelay.local listen 20 # heartbeats, live
python3 tools/relayctl.py --host keyrelay.local disarm
# 3. the mouse, same two terminals
python3 tools/relayctl.py --host keyrelay.local mouse move 120 -40
python3 tools/relayctl.py --host keyrelay.local mouse click --yes
python3 tools/relayctl.py --host keyrelay.local mouse wheel -3
python3 tools/relayctl.py --host keyrelay.local mouse drag 200 0 --button right --yes
python3 tools/relayctl.py --host keyrelay.local mouse release
```
And in the browser, which is where the pointer is actually usable: open the page, arm it,
press **Capture pointer**, move the mouse (the target's cursor moves, the laptop's is
hidden), hold left and move for a drag, press `Esc`. Then arm it, capture, hold a button,
and **close the tab** — the target must stop clicking within 8 s of the silence timeout.
What a correct run proves, and how to tell:
| claim | evidence |
@@ -580,6 +779,13 @@ What a correct run proves, and how to tell:
| target LEDs are read back | `LED CapsLock on` on the target, caps indicator on the page |
| silence disarms | arm it, close the tab, wait 8 s, `status` says `armed: 0` |
| auth is real | `relayctl` with a wrong token fails; the board logs `authfails=1`; 5 tries → `E2` |
| **movement reaches the target as movement** | `watch_target.py` prints a `movement dx … dy …` line per report and the summary's `net position` matches the direction and rough distance you moved — the keyboard half of the watcher cannot show this, which is why the mouse node is watched separately |
| **the button bits are the buttons they claim to be** | `mouse click --yes` gives `BTN Left press/release`; `--button right` gives `BTN Right`. A button that arrives as the wrong one is a bit-order bug and looks identical from the page |
| **a drag is down, move, up in that order** | `BTN Left press`, then `movement` lines, then `BTN Left release` — any other order is a drag that clicks at the start and drags nothing |
| **no stuck buttons** | `still held` absent from the mouse summary at the end. Check it the hard way: hold a button and close the tab |
| **the wheel is notches, not pixels** | `mouse wheel -3` produces `REL_WHEEL -3` three times, and hi-res events do not inflate the count |
| **boot protocol refuses the pointer, not the keyboard** | on a BIOS/KVM that selects boot protocol, keys keep working and `M=` answers `E11`; `console.py status` says `target=boot` |
| **`mouse off` really removes the capability** | `console.py mouse off` + reboot → `lsusb`/evdev show no pointing device, `watch_target.py` notes the missing interface, the page's Capture button is disabled, and keys still work |
`watch_target.py` finds the board by USB id, not device name, so it keeps working on a
different port or a renamed input node, and it accepts both the Logitech identity and the
@@ -635,10 +841,17 @@ AtomS3 Lite → Linux laptop (US layout), board joined the LAN, firmware 1.0.0.
| LED read-back | `LED CapsLock on` then `off`, coming from the *target's* HID output reports |
| **silence disarms** | controller process killed → `armed:0` within 11 s, keys released. **Worth knowing:** that run passed *before* the silence timer existed — killing a process closes the socket, and the socket-close path disarmed. The FIN-less death (laptop slept, network vanished) is what the timer covers, and it is wired now. A check that passes for the wrong reason is still a check that passes, which is exactly the problem |
| single controller | a second client got `E3` / `503` and the board logged `rejected=1` — no takeover |
| **mouse enumeration** | the target created **two** input devices from one HID interface — `keyboard=/dev/input/event19`, `mouse=/dev/input/event20` — with `caps rel: 1943`, i.e. REL_X, REL_Y, REL_WHEEL, REL_HWHEEL and both hi-res axes |
| **mouse movement** | `mouse move 120 -40` → `movement dx +120 dy -40` on the target, exact, one report |
| **mouse buttons** | `click` → `BTN Left press` + `BTN Left release`; `--button right` → `BTN Right`. The bit order the page claims is the bit order the target sees |
| **the wheel** | `mouse wheel -3` → `REL_WHEEL -3`. It took a wrong axis table in the *verifier* to prove this worked — see the note under Tests |
| **drag ordering** | `drag 200 0 --button right` → `BTN Right press`, `movement dx +127`, `movement dx +73`, `BTN Right release`. Both halves of the property at once: the ordering is down-then-move-then-up, and the 200 px left as two reports that stay inside one signed byte |
| **keyboard and mouse together** | keys, chords, a 14-character paste and mouse reports in one session, `drop=0 roll=0`, `keys=3` and `mframes` advancing independently |
| **a controller that dies holding a button** | `BTN Left press` → client `os._exit()` (no close, no disarm) → **`BTN Left release` 9.4 s later**, and a live controller holding the same link with keepalives stayed armed for the 24 s it was tested. This is bug 10, and it is the property the whole release funnel exists to provide |
| lockout | 5 wrong passphrases → `E2`, for 30 s, from the page's own retry loop |
| queues | `drop=0 roll=0` throughout; `queue 0 pending, 0 held, ok` at rest |
### Nine bugs this verification caught (all fixed)
### Ten bugs this verification caught (all fixed)
Worth listing, because each one is a class of bug that HID-over-network tools ship with:
@@ -689,6 +902,43 @@ Worth listing, because each one is a class of bug that HID-over-network tools sh
type. The lesson is the one the whole project keeps relearning: a safety property that
is only written down is not a safety property.
10. **A held mouse button stayed held, forever, and anything on the network could keep it
that way.** Found while verifying the mouse, with the mouse innocent: `releaseEverything()`
queued the release reports (bug 9's fix), and they still did not go out.
Three things stacked up. `closeLink()` ran the release callback only `if (linked())`,
and `linked()` is `_phase == WebSocket && _authed` — so any path that cleared `_authed`
on its way to closing the socket closed it with the keys down and nothing downstream
able to notice. The board then sat `armed=1 mbtn=01` with `phase=none`. And the
backstop that should have caught it, `loop()`'s silence timer, was gated on
`control.linked()` — **a controller that has gone away is precisely the case in which
`linked()` is false**, so the fail-safe designed for a vanished controller could only
ever run while the controller was there. Measured: `BTN Left press`, controller killed,
target still holding the button **26 s later**, and the armed state never expired.
The second half was worse. `acceptNew()` stamped `_lastInboundMs` for *any* incoming
TCP connection, so a read-only `GET /status` — which is served by design while a
controller holds the bridge — reset the silence clock. Any authenticated monitor,
second tab or health probe kept a dead controller's armed state open indefinitely. The
first fix traded that for stamping in `closeLink()`, which moved the reset from
*accept* to *every close* and changed nothing observable; `wasLinked` is the honest
test, because it is true exactly when the thing being closed can leave key state
behind.
Fixed as: `wasLinked = (_phase == Phase::WebSocket)`; the silence clock moved only by
an authenticated controller's own traffic or by that link closing; `controllerSeen()`
distinguishes "no controller since boot" (button-armed, leave it alone) from "one came
and went" (never stay armed on its strength). Verified on hardware with the monitor
running the whole time: `BTN Left press` → `BTN Left release` at 9.4 s, armed 0 and
staying 0 for 37 s of continued polling, while a live controller holding the link with
keepalives stayed armed through the same test.
What to take from it: **never gate a fail-safe on the absence of the thing its
condition detects.** And a backstop needs its own test, because this one passed every
keyboard suite for exactly the reason bug 9 did — the socket-close path happened to
cover the case being checked. `tests/test_relayctl.py` now pins all three sentences in
source, and reinstating either half of the bug fails 3 checks.
### The hazard nobody warns you about
**When the target is the machine you are working on, your keystrokes land on your own