chore(agent): phase roadmap from TODO.md, 3 phases
Convert the three open TODO items into .agents/phases/todo/ directories (Protocol B append — PLAN.md, AGENTS.md and existing phases untouched, so PLAN.md's roadmap table does not list these phases): - 01_keyboard_kiosk_control (TODO.md:3) — mouse-free /watch and /mirror: key router, j/k/Enter/f/g/n/p/1-9/m/?, focus-visible controls, help overlay. - 02_sharp_capture (TODO.md:4) — capture screen pixels instead of window pixels, maintain-resolution per leg, 12 -> 25 Mbit/s default cap, real-desktop size assertion on the viewer. - 03_audio_share (TODO.md:5) — one accumulating MediaStream per tile (the ontrack overwrite dropped the first track), staged-only unmute policy, honest audio states on /mirror, viewer-side inbound-audio assertion. Every task cites its TODO source line. Locked with the user: 25 Mbit/s default, only the staged mirror can be audible, full key map. Note for the user: PLAN.md section 6 still says the per-leg cap defaults to 12 Mbit/s; phase 02 task 03 changes that and this commit does not edit PLAN.md. LLM-Generated: true
This commit is contained in:
@@ -0,0 +1,90 @@
|
||||
# Phase 03 — Audio that actually arrives
|
||||
|
||||
**Source:** `TODO.md` L5 — "Need audio share to work."
|
||||
**Story:** n/a
|
||||
**Context:** `.agents/PLAN.md` §2 A3/A4 (WebRTC mesh, **the sharer always offers and owns the track set**), §3 (one `RTCPeerConnection` per viewer), `SECURITY.md` §4 "Audio", `src/mirror/static/watch.js` (`ensurePeer`, `pc.ontrack`), `share.js` (`pickSurface`, `publish`), `src/mirror/registry.py` L69/197/222–228 and `src/mirror/protocol.py` L57/118/129 (the `audio` flag already crosses the wire)
|
||||
|
||||
## Objective
|
||||
|
||||
When a share carries audio, a viewer can hear it — on the wall, on the stage, and
|
||||
with the keyboard alone — and the UI says honestly whether audio exists at all.
|
||||
|
||||
What is already true, and must stay true: the `audio` boolean is published,
|
||||
relayed and rendered (`registry.py` keeps it on the `Stream`, `stream_updated`
|
||||
carries changes, `paintTile()` shows the 🔇 button only when audio exists). The
|
||||
**server needs no change in this phase.** What is broken is on the client:
|
||||
|
||||
1. **A second `ontrack` throws the first track away.** `watch.js`'s `pc.ontrack`
|
||||
does `tile.video.srcObject = (event.streams && event.streams[0]) || new
|
||||
MediaStream([event.track])`. When the sharer's tracks arrive without a shared
|
||||
stream id (which happens with `addTransceiver(track)` and is common in
|
||||
Firefox), the audio callback builds a *new* one-track `MediaStream` and
|
||||
overwrites the video one. The video keeps playing from the still-attached
|
||||
transceiver, so nothing looks wrong — there is simply never any sound.
|
||||
2. **There is no way to unmute the thing you are watching.** The mute button lives
|
||||
in `.tile .actions`, hover-revealed in the grid (phase 01 makes it reachable),
|
||||
and the staged tile's caption dims to `0.28`.
|
||||
3. **The sharer does not know whether it offered audio.** `pickSurface()` retries
|
||||
without audio on *any* error except `NotAllowedError`, and `publish()` reads
|
||||
`getAudioTracks().length` once — so a source whose audio track ends mid-share
|
||||
leaves every viewer with a dead 🔇 button.
|
||||
|
||||
Policy for this phase (user-confirmed): **only the staged mirror can be audible.**
|
||||
`/watch` mutes everything else when one is unmuted, and unstaging re-mutes.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `01_keyboard_kiosk_control` (todo) — `setMuted(tile, muted)` and the `m` key
|
||||
come from there; this phase extends them rather than inventing a second mute path.
|
||||
- `02_sharp_capture` (todo) — the per-leg `RTCRtpSender` work: the audio leg must
|
||||
not be caught up in the video sender's parameters, and 25 Mbit/s of video headroom
|
||||
is what leaves room for Opus.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_audio_leg_plumbing.md` — one accumulating `MediaStream` per tile, so the
|
||||
audio and video tracks coexist; assert an `m=audio` line really is negotiated.
|
||||
2. `02_staged_audio_playback.md` — the staged-only unmute policy, applied to
|
||||
mouse, keyboard and unstage, with the tile's audio state readable at a glance.
|
||||
3. `03_audio_source_reporting.md` — a precise audio fallback in `pickSurface()`,
|
||||
three honest states on `/mirror`, and republish when an audio track ends.
|
||||
4. `04_audio_e2e_and_docs.md` — a viewer-side assertion that audio crossed, and
|
||||
the `SECURITY.md`/README text that tells users what a browser will actually
|
||||
capture on Linux.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- **Assert on the viewer** (`AGENTS.md` §5): inbound `bytesReceived` for
|
||||
`inbound-rtp`/`kind === "audio"`, `totalAudioEnergy`/`audioLevel`, and the
|
||||
composed `video.srcObject.getAudioTracks().length`. The sharer's panel proves
|
||||
nothing crossed the wire.
|
||||
- The E2E Chromium already starts with
|
||||
`--use-fake-device-for-audio-capture` (`tests/e2e/conftest.py`), but a headless
|
||||
container has **no audio device**, so a real sound assertion belongs to the
|
||||
`realdisplay`-class run and the fake path asserts the SDP and the composed
|
||||
`MediaStream`. Do not fake the WebRTC layer to make a green test.
|
||||
- `tests/e2e/conftest.py` also passes
|
||||
`--autoplay-policy=no-user-gesture-required`: the suite therefore cannot prove
|
||||
the *browser's* autoplay rule, only mirror's own `muted` state. Keep it that way
|
||||
and assert application state, not browser policy — and never add that flag to a
|
||||
test whose point is muting.
|
||||
- The deliberate, user-initiated unmute promised by `SECURITY.md` §4 is preserved:
|
||||
audio never starts without a click or a keypress.
|
||||
- Coverage **>90%** on new/modified code; full gate per task, plus `-m e2e` on
|
||||
every task that touches media.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] With an audio-bearing source, the viewer's inbound audio
|
||||
`bytesReceived` grows and `video.srcObject.getAudioTracks().length === 1`
|
||||
while the video still decodes frames.
|
||||
- [ ] Only the staged mirror can produce sound; unmuting mutes the others and
|
||||
`unstage()` re-mutes. Reachable by mouse **and** by `m`.
|
||||
- [ ] `/mirror` distinguishes `shared` / `silent` / `not offered`, and a source
|
||||
losing its audio mid-share updates every viewer's tile.
|
||||
- [ ] Cancelling the picker still reports a cancellation, not a silent-share
|
||||
fallback.
|
||||
- [ ] `SECURITY.md` §4 "Audio" and README's platform notes describe what shipped,
|
||||
including the Linux/PipeWire reality.
|
||||
- [ ] Full gate: `uv run ruff check . && uv run ruff format --check . && uv run
|
||||
pyright && uv run pytest && MIRROR_REALDISPLAY=1 uv run pytest -m realdisplay`.
|
||||
Reference in New Issue
Block a user