mirror/.agents/phases/todo/03_audio_share/00_phase.md

Phase 03 — Audio that actually arrives

Source: TODO.md L5 — "Need audio share to work." Story: n/a Context: .agents/PLAN.md §2 A3/A4 (WebRTC mesh, the sharer always offers and owns the track set), §3 (one RTCPeerConnection per viewer), SECURITY.md §4 "Audio", src/mirror/static/watch.js (ensurePeer, pc.ontrack), share.js (pickSurface, publish), src/mirror/registry.py L69/197/222–228 and src/mirror/protocol.py L57/118/129 (the audio flag already crosses the wire)

Objective

When a share carries audio, a viewer can hear it — on the wall, on the stage, and with the keyboard alone — and the UI says honestly whether audio exists at all.

What is already true, and must stay true: the audio boolean is published, relayed and rendered (registry.py keeps it on the Stream, stream_updated carries changes, paintTile() shows the 🔇 button only when audio exists). The server needs no change in this phase. What is broken is on the client:

1. A second ontrack throws the first track away. watch.js's pc.ontrack does tile.video.srcObject = (event.streams && event.streams[0]) || new MediaStream([event.track]). When the sharer's tracks arrive without a shared stream id (which happens with addTransceiver(track) and is common in Firefox), the audio callback builds a new one-track MediaStream and overwrites the video one. The video keeps playing from the still-attached transceiver, so nothing looks wrong — there is simply never any sound.
2. There is no way to unmute the thing you are watching. The mute button lives in .tile .actions, hover-revealed in the grid (phase 01 makes it reachable), and the staged tile's caption dims to 0.28.
3. The sharer does not know whether it offered audio. pickSurface() retries without audio on any error except NotAllowedError, and publish() reads getAudioTracks().length once — so a source whose audio track ends mid-share leaves every viewer with a dead 🔇 button.

Policy for this phase (user-confirmed): only the staged mirror can be audible. /watch mutes everything else when one is unmuted, and unstaging re-mutes.

Dependencies

• 01_keyboard_kiosk_control (todo) — setMuted(tile, muted) and the m key come from there; this phase extends them rather than inventing a second mute path.
• 02_sharp_capture (todo) — the per-leg RTCRtpSender work: the audio leg must not be caught up in the video sender's parameters, and 25 Mbit/s of video headroom is what leaves room for Opus.

Tasks

1. 01_audio_leg_plumbing.md — one accumulating MediaStream per tile, so the audio and video tracks coexist; assert an m=audio line really is negotiated.
2. 02_staged_audio_playback.md — the staged-only unmute policy, applied to mouse, keyboard and unstage, with the tile's audio state readable at a glance.
3. 03_audio_source_reporting.md — a precise audio fallback in pickSurface(), three honest states on /mirror, and republish when an audio track ends.
4. 04_audio_e2e_and_docs.md — a viewer-side assertion that audio crossed, and the SECURITY.md/README text that tells users what a browser will actually capture on Linux.

Testing & Quality

• Assert on the viewer (AGENTS.md §5): inbound bytesReceived for inbound-rtp/kind === "audio", totalAudioEnergy/audioLevel, and the composed video.srcObject.getAudioTracks().length. The sharer's panel proves nothing crossed the wire.
• The E2E Chromium already starts with --use-fake-device-for-audio-capture (tests/e2e/conftest.py), but a headless container has no audio device, so a real sound assertion belongs to the realdisplay-class run and the fake path asserts the SDP and the composed MediaStream. Do not fake the WebRTC layer to make a green test.
• tests/e2e/conftest.py also passes --autoplay-policy=no-user-gesture-required: the suite therefore cannot prove the browser's autoplay rule, only mirror's own muted state. Keep it that way and assert application state, not browser policy — and never add that flag to a test whose point is muting.
• The deliberate, user-initiated unmute promised by SECURITY.md §4 is preserved: audio never starts without a click or a keypress.
• Coverage >90% on new/modified code; full gate per task, plus -m e2e on every task that touches media.

Completion Criteria

• With an audio-bearing source, the viewer's inbound audio bytesReceived grows and video.srcObject.getAudioTracks().length === 1 while the video still decodes frames.
• Only the staged mirror can produce sound; unmuting mutes the others and unstage() re-mutes. Reachable by mouse and by m.
• /mirror distinguishes shared / silent / not offered, and a source losing its audio mid-share updates every viewer's tile.
• Cancelling the picker still reports a cancellation, not a silent-share fallback.
• SECURITY.md §4 "Audio" and README's platform notes describe what shipped, including the Linux/PipeWire reality.
• Full gate: uv run ruff check . && uv run ruff format --check . && uv run pyright && uv run pytest && MIRROR_REALDISPLAY=1 uv run pytest -m realdisplay.