phase: 03_audio_share
**Phase 03 final verification pass — complete, 100% green.** - Verified all 4 tasks done in `complete/03_audio_share/`; confirmed implementation in `watch.js` (one composed `MediaStream` per tile, staged-only `setMuted()` policy, `data-audio` state), `share.js` (precise `NotAllowedError` rethrow, 3 honest `#stat-audio` states, mid-share audio republish), conftest flags, and the new `tests/e2e/test_audio_share.py` (11 tests, all **passed** — none skipped). - **Criteria:** (1) viewer inbound audio `bytesReceived` strictly increases + composed stream 1 video/1 audio track while frames still decode — **passed**; (2) staged-only audio via keyboard `m`, mouse button, and `unstage()` re-mute — **passed**; (3) `shared`/`silent`/`not offered` + mid-share audio loss updates panel, roster and tiles — **passed**; (4) cancelled picker reports cancellation, single attempt — **passed**; (5) SECURITY.md §4 Audio, README platform notes, kiosk doc all describe shipped behaviour incl. Linux/PipeWire reality — **verified**; (6) full gate — **passed** (below). - **Gates:** `uv run ruff check . && uv run ruff format --check .` → clean; `uv run pyright` → 0 errors; `uv run pytest` → 245 passed, 4 skipped (only the `MIRROR_REALDISPLAY`-gated X11 tests); `MIRROR_REALDISPLAY=1 uv run pytest -m realdisplay` → 3 passed, 1 skipped (real-capture audio: documented skip naming the exact host/flag reason, per task 04). - **Docs:** README.md reviewed in full against code (env table, CLI flags, routes, kiosk key table, quickstart, audio notes) — consistent, no changes needed. AGENTS.md — all commands/paths/coverage claims verified accurate, no stale statements, no changes needed. - **Defect fixed (pre-existing):** `docs/kiosk-silverblue.md` + `install-kiosk.sh`, `kiosk.conf`, `gnome-kiosk-script` used `mirror.reeselink.com` while README/SECURITY.md use `mirror.reeseapps.com` (the deployment domain, consistent with the fleet); corrected 7 occurrences in 4 files. - No live-infrastructure changes; realdisplay suite ran on private Xvfb only. - **Next pending phase:** `04_reachable_share_cta`.
This commit is contained in:
@@ -0,0 +1,90 @@
|
||||
# Phase 03 — Audio that actually arrives
|
||||
|
||||
**Source:** `TODO.md` L5 — "Need audio share to work."
|
||||
**Story:** n/a
|
||||
**Context:** `.agents/PLAN.md` §2 A3/A4 (WebRTC mesh, **the sharer always offers and owns the track set**), §3 (one `RTCPeerConnection` per viewer), `SECURITY.md` §4 "Audio", `src/mirror/static/watch.js` (`ensurePeer`, `pc.ontrack`), `share.js` (`pickSurface`, `publish`), `src/mirror/registry.py` L69/197/222–228 and `src/mirror/protocol.py` L57/118/129 (the `audio` flag already crosses the wire)
|
||||
|
||||
## Objective
|
||||
|
||||
When a share carries audio, a viewer can hear it — on the wall, on the stage, and
|
||||
with the keyboard alone — and the UI says honestly whether audio exists at all.
|
||||
|
||||
What is already true, and must stay true: the `audio` boolean is published,
|
||||
relayed and rendered (`registry.py` keeps it on the `Stream`, `stream_updated`
|
||||
carries changes, `paintTile()` shows the 🔇 button only when audio exists). The
|
||||
**server needs no change in this phase.** What is broken is on the client:
|
||||
|
||||
1. **A second `ontrack` throws the first track away.** `watch.js`'s `pc.ontrack`
|
||||
does `tile.video.srcObject = (event.streams && event.streams[0]) || new
|
||||
MediaStream([event.track])`. When the sharer's tracks arrive without a shared
|
||||
stream id (which happens with `addTransceiver(track)` and is common in
|
||||
Firefox), the audio callback builds a *new* one-track `MediaStream` and
|
||||
overwrites the video one. The video keeps playing from the still-attached
|
||||
transceiver, so nothing looks wrong — there is simply never any sound.
|
||||
2. **There is no way to unmute the thing you are watching.** The mute button lives
|
||||
in `.tile .actions`, hover-revealed in the grid (phase 01 makes it reachable),
|
||||
and the staged tile's caption dims to `0.28`.
|
||||
3. **The sharer does not know whether it offered audio.** `pickSurface()` retries
|
||||
without audio on *any* error except `NotAllowedError`, and `publish()` reads
|
||||
`getAudioTracks().length` once — so a source whose audio track ends mid-share
|
||||
leaves every viewer with a dead 🔇 button.
|
||||
|
||||
Policy for this phase (user-confirmed): **only the staged mirror can be audible.**
|
||||
`/watch` mutes everything else when one is unmuted, and unstaging re-mutes.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `01_keyboard_kiosk_control` (todo) — `setMuted(tile, muted)` and the `m` key
|
||||
come from there; this phase extends them rather than inventing a second mute path.
|
||||
- `02_sharp_capture` (todo) — the per-leg `RTCRtpSender` work: the audio leg must
|
||||
not be caught up in the video sender's parameters, and 25 Mbit/s of video headroom
|
||||
is what leaves room for Opus.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_audio_leg_plumbing.md` — one accumulating `MediaStream` per tile, so the
|
||||
audio and video tracks coexist; assert an `m=audio` line really is negotiated.
|
||||
2. `02_staged_audio_playback.md` — the staged-only unmute policy, applied to
|
||||
mouse, keyboard and unstage, with the tile's audio state readable at a glance.
|
||||
3. `03_audio_source_reporting.md` — a precise audio fallback in `pickSurface()`,
|
||||
three honest states on `/mirror`, and republish when an audio track ends.
|
||||
4. `04_audio_e2e_and_docs.md` — a viewer-side assertion that audio crossed, and
|
||||
the `SECURITY.md`/README text that tells users what a browser will actually
|
||||
capture on Linux.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- **Assert on the viewer** (`AGENTS.md` §5): inbound `bytesReceived` for
|
||||
`inbound-rtp`/`kind === "audio"`, `totalAudioEnergy`/`audioLevel`, and the
|
||||
composed `video.srcObject.getAudioTracks().length`. The sharer's panel proves
|
||||
nothing crossed the wire.
|
||||
- The E2E Chromium already starts with
|
||||
`--use-fake-device-for-audio-capture` (`tests/e2e/conftest.py`), but a headless
|
||||
container has **no audio device**, so a real sound assertion belongs to the
|
||||
`realdisplay`-class run and the fake path asserts the SDP and the composed
|
||||
`MediaStream`. Do not fake the WebRTC layer to make a green test.
|
||||
- `tests/e2e/conftest.py` also passes
|
||||
`--autoplay-policy=no-user-gesture-required`: the suite therefore cannot prove
|
||||
the *browser's* autoplay rule, only mirror's own `muted` state. Keep it that way
|
||||
and assert application state, not browser policy — and never add that flag to a
|
||||
test whose point is muting.
|
||||
- The deliberate, user-initiated unmute promised by `SECURITY.md` §4 is preserved:
|
||||
audio never starts without a click or a keypress.
|
||||
- Coverage **>90%** on new/modified code; full gate per task, plus `-m e2e` on
|
||||
every task that touches media.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] With an audio-bearing source, the viewer's inbound audio
|
||||
`bytesReceived` grows and `video.srcObject.getAudioTracks().length === 1`
|
||||
while the video still decodes frames.
|
||||
- [ ] Only the staged mirror can produce sound; unmuting mutes the others and
|
||||
`unstage()` re-mutes. Reachable by mouse **and** by `m`.
|
||||
- [ ] `/mirror` distinguishes `shared` / `silent` / `not offered`, and a source
|
||||
losing its audio mid-share updates every viewer's tile.
|
||||
- [ ] Cancelling the picker still reports a cancellation, not a silent-share
|
||||
fallback.
|
||||
- [ ] `SECURITY.md` §4 "Audio" and README's platform notes describe what shipped,
|
||||
including the Linux/PipeWire reality.
|
||||
- [ ] Full gate: `uv run ruff check . && uv run ruff format --check . && uv run
|
||||
pyright && uv run pytest && MIRROR_REALDISPLAY=1 uv run pytest -m realdisplay`.
|
||||
Reference in New Issue
Block a user