Experimental/mirror
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 03_audio_share
Build and Push Containers / test (push) Successful in 35s
Details
Build and Push Containers / build-and-push-app (push) Successful in 11s
Details

Browse Source 
**Phase 03 final verification pass — complete, 100% green.**

- Verified all 4 tasks done in `complete/03_audio_share/`; confirmed implementation in `watch.js` (one composed `MediaStream` per tile, staged-only `setMuted()` policy, `data-audio` state), `share.js` (precise `NotAllowedError` rethrow, 3 honest `#stat-audio` states, mid-share audio republish), conftest flags, and the new `tests/e2e/test_audio_share.py` (11 tests, all **passed** — none skipped).
- **Criteria:** (1) viewer inbound audio `bytesReceived` strictly increases + composed stream 1 video/1 audio track while frames still decode — **passed**; (2) staged-only audio via keyboard `m`, mouse button, and `unstage()` re-mute — **passed**; (3) `shared`/`silent`/`not offered` + mid-share audio loss updates panel, roster and tiles — **passed**; (4) cancelled picker reports cancellation, single attempt — **passed**; (5) SECURITY.md §4 Audio, README platform notes, kiosk doc all describe shipped behaviour incl. Linux/PipeWire reality — **verified**; (6) full gate — **passed** (below).
- **Gates:** `uv run ruff check . && uv run ruff format --check .` → clean; `uv run pyright` → 0 errors; `uv run pytest` → 245 passed, 4 skipped (only the `MIRROR_REALDISPLAY`-gated X11 tests); `MIRROR_REALDISPLAY=1 uv run pytest -m realdisplay` → 3 passed, 1 skipped (real-capture audio: documented skip naming the exact host/flag reason, per task 04).
- **Docs:** README.md reviewed in full against code (env table, CLI flags, routes, kiosk key table, quickstart, audio notes) — consistent, no changes needed. AGENTS.md — all commands/paths/coverage claims verified accurate, no stale statements, no changes needed.
- **Defect fixed (pre-existing):** `docs/kiosk-silverblue.md` + `install-kiosk.sh`, `kiosk.conf`, `gnome-kiosk-script` used `mirror.reeselink.com` while README/SECURITY.md use `mirror.reeseapps.com` (the deployment domain, consistent with the fleet); corrected 7 occurrences in 4 files.
- No live-infrastructure changes; realdisplay suite ran on private Xvfb only.
- **Next pending phase:** `04_reachable_share_cta`.
This commit is contained in:
ducoterra
2026-11-01 22:40:30 -05:00
parent 1da466a412
commit d43eb407f8
39 changed files with 1554 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/e2e/conftest.py
+4
View File
@@ -27,6 +27,10 @@ CHROMIUM_ARGS = [
"--use-fake-ui-for-media-stream",
"--use-fake-device-for-media-stream",
"--use-fake-device-for-audio-capture",
# Deliberate — do not "fix" this away: with this flag the suite cannot
# prove the *browser's* autoplay rule. The audio tests therefore assert
# mirror's own `muted` state (the application's policy), and never claim
# to exercise a real browser's autoplay behaviour. See test_audio_share.py.
"--autoplay-policy=no-user-gesture-required",
# Two Chromium instances must each get their own UDP ports; never proxy.
"--no-sandbox",
tests/e2e/test_audio_share.py
+721
View File
@@ -0,0 +1,721 @@
"""End-to-end: the audio leg is negotiated and composed, not overwritten.
The bug this phase fixes: the viewer replaced ``srcObject`` on every
``ontrack``, so the second track to arrive erased the first — the audio
track's one-track ``MediaStream`` replaced the video's and the picture kept
playing (from the still-attached transceiver) while the sound never came.
The assertions here are therefore on the *viewer*: the composed
``MediaStream`` behind ``video.srcObject`` holds one video track *and* one
audio track, the negotiated SDP agrees about audio in both directions
(``m=audio`` iff the sharer published it), and the picture keeps decoding
distinct frames once the audio leg exists.
The fake-device Chromium (conftest.py) starts with
``--use-fake-device-for-audio-capture`` and on the reference host its
``getDisplayMedia`` does deliver a live "Fake audio" track, but a headless
container has no audio hardware, so a *sound* assertion belongs to the
realdisplay-class run. This suite asserts the SDP and the composed
``MediaStream`` only — and skips, naming the reason, when this
browser/device combination offers no display audio at all. The no-audio
direction simulates a source *without* an audio track (the common case for
a window on Linux) by stripping the track from the captured stream; the
WebRTC offer, the m-lines and the transport stay real.
Task 04 adds the hard arrival assertion, still on the *viewer*: after the
staged tile is unmuted, the peer connection's ``getStats()`` must show an
``inbound-rtp`` audio entry whose ``bytesReceived`` strictly increases
across two samples — Opus frames actually arriving over SRTP — and where
the browser reports ``totalAudioEnergy``/``audioLevel``, a live tone means
at least one of the two is non-zero (the fake capture device produces a
tone, so energy exists when the track exists). The picture is asserted in
the same test: audio arriving while the picture froze would be a different
bug, and this phase must not create it. A skip here is honest and loud:
it names the platform and the exact Chromium flags, never a stubbed peer.
"""
from __future__ import annotations
import re
import sys
import time
import httpx
import pytest
from playwright.sync_api import expect
from .conftest import CHROMIUM_ARGS, frames_matching, start_sharing
pytestmark = [pytest.mark.e2e, pytest.mark.timeout(180)]
def _no_audio_skip() -> str:
"""The one honest exit when the environment has no display audio.
It names the platform and the exact flags the browser was launched
with, so a skip reads as a statement about *this* environment — never
an xfailed-into-green failure, never an assertion on an absent track.
"""
return (
"no display audio track in this environment "
f"(platform={sys.platform}; chromium flags: {'; '.join(CHROMIUM_ARGS)})"
)
@pytest.fixture
def device(base_url, browser, request):
"""Open a page as its own device (isolated context, fresh peer id).
The same fixture `test_live_mirror.py` defines (the project's e2e
convention is one copy per suite file, so fixtures are not imported
across test modules).
"""
opened = []
def _open(path: str = "/", errors: list | None = None):
context = browser.new_context(viewport={"width": 1100, "height": 720})
opened.append(context)
request.addfinalizer(context.close)
page = context.new_page()
if errors is not None:
page.on("pageerror", lambda error: errors.append(str(error)))
page.goto(f"{base_url}{path}", wait_until="domcontentloaded")
expect(page.locator("#net")).to_have_attribute("data-state", "online", timeout=10_000)
return page
return _open
# Test scaffolding (AGENTS.md §6): collect the RTCPeerConnection instances
# watch.js creates so the test can read the viewer's negotiated SDP through
# the one allowed global, ``window.__probe``.
PROBE_INIT = """
window.__probe = {pcs: []};
const RealPC = window.RTCPeerConnection;
window.RTCPeerConnection = function (config) {
const pc = new RealPC(config);
window.__probe.pcs.push(pc);
return pc;
};
window.RTCPeerConnection.prototype = RealPC.prototype;
window.RTCPeerConnection.static = RealPC.static;
"""
# Source-level simulation (the no-audio direction): a surface with no audio
# track is what most windows on Linux deliver. The audio tracks are stopped
# and the stream rebuilt from the video tracks only, *before* the app reads
# it — the WebRTC path (offer, m-lines, SRTP) stays real.
SILENT_SOURCE_INIT = """
const realGetDisplayMedia = navigator.mediaDevices.getDisplayMedia.bind(navigator.mediaDevices);
navigator.mediaDevices.getDisplayMedia = async (constraints) => {
const stream = await realGetDisplayMedia(constraints);
stream.getAudioTracks().forEach((t) => t.stop());
return new MediaStream(stream.getVideoTracks());
};
"""
# Source-level simulation (the `not offered` direction): an engine that cannot
# honour the audio constraint at all (a display stack with no audio backend)
# fails the `audio: true` request with a named constraint error. The app must
# then retry with the constraint dropped and say `not offered` — distinct
# from a source that offered none (`silent`). The wrapper counts which
# constraints were asked for on the function object itself (AGENTS.md §6: no
# new test globals), and the rest of the capture path stays real.
NO_AUDIO_SUPPORT_INIT = """
const realGetDisplayMedia = navigator.mediaDevices.getDisplayMedia.bind(navigator.mediaDevices);
const calls = [];
const ask = async (constraints) => {
calls.push(!!constraints.audio);
if (constraints.audio === true) {
throw new DOMException("audio is not supported by this engine", "NotSupportedError");
}
return realGetDisplayMedia(constraints);
};
ask.calls = calls;
navigator.mediaDevices.getDisplayMedia = ask;
"""
# Source-level simulation (the refusal direction): a cancelled picker, or a
# user who unticked "Share tab audio". The app must rethrow — start() toasts
# the cancellation and the share never begins — instead of silently re-opening
# the picker and sharing a surface the person meant not to.
CANCEL_PICKER_INIT = """
const realGetDisplayMedia = navigator.mediaDevices.getDisplayMedia.bind(navigator.mediaDevices);
const calls = [];
const ask = async (constraints) => {
calls.push(!!constraints.audio);
throw new DOMException("The user cancelled the picker", "NotAllowedError");
};
ask.calls = calls;
navigator.mediaDevices.getDisplayMedia = ask;
"""
def open_viewer(
browser, base_url: str, request, init: str | None = None, errors: list | None = None
):
"""A viewer page with the RTCPeerConnection probe installed."""
context = browser.new_context(viewport={"width": 1100, "height": 720})
request.addfinalizer(context.close)
if init:
context.add_init_script(init)
page = context.new_page()
if errors is not None:
page.on("pageerror", lambda error: errors.append(str(error)))
page.goto(f"{base_url}/watch", wait_until="domcontentloaded")
expect(page.locator("#net")).to_have_attribute("data-state", "online", timeout=10_000)
return page
def wait_for_connected(page, timeout: float = 30.0) -> None:
"""Block until one probed leg is `connected` — the SDP is settled then."""
deadline = time.time() + timeout
states: list = []
while time.time() < deadline:
states = page.evaluate("() => (window.__probe.pcs || []).map(pc => pc.connectionState)")
if "connected" in states:
return
time.sleep(0.3)
raise AssertionError(f"no viewer leg reached `connected`: {states}")
def sdp_audio(page) -> dict:
"""The negotiated SDP's audio section on the viewer's connected leg.
``local`` is the viewer's answer, ``remote`` the sharer's offer: the two
must agree, which is the "iff" of the SDP assertion.
"""
return page.evaluate(
"""() => {
const pc = (window.__probe.pcs || []).find(p => p.connectionState === 'connected');
if (!pc) return {found: false};
const sdp = (d) => (d && d.sdp) || '';
return {
found: true,
local: sdp(pc.localDescription).includes('m=audio'),
remote: sdp(pc.remoteDescription).includes('m=audio'),
};
}"""
)
def composed_stream(page) -> dict:
"""The track mix of the MediaStream the viewer's <video> plays."""
return page.evaluate(
"""() => {
const video = document.querySelector('.tile video');
const so = video && video.srcObject;
if (!so) return {found: false};
return {
found: true,
video: so.getVideoTracks().length,
audio: so.getAudioTracks().length,
};
}"""
)
class TestAudioLegPlumbing:
def test_an_audio_share_reaches_the_viewer_as_one_stream(
self, base_url, browser, request, device
):
"""Audio offered: one composed stream holds both tracks, the SDP
carries ``m=audio`` on both sides, and the picture keeps moving."""
errors: list = []
sharer = device("/mirror", errors)
start_sharing(sharer)
# Did this device combination offer display audio? A headless
# container has no audio hardware and the fake device may then
# deliver a monitor stream without one. If so there is nothing to
# assert: skip, naming the reason — never assert on an absent track.
expect(sharer.locator("#stat-audio")).to_contain_text(
re.compile("^(shared|silent)$"), timeout=10_000
)
audio_state = sharer.locator("#stat-audio").text_content().strip()
if audio_state != "shared":
pytest.skip(_no_audio_skip())
viewer = open_viewer(browser, base_url, request, PROBE_INIT)
expect(viewer.locator(".tile")).to_have_count(1, timeout=15_000)
# The wall's own label says the mirror carries audio (server truth,
# rendered by paintTile) before any media is asserted.
expect(viewer.locator(".tile .sub")).to_contain_text("audio", timeout=5_000)
wait_for_connected(viewer)
# The negotiated SDP agrees about audio: the viewer's answer and the
# sharer's offer both carry the m=audio section.
sdp = sdp_audio(viewer)
assert sdp["found"] and sdp["local"] and sdp["remote"], sdp
# The composed stream behind the viewer's <video>: one MediaStream
# holding both kinds — the direct regression test for the overwrite
# bug, where the audio track's fresh one-track stream replaced the
# video's srcObject and erased it.
composed = composed_stream(viewer)
assert composed["found"], "no srcObject on the viewer's <video>"
assert composed["video"] == 1, composed
assert composed["audio"] == 1, composed
# The picture still decodes distinct frames once the audio track is
# in the composed stream: a "fixed" audio path cannot break the
# picture.
frames = frames_matching(viewer, ".tile video")
assert frames["width"] > 0 and frames["height"] > 0, f"no decoded frame: {frames}"
assert frames["distinct"] >= 2, f"video is frozen after the audio track: {frames}"
# The hard arrival assertion, on the viewer: the deliberate gesture
# (stage, then `m`) is what may make the leg audible, and the
# proof is the leg's own stats — the sharer's preview proves
# nothing crossed the wire.
viewer.keyboard.press("Enter") # nothing selected yet: the first tile
expect(viewer.locator("body")).to_have_class(re.compile("staged"), timeout=5_000)
viewer.keyboard.press("m")
assert viewer.evaluate("() => !document.querySelector('#watch-stage video').muted") is True
def audio_inbound() -> dict:
"""One ``inbound-rtp`` audio sample from the viewer's leg."""
return viewer.evaluate(
"""async () => {
const pcs = (window.__probe && window.__probe.pcs) || [];
const pc = pcs.find((p) => p.connectionState === 'connected');
if (!pc) return {pc: false};
const stats = await pc.getStats();
let audio = null;
stats.forEach((r) => {
if (r.type === 'inbound-rtp' && r.kind === 'audio') audio = r;
});
if (!audio) return {pc: true, entry: null};
const energy = audio.totalAudioEnergy;
const level = audio.audioLevel;
return {
pc: true,
entry: {
bytes: audio.bytesReceived || 0,
energy: Number.isFinite(energy) ? energy : null,
level: Number.isFinite(level) ? level : null,
},
};
}"""
)
first: dict = {}
deadline = time.time() + 30
while time.time() < deadline:
first = audio_inbound()
if first.get("entry"):
break
time.sleep(0.5)
assert first.get("entry"), (
f"the viewer's connected leg has no inbound-rtp audio entry: {first}"
)
time.sleep(4.0) # a stretch long enough for the tone to keep flowing
second = audio_inbound()
assert second.get("entry"), f"the inbound-rtp audio entry vanished: {second}"
assert second["entry"]["bytes"] > first["entry"]["bytes"], (
f"inbound audio bytes did not strictly increase: {first} -> {second}"
)
# Where the browser reports an energy figure, a live tone means
# energy: totalAudioEnergy is cumulative and audioLevel is per-frame,
# so on a leg that is delivering, at least one of the two must be
# non-zero.
reported = [
value
for value in (second["entry"]["energy"], second["entry"]["level"])
if value is not None
]
assert reported, f"the browser reported neither totalAudioEnergy nor audioLevel: {second}"
assert any(value > 0 for value in reported), (
f"audio bytes grew but the browser reports zero energy: {second}"
)
# The picture is asserted in the same test: audio arriving while the
# picture froze would be a different bug, and this phase must not
# create it.
frames = frames_matching(viewer, "#watch-stage video")
assert frames["width"] > 0 and frames["height"] > 0, f"no decoded frame: {frames}"
assert frames["distinct"] >= 2, f"video is frozen with the audio leg live: {frames}"
assert errors == [], f"uncaught browser errors: {errors}"
def test_a_source_without_audio_negotiates_no_audio_section(self, base_url, browser, request):
"""The other side of the iff: the sharer published ``audio: false``,
so neither side's SDP may carry ``m=audio``, the composed stream
holds the video track alone, and the picture still moves."""
context = browser.new_context(viewport={"width": 1100, "height": 720})
request.addfinalizer(context.close)
context.add_init_script(SILENT_SOURCE_INIT)
sharer = context.new_page()
sharer.goto(f"{base_url}/mirror", wait_until="domcontentloaded")
expect(sharer.locator("#net")).to_have_attribute("data-state", "online", timeout=10_000)
start_sharing(sharer)
expect(sharer.locator("#stat-audio")).to_have_text("silent", timeout=10_000)
viewer = open_viewer(browser, base_url, request, PROBE_INIT)
expect(viewer.locator(".tile")).to_have_count(1, timeout=15_000)
expect(viewer.locator(".tile .sub")).to_contain_text("silent", timeout=5_000)
wait_for_connected(viewer)
sdp = sdp_audio(viewer)
assert sdp["found"] and not sdp["local"] and not sdp["remote"], sdp
composed = composed_stream(viewer)
assert composed["found"], "no srcObject on the viewer's <video>"
assert composed["video"] == 1, composed
assert composed["audio"] == 0, composed
frames = frames_matching(viewer, ".tile video")
assert frames["width"] > 0 and frames["height"] > 0, f"no decoded frame: {frames}"
assert frames["distinct"] >= 2, f"video is frozen: {frames}"
class TestStagedAudioPlayback:
"""Task 02: the staged-only unmute policy.
The policy the user chose: the staged mirror is the only one that can be
audible. One deliberate gesture — a click on the tile's audio button or
the ``m`` key — turns it on, which is also what keeps ``SECURITY.md``
§4's promise that sound is always a user-initiated action. The
assertions are on the *viewer's* two ``<video>`` elements (stage and
grid), which is where the exclusivity policy actually lives.
``video.muted`` is application state, not browser policy: the suite runs
with ``--autoplay-policy=no-user-gesture-required`` (conftest.py), so
these tests assert the state as mirror sets it and do not prove a real
browser's autoplay rule.
"""
STAGE_MUTED = "document.querySelector('#watch-stage video').muted"
GRID_MUTED = "document.querySelector('#grid video').muted"
def test_only_the_staged_mirror_can_be_unmuted(self, base_url, browser, request, device):
"""Two live mirrors, keyboard only: staging never unmutes on its own;
``m`` unmutes the staged tile; stepping the stage to the other mirror
and pressing ``m`` again leaves exactly one audible — the staged one,
while the first (now in the grid) is muted."""
errors: list = []
sharer_a = device("/mirror", errors)
start_sharing(sharer_a)
sharer_b = device("/mirror", errors)
start_sharing(sharer_b)
viewer = open_viewer(browser, base_url, request, errors=errors)
expect(viewer.locator(".tile")).to_have_count(2, timeout=15_000)
# Staging is the safe default: nothing unmutes without a gesture.
viewer.keyboard.press("j")
viewer.keyboard.press("Enter")
expect(viewer.locator("body")).to_have_class(re.compile("staged"), timeout=5_000)
time.sleep(2.0) # past any autoplay grace period; the state must not move
assert viewer.evaluate(f"() => {self.STAGE_MUTED}") is True
# `m` is the deliberate gesture: the staged tile becomes unmuted.
viewer.keyboard.press("m")
assert viewer.evaluate(f"() => {self.STAGE_MUTED}") is False
# Stepping the stage (`n`) to the other mirror, then `m` again: the
# first mirror — back in the grid — is muted, the second is not.
# Exclusivity is the behaviour under test, on the two viewer elements.
viewer.keyboard.press("n")
viewer.keyboard.press("m")
assert viewer.evaluate(f"() => {self.STAGE_MUTED}") is False
assert viewer.evaluate(f"() => {self.GRID_MUTED}") is True
assert errors == [], f"uncaught browser errors: {errors}"
def test_unstaging_re_mutes_and_re_staging_does_not_revive_audio(
self, base_url, browser, request, device
):
"""`g` returns the wall to silence, and re-staging the same mirror does
not revive the sound — every audible state must trace back to a
gesture (SECURITY.md §4)."""
errors: list = []
sharer = device("/mirror", errors)
start_sharing(sharer)
viewer = open_viewer(browser, base_url, request, errors=errors)
expect(viewer.locator(".tile")).to_have_count(1, timeout=15_000)
viewer.keyboard.press("Enter") # nothing selected yet: the first tile
expect(viewer.locator("body")).to_have_class(re.compile("staged"), timeout=5_000)
time.sleep(2.0)
assert viewer.evaluate(f"() => {self.STAGE_MUTED}") is True
viewer.keyboard.press("m")
assert viewer.evaluate(f"() => {self.STAGE_MUTED}") is False
# Back to the wall: the tile the stage leaves behind is muted.
viewer.keyboard.press("g")
expect(viewer.locator("body")).not_to_have_class(re.compile("staged"), timeout=5_000)
assert viewer.evaluate(f"() => {self.GRID_MUTED}") is True
# Re-staging the same mirror does not bring the sound back.
viewer.keyboard.press("Enter")
expect(viewer.locator("body")).to_have_class(re.compile("staged"), timeout=5_000)
time.sleep(1.5)
assert viewer.evaluate(f"() => {self.STAGE_MUTED}") is True
assert errors == [], f"uncaught browser errors: {errors}"
def test_the_mute_button_stages_and_unmutes(self, base_url, browser, request, device):
"""The mouse path obeys the same rule as the keyboard: clicking a grid
tile's audio button stages the tile and makes it audible; the button
stays the single source of the button/`m`/policy state, and the wall
goes quiet again on `g`."""
errors: list = []
sharer = device("/mirror", errors)
start_sharing(sharer)
expect(sharer.locator("#stat-audio")).to_contain_text(
re.compile("^(shared|silent)$"), timeout=10_000
)
if sharer.locator("#stat-audio").text_content().strip() != "shared":
pytest.skip(_no_audio_skip())
viewer = open_viewer(browser, base_url, request, errors=errors)
expect(viewer.locator(".tile")).to_have_count(1, timeout=15_000)
# The audio button is un-hidden by paintTile() only when the server
# says the mirror carries audio — that is the wait for the audio.
button = viewer.locator(".tile .audio")
expect(button).to_be_visible(timeout=10_000)
# The grid tile is muted; the click must not unmute it in the grid —
# it stages the tile and only then unmutes it.
button.click()
expect(viewer.locator("body")).to_have_class(re.compile("staged"), timeout=5_000)
stage_tile = viewer.locator("#watch-stage .tile")
expect(stage_tile).to_have_attribute("data-audio", "on")
assert viewer.evaluate("() => !" + self.STAGE_MUTED)
# Button and `m` cannot disagree: the glyph and aria-pressed come from
# setMuted(), so the button shows the live state.
expect(stage_tile.locator(".audio")).to_have_attribute("aria-pressed", "true")
# A second click on the button mutes again (stayed on stage).
stage_tile.locator(".audio").click()
expect(stage_tile).to_have_attribute("data-audio", "off")
# Back to the wall: silent again.
viewer.keyboard.press("g")
expect(viewer.locator("body")).not_to_have_class(re.compile("staged"), timeout=5_000)
assert viewer.evaluate(f"() => {self.GRID_MUTED}") is True
assert errors == [], f"uncaught browser errors: {errors}"
def test_dataset_audio_reflects_the_audio_state(self, base_url, browser, request, device):
"""`data-audio` is what the wall shows without a pointer: `none` for a
silent source, `off` for an audio-bearing source that is muted (both
in the grid and staged), and `on` only after a deliberate unmute.
Tiles are identified by their live track mix, because the stats
ticker overwrites the `.sub` label within a few seconds."""
errors: list = []
sharer = device("/mirror", errors)
start_sharing(sharer)
expect(sharer.locator("#stat-audio")).to_contain_text(
re.compile("^(shared|silent)$"), timeout=10_000
)
if sharer.locator("#stat-audio").text_content().strip() != "shared":
pytest.skip(_no_audio_skip())
# The silent source simulates the common Linux case (a window with no
# audio track); the WebRTC path stays real.
context = browser.new_context(viewport={"width": 1100, "height": 720})
request.addfinalizer(context.close)
context.add_init_script(SILENT_SOURCE_INIT)
silent_sharer = context.new_page()
silent_sharer.goto(f"{base_url}/mirror", wait_until="domcontentloaded")
expect(silent_sharer.locator("#net")).to_have_attribute(
"data-state", "online", timeout=10_000
)
start_sharing(silent_sharer)
expect(silent_sharer.locator("#stat-audio")).to_have_text("silent", timeout=10_000)
viewer = open_viewer(browser, base_url, request, errors=errors)
expect(viewer.locator(".tile")).to_have_count(2, timeout=15_000)
def tile_states() -> list[dict]:
return viewer.evaluate(
"""() => [...document.querySelectorAll('.tile')].map(t => {
const so = t.querySelector('video').srcObject;
return {
stream: t.dataset.stream,
audio: t.dataset.audio,
videoTracks: so ? so.getVideoTracks().length : 0,
audioTracks: so ? so.getAudioTracks().length : 0,
};
})"""
)
states: list[dict] = []
deadline = time.time() + 30
while time.time() < deadline:
states = tile_states()
if len(states) == 2 and sorted(s["audioTracks"] for s in states) == [0, 1]:
break
time.sleep(0.5)
assert len(states) == 2 and sorted(s["audioTracks"] for s in states) == [0, 1], states
silent_tile, audio_tile = sorted(states, key=lambda s: s["audioTracks"])
# The silent source is `none` from the first paint onward.
assert silent_tile["audio"] == "none", states
# The audio-bearing source in the grid: available but muted — `off`.
assert audio_tile["audio"] == "off", states
# Staging does not change the mute state ...
viewer.locator(f'.tile[data-stream="{audio_tile["stream"]}"]').click()
expect(viewer.locator("body")).to_have_class(re.compile("staged"), timeout=5_000)
assert (
viewer.evaluate("() => document.querySelector('#watch-stage .tile').dataset.audio")
== "off"
)
# ...and the deliberate `m` is what makes it `on`.
viewer.keyboard.press("m")
expect(viewer.locator("#watch-stage .tile")).to_have_attribute("data-audio", "on")
assert errors == [], f"uncaught browser errors: {errors}"
class TestAudioSourceReporting:
"""Task 03: the sharer's panel says honestly whether audio exists.
Three states, one per reality: ``shared`` (a live audio track),
``silent`` (the browser honoured the audio constraint and the source
delivered no track) and ``not offered`` (the capture only succeeded
after the audio constraint was dropped). The three-way distinction stays
on the sharer's panel; the roster keeps the single ``audio`` bit, so the
wall's 🔇 button must agree with the panel — visible exactly when the
panel says ``shared``. The panel, the wall and the REST roster must never
disagree.
"""
def test_stat_audio_is_one_of_the_three_states_and_the_wall_agrees(
self, base_url, browser, request, device
):
"""With the fake device, whatever state the source lands in, it is one
of the three documented strings and the wall's button matches: the
server's single audio bit is the third witness that panel and wall
agree on."""
errors: list = []
sharer = device("/mirror", errors)
start_sharing(sharer)
expect(sharer.locator("#stat-audio")).to_contain_text(
re.compile("^(shared|silent|not offered)$"), timeout=10_000
)
stat = sharer.locator("#stat-audio").text_content().strip()
viewer = open_viewer(browser, base_url, request, errors=errors)
expect(viewer.locator(".tile")).to_have_count(1, timeout=15_000)
button = viewer.locator(".tile .audio")
if stat == "shared":
expect(button).to_be_visible()
else:
expect(button).to_be_hidden()
# The roster is the third witness: its single audio bit must equal
# "the panel says shared" — no panel/wall disagreement.
mirror = httpx.get(f"{base_url}/api/streams", timeout=5.0).json()["mirrors"][0]
assert mirror["audio"] is (stat == "shared"), (stat, mirror)
assert errors == [], f"uncaught browser errors: {errors}"
def test_a_source_without_audio_shows_no_button_on_the_wall(self, base_url, browser, request):
"""The common Linux case: the source offers no audio track. The panel
says `silent`, and the wall shows *no* audio button at all — a dead
🔇 on a silent mirror is what this test exists to prevent."""
context = browser.new_context(viewport={"width": 1100, "height": 720})
request.addfinalizer(context.close)
context.add_init_script(SILENT_SOURCE_INIT)
sharer = context.new_page()
sharer.goto(f"{base_url}/mirror", wait_until="domcontentloaded")
expect(sharer.locator("#net")).to_have_attribute("data-state", "online", timeout=10_000)
start_sharing(sharer)
expect(sharer.locator("#stat-audio")).to_have_text("silent", timeout=10_000)
viewer = open_viewer(browser, base_url, request)
expect(viewer.locator(".tile")).to_have_count(1, timeout=15_000)
expect(viewer.locator(".tile .audio")).to_be_hidden()
mirror = httpx.get(f"{base_url}/api/streams", timeout=5.0).json()["mirrors"][0]
assert mirror["audio"] is False, mirror
# The picture is never the casualty: it still decodes distinct frames.
frames = frames_matching(viewer, ".tile video")
assert frames["distinct"] >= 2, frames
def test_an_engine_without_audio_support_retries_and_says_not_offered(
self, base_url, browser, request
):
"""The engine refuses the `audio: true` constraint (the `not offered`
state): the app retries with the constraint dropped, captures anyway,
and the panel and wall both reflect that the audio was never on the
table — the retry happened exactly once, in that order."""
context = browser.new_context(viewport={"width": 1100, "height": 720})
request.addfinalizer(context.close)
context.add_init_script(NO_AUDIO_SUPPORT_INIT)
sharer = context.new_page()
sharer.goto(f"{base_url}/mirror", wait_until="domcontentloaded")
expect(sharer.locator("#net")).to_have_attribute("data-state", "online", timeout=10_000)
start_sharing(sharer)
expect(sharer.locator("#stat-audio")).to_have_text("not offered", timeout=10_000)
# Asked for audio, refused, dropped: exactly one retry, in order.
calls = sharer.evaluate("() => navigator.mediaDevices.getDisplayMedia.calls")
assert calls == [True, False], calls
viewer = open_viewer(browser, base_url, request)
expect(viewer.locator(".tile")).to_have_count(1, timeout=15_000)
expect(viewer.locator(".tile .audio")).to_be_hidden()
frames = frames_matching(viewer, ".tile video")
assert frames["distinct"] >= 2, frames
def test_a_cancelled_picker_still_reports_the_cancellation(self, base_url, browser, request):
"""A refused picker is not a no-audio source: the share never begins,
the toast names the cancellation, and there is *no* silent retry —
a share the user did not approve is worse than one that did not
start."""
context = browser.new_context(viewport={"width": 1100, "height": 720})
request.addfinalizer(context.close)
context.add_init_script(CANCEL_PICKER_INIT)
errors: list = []
sharer = context.new_page()
sharer.on("pageerror", lambda error: errors.append(str(error)))
sharer.goto(f"{base_url}/mirror", wait_until="domcontentloaded")
expect(sharer.locator("#net")).to_have_attribute("data-state", "online", timeout=10_000)
sharer.locator("#start").click()
expect(sharer.locator("#toast")).to_contain_text(
"Sharing was cancelled or blocked.", timeout=5_000
)
# One attempt, with audio — no retry was made in the background.
calls = sharer.evaluate("() => navigator.mediaDevices.getDisplayMedia.calls")
assert calls == [True], calls
# The share never began: the panel is idle, the start button is back.
expect(sharer.locator("body")).to_have_attribute("data-live", "off")
expect(sharer.locator("#start")).to_be_enabled()
assert errors == [], f"uncaught browser errors: {errors}"
def test_ending_audio_mid_share_updates_the_panel_and_the_tiles(
self, base_url, browser, request, device
):
"""The source loses its audio track mid-share, the way the OS would
when the captured surface's audio goes away: the track fires
`ended`, the sharer republishes with the flag flipped, the panel
turns `silent`, and the wall's tile loses its button — no dead 🔇 —
while the picture keeps moving (only the audio leg ended)."""
errors: list = []
sharer = device("/mirror", errors)
start_sharing(sharer)
expect(sharer.locator("#stat-audio")).to_contain_text(
re.compile("^(shared|silent)$"), timeout=10_000
)
if sharer.locator("#stat-audio").text_content().strip() != "shared":
pytest.skip(_no_audio_skip())
viewer = open_viewer(browser, base_url, request, PROBE_INIT, errors=errors)
expect(viewer.locator(".tile")).to_have_count(1, timeout=15_000)
expect(viewer.locator(".tile .audio")).to_be_visible(timeout=10_000)
# End the audio source the way the OS would: the track fires
# `ended`, and the sharer must republish with the flag flipped.
sharer.evaluate(
"() => document.querySelector('#preview').srcObject"
".getAudioTracks().forEach((t) => t.stop())"
)
# The panel flips on the same republish ... (the browser honoured the
# constraint; the source simply has no track left — `silent`).
expect(sharer.locator("#stat-audio")).to_have_text("silent", timeout=10_000)
# ... the roster agrees, and the wall's tile loses its button.
mirror = httpx.get(f"{base_url}/api/streams", timeout=5.0).json()["mirrors"][0]
assert mirror["audio"] is False, mirror
expect(viewer.locator(".tile .audio")).to_be_hidden(timeout=10_000)
# The picture keeps moving: only the audio leg ended.
frames = frames_matching(viewer, ".tile video")
assert frames["distinct"] >= 2, frames
assert errors == [], f"uncaught browser errors: {errors}"
tests/e2e/test_x11_capture.py
+167
View File
@@ -32,14 +32,27 @@ sharer window requested delivered viewer decodes
not by the constraints. On real hardware ``screen`` is the true display
resolution, so a small window on a large display is the case the 90% floor
and the share page's 75% toast are for.
Audio on the real capture path is environment-gated, and the gap is written
down here rather than imagined away. ``--auto-select-desktop-capture-source=
Entire screen`` selects the screen without ticking the picker's "share audio"
box, and a host without an audio device has no track to offer at all — on
such hosts ``test_a_real_desktop_shares_audio_when_the_host_offers_any``
finds no audio track in the captured stream and *skips with that reason*
(never a stubbed peer, never an xfailed-into-green failure). Where a host
does expose an audio source, the test asserts the viewer's inbound-rtp audio
bytes increase across two samples and the picture still decodes distinct
frames, the same way the fake-device suite does.
"""
from __future__ import annotations
import os
import re
import shutil
import socket
import subprocess
import sys
import time
import httpx
@@ -139,6 +152,25 @@ PROBE = """() => {
}"""
# The one global a test is allowed to use (AGENTS.md §6): collect the
# RTCPeerConnection instances watch.js creates so a test can read the
# viewer's inbound-rtp stats. The e2e suite (test_audio_share.py) installs
# the same wrapper; it is repeated here because the e2e convention is one
# copy per suite file. On the module's shared watch page `window.__probe`
# is the frame-sampling promise instead — the two uses never share a page.
PC_PROBE_INIT = """
window.__probe = {pcs: []};
const RealPC = window.RTCPeerConnection;
window.RTCPeerConnection = function (config) {
const pc = new RealPC(config);
window.__probe.pcs.push(pc);
return pc;
};
window.RTCPeerConnection.prototype = RealPC.prototype;
window.RTCPeerConnection.static = RealPC.static;
"""
def _free_display() -> str:
for number in range(90, 120):
path = f"/tmp/.X11-unix/X{number}"
@@ -410,3 +442,138 @@ class TestRealX11Capture:
# And the sharer agrees that someone is watching its desktop.
expect(share_page.locator("#viewer-count")).to_have_text("1", timeout=20_000)
assert errors == [], f"uncaught browser errors: {errors}"
def test_a_real_desktop_shares_audio_when_the_host_offers_any(self, base_url, browsers):
"""The realdisplay counterpart of the audio-arrival assertion.
Where the host exposes an audio source to the real capture path, the
viewer's ``inbound-rtp`` audio bytes must strictly increase across two
samples (Opus frames over SRTP) and the picture must still decode
distinct frames. ``--auto-select-desktop-capture-source=Entire
screen`` selects the screen without ticking the picker's "share
audio" box, and a host without an audio device has no track to offer
at all — in that case the test *skips with the reason written down*
(module docstring), never asserts on an absent track, and never
stubs the peer connection.
"""
errors: list[str] = []
share_page = (
browsers["sharer"].new_context(viewport={"width": 1200, "height": 760}).new_page()
)
share_page.on("pageerror", lambda error: errors.append(str(error)))
share_page.goto(f"{base_url}/mirror", wait_until="domcontentloaded")
expect(share_page.locator("#net")).to_have_attribute("data-state", "online", timeout=20_000)
share_page.locator("#start").click()
expect(share_page.locator("body")).to_have_attribute("data-live", "on", timeout=40_000)
# The honest gate: does the captured stream actually carry an audio
# track? This is what the browser's real capture path delivered.
has_audio = share_page.evaluate(
"""() => {
const so = document.querySelector('#preview').srcObject;
return !!(so && so.getAudioTracks().length);
}"""
)
if not has_audio:
share_page.locator("#stop").click()
share_page.close()
pytest.skip(
"the host offers no audio source to the real capture path "
f"(platform={sys.platform}; --auto-select-desktop-capture-source="
"'Entire screen' does not tick 'share audio', and this host "
"delivers no audio track); see the module docstring"
)
watcher = browsers["watcher"].new_context()
watcher.add_init_script(PC_PROBE_INIT)
watch_page = watcher.new_page()
watch_page.on("pageerror", lambda error: errors.append(str(error)))
watch_page.goto(f"{base_url}/watch", wait_until="domcontentloaded")
expect(watch_page.locator("#net")).to_have_attribute("data-state", "online", timeout=20_000)
expect(watch_page.locator(".tile[data-state='live']")).to_have_count(1, timeout=60_000)
# The deliberate gesture: stage, then unmute.
watch_page.keyboard.press("Enter")
expect(watch_page.locator("body")).to_have_class(re.compile("staged"), timeout=5_000)
watch_page.keyboard.press("m")
assert (
watch_page.evaluate("() => !document.querySelector('#watch-stage video').muted") is True
)
def audio_inbound() -> dict:
"""One ``inbound-rtp`` audio sample from the viewer's leg."""
return watch_page.evaluate(
"""async () => {
const pcs = (window.__probe && window.__probe.pcs) || [];
const pc = pcs.find((p) => p.connectionState === 'connected');
if (!pc) return {pc: false};
const stats = await pc.getStats();
let audio = null;
stats.forEach((r) => {
if (r.type === 'inbound-rtp' && r.kind === 'audio') audio = r;
};
if (!audio) return {pc: true, entry: null};
const energy = audio.totalAudioEnergy;
const level = audio.audioLevel;
return {
pc: true,
entry: {
bytes: audio.bytesReceived || 0,
energy: Number.isFinite(energy) ? energy : null,
level: Number.isFinite(level) ? level : null,
},
};
}"""
)
first: dict = {}
deadline = time.time() + 30
while time.time() < deadline:
first = audio_inbound()
if first.get("entry"):
break
time.sleep(0.5)
assert first.get("entry"), (
f"the viewer's connected leg has no inbound-rtp audio entry: {first}"
)
time.sleep(4.0)
second = audio_inbound()
assert second.get("entry"), f"the inbound-rtp audio entry vanished: {second}"
assert second["entry"]["bytes"] > first["entry"]["bytes"], (
f"inbound audio bytes did not strictly increase: {first} -> {second}"
)
# The picture is asserted in the same test: audio arriving while the
# picture froze would be a different bug. One self-contained sample
# loop — no global (on this page `window.__probe` belongs to the
# peer-connection collector).
frames = watch_page.evaluate(
"""async () => {
const video = document.querySelector('#watch-stage video');
const canvas = document.createElement('canvas');
canvas.width = 96; canvas.height = 60;
const ctx = canvas.getContext('2d', {willReadFrequently: true});
const seen = new Set();
for (let i = 0; i < 10; i++) {
await new Promise((r) => setTimeout(r, 120));
try { ctx.drawImage(video, 0, 0, 96, 60); } catch {}
const d = ctx.getImageData(0, 0, 96, 60).data;
let h = 0;
for (let j = 0; j < d.length; j += 4)
h = ((h * 31) + d[j] + d[j + 1] * 3 + d[j + 2] * 7) >>> 0;
seen.add(h);
}
return {width: video.videoWidth, distinct: seen.size};
}"""
)
assert frames["width"] >= SHARP_FLOOR_PX, (
f"the watcher decoded a {frames['width']}px-wide picture: {frames}"
)
assert frames["distinct"] >= 2, f"the watcher received a frozen frame: {frames}"
# Clean up before the module-scoped fixtures serve the next test.
watch_page.close()
watcher.close()
share_page.locator("#stop").click()
share_page.close()
assert errors == [], f"uncaught browser errors: {errors}"
tests/test_app.py
+121
View File
@@ -54,6 +54,24 @@ def wait_for(socket, kind):
assert len(seen) < 25, f"never saw {kind!r}; got {seen}"
def wait_for_update_with(socket, **stream_fields):
"""`wait_for` for `stream_updated` frames, matched on the stream payload.
`stream_updated` is pushed for audience changes too, so the test must
keep reading until the payload itself says what is being awaited (e.g.
the audio flag actually flipped)."""
seen = []
while True:
message = socket.receive_json()
if message.get("type") == "stream_updated" and all(
message["stream"].get(key) == value for key, value in stream_fields.items()
):
return message["stream"]
assert message.get("type") != "error", f"unexpected error: {message}"
seen.append(message)
assert len(seen) < 25, f"never saw stream_updated with {stream_fields}; got {seen}"
class TestPages:
@pytest.mark.parametrize(
("path", "marker"),
@@ -191,6 +209,53 @@ class TestClientScripts:
r'connectionState\s*===\s*"connected".*applySenderQuality', connected, re.S
), "the cap is not re-applied when the leg becomes connected"
def test_pick_surface_rethrows_cancellation_and_retries_only_no_audio_errors(self):
"""The audio-off retry must be precise. A cancelled picker
(NotAllowedError) is a refusal — in Chrome it is also what unticking
"Share tab audio" produces — and must be *rethrown*: retrying would
silently re-open the picker and share a surface the person meant not
to. Only the errors that genuinely mean "this source has no audio"
(a failed audio *constraint*, not a failed capture) may drop the
audio requirement; a bare retry in a catch-all would hide real
capture failures and turn refusals into silent shares."""
source = (STATIC / "share.js").read_text()
# pickSurface and its named-error set live between these markers; the
# first `catch` in that span is pickSurface's own.
begin = source.index("async function pickSurface")
section = source[begin : source.index("function publish", begin)]
catch = section[section.index("catch") :]
assert "NotAllowedError" in catch, "the catch block must name the refusal error"
assert re.search(r"NotAllowedError[\s\S]*?throw\s+err", catch), (
"a cancelled picker must be rethrown, not retried"
)
for name in (
"NotSupportedError",
"OverconstrainedError",
"NotFoundError",
"TypeError",
):
assert name in section, f"the audio-off retry must name {name}"
assert "return await navigator.mediaDevices.getDisplayMedia" not in catch, (
"the audio-off retry must be conditional on the named errors, "
"not a bare retry in a catch-all"
)
assert re.search(r"audiolessError\(err\)\s*\)\s*throw\s+err", section), (
"an error the named set does not cover must surface, not be retried"
)
def test_no_bare_publish_frame_wipes_the_roster(self):
"""The wire's `publish` is a full-metadata frame. A bare
`{ type: "publish" }` frame parses with the model's defaults and
would wipe width/height/audio off the roster on every surface switch
(the mute/unmute republish path) — the panel and the wall would
disagree from a single event. The full frame is the only publish
share.js may send."""
source = (STATIC / "share.js").read_text()
assert source.count('type: "publish"') == 1, (
"share.js must send exactly one publish frame (the full one); "
"a bare { type: 'publish' } frame would reset the metadata"
)
def test_the_two_pages_share_no_element_ids_but_the_shared_chrome(self):
"""Both pages consume one stylesheet, so a duplicated id means a colliding rule.
@@ -477,6 +542,62 @@ class TestSignalingSocket:
assert (stream["width"], stream["height"]) == (1920, 1080)
assert client.get("/api/streams").json()["mirrors"][0]["displaySurface"] == "window"
def test_the_audio_flag_survives_the_wire_and_a_republish_flips_it(self, client):
"""Regression for the audio work: `audio` is the one bit that tells a
viewer whether to expect sound. A publish with audio on must arrive
with the flag set (and the REST roster must agree); a republish that
drops it must flip it via `stream_updated` for **every** subscribed
viewer; and an `unpublish` must leave the wall empty — a viewer that
joins afterwards sees an empty roster, so the flip cannot leak into a
stream that no longer exists. The policy half is the same one the
capture-metadata test pins; this records it for the audio phase so it
cannot regress."""
with (
client.websocket_connect("/ws") as sharer,
client.websocket_connect("/ws") as one,
client.websocket_connect("/ws") as two,
):
sharer_id = handshake(sharer, "sharer", "A")
handshake(one, "viewer", "one")
handshake(two, "viewer", "two")
sharer.send_json({"type": "publish", "label": "Screen", "audio": True})
assert wait_for(one, "stream_added")["stream"]["audio"] is True
assert wait_for(two, "stream_added")["stream"]["audio"] is True
assert client.get("/api/streams").json()["mirrors"][0]["audio"] is True
one.send_json({"type": "subscribe", "streamId": sharer_id})
two.send_json({"type": "subscribe", "streamId": sharer_id})
wait_for(sharer, "peer_joined")
wait_for(sharer, "peer_joined")
# A shape change (audio off) is a republish: each subscribed
# viewer is re-offered and the flag arrives flipped in
# `stream_updated`. The subscribes already pushed one (audience
# change), so keep reading until the republish's own update
# lands on *each* viewer.
sharer.send_json({"type": "publish", "label": "Screen", "audio": False})
for viewer in (one, two):
assert wait_for_update_with(viewer, audio=False)["audio"] is False
assert client.get("/api/streams").json()["mirrors"][0]["audio"] is False
# Teardown: every viewer is told, and the wall is empty
# afterwards — no stray update may keep advertising the gone
# stream's audio.
sharer.send_json({"type": "unpublish"})
wait_for(one, "stream_removed")
wait_for(two, "stream_removed")
body = client.get("/api/streams").json()
assert body["mirrors"] == []
assert body["stats"]["mirrors"] == 0
# The empty wall is unaffected: a viewer that joins after the
# teardown sees an empty roster, with no audio bit on anything.
with client.websocket_connect("/ws") as late:
late.send_json({"type": "hello", "role": "viewer", "name": "late"})
welcome = late.receive_json()
assert welcome["type"] == "welcome", welcome
assert welcome["streams"] == []
def test_both_viewers_of_one_mirror_see_their_own_leg(self, client):
with (
client.websocket_connect("/ws") as sharer,