chore(agent): phase roadmap from TODO.md, 3 phases
Convert the three open TODO items into .agents/phases/todo/ directories (Protocol B append — PLAN.md, AGENTS.md and existing phases untouched, so PLAN.md's roadmap table does not list these phases): - 01_keyboard_kiosk_control (TODO.md:3) — mouse-free /watch and /mirror: key router, j/k/Enter/f/g/n/p/1-9/m/?, focus-visible controls, help overlay. - 02_sharp_capture (TODO.md:4) — capture screen pixels instead of window pixels, maintain-resolution per leg, 12 -> 25 Mbit/s default cap, real-desktop size assertion on the viewer. - 03_audio_share (TODO.md:5) — one accumulating MediaStream per tile (the ontrack overwrite dropped the first track), staged-only unmute policy, honest audio states on /mirror, viewer-side inbound-audio assertion. Every task cites its TODO source line. Locked with the user: 25 Mbit/s default, only the staged mirror can be audible, full key map. Note for the user: PLAN.md section 6 still says the per-leg cap defaults to 12 Mbit/s; phase 02 task 03 changes that and this commit does not edit PLAN.md. LLM-Generated: true
This commit is contained in:
@@ -0,0 +1,89 @@
|
||||
# Phase 01 — Keyboard-only control of the wall
|
||||
|
||||
**Source:** `TODO.md` L3 — "Need keyboard shortcuts. This will be deployed on a kiosk mode linux client that won't have a mouse. Need to be able to fullscreen a mirror, switch between them easily, etc."
|
||||
**Story:** n/a
|
||||
**Context:** `.agents/PLAN.md` §2 A9 (hand-written ES modules, no framework), A10 (nothing from outside the origin), A13 (Playwright, `channel="chromium"`); `docs/kiosk-silverblue.md` (the target box: GNOME Kiosk + Firefox `--kiosk` pointed at `/watch`, set up over SSH with **no keyboard, no monitor, no console**); `src/mirror/static/watch.js`, `watch.html`, `share.js`, `mirror.html`, `app.css`
|
||||
|
||||
## Objective
|
||||
|
||||
Operate the whole product from a keyboard. On `/watch`: select a mirror, take it
|
||||
fullscreen over the wall, step to the next/previous, jump by number, mute, and get
|
||||
an on-screen list of the keys. On `/mirror`: start and stop a share without a
|
||||
pointer.
|
||||
|
||||
Just as important: **remove the pointer-only affordances**. `.tile .actions` is
|
||||
`opacity: 0` until `:hover` (`app.css` L351–352) and the staged caption fades to
|
||||
`0.28` (L396–409) — on a kiosk with no mouse those controls do not exist, so a
|
||||
mirror can currently be neither un-muted nor exited once it is on stage.
|
||||
|
||||
## Dependencies
|
||||
|
||||
— (none). This is the first phase of the roadmap.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_keyboard_command_router.md` — one focus-aware keydown router on `/watch`
|
||||
plus a keyboard "cursor" over the tiles (`j`/`k`/`↑`/`↓`).
|
||||
2. `02_wall_navigation_commands.md` — `Enter`/`f`, `g`, `n`/`p`, `1`–`9`, `m`
|
||||
wired to the existing `stage()`/`unstage()` plumbing.
|
||||
3. `03_keyboard_reachable_tile_controls.md` — focus-visible affordances: every
|
||||
hover-only control becomes reachable and legible without a mouse.
|
||||
4. `04_help_overlay_and_keyboard_e2e.md` — `?` help overlay, `/mirror` `s`/`x`,
|
||||
a mouse-free Playwright suite, and the kiosk/README keyboard docs.
|
||||
|
||||
## Key map (locked by the user for this phase)
|
||||
|
||||
| Key | Scope | Action |
|
||||
|---|---|---|
|
||||
| `j` `k` `↓` `↑` | `/watch` | move the selection to next / previous tile (wraps) |
|
||||
| `Enter` `f` | `/watch` | stage the selected mirror fullscreen (`f` re-enters fullscreen when already staged) |
|
||||
| `g` | `/watch` | back to the wall (unstage) |
|
||||
| `n` `p` | `/watch` | stage next / previous mirror |
|
||||
| `1`–`9` | `/watch` | stage the Nth mirror in grid order |
|
||||
| `m` | `/watch` | mute / unmute the staged mirror |
|
||||
| `?` | `/watch` | toggle the help overlay |
|
||||
| `Esc` | both | `/watch` unstages (already exists); `/mirror` stops sharing (already exists) |
|
||||
| `s` `x` | `/mirror` | start / stop sharing |
|
||||
|
||||
Never intercepted: any chord containing `Ctrl`/`Cmd`/`Alt`, and every key while an
|
||||
`<input>`, `<textarea>` or `contenteditable` has focus (the `/mirror` display-name
|
||||
field is the one real text box).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- **Browser E2E (`-m e2e`)** is the only honest gate for this phase: the product
|
||||
is "a key produces a visible change in a browser". New suite
|
||||
`tests/e2e/test_keyboard_wall.py` drives everything with `page.keyboard.press()`
|
||||
and never `click()`s — a test that reaches for the mouse proves nothing about a
|
||||
kiosk.
|
||||
- Assert the **observable** result of each key: `document.body.classList`
|
||||
contains/omits `staged`, `#watch-stage .tile[data-stream]` equals the expected
|
||||
id, the tile under selection carries `data-selected`, the staged `<video>.muted`
|
||||
flips. Not "the handler ran".
|
||||
- `tests/test_app.py` guards must stay green, and be extended where they are too
|
||||
narrow: `test_every_element_the_script_reaches_for_exists` (any new `$("#id")`
|
||||
needs the element in its page) and
|
||||
`test_the_two_pages_share_no_element_ids_but_the_shared_chrome` (the help
|
||||
overlay is `/watch`-only; the only allowed cross-page ids remain `net`,
|
||||
`notice`, `toast`).
|
||||
- `TestStylesheet::test_no_rule_reintroduces_a_conflicting_display_for_hidden`
|
||||
constrains the new CSS: no rule may un-hide something `hidden`.
|
||||
- Coverage: **>90%** on new/modified Python (this phase adds none beyond tests);
|
||||
the JS is covered by the E2E assertions above, per `AGENTS.md` §5.
|
||||
- Full gate at the end of every task: `uv run ruff check . && uv run ruff format
|
||||
--check . && uv run pyright && uv run pytest`.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] On `/watch` with two live mirrors, `j`/`k` move a *visible* selection and
|
||||
`Enter` puts the selected mirror on stage — driven only by
|
||||
`page.keyboard.press()`.
|
||||
- [ ] `f` re-enters fullscreen, `g` returns to the wall, `n`/`p` step between
|
||||
mirrors, `1`–`9` jump, `m` toggles the staged mirror's audio, `?` shows and
|
||||
hides the key list, `Esc` exits — all with no pointer events in the test.
|
||||
- [ ] No control on either page is reachable *only* by `:hover`; keyboard focus
|
||||
draws a visible ring (`.tile:focus-visible`) and reveals tile actions.
|
||||
- [ ] Keys typed into `#name` on `/mirror` never trigger a command.
|
||||
- [ ] `uv run pytest -m e2e` green, including the new mouse-free suite;
|
||||
`MIRROR_REALDISPLAY=1 uv run pytest -m realdisplay` unaffected.
|
||||
- [ ] README "Kiosk client" and `docs/kiosk-silverblue.md` document the key map.
|
||||
@@ -0,0 +1,88 @@
|
||||
# Task 01 — One keyboard router with a cursor over the tiles
|
||||
|
||||
**Phase:** `01_keyboard_kiosk_control` · **Source:** `TODO.md:3` — "Need keyboard shortcuts. This will be deployed on a kiosk mode linux client that won't have a mouse. Need to be able to fullscreen a mirror, switch between them easily, etc."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Give `/watch` a single place that decides what a keypress means, and a keyboard
|
||||
"cursor" that can reach every tile. Replaces the two ad-hoc handlers that exist
|
||||
today (a per-tile `Enter`/`Space` listener and a global `Escape` listener) so the
|
||||
later tasks add commands, not more listeners.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/mirror/static/watch.js` — delete the `node.addEventListener("keydown", …)`
|
||||
block inside `makeTile()` (the `Enter`/` ` → `stage()` one) and the standalone
|
||||
`document.addEventListener("keydown", …)` that handles `Escape`. Both move into
|
||||
one router added at the bottom of the file, next to the other global wiring.
|
||||
2. `src/mirror/static/watch.js` — add the cursor to the existing `state` object:
|
||||
`cursor: null` (a stream id, not an index — tiles come and go on
|
||||
`stream_added`/`stream_removed`, the map order is the grid order). Add
|
||||
`orderedTiles()` returning `[...tiles.values()]` (DOM order == insertion order,
|
||||
which is what `renderCounts()` already assumes) and `selectedTile()` returning
|
||||
the tile for `state.cursor`, or `null` when it is stale.
|
||||
3. `src/mirror/static/watch.js` — add `setSelected(streamId)`: clears
|
||||
`data-selected` on the previously selected element, sets it on the new one,
|
||||
stores `state.cursor`, and calls
|
||||
`tile.el.focus({ preventScroll: true })` so focus-follows-selection works
|
||||
without the grid jumping (the tiles are `tabindex="0"` already). When a tile is
|
||||
dropped (`dropTile`) or staged, keep `state.cursor` coherent: point it at the
|
||||
nearest remaining tile, and at the staged tile while `state.staged` is set.
|
||||
4. `src/mirror/static/watch.js` — add `moveSelection(delta)` (wraps:
|
||||
`(index + delta + n) % n`, no-op when `n === 0`) and the router
|
||||
`onWallKey(event)`:
|
||||
* bail out **first** when `event.defaultPrevented`, or when
|
||||
`event.ctrlKey || event.metaKey || event.altKey`, or when `event.target` is
|
||||
`INPUT`/`TEXTAREA`/`SELECT` or `isContentEditable` — the wall has no text box
|
||||
today but `/mirror` does, and the router is copied by task 04;
|
||||
* `switch (event.key)` on `j`/`ArrowDown` → `moveSelection(+1)`, `k`/`ArrowUp`
|
||||
→ `moveSelection(-1)`; leave the other commands unhandled here.
|
||||
* `event.preventDefault()` **only** for keys it actually handled.
|
||||
5. `src/mirror/static/watch.js` — register it once:
|
||||
`document.addEventListener("keydown", onWallKey)` and preserve today's
|
||||
behaviour by handling `Escape` → `unstage()` in the same router (that is the
|
||||
listener being deleted in step 1, so `Escape` must not regress).
|
||||
6. `src/mirror/static/app.css` — a selection ring that is *not* the hover ring:
|
||||
`.tile:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }`
|
||||
and `.tile[data-selected]` with the same visual weight as
|
||||
`.tile[data-state="live"]`'s border, so the selection is visible in a screenshot.
|
||||
7. Comment the router with the rule the kiosk depends on: *no command may require
|
||||
a pointer; no pointer affordance may be the only way to see state.*
|
||||
|
||||
ASSUMPTION: the cursor is a stream id and selection wraps at the ends (no
|
||||
"stop at the last tile"), because a wall with 20 mirrors is navigated in a loop
|
||||
from a remote.
|
||||
ASSUMPTION: `Tab` is left to the browser (it still walks the real focus order —
|
||||
tiles, then the header links); the cursor is an overlay on top of focus, not a
|
||||
replacement for it, so assistive tech keeps working.
|
||||
ASSUMPTION: this task adds only `j`/`k`/arrows/`Esc`. The rest of the key map
|
||||
lands in task 02, and `?` in task 04 — the router is the only place a key is read.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- `tests/test_app.py::TestClientScripts::test_scripts_parse_as_es_modules` —
|
||||
`node --check` must stay clean on `watch.js`.
|
||||
- New E2E in `tests/e2e/test_keyboard_wall.py` (created by task 04; if that file
|
||||
does not exist yet, create it here with the `pytestmark = pytest.mark.e2e`
|
||||
pattern and the `device` fixture style from `tests/e2e/test_live_mirror.py`):
|
||||
open `/mirror`, start sharing, open `/watch`, then
|
||||
`page.keyboard.press("j")` and assert the first tile carries
|
||||
`data-selected`; press `k` and assert it wrapped to the last tile; press `Escape`
|
||||
on a staged mirror and assert `document.body.classList` no longer contains
|
||||
`staged`.
|
||||
- Assert that a synthesised `keydown` with `ctrlKey: true` does **not** move the
|
||||
selection (`page.keyboard.press("Control+j")`).
|
||||
- Coverage: **>90%** on this task's new/modified code (JS: covered by the E2E
|
||||
assertions above; Python: unchanged).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `grep -c 'addEventListener("keydown"' src/mirror/static/watch.js` reports
|
||||
exactly **1** — the router.
|
||||
- [ ] `uv run pytest -m e2e -k keyboard` green for select / wrap / Escape.
|
||||
- [ ] `uv run pytest tests/test_app.py` green (scripts parse, ids intact).
|
||||
- [ ] `uv run ruff check . && uv run ruff format --check . && uv run pyright`
|
||||
clean.
|
||||
- [ ] No behavior change in completed work (staging, fan-out, teardown, refusal
|
||||
notices all still work by mouse).
|
||||
@@ -0,0 +1,80 @@
|
||||
# Task 02 — Wall commands: stage, step, jump, mute
|
||||
|
||||
**Phase:** `01_keyboard_kiosk_control` · **Source:** `TODO.md:3` — "Need keyboard shortcuts … Need to be able to fullscreen a mirror, switch between them easily, etc."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Wire the rest of the key map onto the plumbing that already exists — `stage()`
|
||||
and `unstage()` in `watch.js` already move the tile element and request
|
||||
fullscreen, so this task adds commands, not mechanics.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/mirror/static/watch.js` — extract the mute logic out of the audio button
|
||||
handler in `makeTile()` into `setMuted(tile, muted)`: sets
|
||||
`tile.video.muted`, `tile.video.volume = 1`, `tile.video.play().catch(() => {})`
|
||||
when unmuting, and updates `tile.audioBtn` text/`aria-pressed`/`title`. The
|
||||
button handler becomes a two-liner that calls it. (Phase 03 builds its audio
|
||||
policy on this helper — keep the signature `(tile, muted)`.)
|
||||
2. `src/mirror/static/watch.js` — add the commands to the router from task 01, in
|
||||
one `switch (event.key)`:
|
||||
* `Enter` → `stage(state.cursor, { fullscreen: true })`; if nothing is
|
||||
selected, stage the first tile.
|
||||
* `f` → when already staged, re-request fullscreen
|
||||
(`ui.stage.requestFullscreen({ navigationUI: "hide" }).catch(() => {})`) —
|
||||
a kiosk's window manager can drop out of fullscreen without mirror noticing;
|
||||
when not staged, same as `Enter`.
|
||||
* `g` → `unstage()`.
|
||||
* `n` / `p` → stage the tile after/before the current selection (or after the
|
||||
staged one), wrapping; `stage()` already unstages the previous tile.
|
||||
* `1`–`9` → `stage(orderedTiles()[Number(key) - 1]?.id)` — a no-op with a toast
|
||||
when there is no Nth mirror.
|
||||
* `m` → `setMuted(stagedOrSelectedTile, !(…).video.muted)`.
|
||||
3. `src/mirror/static/watch.js` — every command either does something visible or
|
||||
says why it did not, through the existing `toast()` helper (`"no mirrors to
|
||||
switch to"`, `"no mirror 4"`). A silent no-op is the one outcome a
|
||||
keyboard-only operator cannot debug.
|
||||
4. `src/mirror/static/watch.js` — keep the pointer paths intact: the tile click
|
||||
handler, `.expand`, `.audio` and `.exit` all still work, now sharing the same
|
||||
`stage`/`unstage`/`setMuted` functions.
|
||||
5. Comment the `f` branch with the Firefox `--kiosk` reality from
|
||||
`docs/kiosk-silverblue.md` (`window-config.ini` forces the *window* fullscreen;
|
||||
an element `requestFullscreen()` on top of that is allowed and simply no-ops if
|
||||
the browser refuses).
|
||||
|
||||
ASSUMPTION: `n`/`p` **stage** their target rather than only moving the selection,
|
||||
because the TODO's use case is "switch between them" on a wall — a remote operator
|
||||
cannot reach an `Enter` afterwards without seeing what is selected.
|
||||
ASSUMPTION: keys are matched case-insensitively (`toLowerCase()`), so `Shift+J`
|
||||
and a caps-lock kiosk keyboard still work; `?` (which needs `Shift`) is matched on
|
||||
`event.key === "?"` and handled in task 04, not here.
|
||||
ASSUMPTION: `m` on a mirror whose stream has no audio track still toggles the
|
||||
element's `muted` and toasts `"this mirror has no audio"` — the alternative
|
||||
(nothing happens) is indistinguishable from a broken keyboard.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- E2E in `tests/e2e/test_keyboard_wall.py`, two sharers + one viewer (see
|
||||
`test_several_mirrors_land_on_one_wall` for the shape): `j` `Enter` stages
|
||||
tile A (`#watch-stage .tile[data-stream]` == A), `n` stages B **without**
|
||||
tearing A's peer connection down (assert A's `<video>` element is the same node
|
||||
it was — the element moves, `srcObject` is never reassigned), `g` returns to the
|
||||
wall, `1` stages the first mirror, `m` flips `document.querySelector("#watch-stage video").muted`.
|
||||
- Assert the no-op feedback: with one mirror live, `page.keyboard.press("5")`
|
||||
leaves the wall unchanged **and** `#toast` is visible.
|
||||
- Regression: `uv run pytest -m e2e` must still pass
|
||||
`test_clicking_a_tile_takes_it_on_stage`,
|
||||
`test_exit_only_appears_on_the_tile_that_is_on_stage` and
|
||||
`test_one_mirror_fans_out_to_several_viewers` unchanged.
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] The five commands (`Enter`/`f`, `g`, `n`/`p`, `1`–`9`, `m`) each have an
|
||||
E2E assertion that uses `page.keyboard.press()` only.
|
||||
- [ ] Staged mirror A → `n` → mirror B leaves A's `RTCPeerConnection` alive
|
||||
(verified by the same `<video>` node and its `srcObject`).
|
||||
- [ ] Mouse behaviour is unchanged: `uv run pytest -m e2e` fully green.
|
||||
- [ ] `uv run ruff check . && uv run ruff format --check . && uv run pyright`
|
||||
clean.
|
||||
@@ -0,0 +1,76 @@
|
||||
# Task 03 — Controls that do not need a mouse to be seen
|
||||
|
||||
**Phase:** `01_keyboard_kiosk_control` · **Source:** `TODO.md:3` — "This will be deployed on a kiosk mode linux client that won't have a mouse."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The wall's controls exist in the DOM but are invisible without a pointer, and on
|
||||
a kiosk they are therefore unusable. Make every affordance discoverable and
|
||||
operable by keyboard alone, and make the on-screen state legible in a screenshot
|
||||
(a kiosk is debugged from another machine — see `docs/kiosk-silverblue/dim-probe.py`).
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/mirror/static/app.css` — `.tile .actions` (L351–354) is `opacity: 0`
|
||||
unless `:hover` / `:focus-within`, with a `@media (hover: none)` escape hatch
|
||||
that a desktop Firefox on a mouse-less kiosk does **not** match. Add the
|
||||
keyboard states alongside:
|
||||
`.tile:focus-within .actions, .tile:focus-visible .actions,`
|
||||
`.tile[data-selected] .actions { opacity: 1; }`
|
||||
2. `src/mirror/static/app.css` — the staged caption (L396–409) fades to
|
||||
`opacity: 0.28` and only reaches `1` on `:hover`/`:focus-within`. Make it
|
||||
readable for the keyboard case too (`#watch-stage .tile:focus-within .cap`,
|
||||
`.tile[data-selected] .cap`) and keep the quiet look when nothing is focused.
|
||||
3. `src/mirror/static/app.css` — a visible focus ring for every interactive
|
||||
element the router can land on: `.tile:focus-visible`, `.btn:focus-visible`,
|
||||
`.icon-btn:focus-visible`. Do **not** add a blanket `outline: none` anywhere.
|
||||
4. `src/mirror/static/watch.js` — selection and focus must agree:
|
||||
`setSelected()` (task 01) focuses the element; the browser's own `Tab` focus
|
||||
must in turn move the selection, so add a `focusin` handler on `ui.grid` that
|
||||
calls `setSelected()` for the tile that received focus. Without this, `Tab`
|
||||
and `j`/`k` disagree about where the operator is.
|
||||
5. `src/mirror/static/watch.js` — after `unstage()`, return focus to the tile's
|
||||
grid position (it is re-inserted at `state.stageParent`/`state.stageNext`);
|
||||
after `stage()`, focus the staged tile element so `Esc`/`m`/`g` keep working
|
||||
regardless of what had focus before.
|
||||
6. `src/mirror/static/watch.html` — the tile template: give the `<article>`
|
||||
`role="button"` semantics only if it keeps working for real buttons inside it
|
||||
(the `.audio`/`.expand`/`.exit` `<button>`s are already reachable); prefer
|
||||
keeping the `<article>` and adding `aria-keyshortcuts`-free plain labelling —
|
||||
the `aria-label` set in `paintTile()` already says `Watch <label>`.
|
||||
Keep `tests/test_app.py::test_the_two_pages_share_no_element_ids_but_the_shared_chrome`
|
||||
satisfiable: **add no new ids on this task.**
|
||||
7. `src/mirror/static/app.css` — verify against
|
||||
`TestStylesheet::test_no_rule_reintroduces_a_conflicting_display_for_hidden`:
|
||||
reveal with `opacity`/`visibility`, never with `display`.
|
||||
|
||||
ASSUMPTION: revealing on `data-selected`/`:focus-visible` rather than making
|
||||
`.actions` permanently visible, so the wall keeps its current quiet look for a
|
||||
shared display while remaining operable without a pointer.
|
||||
ASSUMPTION: `@media (hover: none)` is kept (touch phones) and the new selectors
|
||||
are added next to it, not in place of it.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- `uv run pytest tests/test_app.py::TestStylesheet` — both stylesheet guards stay
|
||||
green; add a case there asserting every `.actions` rule that sets `opacity`
|
||||
also has a non-hover trigger (a regex over the stylesheet is enough and is the
|
||||
style the existing tests use).
|
||||
- E2E in `tests/e2e/test_keyboard_wall.py`: with a mirror live, focus a tile with
|
||||
`page.keyboard.press("Tab")` until `document.activeElement` is a tile, then
|
||||
assert `getComputedStyle(tile.querySelector(".actions")).opacity === "1"` and
|
||||
that `.tile:focus-visible` produced a visible outline — with **no**
|
||||
`page.hover()` anywhere in the test.
|
||||
- E2E: `Tab` moves the selection (`data-selected` follows `document.activeElement`).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `grep -n "opacity: 0" src/mirror/static/app.css` shows no control that is
|
||||
revealed *only* by `:hover`.
|
||||
- [ ] A keyboard-only E2E run can reach and read the audio/expand controls of a
|
||||
tile without a single pointer event.
|
||||
- [ ] Screenshot of a focused tile shows an unambiguous focus ring (check the
|
||||
Playwright trace/screenshot artifact).
|
||||
- [ ] `uv run pytest` green; no visual regression on the share page.
|
||||
@@ -0,0 +1,82 @@
|
||||
# Task 04 — Key help, share-page keys, and the mouse-free suite
|
||||
|
||||
**Phase:** `01_keyboard_kiosk_control` · **Source:** `TODO.md:3` — "Need keyboard shortcuts. This will be deployed on a kiosk mode linux client that won't have a mouse. Need to be able to fullscreen a mirror, switch between them easily, etc."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Finish the capability: the operator can discover the keys on screen (`?`), drive
|
||||
`/mirror` from the keyboard, and there is a committed E2E suite that proves the
|
||||
whole wall works with the pointer left at rest.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/mirror/static/watch.html` — add the overlay after `#watch-stage`:
|
||||
`<div class="help" id="wall-help" hidden>` containing a `<dl>` of the key map
|
||||
from `00_phase.md` and a line saying `? closes this`. `/watch`-only id: the
|
||||
only ids two pages may share are `net`, `notice`, `toast`
|
||||
(`tests/test_app.py::test_the_two_pages_share_no_element_ids_but_the_shared_chrome`).
|
||||
2. `src/mirror/static/app.css` — `.help` as a centred, `position: fixed` panel
|
||||
above the wall (`z-index` above `#watch-stage`'s `50`), readable at 3 m on a
|
||||
wall display; reveal/hide via the global `[hidden] { display: none !important }`
|
||||
rule that already exists — do not fight it (see `TestStylesheet`).
|
||||
3. `src/mirror/static/watch.js` — reach for it once (`help: $("#wall-help")`,
|
||||
which the element-existence guard then enforces) and add `toggleHelp()` to the
|
||||
router: `?` (i.e. `event.key === "?"`) toggles it; `Escape` closes the overlay
|
||||
**first** and only unstages when the overlay was not open. Keep focus on the
|
||||
tile so `?` → `?` → `Esc` returns exactly where the operator was.
|
||||
4. `src/mirror/static/share.js` — a small router of its own (the two pages do not
|
||||
share a key handler; duplicating eight lines is cheaper than a new abstraction
|
||||
over `net.js`): ignore `Ctrl`/`Cmd`/`Alt` chords and any key while `#name`
|
||||
has focus, then `s` → `start()`, `x` → `stop()`, and keep the existing
|
||||
`Escape` → `stop()`. A `keydown` is a user activation, so `getDisplayMedia()`
|
||||
from `s` is allowed by the autoplay/permission policy — comment that, because
|
||||
it is the reason `s` works at all.
|
||||
5. `src/mirror/static/share.js` — when `#start` is disabled (a refused address or
|
||||
no secure context) the `s` key must not silently fail: route to the same
|
||||
`showNotice()`/`captureSupport()` path the button click uses (it already does —
|
||||
`start()` checks `captureSupport()` — confirm and cover it).
|
||||
6. `tests/e2e/test_keyboard_wall.py` — the suite tasks 01–03 hang their assertions
|
||||
on. `pytestmark = pytest.mark.e2e`, reuse the `device`/`base_url`/`browser`
|
||||
fixtures and `start_sharing()` helper from `tests/e2e/conftest.py`, and keep the
|
||||
file **pointer-free**: no `click()`, no `hover()`. Cases: select+wrap, stage,
|
||||
`n`/`p` between two mirrors, `1`–`9`, `g`, `m`, `?` open/close, `Esc`
|
||||
overlay-then-stage, and `/mirror` `s`/`x` starting and stopping a real capture.
|
||||
Print `pageerror`/`console` from the first line of each test (`AGENTS.md` §8).
|
||||
7. `README.md` (the "Kiosk client" section, ~L111) and
|
||||
`docs/kiosk-silverblue.md` — a short "Keys" table for the wall, and the note
|
||||
that the kiosk needs no keyboard attached to be operated *or* debugged, but the
|
||||
keys are there when one is plugged in.
|
||||
|
||||
ASSUMPTION: the overlay is static markup in `watch.html` rather than generated
|
||||
from a JS map, so `test_every_element_the_script_reaches_for_exists` keeps
|
||||
verifying it and a translator could later find the strings in the page.
|
||||
ASSUMPTION: `/mirror` gets only `s`/`x`/`Esc`. Adding keys that mirror the wall on
|
||||
the share page would tempt a kiosk to share by accident; sharing always stays an
|
||||
explicit, chosen act.
|
||||
ASSUMPTION: `?` is the only key requiring `Shift` and is matched on `event.key`,
|
||||
so layouts where `?` is a dead key are simply left alone (the overlay's existence
|
||||
is documented in the README for those operators).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- `uv run pytest tests/test_app.py::TestClientScripts` — `node --check` on the
|
||||
changed modules, every new `$("#…")` present in its page, no new cross-page id.
|
||||
- `uv run pytest tests/e2e/test_keyboard_wall.py` — the full keyboard walk, green
|
||||
and pointer-free. Enforce "pointer-free" in the test file itself:
|
||||
`assert "click(" not in Path(__file__).read_text()` in a tiny meta-test, so the
|
||||
guarantee cannot rot.
|
||||
- `uv run pytest -m "not e2e"` fast suite green; `MIRROR_REALDISPLAY=1 uv run
|
||||
pytest -m realdisplay` unaffected (this task touches no capture code).
|
||||
- Coverage: **>90%** on new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `?` opens a readable key list over the wall; `?` or `Esc` closes it; the
|
||||
overlay is the element `watch.js` reaches for and it exists in `watch.html`.
|
||||
- [ ] `s` starts and `x` stops a share on `/mirror`, and `s` in the display-name
|
||||
field types an "s" instead.
|
||||
- [ ] `uv run pytest -m e2e` green, including the pointer-free keyboard suite.
|
||||
- [ ] README and `docs/kiosk-silverblue.md` both list the keys.
|
||||
- [ ] Full gate: `uv run ruff check . && uv run ruff format --check . && uv run
|
||||
pyright && uv run pytest`.
|
||||
@@ -0,0 +1,70 @@
|
||||
# Phase 02 — A sharp picture on the wall
|
||||
|
||||
**Source:** `TODO.md` L4 — "The screen sharing should be sharper, it's a little fuzzy when sharing my display."
|
||||
**Story:** n/a
|
||||
**Context:** `.agents/PLAN.md` §5 (the browser-harness finding *"Chromium scales a monitor capture to the capturing window"*, and the rule that media is asserted on the **viewer's** `<video>`), §6 ("Mesh upload = N × bitrate per sharer … per-leg `maxBitrate` cap"); `src/mirror/static/share.js` (`pickSurface`, `addPeer`), `src/mirror/config.py`, `src/mirror/static/net.js`
|
||||
|
||||
## Objective
|
||||
|
||||
Text on a shared screen is legible on a full-HD wall. Three things make today's
|
||||
picture fuzzy, and each gets its own task:
|
||||
|
||||
1. **The capture is smaller than the screen.** `pickSurface()` requests
|
||||
`{ frameRate: { ideal: 30, max: 60 } }` and *no* dimensions, so Chromium
|
||||
captures the monitor scaled down to the size of the sharing window (PLAN §5
|
||||
records exactly this: `1280×800 → 1200×750`). A window that is half the screen
|
||||
ships a quarter of the pixels, stretched back up on the wall.
|
||||
2. **The encoder is allowed to throw resolution away.** Nothing sets
|
||||
`degradationPreference`, so the default `"balanced"` trades pixels for frame
|
||||
rate the moment the LAN hiccups — the opposite of what a screen of text needs.
|
||||
3. **12 Mbit/s is not enough for the pixels we now send.** A 1440p text-heavy
|
||||
surface at 12 Mbit/s is where macroblocking comes from.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `01_keyboard_kiosk_control` (todo) — no functional dependency, but it runs
|
||||
first and its E2E suites (`tests/e2e/test_keyboard_wall.py`) must still pass
|
||||
here; this phase changes the same two files (`share.js`, `watch.js` untouched).
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_capture_resolution_constraints.md` — ask the picker for the screen's real
|
||||
pixels (`screen × devicePixelRatio`) instead of the window's.
|
||||
2. `02_sender_fidelity.md` — per-leg `RTCRtpSender` parameters:
|
||||
`maintain-resolution`, re-applied after every renegotiation.
|
||||
3. `03_bitrate_default_and_surface.md` — raise the per-leg cap default to
|
||||
25 Mbit/s and keep server, client fallback, `.env.example` and README honest
|
||||
about it.
|
||||
4. `04_sharpness_assertion_e2e.md` — a real-desktop assertion on the **viewer's**
|
||||
decoded size, so "sharper" stops being a feeling.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- The media rule from `AGENTS.md` §5 / PLAN §5 governs this whole phase: assert on
|
||||
the **viewer's** `<video>` (`videoWidth`, `videoHeight`, `framesDecoded`,
|
||||
inbound bitrate), never on the sharer's preview.
|
||||
- The `realdisplay` suite is where sharpness is actually provable
|
||||
(`MIRROR_REALDISPLAY=1 uv run pytest -m realdisplay`) because it captures a real
|
||||
desktop with a real encoder; the fake-device suite proves the constraints do not
|
||||
break the deterministic 1280×720 path.
|
||||
- Python side (`config.py` default) is unit + API tested: `tests/test_config.py`,
|
||||
`tests/test_app.py::TestJsonApi`.
|
||||
- No new external asset, no new dependency, no framework (PLAN A9/A10).
|
||||
- Coverage **>90%** on new/modified code; full gate per task:
|
||||
`uv run ruff check . && uv run ruff format --check . && uv run pyright && uv run pytest`.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `MIRROR_REALDISPLAY=1 uv run pytest -m realdisplay` asserts the viewer's
|
||||
decoded width is ≥ 90% of the Xvfb desktop width (today it only asserts the
|
||||
aspect ratio).
|
||||
- [ ] `/api/config` reports `maxBitrateBps: 25000000` by default and `net.js`'s
|
||||
offline fallback agrees.
|
||||
- [ ] The share page's own stats (`#stat-resolution`) match what the viewer
|
||||
decodes, within the documented downscale the browser applies.
|
||||
- [ ] `uv run pytest` and `MIRROR_REALDISPLAY=1 uv run pytest -m realdisplay`
|
||||
both green; `test_…aspect ratio…` guard still distinguishes real capture
|
||||
from the fake device.
|
||||
- [ ] README "Configuration" documents the new default; the commit message flags
|
||||
that `PLAN.md` §6's "default 12 Mbit/s" figure is now stale for the user to
|
||||
amend (this phase must not edit `PLAN.md` itself).
|
||||
@@ -0,0 +1,84 @@
|
||||
# Task 01 — Capture the screen's pixels, not the window's
|
||||
|
||||
**Phase:** `02_sharp_capture` · **Source:** `TODO.md:4` — "The screen sharing should be sharper, it's a little fuzzy when sharing my display."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Ask `getDisplayMedia()` for the surface's real pixel dimensions. Today the
|
||||
request carries a frame rate and nothing else, so Chromium hands back the monitor
|
||||
rescaled to the *browser window* — the primary cause of the fuzziness.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/mirror/static/share.js` — in `pickSurface()`, extend the `video`
|
||||
constraints object (currently `{ frameRate: { ideal: 30, max: 60 } }`) with
|
||||
dimensions computed from the display, not the window:
|
||||
|
||||
```js
|
||||
const dpr = window.devicePixelRatio || 1;
|
||||
const width = Math.min(Math.round(screen.width * dpr), 3840);
|
||||
const height = Math.min(Math.round(screen.height * dpr), 2160);
|
||||
```
|
||||
|
||||
passed as `width: { ideal: width }, height: { ideal: height }` alongside the
|
||||
existing `frameRate`, keeping `selfBrowserSurface`/`surfaceSwitching`/
|
||||
`monitorTypeSurfaces` untouched.
|
||||
2. `src/mirror/static/share.js` — keep both existing `getDisplayMedia` calls in
|
||||
one place: the `audio: true` attempt and its `audio: false` retry must send the
|
||||
same video constraints, so build the `video` object once and reuse it (this is
|
||||
also what phase 03's audio task relies on; do not change the catch semantics
|
||||
here).
|
||||
3. `src/mirror/static/share.js` — after capture, compare `track.getSettings()`
|
||||
against what was asked. `publish()` already reads the settings for the panel;
|
||||
add a one-line diagnostic: when the delivered width is below ~75% of the
|
||||
requested width, `toast()` once per share —
|
||||
`"Chromium captured N×M: the browser window limits a screen capture, so make the share page bigger for a sharper wall."`
|
||||
Comment the reason (PLAN §5's finding) at the point where it is detected.
|
||||
4. `src/mirror/static/share.js` — keep `video.contentHint = "detail"` exactly
|
||||
where it is (it is the other half of the sharpness story; task 02 adds the
|
||||
sender side).
|
||||
5. Comment in `pickSurface()` stating the rule that the kiosk wall depends on:
|
||||
*a capture is only as sharp as the pixels the sharer's browser agreed to grab,
|
||||
and the request must name them.*
|
||||
|
||||
ASSUMPTION: `ideal`, never `exact`. `exact` fails or hides picker entries on
|
||||
displays/DPRs the browser cannot honour, which would break sharing outright;
|
||||
`ideal` is a preference the browser can decline while still capturing.
|
||||
ASSUMPTION: the request is capped at 3840×2160 so an 8K HiDPI sharer cannot
|
||||
starve a 1080p wall's decoder or its uplink; no new `MIRROR_*` variable is added
|
||||
for this (the existing `MIRROR_MAX_BITRATE` is the tuning knob).
|
||||
ASSUMPTION: `screen.width/height × devicePixelRatio` is used rather than
|
||||
`screen.avail*` (panels shave pixels off) and rather than enumerating
|
||||
`navigator.mediaDevices.enumerateDevices()` (which does not report resolutions
|
||||
for display surfaces).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- E2E (`uv run pytest -m e2e`): with the deterministic fake device
|
||||
(`--use-fake-device-for-media-stream`, always 1280×720) assert the share still
|
||||
starts and the viewer still decodes `1280×720` — i.e. asking for 4K does not
|
||||
break or change the fake path, and `publish()` still reports the real settings.
|
||||
- Unit-ish guard in `tests/test_app.py::TestClientScripts` style: `share.js` must
|
||||
contain no `exact:` constraint key (regex guard), so a future edit cannot
|
||||
reintroduce a hard constraint that breaks pickers.
|
||||
- Real desktop measurement (not a hard assertion yet — task 04 turns it into one):
|
||||
log the requested vs delivered width from
|
||||
`tests/e2e/test_x11_capture.py::…::test_the_real_desktop_is_captured_not_a_synthetic_one`
|
||||
once during development and record the numbers in the commit message.
|
||||
If Chromium still clamps the capture to the window, say so in the commit message
|
||||
so `.agents/PLAN.md` §5 can be amended by the user — do not edit `PLAN.md` here.
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `pickSurface()` requests explicit `width`/`height` `ideal` values derived
|
||||
from `screen × devicePixelRatio` and capped at 3840×2160.
|
||||
- [ ] Both the audio and silent `getDisplayMedia` attempts use identical video
|
||||
constraints (one shared object).
|
||||
- [ ] `uv run pytest -m e2e` green, viewer still decodes 1280×720 on the fake
|
||||
device and frames still differ (`frames_matching`).
|
||||
- [ ] `MIRROR_REALDISPLAY=1 uv run pytest -m realdisplay` green, with the measured
|
||||
requested-vs-delivered numbers written into the commit message.
|
||||
- [ ] `uv run ruff check . && uv run ruff format --check . && uv run pyright`
|
||||
clean.
|
||||
@@ -0,0 +1,87 @@
|
||||
# Task 02 — Tell the encoder to keep the pixels
|
||||
|
||||
**Phase:** `02_sharp_capture` · **Source:** `TODO.md:4` — "The screen sharing should be sharper, it's a little fuzzy when sharing my display."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
`contentHint = "detail"` says *this is text*; nothing yet tells the encoder what
|
||||
to sacrifice. With the default `degradationPreference: "balanced"` a LAN hiccup
|
||||
buys frame rate with resolution — the picture goes soft and stays soft. This task
|
||||
pins resolution per leg, and keeps the per-leg bitrate cap working through
|
||||
renegotiation.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/mirror/static/share.js` — in `addPeer()`, capture the sender returned by
|
||||
`pc.addTransceiver(track, init)` (the `catch` fallback to
|
||||
`pc.addTrack(track, state.stream)` returns a sender too — keep both paths
|
||||
producing one).
|
||||
2. `src/mirror/static/share.js` — add `applySenderQuality(peer)`:
|
||||
* find the video sender on `peer.pc.getSenders()` whose `.track.kind === "video"`;
|
||||
* read `const params = sender.getParameters()`, set
|
||||
`params.degradationPreference = "maintain-resolution"`, and set
|
||||
`params.encodings = (params.encodings?.length ? params.encodings : [{}])`
|
||||
with `maxBitrate = config.maxBitrateBps` on the **first** encoding only;
|
||||
* `await sender.setParameters(params)` inside a `try/catch` that swallows only
|
||||
what it names — Firefox has no `degradationPreference` and `setParameters`
|
||||
rejects there; comment that, since a bare `catch {}` reads like a bug
|
||||
(`AGENTS.md` §7: swallow only what you can name).
|
||||
* guard with `typeof sender.setParameters === "function"` before the call.
|
||||
3. `src/mirror/static/share.js` — call it from `pc.onnegotiationneeded` **after**
|
||||
`setLocalDescription` succeeds (so the encodings exist) and again when
|
||||
`pc.connectionState` becomes `"connected"`: `setParameters` is per-negotiation
|
||||
and a re-offer (surface switch, viewer rejoin) resets the encodings. Note in a
|
||||
comment that `sendEncodings` passed to `addTransceiver` is only the *initial*
|
||||
request.
|
||||
4. `src/mirror/static/share.js` — remove the now-duplicated `init.sendEncodings =
|
||||
[{ maxBitrate: config.maxBitrateBps }]` only if `applySenderQuality` provably
|
||||
covers the same behaviour; otherwise keep both (initial + post-negotiation) and
|
||||
say why in a comment. The cap must never disappear — `AGENTS.md`/PLAN §6 exist
|
||||
because N viewers must not melt one sharer's uplink.
|
||||
5. `src/mirror/static/share.js` — leave `video.contentHint = "detail"` alone, and
|
||||
do not touch the viewer (`watch.js`): this is a sender-side decision, the
|
||||
viewer has nothing to negotiate away (A4: the sharer owns the track set).
|
||||
|
||||
ASSUMPTION: `maintain-resolution` over `"balanced"` — this is a screen-mirroring
|
||||
tool where legible text beats smooth motion, which is the same trade the existing
|
||||
`contentHint = "detail"` already made. Frame rate will drop under congestion;
|
||||
that is the intent, and the sharer's `#stat-perf` line already shows fps so the
|
||||
trade is visible.
|
||||
ASSUMPTION: applied to the video sender only; the audio sender (phase 03) keeps
|
||||
browser defaults.
|
||||
ASSUMPTION: `maxBitrate` stays a *per-leg* cap sourced from `config.maxBitrateBps`
|
||||
(`/api/config`); no per-peer fairness logic — that is PLAN §7 item 4, a later phase.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- E2E (`-m e2e`) with two viewers on one sharer: after both legs reach
|
||||
`"connected"`, read the sender parameters back in the page
|
||||
(`page.evaluate`) and assert `degradationPreference === "maintain-resolution"`
|
||||
and `encodings[0].maxBitrate === <config value>` for each leg. On the
|
||||
full-Chromium channel this is a real assertion, not a stub.
|
||||
- E2E: switch the shared surface (or trigger a renegotiation by adding a second
|
||||
viewer mid-share) and re-assert the parameters survived — this is the bug the
|
||||
"re-apply after negotiation" step exists for.
|
||||
- E2E regression: the two existing fan-out tests
|
||||
(`test_one_mirror_fans_out_to_several_viewers`,
|
||||
`test_several_mirrors_land_on_one_wall`) stay green and still assert distinct
|
||||
decoded frames on the viewer.
|
||||
- Guard in `tests/test_app.py`: `share.js` contains
|
||||
`"maintain-resolution"` exactly once, and `applySenderQuality` is called from
|
||||
both the negotiation path and the connected path (regex over the file, matching
|
||||
the existing test style).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] Every live leg's video sender reports `degradationPreference:
|
||||
"maintain-resolution"` and the configured `maxBitrate`, read back from the
|
||||
browser in a test.
|
||||
- [ ] The parameters survive a renegotiation (asserted).
|
||||
- [ ] The per-leg bitrate cap is still applied (no leg sends more than
|
||||
`maxBitrateBps` in the outbound `getStats()` sample).
|
||||
- [ ] `uv run pytest -m e2e` green; `MIRROR_REALDISPLAY=1 uv run pytest -m
|
||||
realdisplay` green.
|
||||
- [ ] `uv run ruff check . && uv run ruff format --check . && uv run pyright`
|
||||
clean.
|
||||
@@ -0,0 +1,72 @@
|
||||
# Task 03 — A bitrate default that fits the pixels we now send
|
||||
|
||||
**Phase:** `02_sharp_capture` · **Source:** `TODO.md:4` — "The screen sharing should be sharper, it's a little fuzzy when sharing my display."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Tasks 01–02 make the encoder send more real pixels; 12 Mbit/s per leg is not
|
||||
enough for them. Raise the default per-leg cap to **25 Mbit/s** (user-confirmed
|
||||
for this roadmap) and keep every place that number appears in agreement — the
|
||||
server default, the client's offline fallback, `.env.example`, the README.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/mirror/config.py` — `DEFAULT_MAX_BITRATE_BPS = 25_000_000` (was
|
||||
`12_000_000`, L15). It flows to `Settings.max_bitrate_bps` →
|
||||
`client_config()["maxBitrateBps"]` unchanged; no new field.
|
||||
2. `src/mirror/static/net.js` — `loadConfig()`'s offline `fallback` object
|
||||
(L20) hardcodes `maxBitrateBps: 12000000`. Update it to `25000000`. This is the
|
||||
number a browser uses when `/api/config` is unreachable; if it drifts from the
|
||||
server it silently changes picture quality per client, and nothing would fail.
|
||||
3. `.env.example` — `MIRROR_MAX_BITRATE=25000000` (L8) with a one-line comment:
|
||||
per-viewer-leg cap, 25 Mbit/s suits a wired LAN at 1440p, lower it for
|
||||
Wi-Fi sharers.
|
||||
4. `README.md` — the "Configuration" table (~L58) row for
|
||||
`MIRROR_MAX_BITRATE`: new default and the one-sentence trade-off (Wi-Fi sharers
|
||||
with many viewers may need less, not more).
|
||||
5. `tests/test_config.py::TestSettings::test_defaults` — it currently does not pin
|
||||
the bitrate default at all (it asserts port, host, ice_servers, max_streams,
|
||||
cidrs). Add `assert settings.max_bitrate_bps == DEFAULT_MAX_BITRATE_BPS` and
|
||||
import the constant, so the next silent change is a conscious one.
|
||||
6. `tests/test_config.py::TestSettings::test_environment_wins` already covers the
|
||||
override path (`250000`) — leave it.
|
||||
|
||||
ASSUMPTION: 25 Mbit/s per leg (not 50). Chosen with the user for this roadmap:
|
||||
comfortable for 1440p text on a wired LAN while still bounding a sharer's upload
|
||||
when several viewers subscribe. `MIRROR_MAX_BITRATE` remains the escape hatch in
|
||||
both directions, so this is a default change and not a policy change.
|
||||
ASSUMPTION: the ceiling stays *per leg*, exactly as today — a 5-viewer sharer
|
||||
still gets 5 × 25 Mbit/s of headroom available. Aggregate fairness is PLAN §7
|
||||
item 4 ("Capacity & fairness") and is deliberately not smuggled in here.
|
||||
ASSUMPTION: `PLAN.md` §6's "default 12 Mbit/s" text becomes stale with this
|
||||
change; this phase must not edit `PLAN.md` (Protocol B), so the commit message
|
||||
names the stale line for the user to amend.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- `tests/test_config.py` — the new default assertion, plus the existing
|
||||
"bogus number falls back" case still resolves to `DEFAULT_PORT`-style behaviour
|
||||
(unchanged).
|
||||
- `tests/test_app.py::TestJsonApi::test_config_advertises_the_media_defaults`
|
||||
uses the `settings` fixture (`max_bitrate_bps=2_000_000`), so it must stay
|
||||
**unchanged** — that is the point: it proves the value is honoured end to end,
|
||||
not that it equals a magic number. Add nothing there; verify it still passes.
|
||||
- New cross-file guard in `tests/test_app.py::TestClientScripts` (same regex
|
||||
style as the existing script guards): the bitrate default in
|
||||
`src/mirror/config.py` and the fallback literal in `src/mirror/static/net.js`
|
||||
must be equal — one number, two places, one test that notices.
|
||||
- Coverage: **>90%** on new/modified code (`config.py` is fully covered by
|
||||
`tests/test_config.py`).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run python -c "from mirror.config import Settings; print(Settings.from_env().max_bitrate_bps)"`
|
||||
prints `25000000` with no environment set.
|
||||
- [ ] `curl -s localhost:8055/api/config | python -m json.tool` reports
|
||||
`maxBitrateBps: 25000000` on a bare `uv run mirror serve`.
|
||||
- [ ] `grep -n "12000000\|12_000_000"` over `src/`, `.env.example` and
|
||||
`tests/` returns nothing.
|
||||
- [ ] `uv run pytest tests/test_config.py tests/test_app.py` green.
|
||||
- [ ] README + `.env.example` document 25 Mbit/s; commit message names the stale
|
||||
`PLAN.md` §6 line.
|
||||
@@ -0,0 +1,78 @@
|
||||
# Task 04 — Make "sharper" an assertion
|
||||
|
||||
**Phase:** `02_sharp_capture` · **Source:** `TODO.md:4` — "The screen sharing should be sharper, it's a little fuzzy when sharing my display."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Lock the capability in with a test that fails if the picture goes soft again.
|
||||
Today's real-desktop test asserts only the **aspect ratio** — a deliberately weak
|
||||
assertion while the capture was window-sized. Now that the request names real
|
||||
pixels, assert **delivered size on the viewer**, which is the only number that
|
||||
correlates with the fuzziness a person sees.
|
||||
|
||||
## Work
|
||||
|
||||
1. `tests/e2e/test_x11_capture.py` — keep the existing aspect-ratio guard
|
||||
(`…::test_the_real_desktop_is_captured_not_a_synthetic_one`, which
|
||||
distinguishes the 1280×800 Xvfb screen from the 1280×720 fake device) and add a
|
||||
size floor to the same probe: the viewer's `videoWidth` must be at least 90% of
|
||||
`XVFB_GEOMETRY`'s width (≥ 1152 px), not the ~1200/1280 the window-clamped
|
||||
capture produced before phase 02.
|
||||
* Reuse the existing `browser_env()` (strips `WAYLAND_DISPLAY` and
|
||||
`DBUS_SESSION_BUS_ADDRESS` — **do not** add `--use-fake-ui-for-media-stream`,
|
||||
it kills the X11 capturer with `NotReadableError`) and the `browsers` /
|
||||
`share_page` / `watch_page` fixtures. Two browser **processes**, not contexts.
|
||||
* Assert the size on the **viewer's** `<video>`, plus continuing frame
|
||||
movement (`distinct > 1`) in the same probe, so a large-but-frozen picture
|
||||
cannot pass.
|
||||
2. `tests/e2e/test_x11_capture.py` — assert the *published* metadata agrees: read
|
||||
`/api/streams` over `base_url` and require the published `width`/`height` to be
|
||||
within one rounding step of the sharer's `track.getSettings()` and of the
|
||||
viewer's decoded size (the roster must not lie about what is on the wire).
|
||||
3. `tests/e2e/test_live_mirror.py` — add the fake-device counterpart: the viewer
|
||||
decodes exactly `1280×720` from the deterministic device, proving the new
|
||||
`ideal` constraints changed nothing on the synthetic path.
|
||||
4. `tests/e2e/test_live_mirror.py` — an inbound-bitrate sanity check on a staged
|
||||
tile: the viewer's `inbound-rtp.bytesReceived` grows while a share is live, and
|
||||
with the default 25 Mbit/s cap the delivered figure is recorded in the test's
|
||||
docstring (an observation, not an equality assertion — the value is
|
||||
environment-dependent and an over-tight assertion here is a flake factory).
|
||||
5. `tests/e2e/test_x11_capture.py` module docstring — record the measured
|
||||
requested-vs-delivered numbers for this machine, replacing the vague
|
||||
"Chromium scales a monitor capture to the capturing window" note in the test's
|
||||
comment with what was actually observed after phase 02. If the measurement
|
||||
contradicts `.agents/PLAN.md` §5, say so in the commit message for the user;
|
||||
this phase does not edit `PLAN.md`.
|
||||
|
||||
ASSUMPTION: the floor is 90% of the desktop width rather than 100%. Compositors,
|
||||
fractional DPI and the header bar make an exact match environment-specific; 90%
|
||||
still fails loudly on a window-clamped capture (which lands near 60–75%).
|
||||
ASSUMPTION: the size floor is asserted in the `realdisplay` suite only. The
|
||||
fake-device suite cannot prove sharpness (it always returns 1280×720), so it
|
||||
asserts non-regression instead — per `AGENTS.md` §5, media is asserted where it is
|
||||
real.
|
||||
ASSUMPTION: `MIRROR_REALDISPLAY=1` remains opt-in and CI-optional; the sharpness
|
||||
gate is therefore a *release* gate run on the Linux reference host, and the
|
||||
fake-device assertions are the *every-commit* gate.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- `MIRROR_REALDISPLAY=1 uv run pytest -m realdisplay -k sharp` — the new size
|
||||
floor passes on the reference Linux host and **fails** if phase 02 task 01's
|
||||
constraints are reverted (verify this by temporarily removing the constraints;
|
||||
write the observed failure in the commit message).
|
||||
- `uv run pytest -m e2e` — fake-device size + inbound-bytes assertions green.
|
||||
- `uv run pytest` full suite green; coverage gate unchanged (~94%).
|
||||
- No new browser flags beyond what `AGENTS.md` §5 already blesses.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] A real-desktop test asserts viewer-decoded width ≥ 1152 px on a 1280×800
|
||||
Xvfb desktop, and fails when the capture constraints are removed.
|
||||
- [ ] `/api/streams` metadata agrees with the decoded size.
|
||||
- [ ] The fake-device suite still decodes exactly 1280×720 with moving frames.
|
||||
- [ ] Measured numbers recorded in the test docstring + commit message; any
|
||||
`PLAN.md` §5 discrepancy reported to the user rather than edited here.
|
||||
- [ ] Full gate: `uv run ruff check . && uv run ruff format --check . && uv run
|
||||
pyright && uv run pytest && MIRROR_REALDISPLAY=1 uv run pytest -m realdisplay`.
|
||||
@@ -0,0 +1,90 @@
|
||||
# Phase 03 — Audio that actually arrives
|
||||
|
||||
**Source:** `TODO.md` L5 — "Need audio share to work."
|
||||
**Story:** n/a
|
||||
**Context:** `.agents/PLAN.md` §2 A3/A4 (WebRTC mesh, **the sharer always offers and owns the track set**), §3 (one `RTCPeerConnection` per viewer), `SECURITY.md` §4 "Audio", `src/mirror/static/watch.js` (`ensurePeer`, `pc.ontrack`), `share.js` (`pickSurface`, `publish`), `src/mirror/registry.py` L69/197/222–228 and `src/mirror/protocol.py` L57/118/129 (the `audio` flag already crosses the wire)
|
||||
|
||||
## Objective
|
||||
|
||||
When a share carries audio, a viewer can hear it — on the wall, on the stage, and
|
||||
with the keyboard alone — and the UI says honestly whether audio exists at all.
|
||||
|
||||
What is already true, and must stay true: the `audio` boolean is published,
|
||||
relayed and rendered (`registry.py` keeps it on the `Stream`, `stream_updated`
|
||||
carries changes, `paintTile()` shows the 🔇 button only when audio exists). The
|
||||
**server needs no change in this phase.** What is broken is on the client:
|
||||
|
||||
1. **A second `ontrack` throws the first track away.** `watch.js`'s `pc.ontrack`
|
||||
does `tile.video.srcObject = (event.streams && event.streams[0]) || new
|
||||
MediaStream([event.track])`. When the sharer's tracks arrive without a shared
|
||||
stream id (which happens with `addTransceiver(track)` and is common in
|
||||
Firefox), the audio callback builds a *new* one-track `MediaStream` and
|
||||
overwrites the video one. The video keeps playing from the still-attached
|
||||
transceiver, so nothing looks wrong — there is simply never any sound.
|
||||
2. **There is no way to unmute the thing you are watching.** The mute button lives
|
||||
in `.tile .actions`, hover-revealed in the grid (phase 01 makes it reachable),
|
||||
and the staged tile's caption dims to `0.28`.
|
||||
3. **The sharer does not know whether it offered audio.** `pickSurface()` retries
|
||||
without audio on *any* error except `NotAllowedError`, and `publish()` reads
|
||||
`getAudioTracks().length` once — so a source whose audio track ends mid-share
|
||||
leaves every viewer with a dead 🔇 button.
|
||||
|
||||
Policy for this phase (user-confirmed): **only the staged mirror can be audible.**
|
||||
`/watch` mutes everything else when one is unmuted, and unstaging re-mutes.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `01_keyboard_kiosk_control` (todo) — `setMuted(tile, muted)` and the `m` key
|
||||
come from there; this phase extends them rather than inventing a second mute path.
|
||||
- `02_sharp_capture` (todo) — the per-leg `RTCRtpSender` work: the audio leg must
|
||||
not be caught up in the video sender's parameters, and 25 Mbit/s of video headroom
|
||||
is what leaves room for Opus.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_audio_leg_plumbing.md` — one accumulating `MediaStream` per tile, so the
|
||||
audio and video tracks coexist; assert an `m=audio` line really is negotiated.
|
||||
2. `02_staged_audio_playback.md` — the staged-only unmute policy, applied to
|
||||
mouse, keyboard and unstage, with the tile's audio state readable at a glance.
|
||||
3. `03_audio_source_reporting.md` — a precise audio fallback in `pickSurface()`,
|
||||
three honest states on `/mirror`, and republish when an audio track ends.
|
||||
4. `04_audio_e2e_and_docs.md` — a viewer-side assertion that audio crossed, and
|
||||
the `SECURITY.md`/README text that tells users what a browser will actually
|
||||
capture on Linux.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- **Assert on the viewer** (`AGENTS.md` §5): inbound `bytesReceived` for
|
||||
`inbound-rtp`/`kind === "audio"`, `totalAudioEnergy`/`audioLevel`, and the
|
||||
composed `video.srcObject.getAudioTracks().length`. The sharer's panel proves
|
||||
nothing crossed the wire.
|
||||
- The E2E Chromium already starts with
|
||||
`--use-fake-device-for-audio-capture` (`tests/e2e/conftest.py`), but a headless
|
||||
container has **no audio device**, so a real sound assertion belongs to the
|
||||
`realdisplay`-class run and the fake path asserts the SDP and the composed
|
||||
`MediaStream`. Do not fake the WebRTC layer to make a green test.
|
||||
- `tests/e2e/conftest.py` also passes
|
||||
`--autoplay-policy=no-user-gesture-required`: the suite therefore cannot prove
|
||||
the *browser's* autoplay rule, only mirror's own `muted` state. Keep it that way
|
||||
and assert application state, not browser policy — and never add that flag to a
|
||||
test whose point is muting.
|
||||
- The deliberate, user-initiated unmute promised by `SECURITY.md` §4 is preserved:
|
||||
audio never starts without a click or a keypress.
|
||||
- Coverage **>90%** on new/modified code; full gate per task, plus `-m e2e` on
|
||||
every task that touches media.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] With an audio-bearing source, the viewer's inbound audio
|
||||
`bytesReceived` grows and `video.srcObject.getAudioTracks().length === 1`
|
||||
while the video still decodes frames.
|
||||
- [ ] Only the staged mirror can produce sound; unmuting mutes the others and
|
||||
`unstage()` re-mutes. Reachable by mouse **and** by `m`.
|
||||
- [ ] `/mirror` distinguishes `shared` / `silent` / `not offered`, and a source
|
||||
losing its audio mid-share updates every viewer's tile.
|
||||
- [ ] Cancelling the picker still reports a cancellation, not a silent-share
|
||||
fallback.
|
||||
- [ ] `SECURITY.md` §4 "Audio" and README's platform notes describe what shipped,
|
||||
including the Linux/PipeWire reality.
|
||||
- [ ] Full gate: `uv run ruff check . && uv run ruff format --check . && uv run
|
||||
pyright && uv run pytest && MIRROR_REALDISPLAY=1 uv run pytest -m realdisplay`.
|
||||
@@ -0,0 +1,87 @@
|
||||
# Task 01 — Keep both tracks on one MediaStream
|
||||
|
||||
**Phase:** `03_audio_share` · **Source:** `TODO.md:5` — "Need audio share to work."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Fix the bug that makes audio impossible no matter what the sharer offers: the
|
||||
viewer replaces `srcObject` on every `ontrack`, so the second track to arrive
|
||||
erases the first. Then prove an audio section is actually negotiated.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/mirror/static/watch.js` — in `makeTile()`, add `stream: null` to the tile
|
||||
record (next to `pc`, `pending`, `attempts`), and in `ensurePeer()`'s
|
||||
`pc.ontrack` stop assigning a fresh `MediaStream`:
|
||||
|
||||
```js
|
||||
tile.stream = tile.stream || new MediaStream();
|
||||
tile.stream.addTrack(event.track);
|
||||
if (tile.video.srcObject !== tile.stream) tile.video.srcObject = tile.stream;
|
||||
```
|
||||
|
||||
Keep the existing `tile.video.play().catch(() => {})` and its comment. Using
|
||||
`event.streams[0]` when present is *not* kept: mixing "browser's stream" and
|
||||
"our stream" per track is exactly how one track ends up alone.
|
||||
2. `src/mirror/static/watch.js` — handle the track's life: on
|
||||
`event.track.addEventListener("ended", …)` remove it from `tile.stream` and
|
||||
`addTrack` again is unnecessary; the `<video>` follows the live `MediaStream`.
|
||||
Update `tile.audioBtn.hidden` from `tile.stream.getAudioTracks().length` so a
|
||||
source that stops offering audio loses its button.
|
||||
3. `src/mirror/static/watch.js` — reset `tile.stream = null` in `closePeer()` (a
|
||||
retry builds a fresh connection and must not reattach dead tracks) and in
|
||||
`dropTile()`.
|
||||
4. `src/mirror/static/watch.js` — **do not** add `addTransceiver("audio", …)` on
|
||||
the viewer. The viewer answers, and the sharer owns the track set (PLAN A4);
|
||||
answerer-side transceivers reorder m-lines and would fight the locked decision.
|
||||
Leave a comment saying so at `ensurePeer()` — this is the first thing a future
|
||||
reader will be tempted to try.
|
||||
5. `src/mirror/static/watch.js` — keep `muted` as the element's initial state
|
||||
(it is set in `watch.html`'s tile template, which
|
||||
`tests/test_app.py::…test_videos_are_muted_and_inline_so_autoplay_is_not_blocked`
|
||||
guards); this task only fixes *composition*, playback policy is task 02.
|
||||
6. No server change: `registry.py` / `protocol.py` already carry `audio`
|
||||
(`protocol.py` L57 on `publish`, L118/129 into the stream payload;
|
||||
`registry.py` L197/222–228 stores it and decides `stream_updated`). Add a
|
||||
regression test instead of code (below).
|
||||
|
||||
ASSUMPTION: one `MediaStream` per tile, built by us, rather than
|
||||
`event.streams[0]` — Chrome and Firefox disagree about whether `addTransceiver`
|
||||
tracks share a stream id, and the app must behave identically on both (the kiosk
|
||||
is Firefox, the sharers are usually Chrome).
|
||||
ASSUMPTION: a track that ends is removed from the composed stream rather than
|
||||
tearing down the peer connection — the video leg keeps working when a tab's audio
|
||||
goes away, which is the current (correct) behaviour for video-only sources.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Wire-level regression in `tests/test_app.py::TestSignalingSocket` (next to
|
||||
`test_capture_metadata_survives_the_wire`): a `publish` with `audio: true`
|
||||
reaches the viewer as `stream.stream.audio === true`, and a republish with
|
||||
`audio: false` produces a `stream_updated` that flips it — the policy half is
|
||||
already tested, this pins it for the audio work.
|
||||
- E2E in `tests/e2e/test_audio_share.py` (new file, `pytestmark =
|
||||
pytest.mark.e2e`, reuse the `device` fixture from `test_live_mirror.py`):
|
||||
* assert the negotiated SDP agrees about audio —
|
||||
`await pc.localDescription.sdp` on the viewer contains `m=audio` **iff** the
|
||||
sharer published `audio: true`;
|
||||
* assert the composed stream: `video.srcObject.getVideoTracks().length === 1`
|
||||
and (when audio was offered) `getAudioTracks().length === 1` on the
|
||||
**viewer** — the direct regression test for the overwrite bug;
|
||||
* assert the video still decoded distinct frames after the audio track was
|
||||
added (`frames_matching`), so a "fixed" audio path cannot break the picture.
|
||||
- If this browser/device combination offers no display audio at all, `pytest.skip`
|
||||
with a message naming the reason — never assert on an absent track.
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `grep -n "new MediaStream(\[event.track\])" src/mirror/static/watch.js`
|
||||
returns nothing.
|
||||
- [ ] The viewer's `<video>` holds one `MediaStream` containing both kinds while
|
||||
decoded frames keep advancing (E2E).
|
||||
- [ ] The SDP assertion passes in both directions (audio offered / not offered).
|
||||
- [ ] `uv run pytest -m e2e` and `uv run pytest tests/test_app.py` green.
|
||||
- [ ] `uv run ruff check . && uv run ruff format --check . && uv run pyright`
|
||||
clean.
|
||||
@@ -0,0 +1,77 @@
|
||||
# Task 02 — Only the mirror you are watching can make sound
|
||||
|
||||
**Phase:** `03_audio_share` · **Source:** `TODO.md:5` — "Need audio share to work."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Make the audio that now exists actually hearable, with the policy the user chose:
|
||||
**the staged mirror is the only one that can be audible.** One deliberate gesture
|
||||
turns it on — a click or the `m` key — which also keeps `SECURITY.md` §4's promise
|
||||
that sound is always a user-initiated action.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/mirror/static/watch.js` — extend the `setMuted(tile, muted)` helper
|
||||
created in phase 01:
|
||||
* `muted === false` → first `stage(tile.id, { fullscreen: true })` if this tile
|
||||
is not already staged (a wall tile may not become audible in the grid), then
|
||||
mute **every other** tile through the same helper, then unmute this one;
|
||||
* `muted === true` → mute only this tile; never unstages as a side effect.
|
||||
2. `src/mirror/static/watch.js` — `unstage()` sets the departing tile muted, so
|
||||
walking back to the wall is always silent (the kiosk in a shared room depends
|
||||
on this).
|
||||
3. `src/mirror/static/watch.js` — `stage()` never changes mute state by itself:
|
||||
joining a mirror silently is the safe default and matches the element's `muted`
|
||||
attribute in `watch.html`.
|
||||
4. `src/mirror/static/watch.js` — make the state readable without hovering:
|
||||
set `tile.el.dataset.audio = "on" | "off" | "none"` in `paintTile()` (from
|
||||
`stream.audio`) and in `setMuted()`, so CSS can show it and a screenshot from
|
||||
another machine (`docs/kiosk-silverblue/dim-probe.py`) shows what the wall is
|
||||
doing.
|
||||
5. `src/mirror/static/app.css` — a small always-visible speaker mark for a staged
|
||||
tile whose audio is on (`.tile[data-audio="on"]`), and a static dot on grid
|
||||
tiles that have audio available. Reveal by `opacity`/content, never by
|
||||
`display` (`tests/test_app.py::TestStylesheet`), and keep
|
||||
`.icon-btn`'s existing `[hidden]` behaviour intact.
|
||||
6. `src/mirror/static/watch.js` — the audio button keeps calling the same helper;
|
||||
`aria-pressed`, `title` and the 🔊/🔇 glyph all come from `setMuted()` so the
|
||||
button, the `m` key and the policy can never disagree.
|
||||
7. `src/mirror/static/watch.js` — when the staged tile's audio disappears (task 01
|
||||
ends the track), mute it and toast `"This mirror stopped sharing audio."`
|
||||
|
||||
ASSUMPTION: unmuting a grid tile stages it instead of refusing (rather than
|
||||
playing sound out of a small tile). The staged-only rule was chosen to keep a
|
||||
20-tile wall from becoming 20 sound sources; staging is also what a person who
|
||||
wants audio of a mirror is about to do anyway.
|
||||
ASSUMPTION: audio is never restored automatically after `unstage()` or after a
|
||||
reconnect — every audible state must trace back to a gesture (`SECURITY.md` §4).
|
||||
ASSUMPTION: `tile.video.volume` stays at 1; system volume is the operator's job on
|
||||
a kiosk with no mixer.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- E2E in `tests/e2e/test_audio_share.py`: stage mirror A, press `m`, assert
|
||||
`#watch-stage video.muted === false`; stage mirror B, press `m`, assert A's
|
||||
`<video>.muted === true` and B's is `false` — the exclusivity policy is the
|
||||
behaviour under test, and it is asserted on the two viewer elements.
|
||||
- E2E: `unstage()` (via `g` or the Exit button) leaves the tile muted, and a
|
||||
subsequent `stage()` does not resurrect sound on its own.
|
||||
- E2E: clicking a grid tile's 🔇 button stages it and unmutes it (the mouse path
|
||||
must obey the same rule as the keyboard).
|
||||
- E2E: `dataset.audio` reflects reality for a silent source (`"none"`), a staged
|
||||
silent-but-available source (`"off"`) and an audible one (`"on"`).
|
||||
- `uv run pytest tests/test_app.py` — element-id and stylesheet guards stay green
|
||||
with the new CSS and the `data-audio` attribute.
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] Exactly one mirror can ever be unmuted, enforced through `setMuted()` and
|
||||
verified with two live mirrors in a browser test.
|
||||
- [ ] `unstage()` always leaves the wall silent.
|
||||
- [ ] The audio state is legible without a pointer (`data-audio` + the CSS mark).
|
||||
- [ ] Nothing unmutes without a click or a keypress — asserted by staging and
|
||||
waiting, then checking `muted` is still `true`.
|
||||
- [ ] `uv run pytest -m e2e` green; `uv run ruff check . && uv run ruff format
|
||||
--check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,88 @@
|
||||
# Task 03 — Say honestly whether audio exists
|
||||
|
||||
**Phase:** `03_audio_share` · **Source:** `TODO.md:5` — "Need audio share to work."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The sharer page currently promises nothing and hides everything: `pickSurface()`
|
||||
retries without audio on *any* failure except `NotAllowedError`, and `publish()`
|
||||
samples `getAudioTracks().length` exactly once. Make the capture request precise,
|
||||
the panel truthful about which of the three states it is in, and the roster current
|
||||
when audio appears or disappears mid-share.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/mirror/static/share.js` — `pickSurface()`: narrow the fallback. Retry with
|
||||
`audio: false` only for the errors that genuinely mean "this source has no
|
||||
audio": `NotSupportedError`, `OverconstrainedError`, `NotFoundError`, `TypeError`
|
||||
(and a `DOMException` whose message mentions audio). **Rethrow
|
||||
`NotAllowedError`** — in Chrome that is what a user who unticked "Share tab
|
||||
audio" or pressed *Cancel* produces, and retrying would silently re-open the
|
||||
picker and share a screen the person meant not to share. Comment that reason at
|
||||
the `throw`.
|
||||
2. `src/mirror/static/share.js` — remember which attempt succeeded in a local
|
||||
variable (`audioRequested`, `audioAcceptedByBrowser`) so the panel can tell
|
||||
"the browser refused to offer audio" apart from "the source has none".
|
||||
3. `src/mirror/static/share.js` — `#stat-audio` (`ui.audio`) gets three states:
|
||||
`shared` (an audio track exists), `silent` (source offered none), `not offered`
|
||||
(the retry had to drop audio to capture at all). Keep the existing `—` for the
|
||||
idle state set by `renderIdle()`.
|
||||
4. `src/mirror/static/share.js` — track the audio track's life the way the video
|
||||
track's already is: on the audio track's `"ended"` listener call `publish()` so
|
||||
the `audio` flag in the roster flips and every viewer's tile updates (phase 03
|
||||
task 01 relies on this to hide a dead 🔇 button); on `mute`/`unmute` the
|
||||
existing `republish` already fires for the video — extend it to also refresh
|
||||
`#stat-audio`.
|
||||
5. `src/mirror/static/share.js` — `publish()` reads
|
||||
`state.stream.getAudioTracks().length > 0` at call time (it already does);
|
||||
confirm the value is recomputed on *every* publish and never cached in `state`,
|
||||
so there is exactly one source of truth for "is there audio".
|
||||
6. **No new wire field.** The three-way distinction stays in the sharer's panel;
|
||||
the roster keeps the single `audio: bool` it already has (`protocol.py` L57,
|
||||
`registry.py` L197). Adding a multi-word field would require a camelCase alias
|
||||
test (`AGENTS.md` §3) for a distinction the viewers do not need — viewers need
|
||||
"is there audio right now", which the boolean already answers via
|
||||
`stream_updated`.
|
||||
|
||||
ASSUMPTION: rethrowing `NotAllowedError` instead of falling back silently, even
|
||||
though it turns some "no audio" sources into a visible failure — a screen share
|
||||
the user did not consciously approve is worse than a share that did not start, and
|
||||
the toast tells them to pick again without audio.
|
||||
ASSUMPTION: `AudioContext`-based mixing (routing system audio through Web Audio to
|
||||
guarantee a track) is out of scope: it needs a user gesture to unlock, resamples
|
||||
the stream, and buys nothing on the platforms where Chrome already offers the
|
||||
track. Revisit only if a real deployment needs it.
|
||||
ASSUMPTION: no new environment variable gating audio (`MIRROR_DISABLE_AUDIO`); the
|
||||
browser's own picker is the consent mechanism and a second switch would only
|
||||
confuse it.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Wire test in `tests/test_app.py::TestSignalingSocket`: publish `audio: true` →
|
||||
`unpublish`/republish `audio: false` produces `stream_updated` with
|
||||
`audio: false` for every subscribed viewer, and the empty-wall case is unaffected.
|
||||
- Guard in `tests/test_app.py::TestClientScripts` (regex over `share.js`, the
|
||||
existing style): `pickSurface` must rethrow `NotAllowedError` — assert the
|
||||
identifier appears in the catch block and that the audio-off retry is not
|
||||
unconditional (no bare `return await navigator.mediaDevices.getDisplayMedia`
|
||||
inside a catch-all).
|
||||
- E2E in `tests/e2e/test_audio_share.py`: with the fake device, assert `#stat-audio`
|
||||
is one of the three documented strings and that the tile's 🔇 button visibility
|
||||
agrees with `#stat-audio === "shared"` — the panel and the wall cannot disagree.
|
||||
- E2E: with a source that offers no audio, `/watch` shows no audio button at all
|
||||
(`paintTile` hides it) rather than a dead one.
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `pickSurface()` retries without audio only for the named error names, and a
|
||||
cancelled picker still surfaces "Sharing was cancelled or blocked."
|
||||
- [ ] `#stat-audio` shows `shared` / `silent` / `not offered` correctly for all
|
||||
three real cases on the reference Linux host.
|
||||
- [ ] Ending the audio source mid-share updates `#stat-audio` and every viewer's
|
||||
tile within one `stream_updated` (E2E).
|
||||
- [ ] No new field on the wire; `uv run pytest tests/test_protocol.py
|
||||
tests/test_registry.py tests/test_app.py` green unchanged.
|
||||
- [ ] `uv run ruff check . && uv run ruff format --check . && uv run pyright`
|
||||
clean.
|
||||
@@ -0,0 +1,88 @@
|
||||
# Task 04 — Prove the sound crossed, and tell users what a browser will capture
|
||||
|
||||
**Phase:** `03_audio_share` · **Source:** `TODO.md:5` — "Need audio share to work."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Two deliverables that belong together: a viewer-side assertion that audio really
|
||||
travelled over WebRTC, and documentation that tells a Linux user what the browser
|
||||
will and will not hand mirror — the part that made "audio doesn't work" feel
|
||||
undiagnosable.
|
||||
|
||||
## Work
|
||||
|
||||
1. `tests/e2e/test_audio_share.py` — the hard assertion, on the **viewer**:
|
||||
* after the sharer publishes with audio and the viewer unmutes the staged tile,
|
||||
poll `pc.getStats()` and require an `inbound-rtp` entry with
|
||||
`kind === "audio"` whose `bytesReceived` strictly increases across two samples;
|
||||
* where the browser reports it, also require `totalAudioEnergy > 0` or
|
||||
`audioLevel > 0` after the unmute (a fake device produces a tone, so energy
|
||||
exists when the track exists);
|
||||
* keep the existing content assertion (`frames_matching`) in the same test:
|
||||
audio arriving while the picture froze would be a different bug, and this
|
||||
phase must not create it.
|
||||
2. `tests/e2e/test_audio_share.py` — honest skipping, loud reporting: if the
|
||||
browser offers no display-audio track in this environment, `pytest.skip()` with
|
||||
a message naming the platform and the flags used. Never assert on a track that
|
||||
is not there, never `xfail` a real failure into green.
|
||||
3. `tests/e2e/test_x11_capture.py` — the real-desktop counterpart in the
|
||||
`realdisplay` suite (private Xvfb, headed Chromium, `browser_env()` stripping
|
||||
`WAYLAND_DISPLAY`/`DBUS_SESSION_BUS_ADDRESS`, **no**
|
||||
`--use-fake-ui-for-media-stream`): if the host exposes an audio source, assert
|
||||
the inbound audio bytes there too; otherwise record the skip reason in the
|
||||
module docstring so the gap is written down rather than imagined away.
|
||||
4. `tests/e2e/conftest.py` — do **not** add `--autoplay-policy=no-user-gesture-required`
|
||||
to any audio-muting test; it is already in `CHROMIUM_ARGS`, which is why those
|
||||
tests assert mirror's own `muted` state and never claim to prove the browser's
|
||||
autoplay rule. Say that in a comment next to the existing flag so nobody
|
||||
"fixes" it later.
|
||||
5. `SECURITY.md` §4 "Audio" — rewrite for what shipped: the picker offers audio and
|
||||
the app accepts it when the source has any; only the **staged** mirror can be
|
||||
audible; unmuting is always a click or a keypress; unstaging re-mutes; a mirror
|
||||
that stops offering audio says so instead of keeping a dead control. Keep the
|
||||
one-line statement that audio is user-initiated by design.
|
||||
6. `SECURITY.md` §4 — add the platform reality, stated plainly: Chrome on Linux
|
||||
captures **tab** audio reliably and **whole-screen/system** audio only where the
|
||||
platform (PipeWire/PulseAudio, and the picker's "share audio" tickbox) offers
|
||||
it; Firefox's desktop capture does not offer system audio at all on Linux.
|
||||
mirror neither works around nor fakes it — it reports `silent` / `not offered`
|
||||
(`03_audio_source_reporting.md`).
|
||||
7. `README.md` — "Platform notes" (~L73): a short *Audio* entry pointing at the
|
||||
`SECURITY.md` section, plus the operator-facing line "press `m` on the wall to
|
||||
hear the staged mirror".
|
||||
8. `docs/kiosk-silverblue.md` — one line for the wall box: the kiosk is silent
|
||||
until someone presses `m`, and audio depends on what that browser is given.
|
||||
|
||||
ASSUMPTION: audio assertions are environment-gated (`skip`, with a reason) in the
|
||||
fake-device and `realdisplay` suites rather than faked with a stubbed
|
||||
`RTCPeerConnection` — a stubbed peer connection proves the wiring and nothing else,
|
||||
and `AGENTS.md` §5 forbids "unit tests pretending to be WebRTC".
|
||||
ASSUMPTION: documentation names Firefox's Linux limitation instead of trying to
|
||||
detect and paper over it in JS, because the kiosk fleet is Firefox and an operator
|
||||
reading `SECURITY.md` needs to know the wall will be silent by the browser's rule,
|
||||
not mirror's.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- `uv run pytest -m e2e` — the audio test either proves inbound audio bytes
|
||||
increased or skips with a named reason; the rest of the suite is unaffected.
|
||||
- `MIRROR_REALDISPLAY=1 uv run pytest -m realdisplay` — green, with the audio
|
||||
result (pass or documented skip) written into the commit message.
|
||||
- `uv run pytest tests/test_app.py` — the docs edits must not disturb the page /
|
||||
asset guards; no new asset, no CDN (`test_nothing_is_loaded_from_a_cdn`).
|
||||
- Coverage: **>90%** on new/modified code; this task's production-code delta is
|
||||
comments only, so the bar is met by the suites above.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] A viewer-side assertion proves inbound audio bytes increase once the staged
|
||||
mirror is unmuted — or the test skips with a message naming the exact reason.
|
||||
- [ ] The picture still decodes distinct frames in the same test.
|
||||
- [ ] `SECURITY.md` §4 "Audio", README's platform notes and
|
||||
`docs/kiosk-silverblue.md` describe the shipped behaviour, including the
|
||||
Linux system-audio limitation and the `m` key.
|
||||
- [ ] Full gate: `uv run ruff check . && uv run ruff format --check . && uv run
|
||||
pyright && uv run pytest && MIRROR_REALDISPLAY=1 uv run pytest -m realdisplay`.
|
||||
- [ ] No change in completed phases: keyboard navigation, staging, fan-out,
|
||||
teardown and the address gate all behave as before.
|
||||
Reference in New Issue
Block a user