Root cause (owner repro, verified in a real browser 2026-09-06): the five navbar views (Chat, RAG, Sources, Tuning, History) were separate HTML documents, so a navbar click was a REAL cross-document navigation — the chat page unloaded, the in-flight SSE fetch was aborted, and the phase-48 teardown (app/api/chat.py `finally`, "chat: turn cancelled") stopped the model. Observed: send question -> click RAG mid-stream -> click Chat -> the answer never finished: no `query_log` row, and on return a dangling question with no brain record (the pre-token pagehide partial persist skips because `acc` is empty). Phase-48 LOCKED-DECISION REFINEMENT (owner-confirmed 2026-09-06, flagged per AGENTS.md rule 3, not silently deviated): "real navigation cancels the fetch" now means LEAVING THE APP — tab close, external/other-document navigation, the Stop button. In-app navbar switches are client-side view switches and no longer cancel. Fix — Option A (SPA shell), chosen over B (Service Worker owns the stream) and C (server-side turn registry + resume): - frontend/index.html is the shell: ONE `<main id="main">` holds the five `<section class="view">` blocks; hidden views carry BOTH `hidden` and `inert` (WCAG — no focus/keyboard traversal). The shared header, the single `doc-modal-*` skeleton, and the `#app-version` footer each exist exactly once; the per-view copies from the four folded pages are dropped. - New frontend/assets/router.js (vanilla module — no framework, no bundler, No-CDN rule intact): lazy-imports a view module on FIRST show only (mount-once, hide-forever — the chat view's in-flight SSE reader persists across switches; that persistence IS the fix); intercepts same-shell navbar links with preventDefault + history.pushState (never a document load); handles popstate; single writer of `.nav-link` active state (is-active + aria-current), document.title, and the per-view meta description (values carried over from the old pages' heads, brand-resolved at write time). - Each folded page's JS becomes `export async function mount(root)` — root-scoped queries; `initSharedHeader()` dropped (the header boots once in the shell via the chat module; the admin flag comes from the same cached `fetchIsAdmin()` promise — zero extra requests). - app/main.py: a small list-driven route factory serves the shell for /tuning.html, /sources.html, /git-sources.html, /history.html — registered AFTER the API routers and BEFORE the static catch-all (routes-first). The phase-33 caching middleware applies no-cache + `?v=` rewriting unchanged; app/core/caching.py needed NO change (the view paths did not change — pinned by the integration tests). - The four old view .html files are DELETED (one source of truth); deep links to the old URLs keep working (the router picks the view from the pathname); `/?chat=<id>` is unaffected; the Containerfile bundles router.js (inlining the lazy view modules) and drops the folded page files. - app/schemas.py: HistoryTurn.text cap 4000 -> 32000 — the shell keeps long saved answers in the chat, and the old cap (stricter than the 24_000-char total history budget) 422-rejected any second turn in such a chat (found by the phase-42 E2E suite on the shell). Boundaries: login.html, shared.html, doc-edit.html, document.html REMAIN separate documents (flow pages, not navbar tabs); a mid-stream navigation to doc-edit/document.html still cancels per phase 48 (follow-up candidate, out of scope). The SSE API is unchanged. Real departures still cancel the turn — phase 48 intact (pinned by tests/e2e/test_stop_generation.py, unchanged, and by the new suite's real-departure control). Tests: - Phase-20 suite REWRITTEN to the new semantics (tests/e2e/test_sources_midstream_bug.py): a navbar switch no longer cancels — the stream survives the switch and the FULL answer settles; the pagehide partial persist REMAINS for real departures (the partial's exact shape — first streamed chunk prefix, no done metadata — is still pinned there). - NEW story suite tests/e2e/test_nav_switch_keeps_stream.py (mock LLM): the owner repro (send -> RAG mid-stream -> Chat: window sentinel survives = same document, FULL answer, exactly one brain turn in bor.chat.v1, exactly one settled query_log row, auto-saved row matches) + the same mid-stream switch against the other three views + the real-departure-still-cancels control + the no-switch baseline. - tests/unit/test_frontend_router.py: source-level pins of the router invariants (click interceptor targets ONLY same-shell view paths, pushState-only switches, mount-once guard, hidden+inert pair, single-writer active state/title); shell-route integration tests (each folded path serves the shell with no-cache + `?v=` body; a non-view path still 404s); the file-reading unit pins re-pointed at the shell (the four view files are gone — the shell is the source of truth). Verification (this commit): full suite green — 1565 unit+integration tests, app/ coverage 99% (>90% floor); ruff + pyright clean; the phase's E2E suites green in isolation (house protocol, AGENTS.md rule 9). Owner repro verified in a real browser against the real LLM (dev server :8010, headful Chromium): "tell me about everquest" -> RAG mid-stream -> Chat — the answer completed with one brain bubble and no error banner, `query_log` gained exactly one settled row (deflected=True: the dev KB holds no EverQuest docs — the settle, not the topic, is the proof), zero "chat: turn cancelled" lines for that turn; the control (real navigation to /shared.html mid-stream) still cancelled (no settled row, the cancel line logged, the partial persisted on return). Screenshots: .agents/screenshots/76_manual_*. Phase 76 (76_spa_nav_shell) complete — moved to .agents/phases/complete/.
316 lines
14 KiB
Python
316 lines
14 KiB
Python
"""Integration: Containerfile stage-1 (frontend) asset coverage pin (phase 23).
|
|
|
|
HERMETIC — no podman, no network, no database: this suite parses the
|
|
``Containerfile`` and ``frontend/`` as plain text and pins the image-build
|
|
coverage that the TODO L6 bug ("Fix Containerfile build not working")
|
|
demonstrated can silently rot in two independent ways:
|
|
|
|
* the stage-1 ``cp`` line only copies the pages that existed when it was
|
|
written (``document.html`` / ``login.html`` + their scripts +
|
|
``markdown.js`` were all missing from the image), and
|
|
* absolute module imports (``import … from "/assets/header.js"``) break
|
|
the esbuild bundle, so a "fixed" stage 1 can still ship a broken page.
|
|
|
|
The pins (each is one test, per the phase-23 task file):
|
|
|
|
1. every ``frontend/*.html`` page is copied into stage 1's ``/out``
|
|
— exactly (a new page without a ``cp`` entry fails; a ``cp`` of a
|
|
deleted page also fails);
|
|
2. every local ``assets/`` / ``/assets/`` ``src=``/``href=`` reference in
|
|
the pages is produced by a stage-1 line (``esbuild … --outfile`` or
|
|
``cp``) — the missing-``markdown.js``-style gap cannot reappear;
|
|
3. the set of ``type="module"`` page scripts the HTML references equals
|
|
the set of inputs esbuild ``--bundle``s in stage 1;
|
|
4. ``header.js`` is imported relatively by every page script and loaded
|
|
by NO direct ``<script>`` tag (single-evaluation design pin,
|
|
owner-confirmed 2026-08-24);
|
|
5. ``markdown.js`` is a classic script: stage-1 minify line WITHOUT
|
|
``--bundle``, and no top-level ``import``/``export`` in the source
|
|
(the source-level assumption that makes that build line safe);
|
|
6. the frontend stage pins a concrete ``esbuild@X.Y.Z`` (no floating
|
|
version — the exact pinned 0.25.5 is what the diagnosis reproduced
|
|
against).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import re
|
|
from pathlib import Path
|
|
|
|
REPO_ROOT = Path(__file__).resolve().parents[2]
|
|
FRONTEND = REPO_ROOT / "frontend"
|
|
ASSETS = FRONTEND / "assets"
|
|
CONTAINERFILE = REPO_ROOT / "Containerfile"
|
|
|
|
# A local asset reference: src="assets/…" or src/href="/assets/…"
|
|
# (data: URIs, "#main" anchors, and same-origin page links never match).
|
|
_ASSET_REF_RE = re.compile(r"""(?:src|href)\s*=\s*["']((?:/)?assets/[^"']+)["']""")
|
|
_SCRIPT_TAG_RE = re.compile(r"<script\b[^>]*>", re.IGNORECASE)
|
|
_MODULE_ATTR_RE = re.compile(r"""type\s*=\s*["']module["']""", re.IGNORECASE)
|
|
_SRC_ATTR_RE = re.compile(r"""src\s*=\s*["']([^"']+)["']""")
|
|
_TOP_LEVEL_MODULE_RE = re.compile(r"^\s*(?:import\b|export\b)", re.MULTILINE)
|
|
|
|
|
|
def _stage1_lines() -> list[str]:
|
|
"""The physical lines of the ``frontend`` build stage (FROM … up to,
|
|
but not including, the next FROM)."""
|
|
assert CONTAINERFILE.is_file(), f"missing Containerfile: {CONTAINERFILE}"
|
|
lines = CONTAINERFILE.read_text(encoding="utf-8").splitlines()
|
|
start = next(
|
|
i
|
|
for i, ln in enumerate(lines)
|
|
if re.match(r"^FROM\s+\S+\s+AS\s+frontend\b", ln, re.IGNORECASE)
|
|
)
|
|
end = len(lines)
|
|
for j in range(start + 1, len(lines)):
|
|
if re.match(r"^FROM\s", lines[j], re.IGNORECASE):
|
|
end = j
|
|
break
|
|
return lines[start:end]
|
|
|
|
|
|
def _is_esbuild_invocation(ln: str) -> bool:
|
|
"""True if the line actually RUNS esbuild (not just mentions it in a
|
|
comment or in the npm install line)."""
|
|
stripped = ln.lstrip()
|
|
return not stripped.startswith("#") and bool(re.search(r"(?:^|&&)\s*esbuild\b", ln))
|
|
|
|
|
|
def _esbuild_outputs(stage1: list[str]) -> dict[str, bool]:
|
|
"""basename of every asset a stage-1 esbuild line writes → was it
|
|
``--bundle``d? (Only lines with an ``--outfile`` count.)"""
|
|
outputs: dict[str, bool] = {}
|
|
for ln in stage1:
|
|
if not _is_esbuild_invocation(ln):
|
|
continue
|
|
m = re.search(r"--outfile=(\S+)", ln)
|
|
assert m, f"stage-1 esbuild line without --outfile: {ln.strip()}"
|
|
assert m.group(1).startswith("/out/assets/"), (
|
|
f"stage-1 asset must be written under /out/assets (the served "
|
|
f"/assets/ URL space), got {m.group(1)}"
|
|
)
|
|
outputs[m.group(1).rsplit("/", 1)[-1]] = "--bundle" in ln
|
|
return outputs
|
|
|
|
|
|
def _esbuild_bundle_inputs(stage1: list[str]) -> set[str]:
|
|
"""Basenames of the inputs to every stage-1 ``esbuild … --bundle``."""
|
|
inputs: set[str] = set()
|
|
for ln in stage1:
|
|
if _is_esbuild_invocation(ln) and "--bundle" in ln:
|
|
m = re.search(r"esbuild\s+(\S+)", ln)
|
|
assert m, f"stage-1 esbuild --bundle line without an input: {ln.strip()}"
|
|
inputs.add(m.group(1).rsplit("/", 1)[-1])
|
|
return inputs
|
|
|
|
|
|
def _cp_produced_basenames(stage1: list[str]) -> set[str]:
|
|
"""Basenames written into the image by stage-1 ``cp`` lines (the last
|
|
argument is the destination; a ``/``-suffixed one is a directory)."""
|
|
produced: set[str] = set()
|
|
for ln in stage1:
|
|
m = re.search(r"\bcp\s+(.+?)(?:\s*\\)?\s*$", ln)
|
|
if not m:
|
|
continue
|
|
args = m.group(1).split()
|
|
assert len(args) >= 2, f"malformed cp line in stage 1: {ln.strip()}"
|
|
dest = args[-1]
|
|
if dest.endswith("/"):
|
|
produced.update(a.rsplit("/", 1)[-1] for a in args[:-1])
|
|
else: # single-file rename: the image sees the destination name
|
|
produced.add(dest.rsplit("/", 1)[-1])
|
|
return produced
|
|
|
|
|
|
def _local_asset_basenames() -> set[str]:
|
|
"""Every local assets/ reference (basenames) across all frontend pages."""
|
|
refs: set[str] = set()
|
|
for html in sorted(FRONTEND.glob("*.html")):
|
|
text = html.read_text(encoding="utf-8")
|
|
refs.update(m.group(1).rsplit("/", 1)[-1] for m in _ASSET_REF_RE.finditer(text))
|
|
return refs
|
|
|
|
|
|
def _module_script_basenames() -> set[str]:
|
|
"""Basenames of the ``<script type="module" src=…>`` page scripts."""
|
|
names: set[str] = set()
|
|
for html in sorted(FRONTEND.glob("*.html")):
|
|
for tag in _SCRIPT_TAG_RE.findall(html.read_text(encoding="utf-8")):
|
|
if _MODULE_ATTR_RE.search(tag):
|
|
m = _SRC_ATTR_RE.search(tag)
|
|
assert m, f"{html.name}: module <script> without src: {tag}"
|
|
names.add(m.group(1).rsplit("/", 1)[-1])
|
|
return names
|
|
|
|
|
|
# ---------- 1. page coverage ----------
|
|
|
|
|
|
def test_every_html_page_is_copied_into_stage1() -> None:
|
|
"""Every ``frontend/*.html`` page is copied into stage 1's ``/out`` —
|
|
and the ``cp`` set is EXACTLY the pages on disk: a new page without a
|
|
matching cp entry (image 404s) and a cp of a deleted page (stale build
|
|
step) both fail here."""
|
|
stage1 = _stage1_lines()
|
|
on_disk = {p.name for p in FRONTEND.glob("*.html")}
|
|
assert on_disk, "frontend/ contains no .html pages — test is blind"
|
|
|
|
copied: set[str] = set()
|
|
for ln in stage1:
|
|
if ".html" not in ln:
|
|
continue
|
|
m = re.search(r"\bcp\s+(.+?)(?:\s*\\)?\s*$", ln)
|
|
if not m:
|
|
continue
|
|
args = m.group(1).split()
|
|
dest = args[-1]
|
|
assert dest.rstrip("/") == "/out", f"pages must be copied into /out, got {dest}"
|
|
copied.update(a.rsplit("/", 1)[-1] for a in args[:-1] if a.endswith(".html"))
|
|
|
|
missing = on_disk - copied
|
|
extra = copied - on_disk
|
|
assert not missing, (
|
|
f"pages missing from stage 1's cp line (404 in the image): {sorted(missing)}"
|
|
)
|
|
assert not extra, (
|
|
f"stage 1 copies pages that no longer exist in frontend/: {sorted(extra)}"
|
|
)
|
|
|
|
|
|
# ---------- 2. asset-reference coverage ----------
|
|
|
|
|
|
def test_every_local_asset_reference_is_produced() -> None:
|
|
"""Every local assets/… src/href in the pages is produced by a stage-1
|
|
line — esbuild ``--outfile=/out/assets/<name>`` or a ``cp`` of it.
|
|
This is what catches a missing-markdown.js-style gap: a page that
|
|
references an asset stage 1 never builds is a 404 in the image."""
|
|
stage1 = _stage1_lines()
|
|
produced = set(_esbuild_outputs(stage1)) | _cp_produced_basenames(stage1)
|
|
refs = _local_asset_basenames()
|
|
assert refs, "no local assets/ references found in the pages — test is blind"
|
|
missing = refs - produced
|
|
assert not missing, (
|
|
f"local assets referenced by the pages but not produced by stage 1 "
|
|
f"(404 in the image): {sorted(missing)}"
|
|
)
|
|
|
|
|
|
# ---------- 3. page-module ⇄ bundle-input parity ----------
|
|
|
|
|
|
def test_page_module_scripts_are_bundled() -> None:
|
|
"""The set of ``type="module"`` page scripts the HTML references
|
|
(basenames) EQUALS the set of inputs esbuild ``--bundle``s in stage 1
|
|
(today: app.js, sources.js, document.js, login.js). A new page script
|
|
without a bundle entry would 404 in the image; a bundle input whose
|
|
page no longer references it is dead build weight."""
|
|
stage1 = _stage1_lines()
|
|
html_modules = _module_script_basenames()
|
|
bundled = _esbuild_bundle_inputs(stage1)
|
|
assert html_modules, "no module page scripts found in the pages — test is blind"
|
|
assert html_modules == bundled, (
|
|
f"page module scripts {sorted(html_modules)} != esbuild bundle inputs "
|
|
f"{sorted(bundled)} — stage 1 must bundle exactly the pages' modules"
|
|
)
|
|
|
|
|
|
# ---------- 4. single-evaluation design pin ----------
|
|
|
|
|
|
def test_header_module_is_imported_not_directly_loaded() -> None:
|
|
"""Owner-confirmed design (2026-08-24, A4-2): NO page loads header.js
|
|
with a direct ``<script>`` tag — in the image the bundled page script
|
|
already contains the header code, so a raw header.js tag would evaluate
|
|
the module TWICE (duplicate sign-out listener, double init). Every page
|
|
script imports it relatively instead (``from "./header.js"``) — a
|
|
hoisted import that guarantees evaluation order in dev AND in the
|
|
bundle, and the only form esbuild can resolve."""
|
|
htmls = sorted(FRONTEND.glob("*.html"))
|
|
assert htmls, "no frontend pages — test is blind"
|
|
for html in htmls:
|
|
text = html.read_text(encoding="utf-8")
|
|
for tag in _SCRIPT_TAG_RE.findall(text):
|
|
m = _SRC_ATTR_RE.search(tag)
|
|
assert m is None or m.group(1).rsplit("/", 1)[-1] != "header.js", (
|
|
f"{html.name}: direct header.js <script> tag — double-evaluation "
|
|
f"trap in the image: {tag}"
|
|
)
|
|
page_scripts = _module_script_basenames()
|
|
assert page_scripts, "no module page scripts found — test is blind"
|
|
for name in sorted(page_scripts):
|
|
js_path = ASSETS / name
|
|
assert js_path.is_file(), f"page script missing: {js_path}"
|
|
body = js_path.read_text(encoding="utf-8")
|
|
# Phase 76: the shell's router (router.js) is NOT a view module
|
|
# — the shell's shared header boots via the chat module (app.js,
|
|
# which imports header.js), and the view modules the router
|
|
# lazy-imports (tuning.js, …) import header.js themselves, so
|
|
# the single-evaluation contract holds without the router
|
|
# importing it (it owns the view switch, not the header).
|
|
if name == "router.js":
|
|
assert 'from "./header.js"' not in body, (
|
|
"router.js must not import the header module — the chat "
|
|
"module boots the shell's ONE header"
|
|
)
|
|
continue
|
|
assert re.search(r"""from\s+["']\./header\.js["']""", body), (
|
|
f"{name}: must import the shared header module relatively "
|
|
f'("from \\"./header.js\\"")'
|
|
)
|
|
assert '"/assets/header.js"' not in body, (
|
|
f"{name}: absolute header import breaks the esbuild stage-1 bundle"
|
|
)
|
|
|
|
|
|
# ---------- 5. markdown.js is a classic script ----------
|
|
|
|
|
|
def test_markdown_js_is_a_produced_classic_script() -> None:
|
|
"""markdown.js ships minified WITHOUT --bundle (it is a classic global
|
|
script — window.markdownRender — loaded by index.html and
|
|
document.html), and the source has no top-level import/export: that
|
|
source-level fact is exactly what makes the no-bundle build line safe.
|
|
Pinning both sides keeps the assumption honest."""
|
|
stage1 = _stage1_lines()
|
|
outputs = _esbuild_outputs(stage1)
|
|
assert "markdown.js" in outputs, (
|
|
"markdown.js has no stage-1 esbuild --outfile line — index.html and "
|
|
"document.html would 404 the renderer in the image"
|
|
)
|
|
assert outputs["markdown.js"] is False, (
|
|
"markdown.js must NOT be --bundled: it is a classic global script, and "
|
|
"bundling it would strip the window-level globals the pages rely on"
|
|
)
|
|
md_path = ASSETS / "markdown.js"
|
|
assert md_path.is_file(), f"missing {md_path}"
|
|
hits = _TOP_LEVEL_MODULE_RE.findall(md_path.read_text(encoding="utf-8"))
|
|
assert not hits, f"markdown.js has top-level module syntax: {hits!r}"
|
|
|
|
|
|
# ---------- 6. pinned esbuild ----------
|
|
|
|
|
|
def test_esbuild_stays_pinned() -> None:
|
|
"""The frontend stage installs esbuild at a CONCRETE X.Y.Z. The phase-23
|
|
diagnosis was reproduced against the exact pinned 0.25.5; a floating
|
|
version (latest, ^, ~, or no spec) would let an upstream release change
|
|
the bundle behavior with no diff to see."""
|
|
stage1 = _stage1_lines()
|
|
install_lines = [ln for ln in stage1 if re.search(r"\bnpm\s+(?:install|i|add)\b", ln)]
|
|
assert install_lines, "no npm install line in the frontend stage"
|
|
pinned = [
|
|
v for ln in install_lines for v in re.findall(r"\besbuild@(\d+\.\d+\.\d+)\b", ln)
|
|
]
|
|
assert pinned, (
|
|
"the frontend stage must pin esbuild@X.Y.Z — a floating esbuild is how "
|
|
"the bundle behavior rots silently"
|
|
)
|
|
for ln in install_lines:
|
|
for m in re.finditer(r"\besbuild(@\S*)?", ln):
|
|
spec = m.group(1) or ""
|
|
assert re.fullmatch(r"@\d+\.\d+\.\d+", spec), (
|
|
f"esbuild must be installed with a concrete pin, got "
|
|
f"{m.group(0)!r} in: {ln.strip()}"
|
|
)
|