brain-of-reese/tests/e2e/test_pictorial_pdf.py

"""Phase 140 task 04 E2E (Playwright, mock-only): the pictorial PDF —
sparse pages rendered + vision-described into the indexed content.

Run in isolation (DB must be up: ``podman compose up -d db``):

    uv run pytest tests/e2e/test_pictorial_pdf.py -v --no-cov

The dedicated suite for ``140_pictorial_pdf_pages`` (A16 — one Playwright
file per phase, run in isolation): it proves the owner's TODO L5 scenario
end to end on a SYNTHETIC fixture (the owner's real
``ikea_instructions.tar`` is pinned at the integration level, task 03 —
the E2E keeps the deterministic canned-reply premise): a 2-page PDF
built with PyMuPDF (the ``test_pdf_support.py`` recipe) —

* page 1 = a 700+ char prose paragraph (over the 100-char sparseness
  bar — NOT a pictorial candidate),
* page 2 = the 14-char ``Illustration 2`` marker + a drawn VECTOR figure
  (a rectangle + an arrow — the IKEA class: vector-drawn pictograms,
  ZERO embedded raster images, so the phase-122/137 embedded-image
  pipeline sees nothing; the thin text IS a candidate),

tarred as a single-root-member ``.tar`` (the owner's fixture's exact
format — the upload gate admits it) and arriving through the REAL admin
upload flow → the RAG page's Sync button:

* (a) the document appears on the Sources page and the document viewer
  renders the stored content: page 1's prose WITHOUT an illustration
  block, page 2 WITH ``[Page 2 illustration: <the mock's canned
  PICTORIAL_PAGE_ANSWER>]`` (the task-02 ``PICTORIAL_PAGE_MODE`` branch
  — deterministic; the stored content is asserted BYTE-EXACT against
  the fixture's own extraction in the seed — the candidates are computed
  from ``extract_pdf_page_texts`` + ``pictorial_pages`` IN the test, no
  hardcoded page list);
* (b) a chat question that ONLY the illustration answers ("how are the
  two parts joined on page two?" — the phrase lives NOWHERE in the text
  layer) grounds (the described page's embedding retrieves — the mock's
  deterministic token-overlap cosine clears the mock-calibrated 0.30
  gate; FTS corroborates) and, scripted through the house seeded-read
  convention (phase 119 A1 — a zero-read grounded turn chips NOTHING, so
  the PDF must be READ to land in ``done.sources``), the SSE completes
  without deflecting: the ``done`` event's ``sources`` carries the PDF
  and the answer echoes the read result VERBATIM — the
  ``[Page 2 illustration: …]`` block and its phrase reach the model and
  land in the answer;
* (c) the sync's ``detail`` JSON keeps its pre-phase keys EXACTLY (no
  new field — the ``pdfs_pictorial`` / ``pdf_pictorial_pages`` counters
  are log-only, PLAN §9, the phase-133 convention).

House pattern: one module app over the shared E2E Postgres (its own
port, module env — the conftest session app is never started in this
isolated run, so no port clash). The three tests SHARE the module seed
(one source, one document), so the shared registry + KB is truncated
ONCE, after the last test (a module-granularity autouse — a per-test
truncate would wipe the seed from under its siblings), with a
no-running-sync wait so a leaked run cannot race it.

App env: ``BOR_IMPORT_EXTENSIONS`` pinned WITH the ``pdf`` token (the
phase-126 convention — without the token the walk is blind to the file)
and ``BOR_PDF_PICTORIAL`` pinned EXPLICITLY to ``1``: the conftest pins
it OFF in ``os.environ`` for the pre-phase-140 story suites (their
synthetic PDFs carry sparse pages — a default-ON app would move their
byte-exact pins), and this suite is the documented opt-in (the
conftest's phase-140 comment). ``BOR_IMAGES`` is pinned ``false`` — the
embedded-image pipeline is out of scope (the fixture's figure is
vector-drawn; it yields no image doc either way). The byte-exact
content assertion in the seed is the toggle's premise witness: without
the opt-in, page 2 would lack its illustration block and the seed
fails loudly.

MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the described
block is the mock's deterministic ``PICTORIAL_PAGE_MODE`` canned reply
(``PICTORIAL_PAGE_ANSWER``), so the byte-exact content + echo
assertions are exact only against it.

Test → observable mapping (Playwright Mapping Rule):
1. ``test_sources_page_and_viewer_show_the_described_page`` — the
   Sources page reads 1 doc / 3 chunks (2 page-anchored content chunks +
   the summary chunk — no image doc), the drill-down table lists the PDF
   as a plain text row (LOCKED A4 — a text document, no thumbnail), and
   the same-page document viewer (the row's path link) renders the
   stored content under the METADATA title: the page-1 prose (NO
   ``[Page 1 illustration`` block), the page-2 marker + its
   ``[Page 2 illustration: <canned>]`` block, and the Summary panel
   (a non-image doc with a non-empty summary).
2. ``test_chat_question_grounds_and_echoes_the_illustration`` — the raw
   SSE over the scripted seeded-read question (httpx — the ``done``
   event is the wire lens the task pins; the UI flow's premise is the
   module seed): the stream settles ``done`` with ``deflected`` false
   and NO ``error`` frame, EXACTLY one ``tool`` frame (the scripted
   ``read`` on the PDF; the doc is far under the read cap — no
   truncation, so no ``tool_result`` frame), the ``done`` event's
   ``sources`` is EXACTLY the PDF (the citation surface — read docs
   only, phase 119 A1), and the joined deltas echo the read result
   VERBATIM — the ``Document <source>/<path>:`` header and the
   ``[Page 2 illustration: …]`` block with the canned phrase (the
   described page's content reaches the model and lands in the answer).
3. ``test_sync_detail_keeps_the_pre_phase_keys`` — the module run's
   terminal ``detail`` (``GET /api/sync/status`` — the same object the
   UI's "1 added" line renders from) carries the pre-phase key set
   EXACTLY: the ``pdfs_pictorial`` / ``pdf_pictorial_pages`` counters
   never ride the wire (the stats log-only convention), and the run's
   own file counters read 1 file / 1 added / 0 unchanged / 0 pruned /
   0 errors.
"""
from __future__ import annotations

import json
import os
import re
import subprocess
import sys
import tarfile
import time
from collections.abc import Iterator
from pathlib import Path
from typing import Any

import httpx
import pymupdf as fitz  # PyMuPDF — the synthetic-fixture builder (the house recipe)
import pytest
from playwright.sync_api import Browser, Page, expect
from sqlalchemy import select, text

from app.config import Settings
from app.db import SessionLocal
from app.models import Document
from app.rag.pdf import (
    PAGE_SEPARATOR,
    extract_pdf_page_texts,
    pictorial_pages,
)
from e2e.auth_helpers import login
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
from e2e.mock_llm import PICTORIAL_PAGE_ANSWER

REPO = Path(__file__).resolve().parents[2]
SOURCE_NAME = "e2e-pictorial"  # the archive stem (archive_source_name)
DOC_PATH = "instructions.pdf"
PDF_TITLE = "Assembly Instructions"  # the metadata title (LOCKED A2)

#: Page 1 — a 700+ char prose paragraph (over the 100-char sparseness
#: bar — NOT a candidate). Deliberately carries NO "two parts joined"
#: content: the phrase the question targets lives ONLY in the
#: description the vision pass appends to page 2 (the owner's L5
#: scenario — the illustration answers what the text layer cannot).
PAGE_ONE = (
    "This manual covers the complete flat-pack furniture frame, its "
    "shipped hardware kit, and the safety checks a first-time assembler "
    "should run before starting. The box holds the two side panels, the "
    "top board, the adjustable leg set, eight cam locks, fourteen wood "
    "screws, two bolts, and the wall-anchoring strap. Work on a flat "
    "surface with soft flooring, keep the hardware sorted by size, and "
    "read every step through before you begin so you know which panel "
    "is which. The frame stands roughly a metre tall once assembled, "
    "weights under twelve kilograms, and needs no further tools beyond "
    "the included Allen key. Keep the small parts away from children "
    "until the final step is done and the cam locks are fully seated."
)
assert len(PAGE_ONE) > 500, "page 1 must be a 500+ char paragraph (the task's recipe)"

#: Page 2 — the sparse page: the marker line is ALL the text layer
#: carries (14 stripped chars, under the 100-char bar — the candidate);
#: the figure is drawn vector (no raster image).
PAGE_TWO = "Illustration 2"

#: The stored illustration block (LOCKED A4) — the mock's canned
#: ``PICTORIAL_PAGE_MODE`` reply, deterministic.
ILLUSTRATION_BLOCK = f"[Page 2 illustration: {PICTORIAL_PAGE_ANSWER}]"

#: The grounded question (b): the house seeded-read call (phase 119 A1 —
#: a zero-read grounded turn chips NOTHING, so the PDF must be READ to
#: earn its ``done.sources`` citation) + the natural question, whose
#: tokens ("two parts joined page") overlap ONLY the illustrated page's
#: content (the mock's token-overlap cosine grounds; the text layer
#: alone would answer nothing).
QUESTION = (
    f"Read the suggested document: read {SOURCE_NAME}/{DOC_PATH} — "
    "how are the two parts joined on page two?"
)

#: The sync ``detail`` key set EXACTLY as pre-phase 140 (the
#: ``app/api/sync.py`` terminal detail — phase 126/133 shape). The
#: phase-140 counters are log-only and must NOT appear (c).
PRE_PHASE_DETAIL_KEYS = frozenset(
    {
        "files",
        "added",
        "updated",
        "unchanged",
        "pruned",
        "errors",
        "chunks",
        "summaries",
        "summary_errors",
        "dates_updated",
        "overview",
        "sources_version",
    }
)

#: Phase 79 (task 04, full inventory): the conftest session app owns its
#: port in a combined run — this module app binds its own port instead
#: (a same-port second uvicorn dies on bind and would drive the wrong
#: server). Env-overridable.
APP_PORT = int(os.environ.get("E2E_APP_PORT_PICTPDF", "8155"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"

UPLOAD_TIMEOUT_MS = 30_000
SYNC_TIMEOUT_MS = 60_000
SYNCED_LABEL = re.compile(r"^Synced \d{1,2}:\d{2}$")


# ---------------------------------------------------------------------------
# Fixture builder (the ``test_pdf_support.py`` PyMuPDF recipe, the
# owner's class: a dense prose page + a sparse vector-illustrated page)
# ---------------------------------------------------------------------------


def _build_pdf(path: Path) -> None:
    """ONE 2-page PDF (the task's recipe): page 1 = the 700+ char prose
    paragraph (``insert_textbox`` — word-wrapped, no mid-word breaks in
    the extraction), page 2 = the ``Illustration 2`` marker + a drawn
    VECTOR figure (a rectangle + a line arrow with a two-stroke head —
    the IKEA pictogram class: ``get_image_info()`` sees ZERO embedded
    raster images, the phase-122/137 pipeline is blind to it)."""
    doc = fitz.open()
    page1 = doc.new_page()
    rc = page1.insert_textbox(fitz.Rect(72, 72, 523, 772), PAGE_ONE, fontsize=11)
    assert rc >= 0, "the page-1 paragraph must fit its text box"
    page2 = doc.new_page()
    page2.insert_text((72, 72), PAGE_TWO)
    # The vector figure: a rectangle (a part) + an arrow (the join).
    page2.draw_rect(fitz.Rect(100, 120, 300, 220))
    page2.draw_line(fitz.Point(320, 170), fitz.Point(470, 170))
    page2.draw_line(fitz.Point(470, 170), fitz.Point(450, 158))
    page2.draw_line(fitz.Point(470, 170), fitz.Point(450, 182))
    doc.set_metadata({"title": PDF_TITLE})
    doc.save(path)
    doc.close()


@pytest.fixture(scope="module")
def pdf_path(tmp_path_factory: pytest.TempPathFactory) -> Path:
    """The fixture PDF (module-scoped, built once) — the local copy the
    byte-exact content assertion computes against (the app indexes the
    unpacked copy of the SAME bytes)."""
    root = tmp_path_factory.mktemp("bor_pict_pdf")
    pdf = root / DOC_PATH
    _build_pdf(pdf)
    # The premise pins (deterministic — the pass's candidate set and the
    # gate's own contract, computed from the fixture itself): page 1 is
    # prose (no candidate), page 2 is sparse (the candidate).
    page_texts = extract_pdf_page_texts(pdf)
    candidates = pictorial_pages(
        page_texts,
        min_chars=Settings.model_fields["pdf_pictorial_min_chars"].default,
    )
    assert candidates == [2], (
        f"the fixture's sparse-page premise broke — candidates={candidates}"
    )
    return pdf


@pytest.fixture(scope="module")
def tar_path(pdf_path: Path) -> Path:
    """The fixture archive: ONE PDF at the root (the ``.tar`` format —
    the owner's ``ikea_instructions.tar`` class; the upload gate admits
    it) — source ``e2e-pictorial``, document path ``instructions.pdf``."""
    archive = pdf_path.parent / f"{SOURCE_NAME}.tar"
    with tarfile.open(archive, "w") as tf:
        tf.add(pdf_path, arcname=DOC_PATH)
    return archive


# ---------------------------------------------------------------------------
# Module app (the conftest env pattern + the phase-140 knobs)
# ---------------------------------------------------------------------------


def _app_env(mock_llm: int, app_port: int, scratch: Path) -> dict[str, str]:
    """The conftest app env (leak guards included) with the phase-140
    knobs: ``BOR_IMPORT_EXTENSIONS`` pinned to a set carrying the
    ``pdf`` token (the phase-126 gate — an operator's local gitignored
    ``.env`` cannot leak the scope either way), ``BOR_PDF_PICTORIAL``
    pinned EXPLICITLY on (the conftest pins it off in ``os.environ``
    for the pre-phase suites — this suite is the documented opt-in;
    without it the whole premise is void), and ``BOR_IMAGES`` pinned
    off (out of scope — the vector figure yields no image doc anyway)."""
    env = dict(os.environ)
    env.pop("DEBUGPY", None)
    env["BOR_ENVIRONMENT"] = "e2e"
    env["BOR_STATIC_DIR"] = str(REPO / "frontend")
    env["BOR_LLM_BASE_URL"] = (
        "https://aipi.reeseapps.com/v1"
        if USE_REAL_LLM
        else f"http://127.0.0.1:{mock_llm}/v1"
    )
    # Mock-calibrated gate (conftest pattern) — the story's question
    # grounds on these values (see QUESTION).
    env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
    env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
    env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
    # Phase 67: instant retry waits + the code-default budget.
    env["BOR_LLM_RETRY_DELAY"] = "0"
    env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default)
    env.setdefault(
        "BOR_DATABASE_URL",
        "postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
    )
    env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
    env["BOR_SESSION_SECRET"] = SESSION_SECRET
    # Leak guards (conftest pattern).
    env["BOR_GIT_SOURCES"] = ""
    env["BOR_DOCS_REPO"] = ""
    env["BOR_SUGGESTIONS"] = json.dumps(
        Settings.model_fields["suggestions"].default
    )
    env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default
    env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default
    # Phase 140: the import scope WITH the ``pdf`` token + the toggle
    # under test (the EXPLICIT opt-in — see the module docstring) + the
    # suite-private homes.
    env["BOR_IMPORT_EXTENSIONS"] = "md,markdown,txt,pdf"
    env["BOR_PDF_PICTORIAL"] = "1"
    env["BOR_IMAGES"] = "false"
    env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
    env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
    env["BOR_IMAGE_DIR"] = str(scratch / "images")
    return env


@pytest.fixture(scope="module")
def app_server(
    mock_llm: int, tmp_path_factory: pytest.TempPathFactory
) -> Iterator[str]:
    """The app under test (all three tests): one module app with the
    phase-140 env (see :func:`_app_env`)."""
    scratch = tmp_path_factory.mktemp("bor_pict_scratch")
    proc = subprocess.Popen(
        [sys.executable, "-m", "uvicorn", "app.main:app",
         "--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
        cwd=REPO,
        env=_app_env(mock_llm, APP_PORT, scratch=scratch),
    )
    try:
        _wait_http(f"{APP_URL}/api/health")
        yield APP_URL
    finally:
        proc.terminate()
        try:
            proc.wait(timeout=10)
        except subprocess.TimeoutExpired:
            proc.kill()


@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
    return app_server


# ---------------------------------------------------------------------------
# DB + UI helpers
# ---------------------------------------------------------------------------


def _truncate_all() -> None:
    """Fresh KB + registry (the E2E isolation pattern): the suites share
    one Postgres, and a leftover row would corrupt the counts.
    ``sources_meta`` (the KB generation counter) resets with the KB —
    the sync re-creates the row."""
    with SessionLocal() as db:
        db.execute(
            text(
                "TRUNCATE chunks, documents, query_log, steering_notes, "
                "kb_overview, git_sources, folder_summaries, sources_meta"
            )
        )
        db.commit()  # without the commit the TRUNCATE rolls back (the house pattern)


def _health_db(app_url: str) -> bool:
    try:
        return httpx.get(f"{app_url}/api/health", timeout=5).json()["db"] == "up"
    except Exception:  # noqa: BLE001 — unreachable is the skip case
        return False


def _wait_sync_settled(base_url: str, timeout_s: float = 60.0) -> None:
    """No background sync may leak across tests (the run state lives in
    the app's memory, and a still-running run would keep importing into
    the NEXT test's truncated KB): wait for the app's sync status to be
    non-running BEFORE the truncate (the phase-96 pattern)."""
    with httpx.Client(base_url=base_url, timeout=5.0) as client:
        r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
        assert r.status_code == 204, r.text
        deadline = time.monotonic() + timeout_s
        while time.monotonic() < deadline:
            r = client.get("/api/sync/status")
            assert r.status_code == 200, r.text
            if r.json()["state"] != "running":
                return
            time.sleep(0.2)
    raise AssertionError(f"a sync was still running at a test boundary: {base_url}")


@pytest.fixture(scope="module", autouse=True)
def _clean_kb(app_url: str) -> Iterator[None]:
    """Module-end KB cleanup (the phase-96 pattern, module granularity):
    the three tests SHARE the module seed's single source + document,
    so there is NO per-test truncate (it would wipe the seed from under
    the next test — the per-test pattern belongs to the suites where
    every test imports its own source); instead the shared KB is
    flushed ONCE, after the last test — no rows leak into the next
    suite's run. The pre-wait keeps a leaked sync (the module app is
    still up here) from racing the truncate."""
    yield
    _wait_sync_settled(app_url)
    _truncate_all()


def _upload_tar(page: Page, app_url: str, archive: Path) -> None:
    """The real admin upload flow: form login → the git-sources page →
    pick the archive → submit → the terminal result line (phase 90:
    "Uploaded <source> — press Sync sources to import it.")."""
    login(page, app_url, next="/git-sources.html")
    expect(page).to_have_url(app_url + "/git-sources.html", timeout=30_000)
    expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
    page.set_input_files("#archive-upload-file", str(archive))
    page.click("#archive-upload-btn")
    result = page.locator("#archive-upload-result")
    expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
    expect(result).to_have_text(
        f"Uploaded {SOURCE_NAME} — press Sync sources to import it.",
        timeout=UPLOAD_TIMEOUT_MS,
    )


def _sync_via_ui(page: Page, app_url: str, expected_result: str) -> None:
    """The RAG page's "Sync sources" button (the scan): click →
    "Syncing…" → the terminal "Synced HH:MM" label + the fresh counts
    in #sync-result (the never-stale contract)."""
    page.goto(app_url + "/sources.html")
    btn = page.locator("#sync-btn")
    expect(btn).to_be_visible(timeout=30_000)
    expect(page.locator("#sync-label")).to_have_text(
        re.compile(r"^Sync sources$|^Synced \d{1,2}:\d{2}$")
    )
    btn.click()
    expect(btn).to_be_disabled()
    expect(page.locator("#sync-label")).to_have_text("Syncing…")
    expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
    expect(btn).to_be_enabled()
    expect(page.locator("#sync-result")).to_have_text(expected_result)


def _drill_to_source(page: Page, name: str) -> None:
    """Top level → the source (the file table then holds its direct
    files — ours is at the source root)."""
    page.locator("#folders-tbody .folder-link").first.wait_for(state="visible")
    page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
    expect(page.locator("#docs-tbody tr").first).to_be_visible(timeout=30_000)


def _chat_frames(app_url: str, message: str) -> list[dict[str, Any]]:
    """One ``POST /api/chat`` turn as RAW SSE frames (the wire lens —
    the house pattern from ``test_honest_deflection.py``): the signed
    admin client streams the turn and parses the ``data: <json>``
    frames (PLAN §4: ``data: <json>\\n\\n``)."""
    with httpx.Client(base_url=app_url, timeout=120.0) as client:
        r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
        assert r.status_code == 204, r.text
        frames: list[dict[str, Any]] = []
        with client.stream(
            "POST", "/api/chat", json={"message": message}, timeout=120.0
        ) as r:
            assert r.status_code == 200, r.read().decode()
            buf = ""
            for part in r.iter_text():
                buf += part
                while "\n\n" in buf:
                    frame, buf = buf.split("\n\n", 1)
                    if frame.strip().startswith("data:"):
                        frames.append(
                            json.loads(frame.strip().removeprefix("data:").strip())
                        )
        assert buf.strip() == ""  # the stream ends cleanly on a frame boundary
    return frames


# ---------------------------------------------------------------------------
# The module seed: upload + sync the fixture PDF through the real UI,
# once (the story E2E truncate/re-import-per-module pattern)
# ---------------------------------------------------------------------------


@pytest.fixture(scope="module")
def seeded(
    app_url: str,
    browser: Browser,
    tar_path: Path,
    pdf_path: Path,
) -> Iterator[dict[str, str]]:
    """Upload the single-PDF tar through the admin UI, then scan it with
    the RAG page's Sync button (the mock's vision model describes the
    sparse page in-process — the ``PICTORIAL_PAGE_MODE`` canned reply).
    The seed asserts the stored document BYTE-EXACT: the fixture's own
    extraction (page blocks, the page-2 candidate carrying the canned
    illustration block) — the candidates computed from
    ``extract_pdf_page_texts`` + ``pictorial_pages`` IN the fixture (no
    hardcoded page list). This is the module app's toggle premise
    witness too: without the ``BOR_PDF_PICTORIAL`` opt-in, page 2 would
    lack its block and this fails loudly. Yields the doc's identity."""
    if not _health_db(app_url):
        pytest.skip("Postgres not reachable — run `podman compose up -d db` first")

    _truncate_all()
    page = browser.new_page(viewport={"width": 1280, "height": 800})
    try:
        _upload_tar(page, app_url, tar_path)
        # The PDF alone (no embedded raster images — the vector figure
        # yields no image doc): one file, one added document.
        _sync_via_ui(page, app_url, expected_result="1 added")
        with SessionLocal() as db:
            doc = db.scalar(
                select(Document).where(
                    Document.source == SOURCE_NAME, Document.path == DOC_PATH
                )
            )
            assert doc is not None, "the uploaded PDF must become a document"
            # LOCKED A2: one document per PDF, the metadata title; a TEXT
            # document (no image doc, no image path).
            assert doc.title == PDF_TITLE
            assert doc.is_image is False and doc.image_path is None
            # The candidates (computed from the fixture itself — the
            # deterministic gate, LOCKED A1) + the byte-exact stored
            # content: the extraction's page blocks, the described
            # candidate carrying ``[Page N illustration: <canned>]``
            # (LOCKED A4).
            page_texts = extract_pdf_page_texts(pdf_path)
            candidates = pictorial_pages(
                page_texts,
                min_chars=Settings.model_fields["pdf_pictorial_min_chars"].default,
            )
            assert candidates == [2]
            parts = []
            for p, t in page_texts.items():
                block = t
                if p in candidates:
                    block += f"\n\n[Page {p} illustration: {PICTORIAL_PAGE_ANSWER}]"
                parts.append(f"{PAGE_SEPARATOR.format(page=p)}\n{block}")
            expected = "\n\n".join(parts)
            assert doc.content == expected, (
                "the stored content is not the extraction with the "
                "described sparse page — the pictorial pass misfired"
            )
            # The non-markdown doc got its lite summary (the mock's
            # SUMMARY_MODE digest) — the summary chunk + the viewer's
            # Summary panel exist.
            assert doc.summary
    finally:
        page.close()
    yield {"title": PDF_TITLE}


# ---------------------------------------------------------------------------
# 1. Sources page + document viewer (a): the described page renders
# ---------------------------------------------------------------------------


def test_sources_page_and_viewer_show_the_described_page(
    page: Page, app_url: str, seeded: dict[str, str]
) -> None:
    """The KB total is 1 doc / 3 chunks (the PDF: 2 page-anchored
    content chunks + the phase-30 summary chunk — no image doc; the
    vector figure is invisible to the embedded-image pipeline). The
    drill-down table lists the PDF as a plain text row (LOCKED A4), and
    the same-page document viewer (the row's path link, phase 26)
    renders the stored content under the METADATA title: page 1's prose
    WITHOUT an illustration block (not a candidate) and page 2 WITH
    ``[Page 2 illustration: <canned>]`` (the described sparse page —
    LOCKED A4's user surface), plus the Summary panel (a non-image doc
    with a non-empty summary)."""
    login(page, app_url)  # lands on /sources.html
    expect(page.locator("#stat-docs")).to_have_text("1")
    expect(page.locator("#stat-chunks")).to_have_text("3")
    expect(page.locator("#sources-empty")).to_be_hidden()

    _drill_to_source(page, SOURCE_NAME)
    expect(page.locator("#docs-tbody tr")).to_have_count(1)
    row = page.locator("#docs-tbody tr").first
    # A plain text row — no thumbnail box (it is a text document, A4).
    expect(row.locator(".kb-doc-thumb")).to_have_count(0)

    page.locator(f'#docs-tbody a.doc-link:text-is("{DOC_PATH}")').click()
    modal = page.locator("#doc-modal")
    expect(modal).to_be_visible(timeout=30_000)
    expect(page.locator("#doc-modal-title")).to_have_text(PDF_TITLE)

    raw = modal.locator("pre.doc-raw")
    # Page 1 — the prose WITHOUT any illustration block. The viewer
    # renders the content verbatim (``insert_textbox`` wraps mid-
    # sentence, so compare whitespace-normalized).
    expect(raw).to_contain_text("---PAGE-1---")
    rendered = " ".join(raw.inner_text().split())
    assert " ".join(PAGE_ONE.split()) in rendered, (
        "the page-1 prose is not what the viewer renders"
    )
    assert "[Page 1 illustration" not in rendered, (
        "page 1 (over the sparseness bar) must carry NO illustration block"
    )
    # Page 2 — the sparse page WITH the described block (a) — the mock's
    # canned reply, deterministic.
    expect(raw).to_contain_text("---PAGE-2---")
    expect(raw).to_contain_text(PAGE_TWO)
    expect(raw).to_contain_text(ILLUSTRATION_BLOCK)
    # The Summary panel renders (a non-image doc with a non-empty
    # summary); no image block (the PDF is not an image document).
    expect(modal.locator(".doc-summary")).to_have_count(1)
    expect(modal.locator(".doc-image-img")).to_have_count(0)


# ---------------------------------------------------------------------------
# 2. Chat (b): the question only the illustration answers grounds, and
#    the answer echoes the described phrase
# ---------------------------------------------------------------------------


def test_chat_question_grounds_and_echoes_the_illustration(
    app_url: str, seeded: dict[str, str]
) -> None:
    """The raw SSE over the scripted seeded-read question (phase 119 A1
    — a zero-read grounded turn chips NOTHING, so the PDF must be READ
    to earn its ``done.sources`` citation): the turn GROUNDS (no
    deflection — the described page's embedding retrieves: the mock's
    deterministic token-overlap cosine clears the mock-calibrated 0.30
    gate, FTS corroborates), the mock's scripted ``read`` runs (EXACTLY
    one ``tool`` frame — the doc is far under the read cap, so no
    truncation and no ``tool_result`` frame), and the answer ECHOES the
    read result verbatim — the ``[Page 2 illustration: …]`` block and
    its phrase reach the model and land in the rendered answer (the
    described page's content is what a chat turn can ground on — the
    owner's L5 rationale end to end)."""
    frames = _chat_frames(app_url, QUESTION)
    assert frames, "the SSE stream settled with no frames at all"
    assert not any(f.get("type") == "error" for f in frames), (
        f"the turn errored: {[f for f in frames if f.get('type') == 'error']}"
    )

    # The scripted read — exactly one tool call, on the PDF.
    tools = [f for f in frames if f.get("type") == "tool"]
    assert len(tools) == 1, f"expected exactly one tool frame: {tools}"
    assert tools[0]["name"] == "read"
    assert tools[0]["argument"] == f"{SOURCE_NAME}/{DOC_PATH}"
    # The doc is far under the read cap — no truncation frame (the echo
    # below therefore carries the FULL content, the block included).
    assert not any(f.get("type") == "tool_result" for f in frames)

    # The terminal frame: ``done``, NOT deflected.
    done = [f for f in frames if f.get("type") == "done"]
    assert len(done) == 1 and frames[-1] is done[0], (
        f"the stream must settle with exactly one terminal done frame: "
        f"{[f.get('type') for f in frames]}"
    )
    assert done[0]["deflected"] is False, (
        "the question grounded on the described page — a deflection means "
        "the illustrated content did not retrieve"
    )
    # The ``done`` event's ``sources`` — the citation surface (read docs
    # only, phase 119 A1) — carries EXACTLY the PDF.
    refs = done[0]["sources"]
    assert [(r["source"], r["path"]) for r in refs] == [(SOURCE_NAME, DOC_PATH)], (
        f"the done event's sources must be exactly the read PDF: {refs}"
    )
    assert refs[0]["title"] == PDF_TITLE

    # The answer (the joined deltas) echoes the read result VERBATIM —
    # the header line and the described block with the canned phrase.
    answer = "".join(f["text"] for f in frames if f.get("type") == "delta")
    assert "Here's what the read returned:" in answer
    assert f"Document {SOURCE_NAME}/{DOC_PATH}:" in answer
    assert ILLUSTRATION_BLOCK in answer, (
        "the answer does not echo the described illustration block — "
        "the vision pass's content never reached the model"
    )
    assert PICTORIAL_PAGE_ANSWER in answer  # the phrase itself (the task's pin)


# ---------------------------------------------------------------------------
# 3. Sync detail (c): the pre-phase key set is untouched (stats are
#    log-only)
# ---------------------------------------------------------------------------


def test_sync_detail_keeps_the_pre_phase_keys(
    app_url: str, seeded: dict[str, str]
) -> None:
    """The module run's terminal ``detail`` (``GET /api/sync/status`` —
    the same object the UI's "1 added" line renders from) keeps the
    pre-phase key set EXACTLY: the phase-140 counters
    (``pdfs_pictorial`` / ``pdf_pictorial_pages``) are log-only (PLAN
    §9, the phase-133 convention) and must NOT ride the wire, and the
    run's own file counters read 1 file / 1 added / 0 unchanged /
    0 pruned / 0 errors (the single PDF, a fresh KB)."""
    with httpx.Client(base_url=app_url, timeout=30.0) as client:
        r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
        assert r.status_code == 204, r.text
        r = client.get("/api/sync/status")
        assert r.status_code == 200, r.text
        body = r.json()
    assert body["state"] == "success", body
    detail = body["detail"]
    # The stats log-only convention — the pictorial counters never
    # reach the wire.
    assert "pdfs_pictorial" not in detail, (
        "the log-only pdfs_pictorial counter leaked into the sync detail"
    )
    assert "pdf_pictorial_pages" not in detail, (
        "the log-only pdf_pictorial_pages counter leaked into the sync detail"
    )
    # The shape itself — the pre-phase key set EXACTLY (no new field).
    assert set(detail) == set(PRE_PHASE_DETAIL_KEYS), (
        f"the sync detail shape changed — got {sorted(detail)}"
    )
    # The run's own file counters (a fresh KB: one file, one added).
    assert detail["files"] == 1, detail
    assert detail["added"] == 1, detail
    assert detail["unchanged"] == 0, detail
    assert detail["pruned"] == 0, detail
    assert detail["errors"] == 0, detail