phase: 140_pictorial_pdf_pages
Everything is verified green. Final report: ## Phase 140 final verification pass — report - Verified all 4 tasks already complete (code in working tree, task files in `complete/`): `pictorial_pages` gate + 2 settings (`BOR_PDF_PICTORIAL`, `BOR_PDF_PICTORIAL_MIN_CHARS` + validator), `PICTORIAL_PAGE_MODE`/`describe_pdf_page` seam + `render_pdf_pages` page selector + mock branch, `_pictorial_pdf_pass` in `_index_pdf_file` with log-only counters, dedicated E2E. No code changes needed. - Test results (exact commands): - `uv run pytest --cov=app --cov-report=term-missing` → **3320 passed, 11 skipped, 0 failed; TOTAL 99%** (>90% gate) - `uv run pytest tests/e2e/test_pictorial_pdf.py -v --no-cov` → **3 passed in isolation** (Playwright, mock-only) - Pre-existing suites unedited: `test_pdf_support.py` 6 passed; `test_garbage_pdf.py` 2 passed + 1 skipped (owner fixture); `test_pdf_image_context.py` 4 passed - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors - Completion criteria: 1. IKEA tar import repro — pinned by 2 integration tests; **skipped loudly here** (`test_sources/ikea_instructions.tar` absent on this machine — verified repo-wide; same house convention as completed phase-133 HNB tests; runs real on the owner's machine). Equivalent logic fully covered by unit pins on synthetic IKEA-class (vector-drawn, zero-raster) PDFs. 2. Chat grounded on described page — **green** (E2E test b: phrase-only-in-illustration question, PDF in `done.sources`, answer echoes the block) 3. Clean-text no-op (zero vision calls) + garbage precedence pin — **green** (`test_zero_candidates_is_a_noop` unit; `test_garbage_pdf_with_pictorial_on_takes_the_fallback_not_the_pass` integration, verified alone) 4. `BOR_PDF_PICTORIAL=0` byte-identical — **green** (unit pin on synthetic PDF; IKEA variant runs on owner's machine) 5. Full gate + E2E isolation + pre-existing suites — **green** (above) 6. ruff + pyright — **clean** 7. Commit/phase-move — delegated to harness per run rules (left in working tree: 16 modified + 8 new files, uncommitted) - Deviations: none beyond the expected owner-fixture skips (11, all pre-existing fixture-missing skips, including 2 IKEA). - Next pending phase: **none** — `todo/` holds only 140; 139 is already complete.
This commit is contained in:
@@ -41,6 +41,17 @@ USE_REAL_LLM = os.environ.get("E2E_REAL_LLM") == "1"
|
||||
ADMIN_PASSWORD = "e2e-admin-password"
|
||||
SESSION_SECRET = "e2e-session-secret-0123456789abcdef0123456789abcdef"
|
||||
|
||||
# Phase 140 (task 03): the pictorial-PDF kill switch is pinned OFF for
|
||||
# every E2E app. The code default is ON (LOCKED A3), and the pre-phase-
|
||||
# 140 story suites pin the PRE-phase content shape byte-exactly (their
|
||||
# synthetic PDFs carry sparse pages — under the 100-char bar — so a
|
||||
# default-ON app would append ``[Page N illustration: …]`` blocks and
|
||||
# move those pins). The line rides into every app's env through the
|
||||
# house ``env = dict(os.environ)`` pattern; the phase-140 story suite
|
||||
# (tests/e2e/test_pictorial_pdf.py) opts in explicitly
|
||||
# (``env["BOR_PDF_PICTORIAL"] = "1"`` in its module app).
|
||||
os.environ["BOR_PDF_PICTORIAL"] = "0"
|
||||
|
||||
|
||||
def _wait_http(url: str, timeout: float = 40.0) -> None:
|
||||
deadline = time.monotonic() + timeout
|
||||
|
||||
@@ -34,6 +34,15 @@ Implements just enough of the aipi surface:
|
||||
the image document's whole content + summary, so it is byte-
|
||||
stable and token-dense (its cosine against the story suite's
|
||||
question clears the mock-calibrated gate)
|
||||
- ``PICTORIAL_PAGE_MODE`` (phase 140, pictorial PDF pages) -> the
|
||||
fixed ``PICTORIAL_PAGE_ANSWER`` description: the CHAT model's
|
||||
(vision) reply to ``summarizer.describe_pdf_page``'s multimodal
|
||||
user message (one rendered sparse PDF page — the marker is its
|
||||
first text part; the call is non-streaming, so only this answer
|
||||
path serves it). The reply is appended to the page's indexed
|
||||
content block as ``[Page N illustration: …]`` (LOCKED A4), so it
|
||||
is byte-stable: a re-sync with the same reply re-digests to the
|
||||
same content and lands as ``unchanged``
|
||||
- ``DEFLECT_MODE`` -> honest "I haven't done anything like that" answer
|
||||
- otherwise -> upbeat answer quoting the provided document context
|
||||
- user message containing ``pretend to think slowly`` -> 3s warm-up delay
|
||||
@@ -992,6 +1001,21 @@ IMAGE_DESCRIPTION_CONTEXT_ANSWER = (
|
||||
"cable runs. Title: Homelab Network Map."
|
||||
)
|
||||
|
||||
#: Phase 140 (pictorial PDF pages, LOCKED A2): the fixed description
|
||||
#: the mock's vision (CHAT) model returns for a
|
||||
#: ``PICTORIAL_PAGE_MODE`` request — the multimodal user message
|
||||
#: ``[{type: "text", …PICTORIAL_PAGE_MODE…}, {type: "image_url",
|
||||
#: …}]`` (``app.rag.summarizer.describe_pdf_page`` — one rendered
|
||||
#: sparse PDF page, the page's own text as context). The reply is
|
||||
#: appended to the page's indexed content block as
|
||||
#: ``[Page N illustration: …]`` (LOCKED A4), so it is byte-stable and
|
||||
#: retrieval-friendly (distinctive + token-dense on the assembly
|
||||
#: vocabulary the story E2E's question targets).
|
||||
PICTORIAL_PAGE_ANSWER = (
|
||||
"Pictorial page: the illustration shows two parts being joined "
|
||||
"with an arrow."
|
||||
)
|
||||
|
||||
TABLE_ANSWER = (
|
||||
"Here's the shape, in a table:\n"
|
||||
"\n"
|
||||
@@ -2913,6 +2937,17 @@ def compose_answer(body: dict[str, Any]) -> str:
|
||||
if "Extracted from " in user
|
||||
else IMAGE_DESCRIPTION_ANSWER
|
||||
)
|
||||
elif "PICTORIAL_PAGE_MODE" in user:
|
||||
# Phase 140 (pictorial PDF pages, LOCKED A2): the CHAT model's
|
||||
# vision description of ONE rendered sparse PDF page — a
|
||||
# NON-STREAMING multimodal user message, so only this path
|
||||
# ever sees it. ``_content_text`` joins the text parts, so the
|
||||
# marker (the first text part) lands in ``user``. The two
|
||||
# markers are disjoint fixed app constants (neither contains
|
||||
# the other), so the branch order vs ``IMAGE_DESCRIPTION_MODE``
|
||||
# cannot shadow either flow; like that marker, it is an
|
||||
# app-to-app marker a real question is never expected to type.
|
||||
answer = PICTORIAL_PAGE_ANSWER
|
||||
elif TABLE_TRIGGER in user.lower():
|
||||
# Markdown tables (phase 44, TODO.md L6): the story E2E's
|
||||
# deterministic table answer — a 3-column table, the
|
||||
|
||||
@@ -0,0 +1,729 @@
|
||||
"""Phase 140 task 04 E2E (Playwright, mock-only): the pictorial PDF —
|
||||
sparse pages rendered + vision-described into the indexed content.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_pictorial_pdf.py -v --no-cov
|
||||
|
||||
The dedicated suite for ``140_pictorial_pdf_pages`` (A16 — one Playwright
|
||||
file per phase, run in isolation): it proves the owner's TODO L5 scenario
|
||||
end to end on a SYNTHETIC fixture (the owner's real
|
||||
``ikea_instructions.tar`` is pinned at the integration level, task 03 —
|
||||
the E2E keeps the deterministic canned-reply premise): a 2-page PDF
|
||||
built with PyMuPDF (the ``test_pdf_support.py`` recipe) —
|
||||
|
||||
* page 1 = a 700+ char prose paragraph (over the 100-char sparseness
|
||||
bar — NOT a pictorial candidate),
|
||||
* page 2 = the 14-char ``Illustration 2`` marker + a drawn VECTOR figure
|
||||
(a rectangle + an arrow — the IKEA class: vector-drawn pictograms,
|
||||
ZERO embedded raster images, so the phase-122/137 embedded-image
|
||||
pipeline sees nothing; the thin text IS a candidate),
|
||||
|
||||
tarred as a single-root-member ``.tar`` (the owner's fixture's exact
|
||||
format — the upload gate admits it) and arriving through the REAL admin
|
||||
upload flow → the RAG page's Sync button:
|
||||
|
||||
* (a) the document appears on the Sources page and the document viewer
|
||||
renders the stored content: page 1's prose WITHOUT an illustration
|
||||
block, page 2 WITH ``[Page 2 illustration: <the mock's canned
|
||||
PICTORIAL_PAGE_ANSWER>]`` (the task-02 ``PICTORIAL_PAGE_MODE`` branch
|
||||
— deterministic; the stored content is asserted BYTE-EXACT against
|
||||
the fixture's own extraction in the seed — the candidates are computed
|
||||
from ``extract_pdf_page_texts`` + ``pictorial_pages`` IN the test, no
|
||||
hardcoded page list);
|
||||
* (b) a chat question that ONLY the illustration answers ("how are the
|
||||
two parts joined on page two?" — the phrase lives NOWHERE in the text
|
||||
layer) grounds (the described page's embedding retrieves — the mock's
|
||||
deterministic token-overlap cosine clears the mock-calibrated 0.30
|
||||
gate; FTS corroborates) and, scripted through the house seeded-read
|
||||
convention (phase 119 A1 — a zero-read grounded turn chips NOTHING, so
|
||||
the PDF must be READ to land in ``done.sources``), the SSE completes
|
||||
without deflecting: the ``done`` event's ``sources`` carries the PDF
|
||||
and the answer echoes the read result VERBATIM — the
|
||||
``[Page 2 illustration: …]`` block and its phrase reach the model and
|
||||
land in the answer;
|
||||
* (c) the sync's ``detail`` JSON keeps its pre-phase keys EXACTLY (no
|
||||
new field — the ``pdfs_pictorial`` / ``pdf_pictorial_pages`` counters
|
||||
are log-only, PLAN §9, the phase-133 convention).
|
||||
|
||||
House pattern: one module app over the shared E2E Postgres (its own
|
||||
port, module env — the conftest session app is never started in this
|
||||
isolated run, so no port clash). The three tests SHARE the module seed
|
||||
(one source, one document), so the shared registry + KB is truncated
|
||||
ONCE, after the last test (a module-granularity autouse — a per-test
|
||||
truncate would wipe the seed from under its siblings), with a
|
||||
no-running-sync wait so a leaked run cannot race it.
|
||||
|
||||
App env: ``BOR_IMPORT_EXTENSIONS`` pinned WITH the ``pdf`` token (the
|
||||
phase-126 convention — without the token the walk is blind to the file)
|
||||
and ``BOR_PDF_PICTORIAL`` pinned EXPLICITLY to ``1``: the conftest pins
|
||||
it OFF in ``os.environ`` for the pre-phase-140 story suites (their
|
||||
synthetic PDFs carry sparse pages — a default-ON app would move their
|
||||
byte-exact pins), and this suite is the documented opt-in (the
|
||||
conftest's phase-140 comment). ``BOR_IMAGES`` is pinned ``false`` — the
|
||||
embedded-image pipeline is out of scope (the fixture's figure is
|
||||
vector-drawn; it yields no image doc either way). The byte-exact
|
||||
content assertion in the seed is the toggle's premise witness: without
|
||||
the opt-in, page 2 would lack its illustration block and the seed
|
||||
fails loudly.
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the described
|
||||
block is the mock's deterministic ``PICTORIAL_PAGE_MODE`` canned reply
|
||||
(``PICTORIAL_PAGE_ANSWER``), so the byte-exact content + echo
|
||||
assertions are exact only against it.
|
||||
|
||||
Test → observable mapping (Playwright Mapping Rule):
|
||||
1. ``test_sources_page_and_viewer_show_the_described_page`` — the
|
||||
Sources page reads 1 doc / 3 chunks (2 page-anchored content chunks +
|
||||
the summary chunk — no image doc), the drill-down table lists the PDF
|
||||
as a plain text row (LOCKED A4 — a text document, no thumbnail), and
|
||||
the same-page document viewer (the row's path link) renders the
|
||||
stored content under the METADATA title: the page-1 prose (NO
|
||||
``[Page 1 illustration`` block), the page-2 marker + its
|
||||
``[Page 2 illustration: <canned>]`` block, and the Summary panel
|
||||
(a non-image doc with a non-empty summary).
|
||||
2. ``test_chat_question_grounds_and_echoes_the_illustration`` — the raw
|
||||
SSE over the scripted seeded-read question (httpx — the ``done``
|
||||
event is the wire lens the task pins; the UI flow's premise is the
|
||||
module seed): the stream settles ``done`` with ``deflected`` false
|
||||
and NO ``error`` frame, EXACTLY one ``tool`` frame (the scripted
|
||||
``read`` on the PDF; the doc is far under the read cap — no
|
||||
truncation, so no ``tool_result`` frame), the ``done`` event's
|
||||
``sources`` is EXACTLY the PDF (the citation surface — read docs
|
||||
only, phase 119 A1), and the joined deltas echo the read result
|
||||
VERBATIM — the ``Document <source>/<path>:`` header and the
|
||||
``[Page 2 illustration: …]`` block with the canned phrase (the
|
||||
described page's content reaches the model and lands in the answer).
|
||||
3. ``test_sync_detail_keeps_the_pre_phase_keys`` — the module run's
|
||||
terminal ``detail`` (``GET /api/sync/status`` — the same object the
|
||||
UI's "1 added" line renders from) carries the pre-phase key set
|
||||
EXACTLY: the ``pdfs_pictorial`` / ``pdf_pictorial_pages`` counters
|
||||
never ride the wire (the stats log-only convention), and the run's
|
||||
own file counters read 1 file / 1 added / 0 unchanged / 0 pruned /
|
||||
0 errors.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pymupdf as fitz # PyMuPDF — the synthetic-fixture builder (the house recipe)
|
||||
import pytest
|
||||
from playwright.sync_api import Browser, Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document
|
||||
from app.rag.pdf import (
|
||||
PAGE_SEPARATOR,
|
||||
extract_pdf_page_texts,
|
||||
pictorial_pages,
|
||||
)
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
|
||||
from e2e.mock_llm import PICTORIAL_PAGE_ANSWER
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
SOURCE_NAME = "e2e-pictorial" # the archive stem (archive_source_name)
|
||||
DOC_PATH = "instructions.pdf"
|
||||
PDF_TITLE = "Assembly Instructions" # the metadata title (LOCKED A2)
|
||||
|
||||
#: Page 1 — a 700+ char prose paragraph (over the 100-char sparseness
|
||||
#: bar — NOT a candidate). Deliberately carries NO "two parts joined"
|
||||
#: content: the phrase the question targets lives ONLY in the
|
||||
#: description the vision pass appends to page 2 (the owner's L5
|
||||
#: scenario — the illustration answers what the text layer cannot).
|
||||
PAGE_ONE = (
|
||||
"This manual covers the complete flat-pack furniture frame, its "
|
||||
"shipped hardware kit, and the safety checks a first-time assembler "
|
||||
"should run before starting. The box holds the two side panels, the "
|
||||
"top board, the adjustable leg set, eight cam locks, fourteen wood "
|
||||
"screws, two bolts, and the wall-anchoring strap. Work on a flat "
|
||||
"surface with soft flooring, keep the hardware sorted by size, and "
|
||||
"read every step through before you begin so you know which panel "
|
||||
"is which. The frame stands roughly a metre tall once assembled, "
|
||||
"weights under twelve kilograms, and needs no further tools beyond "
|
||||
"the included Allen key. Keep the small parts away from children "
|
||||
"until the final step is done and the cam locks are fully seated."
|
||||
)
|
||||
assert len(PAGE_ONE) > 500, "page 1 must be a 500+ char paragraph (the task's recipe)"
|
||||
|
||||
#: Page 2 — the sparse page: the marker line is ALL the text layer
|
||||
#: carries (14 stripped chars, under the 100-char bar — the candidate);
|
||||
#: the figure is drawn vector (no raster image).
|
||||
PAGE_TWO = "Illustration 2"
|
||||
|
||||
#: The stored illustration block (LOCKED A4) — the mock's canned
|
||||
#: ``PICTORIAL_PAGE_MODE`` reply, deterministic.
|
||||
ILLUSTRATION_BLOCK = f"[Page 2 illustration: {PICTORIAL_PAGE_ANSWER}]"
|
||||
|
||||
#: The grounded question (b): the house seeded-read call (phase 119 A1 —
|
||||
#: a zero-read grounded turn chips NOTHING, so the PDF must be READ to
|
||||
#: earn its ``done.sources`` citation) + the natural question, whose
|
||||
#: tokens ("two parts joined page") overlap ONLY the illustrated page's
|
||||
#: content (the mock's token-overlap cosine grounds; the text layer
|
||||
#: alone would answer nothing).
|
||||
QUESTION = (
|
||||
f"Read the suggested document: read {SOURCE_NAME}/{DOC_PATH} — "
|
||||
"how are the two parts joined on page two?"
|
||||
)
|
||||
|
||||
#: The sync ``detail`` key set EXACTLY as pre-phase 140 (the
|
||||
#: ``app/api/sync.py`` terminal detail — phase 126/133 shape). The
|
||||
#: phase-140 counters are log-only and must NOT appear (c).
|
||||
PRE_PHASE_DETAIL_KEYS = frozenset(
|
||||
{
|
||||
"files",
|
||||
"added",
|
||||
"updated",
|
||||
"unchanged",
|
||||
"pruned",
|
||||
"errors",
|
||||
"chunks",
|
||||
"summaries",
|
||||
"summary_errors",
|
||||
"dates_updated",
|
||||
"overview",
|
||||
"sources_version",
|
||||
}
|
||||
)
|
||||
|
||||
#: Phase 79 (task 04, full inventory): the conftest session app owns its
|
||||
#: port in a combined run — this module app binds its own port instead
|
||||
#: (a same-port second uvicorn dies on bind and would drive the wrong
|
||||
#: server). Env-overridable.
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_PICTPDF", "8155"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
UPLOAD_TIMEOUT_MS = 30_000
|
||||
SYNC_TIMEOUT_MS = 60_000
|
||||
SYNCED_LABEL = re.compile(r"^Synced \d{1,2}:\d{2}$")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixture builder (the ``test_pdf_support.py`` PyMuPDF recipe, the
|
||||
# owner's class: a dense prose page + a sparse vector-illustrated page)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_pdf(path: Path) -> None:
|
||||
"""ONE 2-page PDF (the task's recipe): page 1 = the 700+ char prose
|
||||
paragraph (``insert_textbox`` — word-wrapped, no mid-word breaks in
|
||||
the extraction), page 2 = the ``Illustration 2`` marker + a drawn
|
||||
VECTOR figure (a rectangle + a line arrow with a two-stroke head —
|
||||
the IKEA pictogram class: ``get_image_info()`` sees ZERO embedded
|
||||
raster images, the phase-122/137 pipeline is blind to it)."""
|
||||
doc = fitz.open()
|
||||
page1 = doc.new_page()
|
||||
rc = page1.insert_textbox(fitz.Rect(72, 72, 523, 772), PAGE_ONE, fontsize=11)
|
||||
assert rc >= 0, "the page-1 paragraph must fit its text box"
|
||||
page2 = doc.new_page()
|
||||
page2.insert_text((72, 72), PAGE_TWO)
|
||||
# The vector figure: a rectangle (a part) + an arrow (the join).
|
||||
page2.draw_rect(fitz.Rect(100, 120, 300, 220))
|
||||
page2.draw_line(fitz.Point(320, 170), fitz.Point(470, 170))
|
||||
page2.draw_line(fitz.Point(470, 170), fitz.Point(450, 158))
|
||||
page2.draw_line(fitz.Point(470, 170), fitz.Point(450, 182))
|
||||
doc.set_metadata({"title": PDF_TITLE})
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def pdf_path(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The fixture PDF (module-scoped, built once) — the local copy the
|
||||
byte-exact content assertion computes against (the app indexes the
|
||||
unpacked copy of the SAME bytes)."""
|
||||
root = tmp_path_factory.mktemp("bor_pict_pdf")
|
||||
pdf = root / DOC_PATH
|
||||
_build_pdf(pdf)
|
||||
# The premise pins (deterministic — the pass's candidate set and the
|
||||
# gate's own contract, computed from the fixture itself): page 1 is
|
||||
# prose (no candidate), page 2 is sparse (the candidate).
|
||||
page_texts = extract_pdf_page_texts(pdf)
|
||||
candidates = pictorial_pages(
|
||||
page_texts,
|
||||
min_chars=Settings.model_fields["pdf_pictorial_min_chars"].default,
|
||||
)
|
||||
assert candidates == [2], (
|
||||
f"the fixture's sparse-page premise broke — candidates={candidates}"
|
||||
)
|
||||
return pdf
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def tar_path(pdf_path: Path) -> Path:
|
||||
"""The fixture archive: ONE PDF at the root (the ``.tar`` format —
|
||||
the owner's ``ikea_instructions.tar`` class; the upload gate admits
|
||||
it) — source ``e2e-pictorial``, document path ``instructions.pdf``."""
|
||||
archive = pdf_path.parent / f"{SOURCE_NAME}.tar"
|
||||
with tarfile.open(archive, "w") as tf:
|
||||
tf.add(pdf_path, arcname=DOC_PATH)
|
||||
return archive
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Module app (the conftest env pattern + the phase-140 knobs)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _app_env(mock_llm: int, app_port: int, scratch: Path) -> dict[str, str]:
|
||||
"""The conftest app env (leak guards included) with the phase-140
|
||||
knobs: ``BOR_IMPORT_EXTENSIONS`` pinned to a set carrying the
|
||||
``pdf`` token (the phase-126 gate — an operator's local gitignored
|
||||
``.env`` cannot leak the scope either way), ``BOR_PDF_PICTORIAL``
|
||||
pinned EXPLICITLY on (the conftest pins it off in ``os.environ``
|
||||
for the pre-phase suites — this suite is the documented opt-in;
|
||||
without it the whole premise is void), and ``BOR_IMAGES`` pinned
|
||||
off (out of scope — the vector figure yields no image doc anyway)."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated gate (conftest pattern) — the story's question
|
||||
# grounds on these values (see QUESTION).
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||||
# Phase 67: instant retry waits + the code-default budget.
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# Leak guards (conftest pattern).
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default
|
||||
# Phase 140: the import scope WITH the ``pdf`` token + the toggle
|
||||
# under test (the EXPLICIT opt-in — see the module docstring) + the
|
||||
# suite-private homes.
|
||||
env["BOR_IMPORT_EXTENSIONS"] = "md,markdown,txt,pdf"
|
||||
env["BOR_PDF_PICTORIAL"] = "1"
|
||||
env["BOR_IMAGES"] = "false"
|
||||
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
|
||||
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
|
||||
env["BOR_IMAGE_DIR"] = str(scratch / "images")
|
||||
return env
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(
|
||||
mock_llm: int, tmp_path_factory: pytest.TempPathFactory
|
||||
) -> Iterator[str]:
|
||||
"""The app under test (all three tests): one module app with the
|
||||
phase-140 env (see :func:`_app_env`)."""
|
||||
scratch = tmp_path_factory.mktemp("bor_pict_scratch")
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=_app_env(mock_llm, APP_PORT, scratch=scratch),
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# DB + UI helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh KB + registry (the E2E isolation pattern): the suites share
|
||||
one Postgres, and a leftover row would corrupt the counts.
|
||||
``sources_meta`` (the KB generation counter) resets with the KB —
|
||||
the sync re-creates the row."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources, folder_summaries, sources_meta"
|
||||
)
|
||||
)
|
||||
db.commit() # without the commit the TRUNCATE rolls back (the house pattern)
|
||||
|
||||
|
||||
def _health_db(app_url: str) -> bool:
|
||||
try:
|
||||
return httpx.get(f"{app_url}/api/health", timeout=5).json()["db"] == "up"
|
||||
except Exception: # noqa: BLE001 — unreachable is the skip case
|
||||
return False
|
||||
|
||||
|
||||
def _wait_sync_settled(base_url: str, timeout_s: float = 60.0) -> None:
|
||||
"""No background sync may leak across tests (the run state lives in
|
||||
the app's memory, and a still-running run would keep importing into
|
||||
the NEXT test's truncated KB): wait for the app's sync status to be
|
||||
non-running BEFORE the truncate (the phase-96 pattern)."""
|
||||
with httpx.Client(base_url=base_url, timeout=5.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
deadline = time.monotonic() + timeout_s
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
if r.json()["state"] != "running":
|
||||
return
|
||||
time.sleep(0.2)
|
||||
raise AssertionError(f"a sync was still running at a test boundary: {base_url}")
|
||||
|
||||
|
||||
@pytest.fixture(scope="module", autouse=True)
|
||||
def _clean_kb(app_url: str) -> Iterator[None]:
|
||||
"""Module-end KB cleanup (the phase-96 pattern, module granularity):
|
||||
the three tests SHARE the module seed's single source + document,
|
||||
so there is NO per-test truncate (it would wipe the seed from under
|
||||
the next test — the per-test pattern belongs to the suites where
|
||||
every test imports its own source); instead the shared KB is
|
||||
flushed ONCE, after the last test — no rows leak into the next
|
||||
suite's run. The pre-wait keeps a leaked sync (the module app is
|
||||
still up here) from racing the truncate."""
|
||||
yield
|
||||
_wait_sync_settled(app_url)
|
||||
_truncate_all()
|
||||
|
||||
|
||||
def _upload_tar(page: Page, app_url: str, archive: Path) -> None:
|
||||
"""The real admin upload flow: form login → the git-sources page →
|
||||
pick the archive → submit → the terminal result line (phase 90:
|
||||
"Uploaded <source> — press Sync sources to import it.")."""
|
||||
login(page, app_url, next="/git-sources.html")
|
||||
expect(page).to_have_url(app_url + "/git-sources.html", timeout=30_000)
|
||||
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
|
||||
page.set_input_files("#archive-upload-file", str(archive))
|
||||
page.click("#archive-upload-btn")
|
||||
result = page.locator("#archive-upload-result")
|
||||
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
|
||||
expect(result).to_have_text(
|
||||
f"Uploaded {SOURCE_NAME} — press Sync sources to import it.",
|
||||
timeout=UPLOAD_TIMEOUT_MS,
|
||||
)
|
||||
|
||||
|
||||
def _sync_via_ui(page: Page, app_url: str, expected_result: str) -> None:
|
||||
"""The RAG page's "Sync sources" button (the scan): click →
|
||||
"Syncing…" → the terminal "Synced HH:MM" label + the fresh counts
|
||||
in #sync-result (the never-stale contract)."""
|
||||
page.goto(app_url + "/sources.html")
|
||||
btn = page.locator("#sync-btn")
|
||||
expect(btn).to_be_visible(timeout=30_000)
|
||||
expect(page.locator("#sync-label")).to_have_text(
|
||||
re.compile(r"^Sync sources$|^Synced \d{1,2}:\d{2}$")
|
||||
)
|
||||
btn.click()
|
||||
expect(btn).to_be_disabled()
|
||||
expect(page.locator("#sync-label")).to_have_text("Syncing…")
|
||||
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
|
||||
expect(btn).to_be_enabled()
|
||||
expect(page.locator("#sync-result")).to_have_text(expected_result)
|
||||
|
||||
|
||||
def _drill_to_source(page: Page, name: str) -> None:
|
||||
"""Top level → the source (the file table then holds its direct
|
||||
files — ours is at the source root)."""
|
||||
page.locator("#folders-tbody .folder-link").first.wait_for(state="visible")
|
||||
page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
|
||||
expect(page.locator("#docs-tbody tr").first).to_be_visible(timeout=30_000)
|
||||
|
||||
|
||||
def _chat_frames(app_url: str, message: str) -> list[dict[str, Any]]:
|
||||
"""One ``POST /api/chat`` turn as RAW SSE frames (the wire lens —
|
||||
the house pattern from ``test_honest_deflection.py``): the signed
|
||||
admin client streams the turn and parses the ``data: <json>``
|
||||
frames (PLAN §4: ``data: <json>\\n\\n``)."""
|
||||
with httpx.Client(base_url=app_url, timeout=120.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
frames: list[dict[str, Any]] = []
|
||||
with client.stream(
|
||||
"POST", "/api/chat", json={"message": message}, timeout=120.0
|
||||
) as r:
|
||||
assert r.status_code == 200, r.read().decode()
|
||||
buf = ""
|
||||
for part in r.iter_text():
|
||||
buf += part
|
||||
while "\n\n" in buf:
|
||||
frame, buf = buf.split("\n\n", 1)
|
||||
if frame.strip().startswith("data:"):
|
||||
frames.append(
|
||||
json.loads(frame.strip().removeprefix("data:").strip())
|
||||
)
|
||||
assert buf.strip() == "" # the stream ends cleanly on a frame boundary
|
||||
return frames
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The module seed: upload + sync the fixture PDF through the real UI,
|
||||
# once (the story E2E truncate/re-import-per-module pattern)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def seeded(
|
||||
app_url: str,
|
||||
browser: Browser,
|
||||
tar_path: Path,
|
||||
pdf_path: Path,
|
||||
) -> Iterator[dict[str, str]]:
|
||||
"""Upload the single-PDF tar through the admin UI, then scan it with
|
||||
the RAG page's Sync button (the mock's vision model describes the
|
||||
sparse page in-process — the ``PICTORIAL_PAGE_MODE`` canned reply).
|
||||
The seed asserts the stored document BYTE-EXACT: the fixture's own
|
||||
extraction (page blocks, the page-2 candidate carrying the canned
|
||||
illustration block) — the candidates computed from
|
||||
``extract_pdf_page_texts`` + ``pictorial_pages`` IN the fixture (no
|
||||
hardcoded page list). This is the module app's toggle premise
|
||||
witness too: without the ``BOR_PDF_PICTORIAL`` opt-in, page 2 would
|
||||
lack its block and this fails loudly. Yields the doc's identity."""
|
||||
if not _health_db(app_url):
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
|
||||
_truncate_all()
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 800})
|
||||
try:
|
||||
_upload_tar(page, app_url, tar_path)
|
||||
# The PDF alone (no embedded raster images — the vector figure
|
||||
# yields no image doc): one file, one added document.
|
||||
_sync_via_ui(page, app_url, expected_result="1 added")
|
||||
with SessionLocal() as db:
|
||||
doc = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE_NAME, Document.path == DOC_PATH
|
||||
)
|
||||
)
|
||||
assert doc is not None, "the uploaded PDF must become a document"
|
||||
# LOCKED A2: one document per PDF, the metadata title; a TEXT
|
||||
# document (no image doc, no image path).
|
||||
assert doc.title == PDF_TITLE
|
||||
assert doc.is_image is False and doc.image_path is None
|
||||
# The candidates (computed from the fixture itself — the
|
||||
# deterministic gate, LOCKED A1) + the byte-exact stored
|
||||
# content: the extraction's page blocks, the described
|
||||
# candidate carrying ``[Page N illustration: <canned>]``
|
||||
# (LOCKED A4).
|
||||
page_texts = extract_pdf_page_texts(pdf_path)
|
||||
candidates = pictorial_pages(
|
||||
page_texts,
|
||||
min_chars=Settings.model_fields["pdf_pictorial_min_chars"].default,
|
||||
)
|
||||
assert candidates == [2]
|
||||
parts = []
|
||||
for p, t in page_texts.items():
|
||||
block = t
|
||||
if p in candidates:
|
||||
block += f"\n\n[Page {p} illustration: {PICTORIAL_PAGE_ANSWER}]"
|
||||
parts.append(f"{PAGE_SEPARATOR.format(page=p)}\n{block}")
|
||||
expected = "\n\n".join(parts)
|
||||
assert doc.content == expected, (
|
||||
"the stored content is not the extraction with the "
|
||||
"described sparse page — the pictorial pass misfired"
|
||||
)
|
||||
# The non-markdown doc got its lite summary (the mock's
|
||||
# SUMMARY_MODE digest) — the summary chunk + the viewer's
|
||||
# Summary panel exist.
|
||||
assert doc.summary
|
||||
finally:
|
||||
page.close()
|
||||
yield {"title": PDF_TITLE}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. Sources page + document viewer (a): the described page renders
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_sources_page_and_viewer_show_the_described_page(
|
||||
page: Page, app_url: str, seeded: dict[str, str]
|
||||
) -> None:
|
||||
"""The KB total is 1 doc / 3 chunks (the PDF: 2 page-anchored
|
||||
content chunks + the phase-30 summary chunk — no image doc; the
|
||||
vector figure is invisible to the embedded-image pipeline). The
|
||||
drill-down table lists the PDF as a plain text row (LOCKED A4), and
|
||||
the same-page document viewer (the row's path link, phase 26)
|
||||
renders the stored content under the METADATA title: page 1's prose
|
||||
WITHOUT an illustration block (not a candidate) and page 2 WITH
|
||||
``[Page 2 illustration: <canned>]`` (the described sparse page —
|
||||
LOCKED A4's user surface), plus the Summary panel (a non-image doc
|
||||
with a non-empty summary)."""
|
||||
login(page, app_url) # lands on /sources.html
|
||||
expect(page.locator("#stat-docs")).to_have_text("1")
|
||||
expect(page.locator("#stat-chunks")).to_have_text("3")
|
||||
expect(page.locator("#sources-empty")).to_be_hidden()
|
||||
|
||||
_drill_to_source(page, SOURCE_NAME)
|
||||
expect(page.locator("#docs-tbody tr")).to_have_count(1)
|
||||
row = page.locator("#docs-tbody tr").first
|
||||
# A plain text row — no thumbnail box (it is a text document, A4).
|
||||
expect(row.locator(".kb-doc-thumb")).to_have_count(0)
|
||||
|
||||
page.locator(f'#docs-tbody a.doc-link:text-is("{DOC_PATH}")').click()
|
||||
modal = page.locator("#doc-modal")
|
||||
expect(modal).to_be_visible(timeout=30_000)
|
||||
expect(page.locator("#doc-modal-title")).to_have_text(PDF_TITLE)
|
||||
|
||||
raw = modal.locator("pre.doc-raw")
|
||||
# Page 1 — the prose WITHOUT any illustration block. The viewer
|
||||
# renders the content verbatim (``insert_textbox`` wraps mid-
|
||||
# sentence, so compare whitespace-normalized).
|
||||
expect(raw).to_contain_text("---PAGE-1---")
|
||||
rendered = " ".join(raw.inner_text().split())
|
||||
assert " ".join(PAGE_ONE.split()) in rendered, (
|
||||
"the page-1 prose is not what the viewer renders"
|
||||
)
|
||||
assert "[Page 1 illustration" not in rendered, (
|
||||
"page 1 (over the sparseness bar) must carry NO illustration block"
|
||||
)
|
||||
# Page 2 — the sparse page WITH the described block (a) — the mock's
|
||||
# canned reply, deterministic.
|
||||
expect(raw).to_contain_text("---PAGE-2---")
|
||||
expect(raw).to_contain_text(PAGE_TWO)
|
||||
expect(raw).to_contain_text(ILLUSTRATION_BLOCK)
|
||||
# The Summary panel renders (a non-image doc with a non-empty
|
||||
# summary); no image block (the PDF is not an image document).
|
||||
expect(modal.locator(".doc-summary")).to_have_count(1)
|
||||
expect(modal.locator(".doc-image-img")).to_have_count(0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. Chat (b): the question only the illustration answers grounds, and
|
||||
# the answer echoes the described phrase
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_chat_question_grounds_and_echoes_the_illustration(
|
||||
app_url: str, seeded: dict[str, str]
|
||||
) -> None:
|
||||
"""The raw SSE over the scripted seeded-read question (phase 119 A1
|
||||
— a zero-read grounded turn chips NOTHING, so the PDF must be READ
|
||||
to earn its ``done.sources`` citation): the turn GROUNDS (no
|
||||
deflection — the described page's embedding retrieves: the mock's
|
||||
deterministic token-overlap cosine clears the mock-calibrated 0.30
|
||||
gate, FTS corroborates), the mock's scripted ``read`` runs (EXACTLY
|
||||
one ``tool`` frame — the doc is far under the read cap, so no
|
||||
truncation and no ``tool_result`` frame), and the answer ECHOES the
|
||||
read result verbatim — the ``[Page 2 illustration: …]`` block and
|
||||
its phrase reach the model and land in the rendered answer (the
|
||||
described page's content is what a chat turn can ground on — the
|
||||
owner's L5 rationale end to end)."""
|
||||
frames = _chat_frames(app_url, QUESTION)
|
||||
assert frames, "the SSE stream settled with no frames at all"
|
||||
assert not any(f.get("type") == "error" for f in frames), (
|
||||
f"the turn errored: {[f for f in frames if f.get('type') == 'error']}"
|
||||
)
|
||||
|
||||
# The scripted read — exactly one tool call, on the PDF.
|
||||
tools = [f for f in frames if f.get("type") == "tool"]
|
||||
assert len(tools) == 1, f"expected exactly one tool frame: {tools}"
|
||||
assert tools[0]["name"] == "read"
|
||||
assert tools[0]["argument"] == f"{SOURCE_NAME}/{DOC_PATH}"
|
||||
# The doc is far under the read cap — no truncation frame (the echo
|
||||
# below therefore carries the FULL content, the block included).
|
||||
assert not any(f.get("type") == "tool_result" for f in frames)
|
||||
|
||||
# The terminal frame: ``done``, NOT deflected.
|
||||
done = [f for f in frames if f.get("type") == "done"]
|
||||
assert len(done) == 1 and frames[-1] is done[0], (
|
||||
f"the stream must settle with exactly one terminal done frame: "
|
||||
f"{[f.get('type') for f in frames]}"
|
||||
)
|
||||
assert done[0]["deflected"] is False, (
|
||||
"the question grounded on the described page — a deflection means "
|
||||
"the illustrated content did not retrieve"
|
||||
)
|
||||
# The ``done`` event's ``sources`` — the citation surface (read docs
|
||||
# only, phase 119 A1) — carries EXACTLY the PDF.
|
||||
refs = done[0]["sources"]
|
||||
assert [(r["source"], r["path"]) for r in refs] == [(SOURCE_NAME, DOC_PATH)], (
|
||||
f"the done event's sources must be exactly the read PDF: {refs}"
|
||||
)
|
||||
assert refs[0]["title"] == PDF_TITLE
|
||||
|
||||
# The answer (the joined deltas) echoes the read result VERBATIM —
|
||||
# the header line and the described block with the canned phrase.
|
||||
answer = "".join(f["text"] for f in frames if f.get("type") == "delta")
|
||||
assert "Here's what the read returned:" in answer
|
||||
assert f"Document {SOURCE_NAME}/{DOC_PATH}:" in answer
|
||||
assert ILLUSTRATION_BLOCK in answer, (
|
||||
"the answer does not echo the described illustration block — "
|
||||
"the vision pass's content never reached the model"
|
||||
)
|
||||
assert PICTORIAL_PAGE_ANSWER in answer # the phrase itself (the task's pin)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. Sync detail (c): the pre-phase key set is untouched (stats are
|
||||
# log-only)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_sync_detail_keeps_the_pre_phase_keys(
|
||||
app_url: str, seeded: dict[str, str]
|
||||
) -> None:
|
||||
"""The module run's terminal ``detail`` (``GET /api/sync/status`` —
|
||||
the same object the UI's "1 added" line renders from) keeps the
|
||||
pre-phase key set EXACTLY: the phase-140 counters
|
||||
(``pdfs_pictorial`` / ``pdf_pictorial_pages``) are log-only (PLAN
|
||||
§9, the phase-133 convention) and must NOT ride the wire, and the
|
||||
run's own file counters read 1 file / 1 added / 0 unchanged /
|
||||
0 pruned / 0 errors (the single PDF, a fresh KB)."""
|
||||
with httpx.Client(base_url=app_url, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
# The stats log-only convention — the pictorial counters never
|
||||
# reach the wire.
|
||||
assert "pdfs_pictorial" not in detail, (
|
||||
"the log-only pdfs_pictorial counter leaked into the sync detail"
|
||||
)
|
||||
assert "pdf_pictorial_pages" not in detail, (
|
||||
"the log-only pdf_pictorial_pages counter leaked into the sync detail"
|
||||
)
|
||||
# The shape itself — the pre-phase key set EXACTLY (no new field).
|
||||
assert set(detail) == set(PRE_PHASE_DETAIL_KEYS), (
|
||||
f"the sync detail shape changed — got {sorted(detail)}"
|
||||
)
|
||||
# The run's own file counters (a fresh KB: one file, one added).
|
||||
assert detail["files"] == 1, detail
|
||||
assert detail["added"] == 1, detail
|
||||
assert detail["unchanged"] == 0, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
assert detail["errors"] == 0, detail
|
||||
Reference in New Issue
Block a user