All completion criteria verified. The repository is left functional with all changes in the working tree (no commits made, per the harness protocol). ## Final report — phase 125 `image_anchor_context` (verification pass) All 6 tasks were already complete; this pass verified every criterion and ran the full gates: - **Migration 0023** `documents.image_context` (TEXT nullable, reversible) — applied (`alembic current` = 0023 head); round-trip test (4 passed) exercises real upgrade → downgrade 0022 → upgrade. - **Criterion 1** (referenced image enriched + context line in retrieval): integration `test_markdown_referenced_image_gets_context_and_enriched_chunks` + E2E `test_sync_enriches_the_image_doc_with_markdown_context`, `test_chat_answer_quotes_the_context_line` (wire prompt pinned: `<document>` block leads with the one-line `Referenced in …: "alt"` line) — verified present and passing. - **Criterion 2** (standalone image byte-identical): E2E `test_standalone_*` (NULL `image_context`, block body exactly the bare description, no `Referenced in` anywhere in the prompt) + unit `test_describe_image_without_context_is_byte_identical` — passing. - **Criterion 3** (markdown-first: record stored on markdown doc, image picks up context later): integration `test_markdown_indexed_before_image_stores_record_and_image_picks_it_up` + post-index enrichment pass test (markdown file gone from disk) — passing. - **Gates:** `uv run pytest` → 2938 passed, exit 0 · `uv run pytest --cov=app --cov-report=term-missing` → exit 0, TOTAL **99%** (>90%) · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors, 0 warnings. - **Phase E2E in isolation:** `uv run pytest tests/e2e/test_image_anchor_context.py -v --no-cov` → 4 passed. - **Mock-LLM change regression check:** neighboring image E2E suites green in isolation — `test_smoke` (3), `test_image_documents` (16), `test_chat_image_questions` (17), `test_answer_images` (17). - No defects found; no code changes needed. No live-infrastructure changes (local dev DB only). No commit made (harness commits). - Next pending phase: check `.agents/phases/todo/` after the harness moves this phase.
624 lines
27 KiB
Python
624 lines
27 KiB
Python
"""Phase 125 E2E (Playwright): image-anchor context — the user path.
|
||
|
||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||
|
||
uv run pytest tests/e2e/test_image_anchor_context.py -v --no-cov
|
||
|
||
The suite's app instance runs with ``BOR_IMAGES=true`` and
|
||
``BOR_ANSWER_IMAGES=false`` (module env override — the conftest
|
||
per-suite-app pattern, leak guards included): the fixture archive
|
||
carries ``docs/guide.md`` — a markdown file whose ````
|
||
reference NAMES ``docs/diagram.png`` in the SAME source — so the real
|
||
upload → sync flow drives the whole phase-125 pipeline (LOCKED
|
||
A1/A2): the task-03 disk scan finds the reference at image-index time,
|
||
the vision model receives the context after the locked describe prompt,
|
||
and the image doc's ``image_context`` (and so its ``content``,
|
||
``summary`` and embedded chunks) becomes
|
||
``<context>\\n\n<description>``.
|
||
|
||
* the sync leaves the image doc ENRICHED (the ``Referenced in`` block
|
||
on ``image_context``, the enriched content + chunks, the markdown
|
||
doc's record NULL — the image was indexed first, the task-03 path
|
||
owns that case);
|
||
* a grounded chat turn whose suggested tier carries the enriched
|
||
image doc gets the task-05 context line: the mock (the
|
||
``CONTEXT_LINE_TRIGGER`` branch — the prompt parser extended for
|
||
this phase) QUOTES THE EXACT LINE of the image doc's ``<document>``
|
||
block, so the rendered answer AND the captured wire prompt prove
|
||
the block LEADS with the one-line ``Referenced in
|
||
<source>/<path>: "<alt>"`` rendering before the summary (the stored
|
||
block's shape carries the alt on the next line as a bullet — the
|
||
quote pins the task-05 rendering, not the stored block);
|
||
* a STANDALONE image (a second phase of the same app — the KB is
|
||
re-seeded with a single PNG no markdown references) stays
|
||
byte-identical to pre-phase: ``image_context`` NULL, the bare
|
||
description everywhere, the ``<document>`` block the description
|
||
alone, and the mock's "no reference line" answer (the deterministic
|
||
witness that no context line was in the prompt).
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import base64
|
||
import json
|
||
import os
|
||
import re
|
||
import subprocess
|
||
import sys
|
||
import zipfile
|
||
from collections.abc import Iterator
|
||
from pathlib import Path
|
||
|
||
import httpx
|
||
import pytest
|
||
from playwright.sync_api import Browser, Page, expect
|
||
from sqlalchemy import select, text
|
||
|
||
from app.config import Settings
|
||
from app.db import SessionLocal
|
||
from app.models import Chunk, Document
|
||
from e2e.auth_helpers import login
|
||
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
|
||
from e2e.mock_llm import IMAGE_DESCRIPTION_ANSWER
|
||
|
||
REPO = Path(__file__).resolve().parents[2]
|
||
GIT_SOURCES_URL = "/git-sources.html"
|
||
|
||
#: Phase A — the anchored source: the markdown references the image
|
||
#: from the SAME folder (``diagram.png`` next to ``guide.md`` → the
|
||
#: resolved in-source path ``docs/diagram.png``).
|
||
SOURCE_NAME = "e2e-anchor" # the archive stem (archive_source_name)
|
||
MD_PATH = "docs/guide.md"
|
||
DOC_PATH = "docs/diagram.png"
|
||
|
||
#: Phase B — the standalone source: a single PNG no markdown names.
|
||
STANDALONE_SOURCE_NAME = "e2e-anchor-standalone"
|
||
LOGO_PATH = "logo.png"
|
||
|
||
#: The fixture markdown — the reference on line 5, two context
|
||
#: paragraphs (the task-02 default window).
|
||
GUIDE_MD = (
|
||
"# Backup runbook\n"
|
||
"\n"
|
||
"The restore order diagram explains the backup procedure.\n"
|
||
"\n"
|
||
"\n"
|
||
"\n"
|
||
"Only run the restore script after the checks pass.\n"
|
||
)
|
||
|
||
#: The task-03-shape combined context the same-source disk scan builds
|
||
#: for ``docs/diagram.png`` (one referring markdown, one reference,
|
||
#: the task-02 window as the ``Context:`` body — the ``# Backup
|
||
#: runbook`` heading is the first of the two paragraphs before the
|
||
#: reference; the N=2 window keeps it).
|
||
BLOCK = (
|
||
"Referenced in e2e-anchor/docs/guide.md:\n"
|
||
'- Alt text: "Restore order"\n'
|
||
" Context: # Backup runbook\n"
|
||
"\n"
|
||
"The restore order diagram explains the backup procedure.\n"
|
||
"\n"
|
||
"Only run the restore script after the checks pass."
|
||
)
|
||
|
||
#: The enriched image doc's stored ``content`` (AND ``summary`` — the
|
||
#: phase-122 ``summary = content`` convention): the context FIRST
|
||
#: (for retrieval), then the vision description.
|
||
ENRICHED = f"{BLOCK}\n\n{IMAGE_DESCRIPTION_ANSWER}"
|
||
|
||
#: The content chunk's stored text — the plain-text packer joins the
|
||
#: enriched paragraphs with single newlines (the chunker's
|
||
#: normalization, PLAN §5); the ``is_summary`` chunk keeps the raw
|
||
#: enriched text.
|
||
PACKED = "\n".join(ENRICHED.split("\n\n"))
|
||
|
||
#: The task-05 ONE-LINE rendering of :data:`BLOCK` (the context line
|
||
#: that leads the enriched image doc's ``<document>`` block body):
|
||
#: the referring markdown's identity + the reference's alt, the
|
||
#: ``Context:`` body never rendered.
|
||
CONTEXT_LINE = 'Referenced in e2e-anchor/docs/guide.md: "Restore order"'
|
||
|
||
#: The grounded question — the mock's ``CONTEXT_LINE_TRIGGER`` marker
|
||
#: (the zero-read context-line echo) + the phase-122 grounding phrase
|
||
#: (token-overlap cosine ≈0.47 against the enriched diagram doc and
|
||
#: ≈0.37 against the standalone logo's bare description — both clear
|
||
#: the mock-calibrated gate (0.30), the FTS leg corroborates).
|
||
ANCHOR_QUESTION = (
|
||
"quote the image reference — what is shown in the homelab network diagram?"
|
||
)
|
||
|
||
#: The mock's two deterministic answers for the marker (the exact
|
||
#: branch texts — the phase-125 pins key off them).
|
||
ANSWER_WITH_LINE = (
|
||
f"The notes reference the image thus: {CONTEXT_LINE} "
|
||
"(Deterministic mock answer for E2E.)"
|
||
)
|
||
ANSWER_STANDALONE = (
|
||
"The image stands alone — the notes carry no reference line for it. "
|
||
"(Deterministic mock answer for E2E.)"
|
||
)
|
||
|
||
#: The honest-deflection opening (the gate must NOT fire on the
|
||
#: marker question — a deflected prompt carries no ``<documents>``
|
||
#: block, so the mock would never see the image doc).
|
||
DEFLECT_OPENING = "I haven't done anything like that"
|
||
|
||
UPLOAD_TIMEOUT_MS = 30_000
|
||
SYNC_TIMEOUT_MS = 60_000
|
||
SYNCED_LABEL = re.compile(r"^Synced \d{1,2}:\d{2}$")
|
||
|
||
#: A real 1×1 transparent PNG (the phase-122 fixture — the pipeline
|
||
#: is content-agnostic, the bytes just need to be plausible image
|
||
#: bytes).
|
||
PNG_1X1 = base64.b64decode(
|
||
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
|
||
)
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Module app: images ON (the story) — the conftest env pattern with the
|
||
# phase-122/125 knobs + leak guards.
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def _app_env(mock_llm: int, app_port: int, scratch: Path) -> dict[str, str]:
|
||
"""The conftest app env (leak guards included) with the phase-122
|
||
knobs: ``BOR_IMAGES`` pinned EXPLICITLY true (the story) and
|
||
``BOR_ANSWER_IMAGES`` pinned EXPLICITLY false (the default — the
|
||
auto-cite/figure behavior is phase 124's story, not this phase's,
|
||
and an operator's local gitignored ``.env`` cannot leak it in:
|
||
both toggles are the contract under test's neighbors), the
|
||
image/upload homes in the suite's scratch dir (the app under test
|
||
must not write copies into the owner's real ``~/bor-sources``)."""
|
||
env = dict(os.environ)
|
||
env.pop("DEBUGPY", None)
|
||
env["BOR_ENVIRONMENT"] = "e2e"
|
||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||
env["BOR_LLM_BASE_URL"] = (
|
||
"https://aipi.reeseapps.com/v1"
|
||
if USE_REAL_LLM
|
||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||
)
|
||
# Mock-calibrated gate (conftest pattern) — the marker question
|
||
# grounds on these values (see ANCHOR_QUESTION).
|
||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||
# Phase 67: instant retry waits + the code-default budget.
|
||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||
env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default)
|
||
env.setdefault(
|
||
"BOR_DATABASE_URL",
|
||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||
)
|
||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||
# Leak guards (conftest pattern).
|
||
env["BOR_GIT_SOURCES"] = ""
|
||
env["BOR_DOCS_REPO"] = ""
|
||
env["BOR_SUGGESTIONS"] = json.dumps(Settings.model_fields["suggestions"].default)
|
||
env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default
|
||
env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default
|
||
# Phase 122/124 toggles + the suite-private homes.
|
||
env["BOR_IMAGES"] = "true"
|
||
env["BOR_ANSWER_IMAGES"] = "false"
|
||
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
|
||
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
|
||
env["BOR_IMAGE_DIR"] = str(scratch / "images")
|
||
return env
|
||
|
||
|
||
@pytest.fixture(scope="module")
|
||
def app_server(mock_llm: int, tmp_path_factory: pytest.TempPathFactory) -> Iterator[str]:
|
||
"""The story app — ``BOR_IMAGES=true`` (the module env override;
|
||
the conftest session app is never started in this isolated run,
|
||
so no port clash)."""
|
||
scratch = tmp_path_factory.mktemp("bor_anchor")
|
||
port = int(os.environ.get("E2E_APP_PORT_ANCHOR", "8160"))
|
||
proc = subprocess.Popen(
|
||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
|
||
cwd=REPO,
|
||
env=_app_env(mock_llm, port, scratch=scratch),
|
||
)
|
||
try:
|
||
_wait_http(f"http://127.0.0.1:{port}/api/health")
|
||
yield f"http://127.0.0.1:{port}"
|
||
finally:
|
||
proc.terminate()
|
||
try:
|
||
proc.wait(timeout=10)
|
||
except subprocess.TimeoutExpired:
|
||
proc.kill()
|
||
|
||
|
||
@pytest.fixture(scope="module")
|
||
def app_url(app_server: str) -> str:
|
||
return app_server
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Fixture archives
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
@pytest.fixture(scope="module")
|
||
def anchor_zip(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||
"""Phase A's archive: the markdown that references the image in
|
||
the SAME source (``docs/guide.md`` → ``docs/diagram.png``)."""
|
||
root = tmp_path_factory.mktemp("bor_anchor_zip")
|
||
docs = root / "docs"
|
||
docs.mkdir()
|
||
(docs / "guide.md").write_text(GUIDE_MD, encoding="utf-8")
|
||
(docs / "diagram.png").write_bytes(PNG_1X1)
|
||
archive = root / f"{SOURCE_NAME}.zip"
|
||
with zipfile.ZipFile(archive, "w", zipfile.ZIP_DEFLATED) as zf:
|
||
zf.write(docs / "guide.md", arcname="docs/guide.md")
|
||
zf.write(docs / "diagram.png", arcname="docs/diagram.png")
|
||
return archive
|
||
|
||
|
||
@pytest.fixture(scope="module")
|
||
def standalone_zip(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||
"""Phase B's archive: ONE standalone PNG at the source root — no
|
||
markdown in the source references it (the byte-identical
|
||
pre-phase case)."""
|
||
root = tmp_path_factory.mktemp("bor_anchor_standalone_zip")
|
||
(root / LOGO_PATH).write_bytes(PNG_1X1)
|
||
archive = root / f"{STANDALONE_SOURCE_NAME}.zip"
|
||
with zipfile.ZipFile(archive, "w", zipfile.ZIP_DEFLATED) as zf:
|
||
zf.write(root / LOGO_PATH, arcname=LOGO_PATH)
|
||
return archive
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# DB + UI helpers
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def _truncate_all() -> None:
|
||
"""Fresh KB + registry per phase (the E2E isolation pattern): the
|
||
suites share one Postgres, and a leftover row would corrupt the
|
||
counts. ``sources_meta`` (the KB generation counter) resets with
|
||
the KB — the sync re-creates the row."""
|
||
with SessionLocal() as db:
|
||
db.execute(text(
|
||
"TRUNCATE chunks, documents, query_log, kb_overview, "
|
||
"git_sources, sources_meta, folder_summaries"
|
||
))
|
||
db.commit()
|
||
|
||
|
||
def _health_db(app_url: str) -> bool:
|
||
try:
|
||
return httpx.get(f"{app_url}/api/health", timeout=5).json()["db"] == "up"
|
||
except Exception: # noqa: BLE001 — unreachable is the skip case
|
||
return False
|
||
|
||
|
||
def _upload_zip(page: Page, app_url: str, archive: Path, source: str) -> None:
|
||
"""The real admin upload flow: form login → the git-sources page →
|
||
pick the archive → submit → the terminal result line (phase 90:
|
||
"Uploaded <source> — press Sync sources to import it.")."""
|
||
login(page, app_url, next=GIT_SOURCES_URL)
|
||
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
|
||
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
|
||
page.set_input_files("#archive-upload-file", str(archive))
|
||
page.click("#archive-upload-btn")
|
||
result = page.locator("#archive-upload-result")
|
||
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
|
||
expect(result).to_have_text(
|
||
f"Uploaded {source} — press Sync sources to import it.",
|
||
timeout=UPLOAD_TIMEOUT_MS,
|
||
)
|
||
|
||
|
||
def _sync_via_ui(page: Page, app_url: str, expected_result: str) -> None:
|
||
"""The RAG page's "Sync sources" button (the scan — phase 90 A3):
|
||
click → "Syncing…" → the terminal "Synced HH:MM" label + the fresh
|
||
counts in #sync-result (the never-stale contract)."""
|
||
page.goto(app_url + "/sources.html")
|
||
btn = page.locator("#sync-btn")
|
||
expect(btn).to_be_visible(timeout=30_000)
|
||
expect(page.locator("#sync-label")).to_have_text(
|
||
re.compile(r"^Sync sources$|^Synced \d{1,2}:\d{2}$")
|
||
)
|
||
btn.click()
|
||
expect(btn).to_be_disabled()
|
||
expect(page.locator("#sync-label")).to_have_text("Syncing…")
|
||
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
|
||
expect(btn).to_be_enabled()
|
||
expect(page.locator("#sync-result")).to_have_text(expected_result)
|
||
|
||
|
||
def _captured_systems(mock_llm: int) -> list[str]:
|
||
"""The system prompts of the captured chat requests (the mock's
|
||
ring buffer — the wire-level lens on the EXACT prompt the app
|
||
built; only ``/v1/chat/completions`` bodies are captured, so after
|
||
a reset + one grounded zero-read turn there is exactly one)."""
|
||
bodies = httpx.get(f"http://127.0.0.1:{mock_llm}/v1/e2e/captured", timeout=10).json()
|
||
systems: list[str] = []
|
||
for body in bodies:
|
||
for msg in body.get("messages", []):
|
||
if msg.get("role") == "system" and isinstance(msg.get("content"), str):
|
||
systems.append(msg["content"])
|
||
return systems
|
||
|
||
|
||
def _reset_capture(mock_llm: int) -> None:
|
||
httpx.post(f"http://127.0.0.1:{mock_llm}/v1/e2e/captured/reset", timeout=10)
|
||
|
||
|
||
def _ask(page: Page, app_url: str, question: str) -> None:
|
||
"""The chat user path: login → the question → Send → the final
|
||
brain bubble (the Send button re-enabled = the turn is over)."""
|
||
login(page, app_url, next="/")
|
||
expect(page.locator("#kb-banner")).to_be_hidden()
|
||
page.fill("#message-input", question)
|
||
page.click("#send-btn")
|
||
bubble = page.locator(".msg.brain .bubble").last
|
||
expect(page.locator("#send-btn")).to_be_enabled(timeout=60_000)
|
||
expect(page.locator("#send-label")).to_have_text("Send")
|
||
expect(bubble).to_contain_text("Deterministic mock answer", timeout=30_000)
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# The module seeds: phase A (the anchored archive) and phase B (the
|
||
# standalone archive) — each owns its truncate/re-import window.
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
@pytest.fixture(scope="module")
|
||
def seeded(
|
||
app_url: str,
|
||
browser: Browser,
|
||
anchor_zip: Path,
|
||
) -> Iterator[dict[str, str]]:
|
||
"""Phase A: upload the anchored archive through the admin UI, then
|
||
scan it with the RAG page's Sync button (the mock's vision model
|
||
describes the image in-process, with the markdown context). The
|
||
module app's ``/api/config`` flags are asserted first — the env
|
||
overrides must have reached the app under test, or the whole
|
||
premise of the suite is void."""
|
||
if not _health_db(app_url):
|
||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||
cfg = httpx.get(f"{app_url}/api/config", timeout=5).json()
|
||
assert cfg["images"] is True, "the module app must run with BOR_IMAGES=true"
|
||
|
||
_truncate_all()
|
||
page = browser.new_page(viewport={"width": 1280, "height": 800})
|
||
try:
|
||
_upload_zip(page, app_url, anchor_zip, SOURCE_NAME)
|
||
_sync_via_ui(page, app_url, expected_result="2 added")
|
||
with SessionLocal() as db:
|
||
doc = db.scalar(
|
||
select(Document).where(
|
||
Document.source == SOURCE_NAME, Document.path == DOC_PATH
|
||
)
|
||
)
|
||
assert doc is not None, "the uploaded image must become a document"
|
||
assert doc.is_image is True and doc.image_path is not None
|
||
finally:
|
||
page.close()
|
||
yield {"image_url": f"/api/documents/{doc.id}/image"}
|
||
_truncate_all()
|
||
|
||
|
||
@pytest.fixture(scope="module")
|
||
def standalone(
|
||
app_url: str,
|
||
browser: Browser,
|
||
standalone_zip: Path,
|
||
) -> Iterator[dict[str, str]]:
|
||
"""Phase B: a FRESH KB (the phase-A rows are truncated) seeded
|
||
with the standalone archive — a single PNG no markdown in the
|
||
source references (the byte-identical pre-phase case)."""
|
||
_truncate_all()
|
||
page = browser.new_page(viewport={"width": 1280, "height": 800})
|
||
try:
|
||
_upload_zip(page, app_url, standalone_zip, STANDALONE_SOURCE_NAME)
|
||
_sync_via_ui(page, app_url, expected_result="1 added")
|
||
with SessionLocal() as db:
|
||
doc = db.scalar(
|
||
select(Document).where(
|
||
Document.source == STANDALONE_SOURCE_NAME, Document.path == LOGO_PATH
|
||
)
|
||
)
|
||
assert doc is not None, "the uploaded image must become a document"
|
||
assert doc.is_image is True and doc.image_path is not None
|
||
finally:
|
||
page.close()
|
||
yield {"image_url": f"/api/documents/{doc.id}/image"}
|
||
_truncate_all()
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# 1. Sync: the image doc is enriched with the markdown context
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def test_sync_enriches_the_image_doc_with_markdown_context(
|
||
page: Page, app_url: str, seeded: dict[str, str]
|
||
) -> None:
|
||
"""LOCKED A1/A2 end-to-end: the markdown in the SAME source
|
||
references the image → after the sync the image doc's
|
||
``image_context`` is the ``Referenced in`` block, its ``content``
|
||
(AND ``summary`` — the phase-122 convention) is
|
||
``<context>\\n\n<description>`` (context first — for retrieval —
|
||
then the vision description), the content chunk is embedded from
|
||
the enriched text (the packer's single-newline form) and the
|
||
``is_summary`` chunk mirrors the raw enriched text. The referring
|
||
markdown's OWN ``image_context`` stays NULL: the image was indexed
|
||
BEFORE the markdown (``docs/diagram.png`` sorts first), so its
|
||
reference names an existing image doc — the task-03 path owns that
|
||
case, no pending record is written. The Sources page counts the
|
||
KB: 2 docs, 4 chunks (one content + one summary chunk per doc)."""
|
||
with SessionLocal() as db:
|
||
doc = db.scalar(
|
||
select(Document).where(
|
||
Document.source == SOURCE_NAME, Document.path == DOC_PATH
|
||
)
|
||
)
|
||
assert doc is not None
|
||
# The stored context (task 05's display source).
|
||
assert doc.image_context == BLOCK
|
||
# The enriched content — context FIRST, then the description —
|
||
# IS the summary (the phase-122 image convention).
|
||
assert doc.content == ENRICHED
|
||
assert doc.summary == ENRICHED
|
||
chunks = db.scalars(select(Chunk).where(Chunk.document_id == doc.id)).all()
|
||
content_chunks = [c for c in chunks if not c.is_summary]
|
||
assert [c.content for c in content_chunks] == [PACKED], (
|
||
"the content chunk is embedded from the ENRICHED text"
|
||
)
|
||
summary_chunks = [c for c in chunks if c.is_summary]
|
||
assert len(summary_chunks) == 1 and summary_chunks[0].position == -1
|
||
assert summary_chunks[0].content == ENRICHED
|
||
for c in chunks:
|
||
assert c.embedding is not None and len(c.embedding) == 768
|
||
|
||
# The referring markdown — the image doc existed at its
|
||
# indexing (sort order), so NO pending record was written.
|
||
md = db.scalar(
|
||
select(Document).where(
|
||
Document.source == SOURCE_NAME, Document.path == MD_PATH
|
||
)
|
||
)
|
||
assert md is not None and md.is_image is False
|
||
assert md.image_context is None, "existing image: the record stays NULL"
|
||
|
||
# The KB on the Sources page: 2 docs (the markdown + the image),
|
||
# 4 chunks (one content + one is_summary per doc).
|
||
login(page, app_url)
|
||
expect(page.locator("#stat-docs")).to_have_text("2")
|
||
expect(page.locator("#stat-chunks")).to_have_text("4")
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# 2. Chat: the enriched image doc's <document> block leads with the
|
||
# context line — the answer AND the wire prompt prove it
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def test_chat_answer_quotes_the_context_line(
|
||
page: Page, app_url: str, mock_llm: int, seeded: dict[str, str]
|
||
) -> None:
|
||
"""LOCKED A4 end-to-end: a grounded turn whose suggested tier
|
||
carries the ENRICHED image doc → the ``<document>`` block LEADS
|
||
with the ONE-LINE context line (the task-05 rendering of the
|
||
stored block: the referring markdown's identity + the reference's
|
||
alt, BEFORE the summary — for an enriched image doc the summary
|
||
is the enriched content, so the stored block rides in the body
|
||
too, but the header is the one-line rendering, never the stored
|
||
multi-line header). The mock's marker branch QUOTES THE EXACT
|
||
LINE of the block — so the rendered answer proves the one-line
|
||
rendering was in the prompt (the quoted alt sits on the header
|
||
line — the stored block's shape carries it on the next line as a
|
||
bullet), and the captured wire prompt pins the exact block shape:
|
||
line, blank line, enriched summary."""
|
||
_reset_capture(mock_llm)
|
||
_ask(page, app_url, ANCHOR_QUESTION)
|
||
|
||
bubble = page.locator(".msg.brain .bubble").last
|
||
# The mock's enriched answer — the EXACT context line, quoted.
|
||
expect(bubble).to_contain_text(ANSWER_WITH_LINE, timeout=30_000)
|
||
# Grounded (not deflected) — a deflected prompt would carry no
|
||
# <documents> block at all.
|
||
expect(bubble).not_to_contain_text(DEFLECT_OPENING)
|
||
|
||
# The wire prompt: exactly one chat request (grounded, zero tool
|
||
# calls), and the image doc's <document> block leads with the
|
||
# ONE-LINE context line — then a blank line, then the summary
|
||
# (for an enriched image doc the summary IS the enriched content —
|
||
# the phase-122 ``summary = content`` convention — so the stored
|
||
# block rides along in the body, but the block HEADS with the
|
||
# one-line rendering, never the stored header line).
|
||
systems = _captured_systems(mock_llm)
|
||
assert len(systems) == 1, "one grounded zero-read turn → one chat request"
|
||
system = systems[0]
|
||
start = system.index(f'<document source="{SOURCE_NAME}" path="{DOC_PATH}"')
|
||
body_start = system.index(">\n", start) + 2
|
||
body_end = system.index("\n</document>", body_start)
|
||
block_body = system[body_start:body_end]
|
||
# The exact block shape: the one-line context line (the alt QUOTED
|
||
# on the same line — the stored block's shape carries the alt on
|
||
# the NEXT line as a bullet, so this pins the task-05 rendering),
|
||
# a blank line, then the enriched summary.
|
||
assert block_body == f"{CONTEXT_LINE}\n\n{ENRICHED}"
|
||
# The line is inside <documents>, before the summary's block.
|
||
assert system.index("<documents>") < system.index(CONTEXT_LINE)
|
||
assert system.index(CONTEXT_LINE) < system.index("\n" + BLOCK)
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# 3. Standalone image: the byte-identical pre-phase row
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def test_standalone_image_keeps_null_context_and_bare_description(
|
||
page: Page, app_url: str, standalone: dict[str, str]
|
||
) -> None:
|
||
"""LOCKED A4 (the byte-identity criterion) on the ROW: a standalone
|
||
image (no markdown in the source references it) → ``image_context``
|
||
NULL, ``content`` (AND ``summary``) the BARE vision description,
|
||
every chunk the bare description — exactly the phase-122 shape.
|
||
The Sources page counts 1 doc / 2 chunks (one content + one
|
||
summary)."""
|
||
with SessionLocal() as db:
|
||
doc = db.scalar(
|
||
select(Document).where(
|
||
Document.source == STANDALONE_SOURCE_NAME, Document.path == LOGO_PATH
|
||
)
|
||
)
|
||
assert doc is not None
|
||
assert doc.image_context is None, "standalone: the context stays NULL"
|
||
assert doc.content == IMAGE_DESCRIPTION_ANSWER, "standalone: the bare description"
|
||
assert doc.summary == IMAGE_DESCRIPTION_ANSWER
|
||
chunks = db.scalars(select(Chunk).where(Chunk.document_id == doc.id)).all()
|
||
assert len(chunks) == 2
|
||
assert all(c.content == IMAGE_DESCRIPTION_ANSWER for c in chunks)
|
||
for c in chunks:
|
||
assert c.embedding is not None and len(c.embedding) == 768
|
||
|
||
login(page, app_url)
|
||
expect(page.locator("#stat-docs")).to_have_text("1")
|
||
expect(page.locator("#stat-chunks")).to_have_text("2")
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# 4. Standalone image: the <document> block is byte-identical in the
|
||
# prompt — no context line, the bare description alone
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def test_standalone_answer_carries_no_context_line(
|
||
page: Page, app_url: str, mock_llm: int, standalone: dict[str, str]
|
||
) -> None:
|
||
"""LOCKED A4 (the byte-identity criterion) in the PROMPT: the SAME
|
||
marker question against the standalone KB → the image doc's
|
||
``<document>`` block body is EXACTLY the bare description (no
|
||
context line, no stray blank — the pre-phase block), and the mock
|
||
— which quotes the block's first line when it is a ``Referenced
|
||
in`` line — answers that the image carries no reference line: the
|
||
deterministic witness that the line was NOT in the prompt.
|
||
``Referenced in`` does not appear ANYWHERE in the system prompt."""
|
||
_reset_capture(mock_llm)
|
||
_ask(page, app_url, ANCHOR_QUESTION)
|
||
|
||
bubble = page.locator(".msg.brain .bubble").last
|
||
expect(bubble).to_contain_text(ANSWER_STANDALONE, timeout=30_000)
|
||
expect(bubble).not_to_contain_text(DEFLECT_OPENING)
|
||
|
||
systems = _captured_systems(mock_llm)
|
||
assert len(systems) == 1, "one grounded zero-read turn → one chat request"
|
||
system = systems[0]
|
||
start = system.index(
|
||
f'<document source="{STANDALONE_SOURCE_NAME}" path="{LOGO_PATH}"'
|
||
)
|
||
body_start = system.index(">\n", start) + 2
|
||
body_end = system.index("\n</document>", body_start)
|
||
block_body = system[body_start:body_end]
|
||
# The pre-phase block body: the description alone, verbatim.
|
||
assert block_body == IMAGE_DESCRIPTION_ANSWER
|
||
assert "Referenced in" not in system
|