phase: 124_image_chips_and_answers
All green. Final report: **Phase 124 final verification pass — all criteria verified, no defects found (nothing to fix)** - Verified the implemented phase (all 7 tasks done, changes in working tree): `BOR_ANSWER_IMAGES` toggle, `show_image` tool + refusals + `holder.shown_images`, toggle-gated prompt clause, `answer_image` frame field (None-omitting serializer) + `ChatMessage` persistence, `merge_cited_docs` auto-cite bug fix, answer-figure UI (live/restore/shared/CSS), mock triggers + E2E suite. Also confirmed the bytes route → public change matches the phase-123 shared-chat precedent (task 06 design). - `uv run pytest --cov=app --cov-report=term-missing` → **2848 passed**, TOTAL **99%** (>90% gate) - `bash .agents/validate.sh` → **validation OK** (tests + coverage + ruff + pyright) - `uv run ruff check . && uv run pyright` → clean, 0 errors - `uv run pytest tests/e2e/test_answer_images.py -v --no-cov` (isolated) → **5/5 passed** - Mock-inert spot checks: `tests/e2e/test_image_documents.py` 4/4, `tests/e2e/test_agent_document_tools.py` 4/4 **Completion criteria:** ① zero-read suggested-tier image doc chips with figure — verified (unit matrix, integration `test_zero_read_grounded_turn_auto_cites_suggested_image_doc`, E2E bug-fix flows); ② show_image → answer figure below text/above chips, second call refused, persists + restore + shared — verified (unit handler matrix, integration round-trip, E2E feature flow); ③ default-off byte-identical (3 tools, no clause, no key) — verified (unit + integration + E2E toggle-off); ④ full suite green / >90% / lint clean — verified above; ⑤ commit + phase move — harness responsibility (changes left uncommitted in working tree, no live-infrastructure changes made). **Next pending phase:** none — `todo/` holds only this phase; the pipeline is clear once the harness moves it.
This commit is contained in:
@@ -467,6 +467,56 @@ Implements just enough of the aipi surface:
|
||||
the trigger phrase is disjoint from every other trigger's; verified
|
||||
2026-09-16: no existing E2E question or fixture file contains the
|
||||
phrase, so every other suite is unaffected.
|
||||
- user message containing ``show me the picture of``
|
||||
(``SHOW_IMAGE_TRIGGER``, phase 124 task 07 — the image-chips-and-
|
||||
answers suite ``tests/e2e/test_answer_images.py``) **and** the
|
||||
system prompt contains ``show_image`` (the tool clause's presence
|
||||
— the ``BOR_ANSWER_IMAGES`` toggle's server-side marker: the
|
||||
clause rides inside the HIGH prompt's ``<tools>`` section ONLY
|
||||
when the toggle is on, so on a default-OFF app the trigger NEVER
|
||||
fires and the mock behaves exactly as pre-phase for every other
|
||||
suite) -> the deterministic SHOW_IMAGE flow, discriminated
|
||||
statelessly from the tool results (streaming only): the mock
|
||||
picks the IMAGE document seeded in the ``<documents>`` blocks —
|
||||
the block whose ``path`` attribute ends in an image extension
|
||||
(``app.rag.summarizer.IMAGE_MIMES``' family — the suite's
|
||||
single-PNG fixture guarantees exactly one such block):
|
||||
* request 1 (``tools`` offered, no confirmed show yet): stream
|
||||
ONLY ``tool_calls`` deltas — ``show_image`` with
|
||||
``{"path": <combined source/path>}`` (synthetic id
|
||||
``call_show``, ``finish_reason: "tool_calls"``, no content) —
|
||||
NO ``read`` (the ZERO-READ turn: the image doc's seed block
|
||||
already carries its entire content, so a real model never
|
||||
reads it — the zero-read shape is what makes the bug-fix's
|
||||
chip assertion meaningful);
|
||||
* the follow-up request (a ``tool``-role result carrying the
|
||||
agent's confirming line ``Image shown to the user`` —
|
||||
``_SHOW_CONFIRM_MARKER``) in the messages: the deterministic
|
||||
FINAL answer — a short sentence ending with the ``<documents>``
|
||||
block's tail echo (the phase-24 END_OF_NOTES echo convention —
|
||||
it proves the seed was there), NO further tool calls.
|
||||
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint trigger
|
||||
phrase — the phase-71/72/94/118 ordering convention); verified
|
||||
2026-09-26: no existing E2E question or fixture file contains the
|
||||
phrase, so every other suite is unaffected.
|
||||
- user message containing ``which diagram is in your notes``
|
||||
(``DIAGRAM_TRIGGER``, phase 124 task 07 — the BUG-FIX-ONLY
|
||||
trigger, same suite) -> the deterministic ZERO-READ answer: the
|
||||
mock answers straight from the seed with NO tool calls at all
|
||||
(no ``show_image``, no ``read`` — unlike the ``SHOW_IMAGE_
|
||||
TRIGGER`` flow it needs NO clause, so it also fires on a default-
|
||||
OFF app: the chips fix is not gated by ``BOR_ANSWER_IMAGES``,
|
||||
it belongs to the ``BOR_IMAGES``-indexed docs) and QUOTES THE
|
||||
TAIL of the seeded image doc's ``<document>`` block (the same
|
||||
image-doc pick as above): proving the description WAS in the
|
||||
prompt (for an image doc the seed block IS its entire content)
|
||||
and that its chip appears with NO tool calls at all (the phase-
|
||||
124 auto-cite — the fix for the owner's report). Checked BEFORE
|
||||
the ``DEFLECT_MODE`` branch (the marker lives in the user message
|
||||
— the ``TABLE_TRIGGER``/``HISTORY_TRIGGER`` convention), so a
|
||||
marker question always gets the echo whatever the honesty gate
|
||||
says. Verified 2026-09-26: no existing E2E question or fixture
|
||||
file contains the phrase, so every other suite is unaffected.
|
||||
|
||||
Failure injection (phase 67, LLM retry, TODO.md L3) — deterministic
|
||||
dead-endpoint behavior for the retry E2E suite (``tests/e2e/
|
||||
@@ -580,6 +630,7 @@ from fastapi.responses import JSONResponse, StreamingResponse
|
||||
|
||||
from app.rag.agent import ALREADY_IN_CONTEXT, CORRECTION_INSTRUCTION
|
||||
from app.rag.folder_summaries import FOLDER_HEADER_PREFIX # phase 94: the mock's key
|
||||
from app.rag.summarizer import IMAGE_MIMES # phase 124: the image-doc extension family
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
@@ -1756,6 +1807,117 @@ _SUMMARY_SEED_READ_CALL_RE = re.compile(
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 124 (task 07, image chips and answers, TODO L3): the scripted
|
||||
# SHOW_IMAGE flow + the zero-read bug-fix trigger — see the module
|
||||
# docstring
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: A user message containing this substring (case-insensitive) —
|
||||
#: combined with the ``show_image`` prompt clause in the system
|
||||
#: prompt (the ``BOR_ANSWER_IMAGES`` toggle's server-side marker —
|
||||
#: the clause rides inside the HIGH prompt's ``<tools>`` section ONLY
|
||||
#: when the toggle is on) — drives the deterministic SHOW_IMAGE flow
|
||||
#: (the mock shows the seeded image document and answers straight
|
||||
#: from the seed — ZERO ``read`` calls). On a default-OFF app the
|
||||
#: clause is absent, so the trigger NEVER fires there and the mock
|
||||
#: behaves exactly as pre-phase for every other suite. Disjoint from
|
||||
#: every other trigger phrase — the phase-71/72/94/118 ordering
|
||||
#: convention; verified 2026-09-26: no existing E2E question or
|
||||
#: fixture file contains the phrase, so every other suite is
|
||||
#: unaffected.
|
||||
SHOW_IMAGE_TRIGGER = "show me the picture of"
|
||||
|
||||
#: The phase-124 BUG-FIX-ONLY trigger (case-insensitive): the mock
|
||||
#: answers straight from the seed with ZERO tool calls and QUOTES THE
|
||||
#: TAIL of the seeded image doc's ``<document>`` block — proving the
|
||||
#: description was in the prompt (for an image doc the seed block IS
|
||||
#: its entire content) and that its chip appears with NO tool calls
|
||||
#: at all (the phase-124 auto-cite). Unlike ``SHOW_IMAGE_TRIGGER``
|
||||
#: it needs NO clause — the chips fix is not gated by
|
||||
#: ``BOR_ANSWER_IMAGES`` (it belongs to the ``BOR_IMAGES``-indexed
|
||||
#: docs), so it also fires on a default-OFF app. Checked BEFORE the
|
||||
#: ``DEFLECT_MODE`` branch in :func:`compose_answer` (the marker
|
||||
#: lives in the user message — the TABLE/HISTORY convention), so a
|
||||
#: marker question always gets the echo whatever the honesty gate
|
||||
#: says.
|
||||
DIAGRAM_TRIGGER = "which diagram is in your notes"
|
||||
|
||||
#: The agent's ``show_image`` SUCCESS result's confirming first line
|
||||
#: (``app.rag.agent``'s ``show_image`` branch — the line ``read``'s
|
||||
#: result never carries): the SHOW_IMAGE flow's follow-up request
|
||||
#: keys on it in a ``tool``-role result (stateless discrimination —
|
||||
#: the house marker-flow convention).
|
||||
_SHOW_CONFIRM_MARKER = "Image shown to the user"
|
||||
|
||||
#: The image-doc extension family (``app.rag.summarizer.IMAGE_MIMES``
|
||||
#: — phase 122's ext→mime map, imported so the mock can never drift
|
||||
#: from the app's image family): a seeded ``<documents>`` block is
|
||||
#: the IMAGE document when its ``path`` attribute ends in one of
|
||||
#: these (the suite's single-PNG fixture guarantees exactly one such
|
||||
#: block — the pick is deterministic).
|
||||
_IMAGE_PATH_SUFFIXES = tuple(IMAGE_MIMES)
|
||||
|
||||
|
||||
def _seeded_image_doc(system: str) -> tuple[str, str, str] | None:
|
||||
"""The seeded image doc's ``(source, path, content)`` — or ``None``.
|
||||
|
||||
Phase 124 (task 07): the ``<documents>`` block (the
|
||||
:data:`_DOCUMENT_BLOCK_RE` machinery) whose ``path`` attribute
|
||||
ends in an image extension (:data:`_IMAGE_PATH_SUFFIXES`) is the
|
||||
seeded image document — for an image doc the block's content IS
|
||||
the document's entire text (the vision description — content ==
|
||||
summary, phase 122 task 03). Both of this phase's triggers pick
|
||||
the doc this way (the suite's single-PNG fixture guarantees
|
||||
exactly one such block; ``None`` = a text-only KB, the premise
|
||||
broke — the caller degrades loudly).
|
||||
"""
|
||||
for block in _DOCUMENT_BLOCK_RE.finditer(system):
|
||||
if block.group("path").lower().endswith(_IMAGE_PATH_SUFFIXES):
|
||||
return (block.group("source"), block.group("path"), block.group("content"))
|
||||
return None
|
||||
|
||||
|
||||
def _show_image_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
"""Classify a phase-124 SHOW_IMAGE request (see the module
|
||||
docstring). Stateless over the messages, like the other marker
|
||||
flows:
|
||||
|
||||
* ``("call", combined)`` — the system prompt carries the
|
||||
``show_image`` clause (the toggle-ON gate) and no ``tool``-role
|
||||
result has confirmed the show yet: stream ONLY the
|
||||
``show_image`` tool-call delta on the seeded image doc's
|
||||
combined ``source/path`` (synthetic id ``call_show``,
|
||||
``finish_reason: "tool_calls"``, no content) — NO ``read`` (the
|
||||
zero-read turn).
|
||||
* ``("answer",)`` — a ``tool``-role result carrying the agent's
|
||||
confirming line (:data:`_SHOW_CONFIRM_MARKER`) is in the
|
||||
messages: the deterministic FINAL answer — a short sentence
|
||||
ending with the ``<documents>`` block's tail echo (the phase-24
|
||||
END_OF_NOTES echo convention — it proves the seed was there),
|
||||
NO further tool calls.
|
||||
* ``None`` — not the flow: the trigger phrase is absent, the
|
||||
clause is absent (a default-OFF app — the mock behaves exactly
|
||||
as pre-phase), the turn is deflected (no ``<documents>`` seed
|
||||
— a deflection prompt never carries one), or no seeded image
|
||||
doc exists (a text-only KB — the premise broke; the suite fails
|
||||
loudly on the missing figure).
|
||||
"""
|
||||
if SHOW_IMAGE_TRIGGER not in _user(body).lower():
|
||||
return None
|
||||
system = _system(body)
|
||||
if "show_image" not in system:
|
||||
return None
|
||||
doc = _seeded_image_doc(system)
|
||||
if doc is None:
|
||||
return None
|
||||
if any(_SHOW_CONFIRM_MARKER in r for r in _tool_results(body)):
|
||||
return ("answer",)
|
||||
if not body.get("tools"):
|
||||
return None
|
||||
return ("call", f"{doc[0]}/{doc[1]}")
|
||||
|
||||
|
||||
def _summary_seed_read_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
"""Classify a phase-118 scripted summary-read request (see the
|
||||
module docstring). The question carries the scripted call (``read
|
||||
@@ -2280,6 +2442,31 @@ def compose_answer(body: dict[str, Any]) -> str:
|
||||
# message, a deflection prompt never carries it, so a marker
|
||||
# question always gets the echo whatever the gate says.
|
||||
answer = _history_echo(body)
|
||||
elif DIAGRAM_TRIGGER in user.lower():
|
||||
# Phase 124 (task 07): the BUG-FIX-ONLY trigger — the mock
|
||||
# answers straight from the seed with ZERO tool calls (no
|
||||
# ``show_image``, no ``read`` — the clause is NOT required,
|
||||
# so the flow also fires on a default-OFF app: the chips fix
|
||||
# belongs to the ``BOR_IMAGES``-indexed docs) and quotes the
|
||||
# TAIL of the seeded image doc's ``<document>`` block (the
|
||||
# same pick as the SHOW_IMAGE flow — the block's content is
|
||||
# the image doc's entire text, the vision description):
|
||||
# proving the description WAS in the prompt and that the chip
|
||||
# appears with NO tool calls at all (the phase-124 auto-cite
|
||||
# — the fix for the owner's report). Checked BEFORE the
|
||||
# DEFLECT_MODE branch (the marker lives in the user message —
|
||||
# the TABLE/HISTORY convention), so a marker question always
|
||||
# gets the echo whatever the honesty gate says. A text-only KB
|
||||
# (no seeded image doc) degrades to the plain line — the
|
||||
# suite's premise broke; it fails loudly on the missing quote.
|
||||
image_doc = _seeded_image_doc(system)
|
||||
answer = (
|
||||
f"Your notes carry that diagram — it reads: “{image_doc[2][-160:]}” "
|
||||
"(Deterministic mock answer for E2E.)"
|
||||
if image_doc
|
||||
else "Your notes don't carry that diagram — there is nothing to show. "
|
||||
"(Deterministic mock answer for E2E.)"
|
||||
)
|
||||
elif "DEFLECT_MODE" in system:
|
||||
answer = (
|
||||
"Ah — I haven't done anything like that, so I don't want to make stuff up! "
|
||||
@@ -2911,6 +3098,42 @@ def chat_completions(body: dict[str, Any]) -> Any:
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 124 (task 07, image chips and answers, TODO L3): the
|
||||
# scripted SHOW_IMAGE flow — the mock shows the seeded image
|
||||
# document (the ``<documents>`` block with the image-
|
||||
# extension path) and answers straight from the seed: ZERO
|
||||
# ``read`` calls (the zero-read turn that makes the bug-fix's
|
||||
# chip assertion meaningful). Clause-gated — the ``show_image``
|
||||
# teaching line in the system prompt is the
|
||||
# ``BOR_ANSWER_IMAGES`` toggle's presence marker, so on a
|
||||
# default-OFF app the trigger never fires and the mock stays
|
||||
# byte-identical to pre-phase for every other suite. Checked
|
||||
# BEFORE the plain TOOLS_TRIGGER flow (disjoint trigger
|
||||
# phrase — the phase-71/72/94/118 ordering convention; the
|
||||
# trigger needs the clause, so deflected turns never hit it).
|
||||
show_image = _show_image_flow(body)
|
||||
if show_image is not None:
|
||||
if show_image[0] == "call":
|
||||
stream = _tool_call_stream(
|
||||
"show_image", {"path": show_image[1]}, "call_show"
|
||||
)
|
||||
else: # "answer" — the final answer + the tail echo
|
||||
block = _DOCUMENTS_BLOCK_RE.search(_system(body))
|
||||
tail = (block.group(0) if block else _context(body))[-160:]
|
||||
stream = _sse_stream(
|
||||
_apply_max_tokens(
|
||||
"Here is the picture you asked about — …and the very "
|
||||
f"end of my notes reads: “{tail}” "
|
||||
"(Deterministic mock answer for E2E.)",
|
||||
body.get("max_tokens"),
|
||||
),
|
||||
0.0,
|
||||
)
|
||||
return StreamingResponse(
|
||||
stream,
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 109 (task 03, never-frozen-turn story suite): the
|
||||
# deterministic REPORTED-REPRO turn (delta → tool →
|
||||
# thinking-after-delta — TODO.md L3): the scripted first answer,
|
||||
|
||||
@@ -0,0 +1,971 @@
|
||||
"""Phase 124 E2E (Playwright): image source chips + LLM-chosen answer images.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_answer_images.py -v --no-cov
|
||||
|
||||
The suite proves BOTH halves of TODO.md L3 on the real user path, plus
|
||||
the toggle-off contract — two module apps on dedicated ports (the
|
||||
phase-122 ``test_image_documents.py`` pattern), sharing one deterministic
|
||||
mock LLM and the shared e2e Postgres (truncate/re-import per app):
|
||||
|
||||
* **App A (feature ON)** — ``BOR_IMAGES=true`` +
|
||||
``BOR_ANSWER_IMAGES=true``: the standalone PNG arrives through the
|
||||
REAL admin upload flow + the RAG page's "Sync sources" (the mock's
|
||||
vision model writes the byte-stable description), then:
|
||||
- **Feature flow** — the ``show me the picture of …`` trigger (the
|
||||
mock's phase-124 scripted trigger, ``tests/e2e/mock_llm.py``): the
|
||||
mock scripts ONLY a ``show_image`` call (NO ``read`` — the ZERO-READ
|
||||
turn, the shape that makes the chip assertion meaningful). The
|
||||
brain bubble shows the ANSWER FIGURE (``.answer-image`` — ``<img>``
|
||||
from the bytes route, ``loading="lazy"``, below the text and above
|
||||
the chip row per A4, alt/caption settling to the summary), the
|
||||
answer TEXT is still streamed in full (the tail echo of the
|
||||
``<documents>`` seed proves the description was in the prompt), the
|
||||
source-chip row carries the image doc's chip + its phase-122
|
||||
figure (the AUTO-CITE proof — the mock made zero ``read`` calls,
|
||||
asserted on the mock's request capture), and the "Nearby docs" row
|
||||
does not repeat the doc. The wire pin: the first model request
|
||||
offered FOUR tools (the three base + ``show_image``) and the turn
|
||||
executed exactly one ``show_image`` — the saved brain record
|
||||
persists the ``answer_image`` ref.
|
||||
- **Restore + share** — a second feature turn, then a page RELOAD:
|
||||
the restored brain bubble shows the figure again (the
|
||||
``answer_image`` persistence through the ``bor.chat.v1`` restore
|
||||
path), and the shared link (the existing chat-page Share flow, a
|
||||
fresh anonymous context) renders the figure on the shared page.
|
||||
- **Bug-fix-only flow** — the ``which diagram is in your notes``
|
||||
trigger (the mock answers straight from the seed with ZERO tool
|
||||
calls — no ``show_image``, no ``read``): the source-chip row STILL
|
||||
carries the image doc's chip + figure (the bug fix on the pure
|
||||
zero-read path — the assertion the owner's report describes), the
|
||||
answer text quotes the seeded description's tail, and NO answer
|
||||
figure renders (the model never called the tool).
|
||||
* **App B (default off)** — ``BOR_IMAGES=true``, ``BOR_ANSWER_IMAGES``
|
||||
pinned to the code default ``false`` (EXPLICITLY — process env ranks
|
||||
above an operator's local gitignored ``.env``, so the default-off
|
||||
contract under test cannot leak in, the phase-122 default-app
|
||||
convention; the production default is unset = the same ``False``):
|
||||
the same feature trigger question → the clause is absent from the
|
||||
prompt, so the mock's trigger NEVER fires — the mock never sees
|
||||
``show_image`` among the offered tools (the request capture carries
|
||||
exactly the THREE base tools), the brain bubble has NO answer
|
||||
figure, and the saved brain record carries NO ``answer_image`` key
|
||||
(the byte-identical-off contract). The bug-fix trigger STILL chips
|
||||
the image doc (the chips fix is NOT gated by ``BOR_ANSWER_IMAGES`` —
|
||||
it belongs to the ``BOR_IMAGES``-indexed docs, per the phase design).
|
||||
|
||||
DB isolation: the suites share one Postgres; ``saved_chats`` survives
|
||||
the truncation (the house pattern), so every question text here is
|
||||
UNIQUE (its auto-title is therefore unique) and the row lookups key on
|
||||
it. App B's module seed truncates the KB tables App A's tests used —
|
||||
App B's tests run AFTER App A's (file order) and never look back.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import zipfile
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Browser, Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
|
||||
from e2e.mock_llm import IMAGE_DESCRIPTION_ANSWER
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
GIT_SOURCES_URL = "/git-sources.html"
|
||||
SOURCE_NAME = "e2e-answer-img" # the archive stem (archive_source_name)
|
||||
DOC_PATH = "pic.png"
|
||||
|
||||
#: The mock's deterministic answer tail (every composed answer ends
|
||||
#: with it — the turn-settled wait signal, the house convention).
|
||||
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
|
||||
#: The TAIL of the fixture description (``IMAGE_DESCRIPTION_ANSWER`` —
|
||||
#: the image doc's entire content): both phase-124 mock answers quote
|
||||
#: it (the feature flow's ``<documents>`` tail echo, the bug-fix flow's
|
||||
#: block-content tail) — asserting it proves the SEED was in the
|
||||
#: prompt and reached the rendered answer.
|
||||
DESCRIPTION_TAIL = "Title: Homelab Network Map."
|
||||
|
||||
#: App A (feature ON) — the ``show_image`` trigger question (cosine
|
||||
#: 0.4552 against the mock's token-overlap embeddings, well over the
|
||||
#: mock-calibrated 0.30 gate). Unique auto-title (the DB is shared).
|
||||
FEATURE_QUESTION = "Show me the picture of the homelab network diagram?"
|
||||
#: The restore/share flow's variant (the trigger substring is kept —
|
||||
#: cosine 0.4358, still over the gate).
|
||||
RESTORE_QUESTION = "Show me the picture of the homelab network diagram? (restore)"
|
||||
#: App B (default off) — the feature trigger's variant (the clause is
|
||||
#: absent, so the mock answers with the plain composed answer — cosine
|
||||
#: 0.4187, the turn still grounds).
|
||||
FEATURE_OFF_QUESTION = (
|
||||
"Show me the picture of the homelab network diagram? (toggle off)"
|
||||
)
|
||||
#: The bug-fix-only trigger question (cosine 0.3410): the mock answers
|
||||
#: straight from the seed with ZERO tool calls and quotes the image
|
||||
#: doc's block tail — the chip must appear anyway (the auto-cite).
|
||||
DIAGRAM_QUESTION = "Which diagram is in your notes about the homelab network?"
|
||||
#: App B's bug-fix variant (the chips fix is not toggle-gated — cosine
|
||||
#: 0.3113, still over the mock-calibrated gate).
|
||||
DIAGRAM_OFF_QUESTION = (
|
||||
"Which diagram is in your notes about the homelab network? (default off)"
|
||||
)
|
||||
|
||||
#: A real 1×1 transparent PNG (the unit/integration suites' fixture —
|
||||
#: the pipeline is content-agnostic, the well-formed bytes keep the
|
||||
#: upload + serve + render path honest).
|
||||
PNG_1X1 = base64.b64decode(
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
|
||||
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
|
||||
)
|
||||
|
||||
UPLOAD_TIMEOUT_MS = 30_000
|
||||
SYNC_TIMEOUT_MS = 60_000
|
||||
SYNCED_LABEL = re.compile(r"^Synced \d{1,2}:\d{2}$")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Module apps: feature ON (App A) and default-OFF (App B). Each owns
|
||||
# its port + its scratch upload/source/image dirs.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _app_env(
|
||||
mock_llm: int, app_port: int, *, answer_images: str, scratch: Path
|
||||
) -> dict[str, str]:
|
||||
"""The conftest app env (leak guards included) with the phase-124
|
||||
knobs: BOTH apps index images (``BOR_IMAGES=true`` — the fixture's
|
||||
PNG must become a document on either), and ``BOR_ANSWER_IMAGES``
|
||||
is pinned EXPLICITLY per app (``true`` for App A, ``false`` for
|
||||
App B — process env ranks above an operator's local gitignored
|
||||
``.env``, so the contract under test cannot leak in either
|
||||
direction, the phase-122 default-app convention; App B's explicit
|
||||
``false`` IS the production default — unset is ``False``). The
|
||||
image/upload homes are in the suite's scratch dir (the app under
|
||||
test must not write image copies into the owner's real
|
||||
``~/bor-sources``)."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated gate (conftest pattern) — both trigger questions
|
||||
# ground on these values (see the question constants above).
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||||
# Phase 67: instant retry waits + the code-default budget.
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# Leak guards (conftest pattern).
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default
|
||||
# Phase 124: both apps index images; only the ANSWER side differs.
|
||||
env["BOR_IMAGES"] = "true"
|
||||
env["BOR_ANSWER_IMAGES"] = answer_images
|
||||
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
|
||||
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
|
||||
env["BOR_IMAGE_DIR"] = str(scratch / "images")
|
||||
return env
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int, tmp_path_factory: pytest.TempPathFactory) -> Iterator[str]:
|
||||
"""App A (feature ON) — ``BOR_IMAGES=true`` +
|
||||
``BOR_ANSWER_IMAGES=true`` (the conftest session app is never
|
||||
started in this isolated run, so no port clash)."""
|
||||
scratch = tmp_path_factory.mktemp("bor_answer_img_on")
|
||||
port = int(os.environ.get("E2E_APP_PORT_ANSWER_IMG_ON", "8162"))
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=_app_env(mock_llm, port, answer_images="true", scratch=scratch),
|
||||
)
|
||||
try:
|
||||
_wait_http(f"http://127.0.0.1:{port}/api/health")
|
||||
yield f"http://127.0.0.1:{port}"
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def default_app_server(
|
||||
mock_llm: int, tmp_path_factory: pytest.TempPathFactory
|
||||
) -> Iterator[str]:
|
||||
"""App B (default off) — ``BOR_IMAGES=true`` +
|
||||
``BOR_ANSWER_IMAGES=false`` pinned explicitly (the production
|
||||
default — the toggle-off contract; only the negative tests start
|
||||
it, and they run AFTER App A's — file order)."""
|
||||
scratch = tmp_path_factory.mktemp("bor_answer_img_off")
|
||||
port = int(os.environ.get("E2E_APP_PORT_ANSWER_IMG_OFF", "8163"))
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=_app_env(mock_llm, port, answer_images="false", scratch=scratch),
|
||||
)
|
||||
try:
|
||||
_wait_http(f"http://127.0.0.1:{port}/api/health")
|
||||
yield f"http://127.0.0.1:{port}"
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def default_app_url(default_app_server: str) -> str:
|
||||
return default_app_server
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def zip_path(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The fixture archive: ONE standalone PNG at the root — source
|
||||
``e2e-answer-img``, document path ``pic.png`` (the mock's seeded-
|
||||
image-doc pick is deterministic: the only image-extension block)."""
|
||||
root = tmp_path_factory.mktemp("bor_answer_img_zip")
|
||||
(root / DOC_PATH).write_bytes(PNG_1X1)
|
||||
archive = root / f"{SOURCE_NAME}.zip"
|
||||
with zipfile.ZipFile(archive, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
zf.write(root / DOC_PATH, arcname=DOC_PATH)
|
||||
return archive
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# DB + UI + mock-capture helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh KB + registry (the E2E isolation pattern): the suites
|
||||
share one Postgres, and a leftover row would corrupt the counts.
|
||||
``sources_meta`` (the KB generation counter) resets with the KB —
|
||||
the sync re-creates the row. ``saved_chats`` is never touched by
|
||||
the truncation — the unique question texts keep the row lookups
|
||||
unambiguous, and :func:`_cleanup_saved_rows` owns this suite's
|
||||
rows (re-runnable on the shared DB)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text(
|
||||
"TRUNCATE chunks, documents, query_log, kb_overview, "
|
||||
"git_sources, sources_meta, folder_summaries"
|
||||
))
|
||||
db.commit()
|
||||
|
||||
|
||||
def _health_db(app_url: str) -> bool:
|
||||
try:
|
||||
return httpx.get(f"{app_url}/api/health", timeout=5).json()["db"] == "up"
|
||||
except Exception: # noqa: BLE001 — unreachable is the skip case
|
||||
return False
|
||||
|
||||
|
||||
def _mock_capture(mock_llm: int) -> list[dict[str, Any]]:
|
||||
"""The mock's recent chat-completions request bodies (oldest
|
||||
first) — the wire lens on the app's tool OFFER and the turn's
|
||||
tool CALLS (the mock is the only E2E lens on the LLM's context)."""
|
||||
r = httpx.get(f"http://127.0.0.1:{mock_llm}/v1/e2e/captured", timeout=5)
|
||||
assert r.status_code == 200
|
||||
return r.json()
|
||||
|
||||
|
||||
def _mock_capture_reset(mock_llm: int) -> None:
|
||||
r = httpx.post(f"http://127.0.0.1:{mock_llm}/v1/e2e/captured/reset", timeout=5)
|
||||
assert r.status_code == 200
|
||||
|
||||
|
||||
def _executed_tool_calls(captured: list[dict[str, Any]]) -> list[str]:
|
||||
"""Every tool call the MODEL made across *captured* requests (the
|
||||
assistant ``tool_calls`` messages), in order — the executed-call
|
||||
lens (a refusal still appears here; it is the model's call)."""
|
||||
return [
|
||||
tc["function"]["name"]
|
||||
for body in captured
|
||||
for m in body["messages"]
|
||||
if m.get("role") == "assistant" and m.get("tool_calls")
|
||||
for tc in m["tool_calls"]
|
||||
]
|
||||
|
||||
|
||||
def _offered_tool_names(body: dict[str, Any]) -> list[str]:
|
||||
return [t["function"]["name"] for t in (body.get("tools") or [])]
|
||||
|
||||
|
||||
def _upload_zip(page: Page, app_url: str, archive: Path) -> None:
|
||||
"""The real admin upload flow: form login → the git-sources page →
|
||||
pick the archive → submit → the terminal result line (phase 90:
|
||||
"Uploaded <source> — press Sync sources to import it.")."""
|
||||
login(page, app_url, next=GIT_SOURCES_URL)
|
||||
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
|
||||
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
|
||||
page.set_input_files("#archive-upload-file", str(archive))
|
||||
page.click("#archive-upload-btn")
|
||||
result = page.locator("#archive-upload-result")
|
||||
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
|
||||
expect(result).to_have_text(
|
||||
f"Uploaded {SOURCE_NAME} — press Sync sources to import it.",
|
||||
timeout=UPLOAD_TIMEOUT_MS,
|
||||
)
|
||||
|
||||
|
||||
def _sync_via_ui(page: Page, app_url: str, expected_result: str) -> None:
|
||||
"""The RAG page's "Sync sources" button (the scan — phase 90 A3):
|
||||
click → "Syncing…" → the terminal "Synced HH:MM" label + the fresh
|
||||
counts in #sync-result (the never-stale contract)."""
|
||||
page.goto(app_url + "/sources.html")
|
||||
btn = page.locator("#sync-btn")
|
||||
expect(btn).to_be_visible(timeout=30_000)
|
||||
expect(page.locator("#sync-label")).to_have_text(
|
||||
re.compile(r"^Sync sources$|^Synced \d{1,2}:\d{2}$")
|
||||
)
|
||||
btn.click()
|
||||
expect(btn).to_be_disabled()
|
||||
expect(page.locator("#sync-label")).to_have_text("Syncing…")
|
||||
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
|
||||
expect(btn).to_be_enabled()
|
||||
expect(page.locator("#sync-result")).to_have_text(expected_result)
|
||||
|
||||
|
||||
def _seed_image_doc(
|
||||
app_url: str, browser: Browser, zip_path: Path
|
||||
) -> dict[str, str]:
|
||||
"""Upload the single-PNG zip through the admin UI, then scan it
|
||||
with the RAG page's Sync button (the mock's vision model describes
|
||||
the image in-process). Returns the seeded doc's identity (the
|
||||
bytes route + the description + the title) for the render
|
||||
assertions. The app's ``/api/config`` flag is asserted first — the
|
||||
env override must have reached the app under test, or the whole
|
||||
premise of the suite is void."""
|
||||
cfg = httpx.get(f"{app_url}/api/config", timeout=5).json()
|
||||
assert cfg["images"] is True, "the module app must run with BOR_IMAGES=true"
|
||||
_truncate_all()
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 800})
|
||||
try:
|
||||
_upload_zip(page, app_url, zip_path)
|
||||
_sync_via_ui(page, app_url, expected_result="1 added")
|
||||
with SessionLocal() as db:
|
||||
doc = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE_NAME, Document.path == DOC_PATH
|
||||
)
|
||||
)
|
||||
assert doc is not None, "the uploaded image must become a document"
|
||||
assert doc.is_image is True and doc.image_path is not None
|
||||
# content == summary == the mock's description (the ONLY
|
||||
# embedded text of the doc — the pipeline contract, phase 122).
|
||||
assert doc.content == IMAGE_DESCRIPTION_ANSWER
|
||||
assert doc.summary == IMAGE_DESCRIPTION_ANSWER
|
||||
return {
|
||||
"image_url": f"/api/documents/{doc.id}/image",
|
||||
"description": IMAGE_DESCRIPTION_ANSWER,
|
||||
"title": doc.title,
|
||||
}
|
||||
finally:
|
||||
page.close()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def seeded(
|
||||
app_url: str,
|
||||
browser: Browser,
|
||||
zip_path: Path,
|
||||
) -> Iterator[dict[str, str]]:
|
||||
"""App A's KB: the fixture image doc through the real UI path."""
|
||||
if not _health_db(app_url):
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
_cleanup_saved_rows() # a previous run's rows would be stale (new doc id)
|
||||
yield _seed_image_doc(app_url, browser, zip_path)
|
||||
_cleanup_saved_rows()
|
||||
_truncate_all()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def seeded_off(
|
||||
default_app_url: str,
|
||||
browser: Browser,
|
||||
zip_path: Path,
|
||||
) -> Iterator[dict[str, str]]:
|
||||
"""App B's KB: the SAME fixture image doc through the SAME real
|
||||
UI path (images indexed, the answer feature OFF). Runs AFTER all
|
||||
of App A's tests (file order) — its truncation wipes App A's KB
|
||||
tables, and App B's tests never look back."""
|
||||
if not _health_db(default_app_url):
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
_cleanup_saved_rows()
|
||||
yield _seed_image_doc(default_app_url, browser, zip_path)
|
||||
_cleanup_saved_rows()
|
||||
_truncate_all()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Saved-chat + share helpers (the test_share_chat.py conventions)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _admin_cookies(page: Page) -> dict[str, str]:
|
||||
"""The signed session cookies the browser holds after a form login
|
||||
— used to call the admin API with plain httpx (the test's API side
|
||||
sees exactly what the signed-in browser sees)."""
|
||||
return {
|
||||
c["name"]: c["value"]
|
||||
for c in page.context.cookies()
|
||||
if "name" in c and "value" in c
|
||||
}
|
||||
|
||||
|
||||
def _auto_title(question: str) -> str:
|
||||
"""The phase-50 auto-title convention: the first question,
|
||||
whitespace-collapsed, capped at 120 chars."""
|
||||
return " ".join(question.split())[:120]
|
||||
|
||||
|
||||
#: This suite's auto-titles (the shared e2e Postgres keeps
|
||||
#: ``saved_chats`` across runs — the test_share_chat cleanup
|
||||
#: concern): a re-run must not find the previous run's row (same
|
||||
#: title, but the KB re-seed minted a NEW doc id — the row's
|
||||
#: ``answer_image.image_url`` would no longer match the seed).
|
||||
_SUITE_TITLES = tuple(
|
||||
_auto_title(q)
|
||||
for q in (
|
||||
FEATURE_QUESTION,
|
||||
RESTORE_QUESTION,
|
||||
FEATURE_OFF_QUESTION,
|
||||
DIAGRAM_QUESTION,
|
||||
DIAGRAM_OFF_QUESTION,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _cleanup_saved_rows() -> None:
|
||||
"""Delete THIS suite's saved rows by title (the house pattern —
|
||||
test_share_chat deletes its own rows; here the module-boundary
|
||||
fixture calls it, so a re-run starts clean). Best-effort: a
|
||||
failed delete (DB unreachable — the skip case) never masks the
|
||||
skip; a failed delete with the DB up just re-creates the stale-
|
||||
row premise, and the suite fails loudly on the row assertion."""
|
||||
try:
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text("DELETE FROM saved_chats WHERE title = ANY(:titles)"),
|
||||
{"titles": list(_SUITE_TITLES)},
|
||||
)
|
||||
db.commit()
|
||||
except Exception: # noqa: BLE001 — DB unreachable: the skip handles it
|
||||
pass
|
||||
|
||||
|
||||
def _wait_saved_row(
|
||||
app_url: str,
|
||||
cookies: dict[str, str],
|
||||
title: str,
|
||||
messages: int = 2,
|
||||
) -> dict[str, Any]:
|
||||
"""Wait for the auto-saved row (phase 55: auto-saves are SILENT —
|
||||
A2 — so there is no status line to wait on). The upsert is
|
||||
fire-and-forget from the UI's point of view, so poll the admin
|
||||
list until the row with the conversation's auto-title appears with
|
||||
the expected message count."""
|
||||
deadline = time.monotonic() + 15
|
||||
last: dict[str, Any] | None = None
|
||||
while time.monotonic() < deadline:
|
||||
r = httpx.get(f"{app_url}/api/chats", timeout=10, cookies=cookies)
|
||||
assert r.status_code == 200
|
||||
rows = r.json()["chats"]
|
||||
last = next((c for c in rows if c["title"] == title), None)
|
||||
if last is not None and last["message_count"] >= messages:
|
||||
return last
|
||||
time.sleep(0.2)
|
||||
raise AssertionError(f"no auto-saved row for {title!r} (last: {last!r})")
|
||||
|
||||
|
||||
def _chat_messages(
|
||||
app_url: str, cookies: dict[str, str], chat_id: str
|
||||
) -> list[dict[str, Any]]:
|
||||
r = httpx.get(f"{app_url}/api/chats/{chat_id}", timeout=10, cookies=cookies)
|
||||
assert r.status_code == 200
|
||||
return r.json()["messages"]
|
||||
|
||||
|
||||
def _brain_record(messages: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
brains = [m for m in messages if m.get("who") == "brain"]
|
||||
assert brains, f"no brain record in the saved conversation: {messages!r}"
|
||||
return brains[-1]
|
||||
|
||||
|
||||
def _grant_clipboard(page: Page, app_url: str) -> None:
|
||||
"""Grant the async-clipboard permissions on the admin context
|
||||
(test_share_chat.py — headless Chromium still requires the grant
|
||||
before ``writeText`` resolves on a secure origin)."""
|
||||
page.context.grant_permissions(
|
||||
["clipboard-read", "clipboard-write"], origin=app_url
|
||||
)
|
||||
|
||||
|
||||
def _click_share(app_url: str, page: Page) -> None:
|
||||
"""Press the chat page's Share pill and pin the owner-locked
|
||||
outcome on the live region (the test_share_chat.py branch —
|
||||
clipboard success vs the inline-link fallback)."""
|
||||
page.locator("#share-chat-btn").click()
|
||||
if page.evaluate("() => !!navigator.clipboard"):
|
||||
expect(page.locator("#send-status")).to_have_text(
|
||||
"Share link copied.", timeout=15_000
|
||||
)
|
||||
else:
|
||||
expect(page.locator("#send-status")).to_have_text(
|
||||
"Share link ready — copy it from the field.", timeout=15_000
|
||||
)
|
||||
expect(page.locator(".share-link-fallback")).to_be_visible()
|
||||
|
||||
|
||||
def _assert_answer_figure(
|
||||
page: Page,
|
||||
app_url: str,
|
||||
seeded: dict[str, str],
|
||||
*,
|
||||
caption: str,
|
||||
last_brain: str = ".msg.brain",
|
||||
) -> None:
|
||||
"""The A4 answer-figure contract in one place (the live, restore,
|
||||
and shared surfaces share the shape — the shared page's figure is
|
||||
a plain ``<figure>`, the chat page's an anchor, but the inner
|
||||
``img``/caption classes are the same): exactly ONE ``.answer-image``
|
||||
in the brain bubble, its ``<img>`` from the bytes route
|
||||
(``loading="lazy"``, visible — the bytes really serve), and the
|
||||
caption/alt settled to *caption* (the summary on the admin
|
||||
surfaces, the title on the guest one)."""
|
||||
fig = page.locator(f"{last_brain} .answer-image")
|
||||
expect(fig).to_have_count(1)
|
||||
img = fig.locator("img")
|
||||
expect(img).to_have_count(1)
|
||||
expect(img).to_be_visible(timeout=15_000)
|
||||
expect(img).to_have_attribute("src", seeded["image_url"])
|
||||
expect(img).to_have_attribute("loading", "lazy")
|
||||
expect(fig.locator(".answer-image-caption")).to_have_text(
|
||||
caption, timeout=15_000
|
||||
)
|
||||
# The served bytes are a real image (the public bytes route).
|
||||
resp = httpx.get(app_url + seeded["image_url"], timeout=10)
|
||||
assert resp.status_code == 200
|
||||
assert resp.headers["content-type"].startswith("image/")
|
||||
assert resp.content == PNG_1X1
|
||||
|
||||
|
||||
def _assert_image_chip(
|
||||
page: Page, seeded: dict[str, str], *, last_brain: str = ".msg.brain"
|
||||
) -> None:
|
||||
"""The citation chip the AUTO-CITE earns (phase 124 task 05): the
|
||||
image doc's text chip + its phase-122 compact figure (``<img>``
|
||||
from the bytes route, alt settling to the summary — the admin
|
||||
surface; the shared page's chip is plain text and asserted
|
||||
separately)."""
|
||||
chip = page.locator(f"{last_brain} .source-chip")
|
||||
expect(chip).to_have_count(1)
|
||||
expect(chip).to_have_text(f"{SOURCE_NAME}/{DOC_PATH}")
|
||||
fig = page.locator(f"{last_brain} .source-image")
|
||||
expect(fig).to_have_count(1)
|
||||
img = fig.locator(".source-image-img")
|
||||
expect(img).to_be_visible(timeout=15_000)
|
||||
expect(img).to_have_attribute("src", seeded["image_url"])
|
||||
expect(img).to_have_attribute("alt", seeded["description"], timeout=15_000)
|
||||
|
||||
|
||||
def _body_child_classes(page: Page) -> list[str]:
|
||||
"""The LAST brain bubble's ``.msg-body`` direct children's class
|
||||
names, in DOM order — the A4 position pin (bubble < answer-image
|
||||
< msg-meta)."""
|
||||
return page.evaluate(
|
||||
"""() => {
|
||||
const body = [...document.querySelectorAll(".msg.brain")].pop()
|
||||
.querySelector(".msg-body");
|
||||
return [...body.children].map((el) => el.className);
|
||||
}"""
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. App A (feature ON) — the feature flow: the scripted show_image
|
||||
# turn (ZERO reads) renders the answer figure + the auto-cited chip
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_feature_flow_answer_figure_chips_and_zero_reads(
|
||||
page: Page, app_url: str, seeded: dict[str, str], mock_llm: int
|
||||
) -> None:
|
||||
"""The ``show me the picture of …`` trigger: the mock scripts ONLY
|
||||
a ``show_image`` call (NO ``read`` — the zero-read turn). Asserts:
|
||||
(1) the brain bubble's ``.msg-body`` carries ``.answer-image`` with
|
||||
an ``<img>`` from the bytes route (200, image Content-Type) and
|
||||
``loading="lazy"``; (2) the figure sits BEFORE the ``.msg-meta``
|
||||
source-chip row (A4); (3) the answer TEXT is non-empty and quotes
|
||||
the seed's tail (the "summaries still provided" contract + the
|
||||
echo proving the seed was in the prompt); (4) the source-chip row
|
||||
carries the image doc's chip + its figure (the AUTO-CITE proof —
|
||||
the mock made ZERO ``read`` calls: only ``show_image`` — asserted
|
||||
on the wire); (5) the "Nearby docs" row does NOT repeat the doc.
|
||||
Plus the persistence pin: the saved brain record carries the
|
||||
``answer_image`` ref (the task-04 frame field, round-tripped)."""
|
||||
login(page, app_url, next="/")
|
||||
expect(page.locator("#kb-banner")).to_be_hidden()
|
||||
|
||||
_mock_capture_reset(mock_llm)
|
||||
page.fill("#message-input", FEATURE_QUESTION)
|
||||
page.click("#send-btn")
|
||||
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
# The answer TEXT is non-empty (the "summaries still provided"
|
||||
# contract) — the tail echo lands LAST (the terminal signal): it
|
||||
# quotes the <documents> seed's tail, i.e. the description.
|
||||
expect(bubble).to_contain_text(DESCRIPTION_TAIL, timeout=30_000)
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send")
|
||||
|
||||
# (1) The answer figure: the bytes route, lazy, alt/caption the
|
||||
# summary (the fetchContentSummary pattern settles both).
|
||||
_assert_answer_figure(page, app_url, seeded, caption=seeded["description"])
|
||||
|
||||
# (2) A4 position: below the answer text, above the chip row.
|
||||
classes = _body_child_classes(page)
|
||||
assert "bubble" in classes, f"no answer bubble in .msg-body: {classes}"
|
||||
assert "answer-image" in classes, f"no .answer-image in .msg-body: {classes}"
|
||||
assert "msg-meta" in classes, f"no chip row in .msg-body: {classes}"
|
||||
assert (
|
||||
classes.index("bubble")
|
||||
< classes.index("answer-image")
|
||||
< classes.index("msg-meta")
|
||||
), f"A4 order violated (bubble < answer-image < msg-meta): {classes}"
|
||||
|
||||
# (4) The chip row carries the image doc's chip + its figure —
|
||||
# the AUTO-CITE (the model never read it; the seed IS its
|
||||
# entire content, so suggested == read for it).
|
||||
_assert_image_chip(page, seeded)
|
||||
|
||||
# (5) The "Nearby docs" row does NOT repeat the image doc — the
|
||||
# related tier excludes cited members (the single-doc KB has
|
||||
# no related tier at all: the row is absent).
|
||||
expect(page.locator(".msg.brain .related-docs")).to_have_count(0)
|
||||
|
||||
# The WIRE pin: exactly two model calls (the show_image round +
|
||||
# the final answer), the FIRST request offered the FOUR tools
|
||||
# (three base + show_image — the toggle-ON offer gate), and the
|
||||
# turn executed EXACTLY one tool call: show_image — NO read (the
|
||||
# zero-read turn that makes the chip assertion meaningful).
|
||||
captured = _mock_capture(mock_llm)
|
||||
assert len(captured) == 2, (
|
||||
f"expected exactly 2 model calls (call + answer), got {len(captured)}"
|
||||
)
|
||||
assert _offered_tool_names(captured[0]) == ["ls", "read", "grep", "show_image"]
|
||||
assert _executed_tool_calls(captured) == ["show_image"]
|
||||
tool_results = [
|
||||
m.get("content") or ""
|
||||
for m in captured[1]["messages"]
|
||||
if m.get("role") == "tool"
|
||||
]
|
||||
assert any("Image shown to the user" in r for r in tool_results), (
|
||||
f"the confirming tool result never reached the model: {tool_results}"
|
||||
)
|
||||
|
||||
# The persistence pin: the auto-saved brain record carries the
|
||||
# answer_image ref (source/path/title/image_url — the task-04
|
||||
# frame field; the restore + shared flows assert on its render).
|
||||
row = _wait_saved_row(app_url, _admin_cookies(page), _auto_title(FEATURE_QUESTION))
|
||||
record = _brain_record(_chat_messages(app_url, _admin_cookies(page), str(row["id"])))
|
||||
assert record.get("answer_image") == {
|
||||
"source": SOURCE_NAME,
|
||||
"path": DOC_PATH,
|
||||
"title": seeded["title"],
|
||||
"image_url": seeded["image_url"],
|
||||
}, f"the brain record's answer_image is wrong: {record.get('answer_image')!r}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. App A — reload restores the figure; the shared page shows it
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_reload_restores_figure_and_shared_page_shows_it(
|
||||
page: Page,
|
||||
app_url: str,
|
||||
seeded: dict[str, str],
|
||||
mock_llm: int,
|
||||
browser: Browser,
|
||||
) -> None:
|
||||
"""A second feature turn (a unique question — unique auto-title on
|
||||
the shared DB), then a page RELOAD: the restored brain bubble
|
||||
shows the figure again (the ``answer_image`` persistence through
|
||||
the ``bor.chat.v1`` restore path — pixel-parity with the live
|
||||
render), and the existing Share flow's link — opened in a FRESH
|
||||
anonymous context — renders the figure on the shared page (the
|
||||
bytes route is public; the guest caption keeps the title — the
|
||||
content endpoint is user-gated, so the summary fetch is skipped
|
||||
there)."""
|
||||
login(page, app_url, next="/")
|
||||
expect(page.locator("#kb-banner")).to_be_hidden()
|
||||
|
||||
_mock_capture_reset(mock_llm)
|
||||
page.fill("#message-input", RESTORE_QUESTION)
|
||||
page.click("#send-btn")
|
||||
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(DESCRIPTION_TAIL, timeout=30_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send")
|
||||
_assert_answer_figure(page, app_url, seeded, caption=seeded["description"])
|
||||
_assert_image_chip(page, seeded)
|
||||
|
||||
# The auto-save settled (the row exists with both messages) before
|
||||
# the reload — the Share click later shares the linked row, or
|
||||
# re-creates it unlinked (either path is owner-locked; the row
|
||||
# lookup keys on the unique auto-title).
|
||||
cookies = _admin_cookies(page)
|
||||
_wait_saved_row(app_url, cookies, _auto_title(RESTORE_QUESTION))
|
||||
|
||||
# RELOAD: the restored brain bubble shows the figure again — the
|
||||
# localStorage restore (rememberBrainTurn's save point ran
|
||||
# synchronously in the done handler, before the send button
|
||||
# re-enabled, so the record is durable before we reload).
|
||||
page.reload()
|
||||
expect(page.locator("#kb-banner")).to_be_hidden()
|
||||
restored = page.locator(".msg.brain .bubble").last
|
||||
expect(restored).to_contain_text(DESCRIPTION_TAIL, timeout=30_000)
|
||||
expect(restored).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
_assert_answer_figure(page, app_url, seeded, caption=seeded["description"])
|
||||
_assert_image_chip(page, seeded)
|
||||
|
||||
# SHARE: the existing chat-page Share flow → a fresh anonymous
|
||||
# context sees the figure (and the plain-text chip).
|
||||
_grant_clipboard(page, app_url)
|
||||
_click_share(app_url, page)
|
||||
# The share landed on the pre-existing row (the linked branch —
|
||||
# the idempotent share) or a fresh row (the unlinked save-and-
|
||||
# share) — either carries the token; the unique auto-title keeps
|
||||
# the lookup unambiguous on the shared DB.
|
||||
r = httpx.get(f"{app_url}/api/chats", timeout=10, cookies=cookies)
|
||||
rows = [
|
||||
c for c in r.json()["chats"]
|
||||
if c["title"] == _auto_title(RESTORE_QUESTION)
|
||||
]
|
||||
token_row = next((c for c in rows if c.get("share_url")), None)
|
||||
assert token_row is not None and token_row.get("share_url"), (
|
||||
f"no share_url on any shared row: {rows!r}"
|
||||
)
|
||||
share_link = str(token_row["share_url"])
|
||||
assert share_link.startswith("/shared/")
|
||||
|
||||
anon = browser.new_page(viewport={"width": 1280, "height": 800})
|
||||
try:
|
||||
anon.goto(app_url + share_link)
|
||||
expect(anon.locator("#shared-title")).to_have_text(
|
||||
_auto_title(RESTORE_QUESTION), timeout=30_000
|
||||
)
|
||||
expect(anon.locator(".msg.brain .bubble").last).to_contain_text(
|
||||
DESCRIPTION_TAIL, timeout=30_000
|
||||
)
|
||||
# The shared page renders the figure — the guest caption keeps
|
||||
# the TITLE (no summary fetch — the content endpoint is
|
||||
# user-gated), the img is from the public bytes route.
|
||||
_assert_answer_figure(anon, app_url, seeded, caption=seeded["title"])
|
||||
# The shared chip is PLAIN TEXT (guests cannot open documents):
|
||||
chip = anon.locator(".msg.brain .source-chip")
|
||||
expect(chip).to_have_count(1)
|
||||
expect(chip).to_have_text(f"{SOURCE_NAME}/{DOC_PATH}")
|
||||
# The shared page is zero-controls: the figure is a plain
|
||||
# <figure>, never an anchor (the owner-locked contract).
|
||||
assert anon.locator(".msg.brain a.answer-image").count() == 0
|
||||
finally:
|
||||
anon.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. App A — the bug-fix-only flow: ZERO tool calls, the chip STILL
|
||||
# appears (the pure zero-read path — the owner's report)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_bugfix_only_flow_zero_tool_calls_still_chips(
|
||||
page: Page, app_url: str, seeded: dict[str, str], mock_llm: int
|
||||
) -> None:
|
||||
"""The ``which diagram is in your notes`` trigger: the mock
|
||||
answers straight from the seed with ZERO tool calls (no
|
||||
``show_image`` — the clause's flow is not triggered — and no
|
||||
``read``): the source-chip row STILL carries the image doc's chip
|
||||
+ figure (the bug fix on the pure zero-read path — the assertion
|
||||
the owner's report describes), the answer text quotes the seeded
|
||||
description's tail (the description WAS in the prompt), and NO
|
||||
answer figure renders (the model never called the tool)."""
|
||||
login(page, app_url, next="/")
|
||||
expect(page.locator("#kb-banner")).to_be_hidden()
|
||||
|
||||
_mock_capture_reset(mock_llm)
|
||||
page.fill("#message-input", DIAGRAM_QUESTION)
|
||||
page.click("#send-btn")
|
||||
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
# The answer quotes the seeded image doc's block tail (the
|
||||
# description's end — the seed was in the prompt) and settles.
|
||||
expect(bubble).to_contain_text(DESCRIPTION_TAIL, timeout=30_000)
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send")
|
||||
|
||||
# ZERO tool calls — on the wire (exactly ONE model call, no
|
||||
# tool_calls message at all) and in the DOM (no tool record).
|
||||
captured = _mock_capture(mock_llm)
|
||||
assert len(captured) == 1, (
|
||||
f"the bug-fix flow must make exactly 1 model call, got {len(captured)}"
|
||||
)
|
||||
assert _executed_tool_calls(captured) == [], (
|
||||
f"the bug-fix flow must make ZERO tool calls: {_executed_tool_calls(captured)}"
|
||||
)
|
||||
expect(page.locator(".msg.brain .tool-calls-disclosure")).to_have_count(0)
|
||||
|
||||
# NO answer figure (the model never called show_image) — but the
|
||||
# chip row STILL carries the image doc's chip + figure (the auto-
|
||||
# cite: suggested == read for an image doc, no read performed).
|
||||
expect(page.locator(".msg.brain .answer-image")).to_have_count(0)
|
||||
_assert_image_chip(page, seeded)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. App B (default off) — the toggle-off contract: no show_image
|
||||
# offered, no answer figure, no answer_image key in the record
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_toggle_off_flow_no_tool_no_answer_image_key(
|
||||
page: Page,
|
||||
default_app_url: str,
|
||||
seeded_off: dict[str, str],
|
||||
mock_llm: int,
|
||||
) -> None:
|
||||
"""The same feature trigger question on the default-OFF app: the
|
||||
prompt clause is ABSENT, so the mock's trigger never fires (the
|
||||
plain composed answer streams instead) — and the contract holds
|
||||
at every surface: the mock NEVER sees ``show_image`` among the
|
||||
offered tools (the request capture carries exactly the THREE base
|
||||
tools), the brain bubble has NO ``.answer-image``, and the saved
|
||||
brain record carries NO ``answer_image`` key (the byte-identical-
|
||||
off contract — the key is absent, never null)."""
|
||||
login(page, default_app_url, next="/")
|
||||
expect(page.locator("#kb-banner")).to_be_hidden()
|
||||
|
||||
_mock_capture_reset(mock_llm)
|
||||
page.fill("#message-input", FEATURE_OFF_QUESTION)
|
||||
page.click("#send-btn")
|
||||
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send")
|
||||
|
||||
# No answer figure anywhere in the brain bubble.
|
||||
expect(page.locator(".msg.brain .answer-image")).to_have_count(0)
|
||||
|
||||
# The WIRE pin: the offer list is EXACTLY the three base tools
|
||||
# (no show_image — the toggle-OFF gate on the offer), and the
|
||||
# turn made no tool calls at all (the mock's feature trigger is
|
||||
# clause-gated — it never fired, so the plain answer streamed in
|
||||
# one call).
|
||||
captured = _mock_capture(mock_llm)
|
||||
assert len(captured) == 1, (
|
||||
f"expected exactly 1 model call (no tool round), got {len(captured)}"
|
||||
)
|
||||
assert _offered_tool_names(captured[0]) == ["ls", "read", "grep"], (
|
||||
f"show_image leaked into the default-off offer: "
|
||||
f"{_offered_tool_names(captured[0])}"
|
||||
)
|
||||
assert _executed_tool_calls(captured) == []
|
||||
|
||||
# The saved brain record carries NO answer_image key (ABSENT — the
|
||||
# byte-identical-off contract; the phase-113 422-silence pin is
|
||||
# the positive half of this, in the App A flow + the integration
|
||||
# suite). The chip, though, IS still auto-cited (the chips fix is
|
||||
# not toggle-gated — asserted in the next test).
|
||||
row = _wait_saved_row(
|
||||
default_app_url,
|
||||
_admin_cookies(page),
|
||||
_auto_title(FEATURE_OFF_QUESTION),
|
||||
)
|
||||
record = _brain_record(
|
||||
_chat_messages(default_app_url, _admin_cookies(page), str(row["id"]))
|
||||
)
|
||||
assert "answer_image" not in record, (
|
||||
f"the default-off brain record must NOT carry answer_image: "
|
||||
f"{record.get('answer_image')!r}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 5. App B (default off) — the bug-fix trigger STILL chips the image
|
||||
# doc (the chips fix belongs to BOR_IMAGES, not BOR_ANSWER_IMAGES)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_toggle_off_bugfix_trigger_still_chips(
|
||||
page: Page, default_app_url: str, seeded_off: dict[str, str], mock_llm: int
|
||||
) -> None:
|
||||
"""The bug-fix trigger on the default-OFF app (the clause is
|
||||
absent — the trigger does NOT require it): ZERO tool calls, the
|
||||
answer quotes the seeded description's tail, and the source-chip
|
||||
row STILL carries the image doc's chip + figure — the chips fix
|
||||
is NOT gated by ``BOR_ANSWER_IMAGES`` (it belongs to the
|
||||
``BOR_IMAGES``-indexed docs, per the phase design)."""
|
||||
login(page, default_app_url, next="/")
|
||||
expect(page.locator("#kb-banner")).to_be_hidden()
|
||||
|
||||
_mock_capture_reset(mock_llm)
|
||||
page.fill("#message-input", DIAGRAM_OFF_QUESTION)
|
||||
page.click("#send-btn")
|
||||
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(DESCRIPTION_TAIL, timeout=30_000)
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send")
|
||||
|
||||
captured = _mock_capture(mock_llm)
|
||||
assert len(captured) == 1
|
||||
assert _executed_tool_calls(captured) == []
|
||||
expect(page.locator(".msg.brain .tool-calls-disclosure")).to_have_count(0)
|
||||
expect(page.locator(".msg.brain .answer-image")).to_have_count(0)
|
||||
_assert_image_chip(page, seeded_off)
|
||||
Reference in New Issue
Block a user